fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere

- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
  `formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
  file paths, and the job-failed, refused, edit-failed and backfill lines use
  it. Done lines log the asset id, not the title-built path. The PII contract
  now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
  failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
  is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
  set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
  a process's last batch closes the repeat window, so a short-lived download
  child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
  `download.job.failed` with a cause; they left no line at all before. The
  crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
  and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
  not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
  carry reason `transport`. Subscribe recognises a duplicate by reason code
  only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
  directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
  were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
This commit is contained in:
2026-09-18 03:16:15 +02:00
parent 04b58e5c6f
commit 54d19cc386
22 changed files with 834 additions and 141 deletions
+33 -11
View File
@@ -26,10 +26,29 @@ from app import logs
log = logs.get_logger(__name__)
# yt-dlp puts the full source URL in most of its lines, and a source URL is a row of someone's
# download history. The job id is already on every line (`logs.set_job` in the download child), so
# the URL adds nothing an operator needs and plenty they should not keep.
# yt-dlp names what it is working on in three shapes, and each one is a row of someone's download
# history: the source URL, the `[extractor] <id>:` prefix on nearly every warning and error (for
# the generic extractor the "id" is the URL's own basename), and the file path it writes to, which
# `storage.rel_path` builds from the channel and the title. The job id is already on every line
# (`logs.set_job` in the download child), so none of the three adds anything an operator needs.
# What stays: the host (it says which extractor is failing) and the extractor tag.
_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
# `Destination:` is yt-dlp's own word in `[download] Destination: <path>`, not an id — the path
# after it is what the third pattern takes out.
_ID_PREFIX_RE = re.compile(r"(\[[\w:.-]+\] )(?!Destination:)[^\s:\[\]]+:")
# Two segments or more, and not glued to a word: `www.youtube.com/…` is what the URL pattern
# leaves behind, and it must survive this one.
_PATH_RE = re.compile(r"(?<![\w.…])/(?:[^/\s\"']+/)+[^/\s\"']*")
def log_safe(text: object) -> str:
"""yt-dlp's text (its own lines, or a DownloadError's message) with the parts that name a
person's download taken out. Every log line that carries yt-dlp's words goes through here,
not just the adapter's: yt-dlp raises the same text it has just logged, so a line that skipped
it would put back one line later exactly what the adapter took out."""
text = _URL_RE.sub(lambda m: f"{m.group(1)}/…", str(text))
text = _ID_PREFIX_RE.sub(r"\1…:", text)
return _PATH_RE.sub("…", text)
class _YdlLog:
@@ -41,19 +60,22 @@ class _YdlLog:
2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
`warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
`debug` goes to TRACE — it is several lines per second during a download, and TRACE is the
level a user turns on for their own session. Everything else keeps its level."""
`debug` goes to TRACE: it is several lines per second during a download, so it shows only
when the whole process runs at TRACE. (Not under a user's own Teljes consent — that lives in
the API process's request scope, and the download child that runs yt-dlp has none.)
@staticmethod
def _clean(msg: str) -> str:
return _URL_RE.sub(lambda m: f"{m.group(1)}/…", str(msg))
`error` goes to WARNING, not ERROR. yt-dlp calls it from `trouble()` just before it raises,
on EVERY failed attempt — including the ones the pipeline then recovers from with its fallback
client set. Whether the job failed is the pipeline's verdict (`download.job.failed`), and that
is the one line whose level should say so."""
def _emit(self, level: int, msg: str) -> None:
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
# a record that has args, this way the question never arises — and it is the shape ruff's
# `G` rules ask for.
log.log(level, "%s", self._clean(msg), extra=logs.event("download.ytdlp"))
# `G` rules ask for. (The sink's flood limiter groups a bare `%s` by the text itself, so
# distinct yt-dlp warnings do not share one bucket — `logsink._template`.)
log.log(level, "%s", log_safe(msg), extra=logs.event("download.ytdlp"))
def debug(self, msg):
self._emit(logs.TRACE, msg)
@@ -65,7 +87,7 @@ class _YdlLog:
self._emit(logging.WARNING, msg)
def error(self, msg):
self._emit(logging.ERROR, msg)
self._emit(logging.WARNING, msg)
_YDL_LOG = _YdlLog()