fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere

- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
  `formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
  file paths, and the job-failed, refused, edit-failed and backfill lines use
  it. Done lines log the asset id, not the title-built path. The PII contract
  now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
  failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
  is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
  set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
  a process's last batch closes the repeat window, so a short-lived download
  child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
  `download.job.failed` with a cause; they left no line at all before. The
  crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
  and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
  not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
  carry reason `transport`. Subscribe recognises a duplicate by reason code
  only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
  directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
  were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
This commit is contained in:
2026-09-18 03:16:15 +02:00
parent 04b58e5c6f
commit 54d19cc386
22 changed files with 834 additions and 141 deletions
+27 -7
View File
@@ -29,6 +29,7 @@ import atexit
import json
import logging
import queue
import re
import sys
import threading
import time
@@ -137,6 +138,21 @@ def _notice(code: str, msg: str, **attrs: Any) -> dict[str, Any]:
)
_DIGITS = re.compile(r"\d+")
def _template(record: logging.LogRecord) -> str:
"""The flood limiter's grouping key: the message template. Except when the template is a bare
`%s` — a pass-through for someone else's text (yt-dlp's lines, `formats._YdlLog`), where the
template says nothing and every distinct line would share one bucket. Then the text itself
groups, with its numbers folded, so `Retrying fragment 5 (1/10)` and `(2/10)` are still one
repeat and a different warning is not crowded out by them."""
msg = str(record.msg)
if msg != "%s" or not record.args:
return msg
return _DIGITS.sub("#", record.getMessage())[:200]
class _Repeats:
"""Per-template admission for one minute at a time. Used by the writer thread only (unlocked)."""
@@ -155,9 +171,11 @@ class _Repeats:
self.seen[key] = (count or 0) + 1
return self.seen[key] <= self.per_window
def roll(self, now: float) -> list[dict[str, Any]]:
"""Close the window if it is over: one notice per template that went past the limit."""
if now - self.started < self.window:
def roll(self, now: float, *, final: bool = False) -> list[dict[str, Any]]:
"""Close the window if it is over — or, `final`, because the process is: one notice per
template that went past the limit. Without `final` a process that lives under a minute
(every download child) would drop its repeats and never say so."""
if not final and now - self.started < self.window:
return []
notices = [
_notice(
@@ -211,7 +229,7 @@ class LogSink(logging.Handler):
key = (
None
if row["event"] == "http.request"
else (record.name, str(record.msg), record.levelno)
else (record.name, _template(record), record.levelno)
)
try:
self._queue.put_nowait((key, row))
@@ -222,9 +240,10 @@ class LogSink(logging.Handler):
self._dropped += 1
# --- writer side ---------------------------------------------------------------------------
def write_pending(self, wait: float = 0.0) -> int:
def write_pending(self, wait: float = 0.0, *, final: bool = False) -> int:
"""Write one batch: whatever is queued (up to BATCH_MAX), waiting up to `wait` seconds for the
first item. Returns the number of rows written, notices included."""
first item. Returns the number of rows written, notices included. `final` closes the
repeat window early — the last batch before the process exits."""
rows: list[dict[str, Any]] = []
try:
item = self._queue.get(timeout=wait) if wait > 0 else self._queue.get_nowait()
@@ -242,7 +261,7 @@ class LogSink(logging.Handler):
# silent-ok: the batch ends here because the queue ran dry, which is the normal end.
except queue.Empty:
item = None
rows.extend(self._repeats.roll(time.monotonic()))
rows.extend(self._repeats.roll(time.monotonic(), final=final))
with self._dropped_lock:
dropped, self._dropped = self._dropped, 0
if dropped:
@@ -276,6 +295,7 @@ class LogSink(logging.Handler):
self.write_pending(wait=FLUSH_SECONDS)
while self.write_pending(): # drain what arrived before the stop
pass
self.write_pending(final=True) # and the repeat notices of a window cut short
def start(self) -> None:
self._writer = threading.Thread(target=self._run, name="log-sink", daemon=True)