fix(logs): deep-review round 2 — a stack names code, not data, and survives a failed write

- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a
  message that is the whole story). A plain RuntimeError, RecursionError or OSError
  is also what a bug of ours raises, and its stack is the only lead.
- The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError
  under a DB error no longer prints yt-dlp's text unscrubbed.
- Engine hide_parameters: a DB error's message no longer carries the title or path
  it was writing.
- The download and edit failure lines go out even when their DB write fails.
- The worker's child_exit/crash/supervisor lines say download.job.failed only when
  the job failed there; a crash the DB will not record no longer ends the thread.
  Stopping a paused child is download.child.stopped, not a second abort.
- log_safe also covers generic ids with spaces, playlist titles and bare ids.
- The ffmpeg image steps log the failure kind, not the argv with its path.
- The consent contract accepts only the body of a plain consented_trace() check,
  and any level that may be TRACE.
- Tests for each; every one mutation-checked.
This commit is contained in:
2026-09-18 22:50:37 +02:00
parent 54d19cc386
commit 0b672ed6e7
14 changed files with 729 additions and 138 deletions
+49 -1
View File
@@ -37,7 +37,9 @@ import re
import sys
import threading
import time
import traceback
import uuid
from collections.abc import Callable
from datetime import datetime, timezone
from typing import Any
from urllib.parse import parse_qsl
@@ -263,11 +265,53 @@ def redact(value: Any, *, key: str = "") -> Any:
# --- formatters ----------------------------------------------------------------------------
_SCRUBBED_STACK = "scrubbed_stack"
def scrubbed_stack(exc: BaseException, scrub: Callable[[BaseException], str]) -> dict[str, str]:
"""`extra=` fragment: `exc`'s traceback, rendered where an `exc_info` stack goes, with the
message of every exception in it put through `scrub`.
For a failure whose message can name someone's data (a download's URL, a file path, a video
id). The frames name code and are kept whole; the text at the foot of each link is the data.
A plain `exc_info` renders the chain too — the `__cause__`/`__context__` a DB error raised
inside `except DownloadError` carries — so scrubbing only the line's own message is not
enough. Merge it into the call's `extra`: `{**event(...), **scrubbed_stack(...)}`."""
links: list[BaseException] = []
link: BaseException | None = exc
while link is not None and all(link is not seen for seen in links):
links.append(link)
link = link.__cause__ or (None if link.__suppress_context__ else link.__context__)
parts: list[str] = []
for i in range(len(links) - 1, -1, -1): # the oldest first, as Python prints a chain
link = links[i]
kind = type(link)
name = kind.__qualname__
if kind.__module__ not in ("builtins", "__main__"):
name = f"{kind.__module__}.{name}"
try:
message = scrub(link)
# silent-ok: the message is what could not be rendered, and the placeholder says so — the
# caller is in the middle of reporting one failure and must not get a second from here.
except Exception: # noqa: BLE001 — a `__str__` that raises, of any shape
message = "<unprintable>"
frames = "".join(traceback.format_tb(link.__traceback__))
head = f"Traceback (most recent call last):\n{frames}" if frames else ""
parts.append(f"{head}{name}: {message}" if message else f"{head}{name}")
if i:
parts.append(
"\nThe above exception was the direct cause of the following exception:\n"
if links[i - 1].__cause__ is link
else "\nDuring handling of the above exception, another exception occurred:\n"
)
return {_SCRUBBED_STACK: "\n".join(parts)}
# Attributes every LogRecord carries; anything else a caller passed via `extra=` is a structured
# attribute and is merged into `attrs` (so `extra={"job_id": 7}` works as well as `event(...)`).
_RECORD_STANDARD = frozenset(
vars(logging.LogRecord("", 0, "", 0, "", None, None)).keys()
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated"}
| {"message", "asctime", "taskName", "event", "attrs", PERSIST, "elevated", _SCRUBBED_STACK}
)
@@ -348,6 +392,8 @@ def record_fields(record: logging.LogRecord) -> dict[str, Any]:
fields["attrs"] = redact(attrs)
if record.exc_info:
fields["exc"] = redact_text(logging.Formatter().formatException(record.exc_info))
elif getattr(record, _SCRUBBED_STACK, None):
fields["exc"] = redact_text(getattr(record, _SCRUBBED_STACK))
elif record.exc_text:
# A QueueHandler-style hand-off formats the traceback into exc_text and drops exc_info.
fields["exc"] = redact_text(record.exc_text)
@@ -381,6 +427,8 @@ class TextFormatter(logging.Formatter):
line += f" {json.dumps(redact(attrs), ensure_ascii=False, default=str)}"
if record.exc_info:
line += "\n" + redact_text(self.formatException(record.exc_info))
elif getattr(record, _SCRUBBED_STACK, None):
line += "\n" + redact_text(getattr(record, _SCRUBBED_STACK))
return line