feat(logs): give the log the events that no request line already carries
The gap this sprint was scoped against turned out to be smaller than the survey said, and in a way worth writing down: `log_requests` already records every HTTP call with its route template, status, duration, request id and user id, and the audit log already records every admin act on someone else's account. Putting a line in each of routes/downloads.py's twenty mutating endpoints would have added a second copy of the first and nothing else. So this instruments what neither of them can see. yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting: with no `logger` its whole account of a download went to the child's stdout, so "Requested format is not available", a signature-extraction failure or a POT refusal reached us only as whatever text ended up in the raised exception — the real reason a download failed was routinely one line the app never kept. Measured against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls `debug` for extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being several lines a second, so the noisy half is the half the user's own consent gates. The full source URL is trimmed to its host on the way in — it is a row of someone's download history, and `logs.set_job` already puts the job id on every one of these lines. The option is set AFTER the adapter merge, like `cookiefile` and the size cap: a source adapter must not be able to take away the one account of what went wrong that a failed job leaves behind. An HLS session now says which pipeline it got (transcode/remux, hw, tonemap, variants, offset) — everything downstream is about that choice, and it was only inferable from an ffmpeg command line nobody keeps. A session culled by the concurrent-transcode cap says so: somebody's playback stops there, the route turns it into a retryable 404, and "my film keeps rebuffering" had no other explanation anywhere. The HW→SW fallback keeps its line and gains an event code. A live search refused by a quota cap says WHICH cap: the 429 is in the request line either way, but the per-user daily cap is answered by tomorrow and the shared budget by the admin. The download and edit job lifecycle lines gain event codes, which is what E22.6's console filters on — the job id was already a structured field. tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth (a `%` in a progress line) survived its mutation — stdlib only applies `%` formatting to a record that has args — so it is gone rather than kept as a case nothing can break; the reasoning moved into the comment on `_emit`.
This commit is contained in:
@@ -19,11 +19,57 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
from app import logs
|
||||
|
||||
log = logs.get_logger(__name__)
|
||||
|
||||
# yt-dlp puts the full source URL in most of its lines, and a source URL is a row of someone's
|
||||
# download history. The job id is already on every line (`logs.set_job` in the download child), so
|
||||
# the URL adds nothing an operator needs and plenty they should not keep.
|
||||
_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
|
||||
|
||||
|
||||
class _YdlLog:
|
||||
"""yt-dlp's logger protocol, pointed at ours.
|
||||
|
||||
Without it yt-dlp explains itself to nobody: `quiet` sends its output to a stdout we do not
|
||||
read, so "Requested format is not available", a signature-extraction failure or a POT refusal
|
||||
reached us only as whatever text ended up in the raised exception. Measured against yt-dlp
|
||||
2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
|
||||
`warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
|
||||
|
||||
`debug` goes to TRACE — it is several lines per second during a download, and TRACE is the
|
||||
level a user turns on for their own session. Everything else keeps its level."""
|
||||
|
||||
@staticmethod
|
||||
def _clean(msg: str) -> str:
|
||||
return _URL_RE.sub(lambda m: f"{m.group(1)}/…", str(msg))
|
||||
|
||||
def _emit(self, level: int, msg: str) -> None:
|
||||
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
|
||||
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
|
||||
# a record that has args, this way the question never arises — and it is the shape ruff's
|
||||
# `G` rules ask for.
|
||||
log.log(level, "%s", self._clean(msg), extra=logs.event("download.ytdlp"))
|
||||
|
||||
def debug(self, msg):
|
||||
self._emit(logs.TRACE, msg)
|
||||
|
||||
def info(self, msg):
|
||||
self._emit(logging.DEBUG, msg)
|
||||
|
||||
def warning(self, msg):
|
||||
self._emit(logging.WARNING, msg)
|
||||
|
||||
def error(self, msg):
|
||||
self._emit(logging.ERROR, msg)
|
||||
|
||||
|
||||
_YDL_LOG = _YdlLog()
|
||||
|
||||
# Fields that affect the produced file — and therefore the cache identity.
|
||||
_SIG_KEYS = (
|
||||
"mode",
|
||||
@@ -191,6 +237,12 @@ def build_ydl_opts(
|
||||
if extra:
|
||||
opts.update(extra)
|
||||
|
||||
# With a logger attached, `quiet` and `no_warnings` stop deciding whether yt-dlp explains itself
|
||||
# and only decide where — so both stay on above and the explanation comes here instead of to a
|
||||
# stdout nothing reads. AFTER the adapter merge, like `cookiefile` and the size cap: an adapter
|
||||
# must not be able to silence the one account of what went wrong that a failed job leaves.
|
||||
opts["logger"] = _YDL_LOG
|
||||
|
||||
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
|
||||
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
|
||||
#
|
||||
|
||||
Reference in New Issue
Block a user