peter
04b58e5c6f
The gap this sprint was scoped against turned out to be smaller than the survey said, and in a way worth writing down: `log_requests` already records every HTTP call with its route template, status, duration, request id and user id, and the audit log already records every admin act on someone else's account. Putting a line in each of routes/downloads.py's twenty mutating endpoints would have added a second copy of the first and nothing else. So this instruments what neither of them can see. yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting: with no `logger` its whole account of a download went to the child's stdout, so "Requested format is not available", a signature-extraction failure or a POT refusal reached us only as whatever text ended up in the raised exception — the real reason a download failed was routinely one line the app never kept. Measured against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls `debug` for extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being several lines a second, so the noisy half is the half the user's own consent gates. The full source URL is trimmed to its host on the way in — it is a row of someone's download history, and `logs.set_job` already puts the job id on every one of these lines. The option is set AFTER the adapter merge, like `cookiefile` and the size cap: a source adapter must not be able to take away the one account of what went wrong that a failed job leaves behind. An HLS session now says which pipeline it got (transcode/remux, hw, tonemap, variants, offset) — everything downstream is about that choice, and it was only inferable from an ffmpeg command line nobody keeps. A session culled by the concurrent-transcode cap says so: somebody's playback stops there, the route turns it into a retryable 404, and "my film keeps rebuffering" had no other explanation anywhere. The HW→SW fallback keeps its line and gains an event code. A live search refused by a quota cap says WHICH cap: the 429 is in the request line either way, but the per-user daily cap is answered by tomorrow and the shared budget by the admin. The download and edit job lifecycle lines gain event codes, which is what E22.6's console filters on — the job id was already a structured field. tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth (a `%` in a progress line) survived its mutation — stdlib only applies `%` formatting to a record that has args — so it is gone rather than kept as a case nothing can break; the reasoning moved into the comment on `_emit`.
109 lines
3.8 KiB
Python
109 lines
3.8 KiB
Python
"""E22.5 S4 — yt-dlp explains itself to our log, not to a stdout nobody reads.
|
|
|
|
`quiet: True` was doing more than quieting: with no `logger`, yt-dlp's whole account of a download
|
|
went to the child process's stdout, so "Requested format is not available", a signature-extraction
|
|
failure or a POT refusal reached us only as whatever text happened to end up in the raised
|
|
exception. A failed download's real reason was routinely one line the app never kept.
|
|
|
|
Measured against yt-dlp 2026.07.04 (the pinned version): with `logger` set, yt-dlp calls `debug`
|
|
for its extraction and progress lines and `warning`/`error` for the rest, and neither `quiet` nor
|
|
`no_warnings` gates those calls. The cases below pin what we do with them.
|
|
"""
|
|
|
|
import json
|
|
import logging
|
|
|
|
import pytest
|
|
|
|
from app import logs
|
|
from app.downloads import formats
|
|
|
|
URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ&list=PLsomething"
|
|
|
|
|
|
class _Sink(logging.Handler):
|
|
"""The stdout handler's own filter and formatter, so a case sees the bytes a deployment writes."""
|
|
|
|
def __init__(self):
|
|
super().__init__(level=logging.NOTSET)
|
|
self.lines: list[str] = []
|
|
self.setFormatter(logs.JsonFormatter())
|
|
self.addFilter(logs.ContextFilter())
|
|
|
|
def emit(self, record):
|
|
self.lines.append(self.format(record))
|
|
|
|
def objects(self) -> list[dict]:
|
|
return [json.loads(line) for line in self.lines]
|
|
|
|
|
|
@pytest.fixture
|
|
def sink():
|
|
logger = logging.getLogger("siftlode")
|
|
handler = _Sink()
|
|
previous = logger.level
|
|
logger.addHandler(handler)
|
|
logger.setLevel(logs.TRACE)
|
|
try:
|
|
yield handler
|
|
finally:
|
|
logger.removeHandler(handler)
|
|
logger.setLevel(previous)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("method", "level"),
|
|
[
|
|
# Several lines per second during a download, and TRACE is the level a user turns on for
|
|
# their own session — so the noisy half is exactly the half the consent gates.
|
|
("debug", "TRACE"),
|
|
("info", "DEBUG"),
|
|
("warning", "WARNING"),
|
|
("error", "ERROR"),
|
|
],
|
|
)
|
|
def test_each_ytdlp_level_lands_where_it_belongs(sink, method, level):
|
|
getattr(formats._YDL_LOG, method)("something happened")
|
|
|
|
(line,) = sink.objects()
|
|
assert line["level"] == level
|
|
assert line["event"] == "download.ytdlp"
|
|
assert "something happened" in line["msg"]
|
|
|
|
|
|
def test_the_source_url_does_not_travel_with_the_explanation(sink):
|
|
formats._YDL_LOG.error(f"ERROR: [youtube] {URL}: Requested format is not available")
|
|
|
|
(line,) = sink.objects()
|
|
# The host survives — it is what says which extractor is failing. The rest is a row of this
|
|
# person's download history, and the job id already names the job on every one of these lines.
|
|
assert "www.youtube.com/…" in line["msg"]
|
|
assert "dQw4w9WgXcQ" not in json.dumps(line)
|
|
assert "list=PLsomething" not in json.dumps(line)
|
|
assert "Requested format is not available" in line["msg"]
|
|
|
|
|
|
def test_a_line_with_no_url_is_left_alone(sink):
|
|
formats._YDL_LOG.warning("nsig extraction failed: Some formats may be missing")
|
|
|
|
(line,) = sink.objects()
|
|
assert "nsig extraction failed: Some formats may be missing" in line["msg"]
|
|
|
|
|
|
def test_the_download_options_carry_the_logger():
|
|
opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None)
|
|
|
|
assert opts["logger"] is formats._YDL_LOG
|
|
|
|
|
|
def test_a_source_adapter_cannot_silence_it():
|
|
"""`extra` is the adapter's own options (YouTube's player-client/POT strategy), merged wholesale.
|
|
The logger is set after that merge for the same reason `cookiefile` and the size cap are: an
|
|
adapter must not be able to take away the one account of what went wrong that a failed job
|
|
leaves behind."""
|
|
opts = formats.build_ydl_opts(
|
|
{}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}
|
|
)
|
|
|
|
assert opts["logger"] is formats._YDL_LOG
|