fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
This commit is contained in:
@@ -59,7 +59,10 @@ def sink():
|
||||
("debug", "TRACE"),
|
||||
("info", "DEBUG"),
|
||||
("warning", "WARNING"),
|
||||
("error", "ERROR"),
|
||||
# yt-dlp calls `error` from `trouble()` just before it raises, on every failed ATTEMPT —
|
||||
# the pipeline's fallback client set then often succeeds. The job's verdict is the
|
||||
# pipeline's `download.job.failed`, not this.
|
||||
("error", "WARNING"),
|
||||
],
|
||||
)
|
||||
def test_each_ytdlp_level_lands_where_it_belongs(sink, method, level):
|
||||
@@ -83,6 +86,45 @@ def test_the_source_url_does_not_travel_with_the_explanation(sink):
|
||||
assert "Requested format is not available" in line["msg"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("line", "kept", "gone"),
|
||||
[
|
||||
# The shape nearly every yt-dlp warning and error has: no URL, the id as a prefix.
|
||||
(
|
||||
"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable",
|
||||
"ERROR: [youtube] …: Video unavailable",
|
||||
"dQw4w9WgXcQ",
|
||||
),
|
||||
# The generic extractor's "id" is the URL's own basename.
|
||||
(
|
||||
"WARNING: [generic] my-private-holiday-clip: Falling back on generic information",
|
||||
"WARNING: [generic] …: Falling back on generic information",
|
||||
"my-private-holiday-clip",
|
||||
),
|
||||
# `Destination` is yt-dlp's word, not an id; the path after it is built from the title.
|
||||
(
|
||||
"[download] Destination: /downloads/Some_Channel/Season_2024/A_Private_Title_a7.mp4",
|
||||
"[download] Destination: …",
|
||||
"A_Private_Title",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_what_names_the_download_comes_out_of_real_ytdlp_shapes(sink, line, kept, gone):
|
||||
formats._YDL_LOG.warning(line)
|
||||
|
||||
(obj,) = sink.objects()
|
||||
assert obj["msg"] == kept
|
||||
assert gone not in json.dumps(obj)
|
||||
|
||||
|
||||
def test_a_download_error_is_scrubbed_the_same_way_wherever_it_is_logged():
|
||||
"""yt-dlp raises the text it has just logged, and the pipeline logs the exception again one
|
||||
line later (`download.job.failed`). The same scrub, or that line puts the URL back."""
|
||||
text = "ERROR: Unsupported URL: https://example.org/private/path?id=123"
|
||||
|
||||
assert formats.log_safe(text) == "ERROR: Unsupported URL: example.org/…"
|
||||
|
||||
|
||||
def test_a_line_with_no_url_is_left_alone(sink):
|
||||
formats._YDL_LOG.warning("nsig extraction failed: Some formats may be missing")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user