fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
`formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
file paths, and the job-failed, refused, edit-failed and backfill lines use
it. Done lines log the asset id, not the title-built path. The PII contract
now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
a process's last batch closes the repeat window, so a short-lived download
child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
`download.job.failed` with a cause; they left no line at all before. The
crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
carry reason `transport`. Subscribe recognises a duplicate by reason code
only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
This commit is contained in:
@@ -90,6 +90,40 @@ def test_a_flood_of_one_message_is_stored_up_to_the_limit_and_then_summarised(st
|
||||
assert notice.attrs["template"] == "RSS poll skipped for channel %s: HTTP 404"
|
||||
|
||||
|
||||
def test_a_pass_through_template_groups_by_its_text_not_by_the_bare_placeholder(stored):
|
||||
"""`"%s"` is how the yt-dlp adapter hands over yt-dlp's own line, and as a key it says nothing:
|
||||
every distinct warning would share one bucket and the thirty-first — whatever it said — would be
|
||||
dropped as a "repeat" of the first. The text groups instead, numbers folded, so a retry storm is
|
||||
still one repeat and the warning that explains the failure still gets through."""
|
||||
sink, log, rows = stored
|
||||
limit = logsink.REPEATS_PER_MINUTE
|
||||
for i in range(limit + 10):
|
||||
log.warning("%s", f"[download] Got error: timed out. Retrying fragment {i} (1/10)...")
|
||||
log.warning("%s", "[youtube] …: Requested format is not available")
|
||||
stored_rows = rows()
|
||||
assert len(stored_rows) == limit + 1
|
||||
assert stored_rows[-1].msg == "[youtube] …: Requested format is not available"
|
||||
sink._repeats.started -= logsink.REPEAT_WINDOW_SECONDS
|
||||
notice = rows()[-1]
|
||||
assert notice.attrs["suppressed"] == 10
|
||||
assert notice.attrs["template"].startswith(
|
||||
"[download] Got error: timed out. Retrying fragment #"
|
||||
)
|
||||
|
||||
|
||||
def test_a_process_that_ends_inside_the_minute_still_says_what_it_dropped(stored):
|
||||
"""Every download child lives well under the limiter's minute. Its last batch closes the window
|
||||
early, or the repeats it dropped would go unreported."""
|
||||
sink, log, rows = stored
|
||||
for i in range(logsink.REPEATS_PER_MINUTE + 3):
|
||||
log.warning("RSS poll skipped for channel %s: HTTP 404", f"UC{i}")
|
||||
rows()
|
||||
sink.write_pending(final=True)
|
||||
notice = rows()[-1]
|
||||
assert notice.event == "log.repeats_suppressed"
|
||||
assert notice.attrs["suppressed"] == 3
|
||||
|
||||
|
||||
def test_request_lines_are_never_treated_as_a_flood(stored):
|
||||
_sink, log, rows = stored
|
||||
for _ in range(logsink.REPEATS_PER_MINUTE + 5):
|
||||
|
||||
Reference in New Issue
Block a user