peter
358d722ce9
Round two of the review loop, on round one's own diff. Two of its findings were regressions the fixes themselves introduced, which is exactly what the loop is for: **A per-user refusal was writing onto a shared row.** MediaAssets are deduped across users, and the generic failure handler puts the message on the asset — right for an upstream failure, wrong for "your storage is full". Two users queuing the same video share one asset, so the first user hitting their cap failed the second user's job with the first one's private reason, telling someone with an empty library to free up space. The refusal now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to `pending` for whoever else holds it. Pinned by a test that drives `_download` directly — the refusal happens before extraction, so nothing needs stubbing. **The restore scripts restarted the app, undoing the restore.** `docker compose start` re-runs the container command, and that runs `alembic upgrade head` — so on the rollback path this documents as recommended (restore the dump, THEN pin the older tag) the newer image immediately re-applied the migrations the restore had just undone, and on a FAILED restore it put the app back on a half-restored schema. Both are the opposite of why the stop was added. Nothing is restarted automatically now; the scripts say what to run and when, and the docs' snippets say the same. **Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders that failure inline, next to a Retry button — three recoveries for one failure), and `useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out card's only banner, where nothing renders the reset flag to replace it. The two states are independent again; what changes is that `dismiss` peels ONE layer, which is what the original finding was actually about. Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403 detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed` since "bad port" misdescribes it; three docstrings narrowed to what the code actually does (the worker's byte check is per-job, not an aggregate ledger, and not on the edit path; the redirect residual leaks a reachability oracle, which the note had denied; the invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw` can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were missing for the round-one fixes — the account panel's error paths, the invalidation set, and the shared-asset case above. Gate green: vitest 685, pytest 950.
129 lines
6.9 KiB
Python
129 lines
6.9 KiB
Python
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
|
|
|
|
The worker stores the raw exception text (truncated) for technical detail, but a raw
|
|
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
|
|
to one of a small set of category codes; the frontend turns the code into a localized, human
|
|
explanation (and decides whether a retry makes sense). The raw text stays available behind a
|
|
"details" disclosure. This is the same string-matching approach the worker already trusts for
|
|
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
|
|
"""
|
|
|
|
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
|
|
#
|
|
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
|
|
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
|
|
# rule drifting apart is exactly how a category silently becomes unreachable.
|
|
#
|
|
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
|
|
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
|
|
# does, it must carry the original code rather than re-deriving one from this text.
|
|
#
|
|
# Other worker-raised texts (ffmpeg, "produced no output file", the urlguard refusal) are still
|
|
# inline literals at their raise sites — pinned by markers here, but not owned here.
|
|
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
|
|
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
|
|
|
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
|
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
|
|
_RULES: tuple[tuple[str, str], ...] = (
|
|
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
|
|
# first — it is our wording, so it can never be a false match on an upstream message.
|
|
("download storage is full", "quota_exceeded"),
|
|
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
|
|
("sign in to confirm your age", "login_required"),
|
|
("confirm your age", "login_required"),
|
|
("sign in to confirm you're not a bot", "login_required"),
|
|
("not a bot", "login_required"),
|
|
("sign in to confirm", "login_required"),
|
|
("members-only", "login_required"),
|
|
("available to this channel's members", "login_required"),
|
|
("join this channel", "login_required"),
|
|
("this video is only available to", "login_required"),
|
|
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
|
|
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
|
|
("authentication is required", "login_required"),
|
|
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
|
|
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
|
|
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
|
|
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
|
|
("log in to", "login_required"),
|
|
("login required", "login_required"),
|
|
("you must be logged in", "login_required"),
|
|
("only available for registered users", "login_required"),
|
|
("use --cookies", "login_required"),
|
|
("--cookies-from-browser", "login_required"),
|
|
# Geo-restricted.
|
|
("in your country", "geo_blocked"),
|
|
("available in your location", "geo_blocked"),
|
|
("not available from your location", "geo_blocked"),
|
|
("geo restricted", "geo_blocked"),
|
|
# An edit's source download went away — our own wording, and MORE specific than the generic
|
|
# "no longer available" below, so it must be matched first.
|
|
("source download is no longer available", "source_missing"),
|
|
("source file is missing", "source_missing"),
|
|
# Private / deleted / removed / terminated.
|
|
("private video", "unavailable"),
|
|
("video unavailable", "unavailable"),
|
|
("no longer available", "unavailable"),
|
|
("has been removed", "unavailable"),
|
|
("removed by the uploader", "unavailable"),
|
|
("account associated with this video has been terminated", "unavailable"),
|
|
("this video is not available", "unavailable"),
|
|
("this video has been removed", "unavailable"),
|
|
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
|
|
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
|
|
("unsupported url", "unsupported_host"),
|
|
("no video could be found in this url", "unsupported_host"),
|
|
# DRM.
|
|
("drm protected", "drm"),
|
|
("drm-protected", "drm"),
|
|
# No usable media format.
|
|
("requested format is not available", "format_unavailable"),
|
|
("only images are available", "format_unavailable"),
|
|
("no video formats found", "format_unavailable"),
|
|
("unable to extract", "format_unavailable"),
|
|
# Our own post-processing (ffmpeg) step failed.
|
|
("ffmpeg failed", "postprocess"),
|
|
("ffmpeg stalled", "postprocess"),
|
|
("postprocessing", "postprocess"),
|
|
("produced no output file", "postprocess"),
|
|
("no output file", "postprocess"),
|
|
# Network / transport.
|
|
("unable to download", "network"),
|
|
("http error", "network"),
|
|
("timed out", "network"),
|
|
("read timed out", "network"),
|
|
("temporary failure in name resolution", "network"),
|
|
("connection reset", "network"),
|
|
("connection refused", "network"),
|
|
("failed to resolve", "network"),
|
|
)
|
|
|
|
|
|
def classify(msg: str | None) -> str:
|
|
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
|
|
if not msg:
|
|
return "unknown"
|
|
low = msg.lower()
|
|
for marker, category in _RULES:
|
|
if marker in low:
|
|
return category
|
|
return "unknown"
|
|
|
|
|
|
def user_error_text(source_kind: str, msg: str, code: str) -> str:
|
|
"""The failure text to STORE on the job, given the raw message and its category.
|
|
|
|
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
|
|
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
|
|
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
|
|
stops the fetch aiming at our own network; this stops the remaining readout.
|
|
|
|
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
|
|
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
|
|
function rather than an inline branch in the worker so it can be pinned by a test — the raw
|
|
text leaking back is a security regression that would otherwise pass a green suite."""
|
|
if source_kind == "url" and code == "network":
|
|
return NETWORK_REDACTED_MESSAGE
|
|
return msg
|