peter
fe38dfbcca
The deep review of the branch raised 47 findings, 18 of which survived adversarial verification. Grouped by what they actually were: **The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one user over quota was the only one downloading with no ceiling at all. `check_enqueue` could not catch it: it ran while the job was queued and the user was still under the cap. The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision lives, and where it can be tested), and the worker now fails the job outright rather than proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help until space is freed. **Silent failures in the new account panel.** All three actions went through `quiet: true` endpoints, which suppress the global error dialog on the promise that the caller reports inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On the broken-SMTP instance this panel exists for, an admin would read silence as "the mail went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`, shared with the pending-access queue, so the two copies of these mutations cannot drift apart again — which is how half of this was lost in the first place. **A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps back one migration, but a release routinely ships several (0.80.0 shipped four): it drops the newest one's columns and leaves the schema at a revision the older image also cannot resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is now the recommended path, and the alembic route asks the target version for its own head revision instead of counting backwards. Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as a 500; the pull cooldown extended to the sibling route spending the same QuotaAction; restore stops the app before dropping its tables; the PowerShell cleanup can no longer decide the script's verdict on 5.1, nor report the wrong exit code; the documented one-liners hardened to match the scripts; the registry's retention window stated where pinning is advertised; the two banners made mutually exclusive so neither can shadow the other; `hover:bg-hover` (a utility this project never generated) replaced by the `selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to the two callers that discarded it. Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset banner UAT'd on localdev.
121 lines
6.4 KiB
Python
121 lines
6.4 KiB
Python
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
|
|
|
|
The worker stores the raw exception text (truncated) for technical detail, but a raw
|
|
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
|
|
to one of a small set of category codes; the frontend turns the code into a localized, human
|
|
explanation (and decides whether a retry makes sense). The raw text stays available behind a
|
|
"details" disclosure. This is the same string-matching approach the worker already trusts for
|
|
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
|
|
"""
|
|
|
|
# Failure texts the WORKER itself raises, as opposed to ones yt-dlp/ffmpeg produce. They live here,
|
|
# beside the rules that classify them, because a message and its category drifting apart is exactly
|
|
# how a category silently becomes unreachable — the rule below matches a fragment of each, and
|
|
# test_download_errors pins that they still classify as intended.
|
|
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
|
|
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
|
|
|
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
|
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
|
|
_RULES: tuple[tuple[str, str], ...] = (
|
|
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
|
|
# first — it is our wording, so it can never be a false match on an upstream message.
|
|
("download storage is full", "quota_exceeded"),
|
|
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
|
|
("sign in to confirm your age", "login_required"),
|
|
("confirm your age", "login_required"),
|
|
("sign in to confirm you're not a bot", "login_required"),
|
|
("not a bot", "login_required"),
|
|
("sign in to confirm", "login_required"),
|
|
("members-only", "login_required"),
|
|
("available to this channel's members", "login_required"),
|
|
("join this channel", "login_required"),
|
|
("this video is only available to", "login_required"),
|
|
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
|
|
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
|
|
("authentication is required", "login_required"),
|
|
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
|
|
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
|
|
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
|
|
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
|
|
("log in to", "login_required"),
|
|
("login required", "login_required"),
|
|
("you must be logged in", "login_required"),
|
|
("only available for registered users", "login_required"),
|
|
("use --cookies", "login_required"),
|
|
("--cookies-from-browser", "login_required"),
|
|
# Geo-restricted.
|
|
("in your country", "geo_blocked"),
|
|
("available in your location", "geo_blocked"),
|
|
("not available from your location", "geo_blocked"),
|
|
("geo restricted", "geo_blocked"),
|
|
# An edit's source download went away — our own wording, and MORE specific than the generic
|
|
# "no longer available" below, so it must be matched first.
|
|
("source download is no longer available", "source_missing"),
|
|
("source file is missing", "source_missing"),
|
|
# Private / deleted / removed / terminated.
|
|
("private video", "unavailable"),
|
|
("video unavailable", "unavailable"),
|
|
("no longer available", "unavailable"),
|
|
("has been removed", "unavailable"),
|
|
("removed by the uploader", "unavailable"),
|
|
("account associated with this video has been terminated", "unavailable"),
|
|
("this video is not available", "unavailable"),
|
|
("this video has been removed", "unavailable"),
|
|
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
|
|
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
|
|
("unsupported url", "unsupported_host"),
|
|
("no video could be found in this url", "unsupported_host"),
|
|
# DRM.
|
|
("drm protected", "drm"),
|
|
("drm-protected", "drm"),
|
|
# No usable media format.
|
|
("requested format is not available", "format_unavailable"),
|
|
("only images are available", "format_unavailable"),
|
|
("no video formats found", "format_unavailable"),
|
|
("unable to extract", "format_unavailable"),
|
|
# Our own post-processing (ffmpeg) step failed.
|
|
("ffmpeg failed", "postprocess"),
|
|
("ffmpeg stalled", "postprocess"),
|
|
("postprocessing", "postprocess"),
|
|
("produced no output file", "postprocess"),
|
|
("no output file", "postprocess"),
|
|
# Network / transport.
|
|
("unable to download", "network"),
|
|
("http error", "network"),
|
|
("timed out", "network"),
|
|
("read timed out", "network"),
|
|
("temporary failure in name resolution", "network"),
|
|
("connection reset", "network"),
|
|
("connection refused", "network"),
|
|
("failed to resolve", "network"),
|
|
)
|
|
|
|
|
|
def classify(msg: str | None) -> str:
|
|
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
|
|
if not msg:
|
|
return "unknown"
|
|
low = msg.lower()
|
|
for marker, category in _RULES:
|
|
if marker in low:
|
|
return category
|
|
return "unknown"
|
|
|
|
|
|
def user_error_text(source_kind: str, msg: str, code: str) -> str:
|
|
"""The failure text to STORE on the job, given the raw message and its category.
|
|
|
|
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
|
|
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
|
|
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
|
|
stops the fetch aiming at our own network; this stops the remaining readout.
|
|
|
|
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
|
|
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
|
|
function rather than an inline branch in the worker so it can be pinned by a test — the raw
|
|
text leaking back is a security regression that would otherwise pass a green suite."""
|
|
if source_kind == "url" and code == "network":
|
|
return NETWORK_REDACTED_MESSAGE
|
|
return msg
|