feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do

A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.

- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
  (all-of markers, cookie state, link fragment) and falls back to the category.
  First entries: facebook.login_wall, facebook.cookie_rejected,
  facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
  failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
  with it; resume and redownload clear code and detail (redownload never
  cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
  "Download failed" where the first job said why. It is diagnosed again, with
  the cookie state from credentials.is_authed_asset (now shared with the
  cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
  copy, else the category. Copy in EN and HU; a contract test pins every
  backend entry to both locales.
This commit is contained in:
2026-09-13 21:53:25 +02:00
parent ca5e446fbd
commit 18c11ca437
20 changed files with 582 additions and 13 deletions
+104 -4
View File
@@ -1,4 +1,5 @@
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code — and, where a site's
failure has been measured, into that site's entry of the failure catalog.
The worker stores the raw exception text (truncated) for technical detail, but a raw
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
@@ -6,17 +7,36 @@ to one of a small set of category codes; the frontend turns the code into a loca
explanation (and decides whether a retry makes sense). The raw text stays available behind a
"details" disclosure. This is the same string-matching approach the worker already trusts for
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
**The failure catalog** (`_CATALOG`, E11.5). A category is generic: "Sign-in required" cannot tell a
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
its own sentence for — why it failed on that site, and what the user can do. `diagnose()` tries the
catalog first and falls back to the category. How an entry is added, every time a failure is reported:
1. reproduce it on dev (with and without the user's cookie) and keep the verbatim yt-dlp text;
2. add the entry below, scoped as narrowly as the measurement allows (a marker that is yt-dlp's
generic wording for every site must never match without its host);
3. pin the real text in `tests/test_download_errors.py`;
4. add `downloads.errors.fail.catalog.<detail>.{short,body}` in EN and HU (a contract test checks it;
the group is not called `detail`, which `actionErrors.contract` reads as a raw `HttpError.detail`).
"""
from dataclasses import dataclass
from typing import NamedTuple
from urllib.parse import urlparse
from app.downloads.credentials import host_key_for_url
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
# does, it must carry the original code rather than re-deriving one from this text.
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
# text (`worker._process_job`), so it has a rule of its own that gives the category back.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
@@ -30,6 +50,8 @@ _RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -101,6 +123,62 @@ _RULES: tuple[tuple[str, str], ...] = (
)
class Diagnosis(NamedTuple):
"""What a failure was: the generic `code` (category) and, when a catalog entry matched, its
host-specific `detail`."""
code: str
detail: str | None = None
@dataclass(frozen=True)
class _Entry:
"""One measured failure of one site. It matches when the download URL belongs to `host`
(`credentials.SUPPORTED_HOSTS`), every one of `markers` is in the message, the cookie state is
`cookie` (None: either), and — when `link` is given — one of those fragments is in the URL path."""
host: str
markers: tuple[str, ...]
code: str
detail: str
cookie: bool | None = None
link: tuple[str, ...] = ()
_CATALOG: tuple[_Entry, ...] = (
# Facebook answers a signed-out client with a page yt-dlp cannot parse — an 18+ reel, a closed
# group's post (measured 2026-09-13: prod jobs 145, 156, 256 download once the owner's cookie is
# attached). "Cannot parse data" is yt-dlp's generic wording for a broken extractor on EVERY site,
# which is why the entry is scoped to Facebook and never becomes a general rule.
_Entry(
host="facebook",
markers=("[facebook]", "cannot parse data"),
code="login_required",
detail="facebook.login_wall",
cookie=False,
),
# The same text WITH a cookie: an expired / signed-out cookie, or a page that also needs
# impersonation (prod job 257). Telling this user to add cookies would be wrong — they have.
_Entry(
host="facebook",
markers=("[facebook]", "cannot parse data"),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post whose video yt-dlp does not reach from the post's own link, although the video's
# own link downloads (prod job 66, with the cookie). Only for group links: a reel with no formats
# is a different failure.
_Entry(
host="facebook",
markers=("[facebook]", "no video formats found"),
code="format_unavailable",
detail="facebook.group_post_video",
link=("/groups/",),
),
)
def classify(msg: str | None) -> str:
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
if not msg:
@@ -112,6 +190,28 @@ def classify(msg: str | None) -> str:
return "unknown"
def diagnose(
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
) -> Diagnosis:
"""The catalog entry for this failure of this link, else just its category (see the module doc).
`url` is the address yt-dlp was given; `cookie_attached` whether the acting user's cookie went
with it. Without a URL only the category can be known."""
host = host_key_for_url(url) if msg and url else None
if host is not None:
low = msg.lower() # type: ignore[union-attr] — `host` is only set when `msg` is truthy
path = urlparse(url).path.lower()
for e in _CATALOG:
if (
e.host == host
and all(m in low for m in e.markers)
and (e.cookie is None or e.cookie == cookie_attached)
and (not e.link or any(fragment in path for fragment in e.link))
):
return Diagnosis(e.code, e.detail)
return Diagnosis(classify(msg))
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.