peter
90ee34a1b4
Given a group post's link, yt-dlp either found no formats or took another video from the page's feed data, so a "successful" retry of such a post was a wrong download. With the user's cookie the Facebook adapter now reads the post page, takes the single video node in the script blocks that carry the post id, and downloads that video's watch link (the Chrome retry does the rest). No single video in the post fails with its own message and the group-post copy, reworded to say the post has no video; the post link is never handed to yt-dlp as a fallback. Without a cookie nothing changes.
244 lines
12 KiB
Python
244 lines
12 KiB
Python
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code — and, where a site's
|
|
failure has been measured, into that site's entry of the failure catalog.
|
|
|
|
The worker stores the raw exception text (truncated) for technical detail, but a raw
|
|
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
|
|
to one of a small set of category codes; the frontend turns the code into a localized, human
|
|
explanation (and decides whether a retry makes sense). The raw text stays available behind a
|
|
"details" disclosure. This is the same string-matching approach the worker already trusts for
|
|
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
|
|
|
|
**The failure catalog** (`_CATALOG`, E11.5). A category is generic: "Sign-in required" cannot tell a
|
|
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
|
|
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
|
|
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
|
|
its own sentence for — why it failed on that site, and what the user can do. `diagnose()` tries the
|
|
catalog first and falls back to the category. How an entry is added, every time a failure is reported:
|
|
1. reproduce it on dev (with and without the user's cookie) and keep the verbatim yt-dlp text;
|
|
2. add the entry below, scoped as narrowly as the measurement allows (a marker that is yt-dlp's
|
|
generic wording for every site must never match without its host);
|
|
3. pin the real text in `tests/test_download_errors.py`;
|
|
4. add `downloads.errors.fail.catalog.<detail>.{short,body}` in EN and HU (a contract test checks it;
|
|
the group is not called `detail`, which `actionErrors.contract` reads as a raw `HttpError.detail`).
|
|
"""
|
|
|
|
from dataclasses import dataclass
|
|
from typing import NamedTuple
|
|
from urllib.parse import urlparse
|
|
|
|
from app.downloads.credentials import host_key_for_url
|
|
|
|
# Worker-authored strings that this module owns because each is coupled to `_RULES`.
|
|
#
|
|
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
|
|
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
|
|
# rule drifting apart is exactly how a category silently becomes unreachable.
|
|
#
|
|
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
|
|
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
|
|
# text (`worker._process_job`), so it has a rule of its own that gives the category back.
|
|
#
|
|
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
|
|
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
|
|
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
|
|
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
|
|
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
|
|
# POST_WITHOUT_VIDEO_MESSAGE is raised by the worker when a source adapter read a post's own page and
|
|
# found no single video in it (`SourceAdapter.resolve_url` → None; a Facebook group post, E11.5 S4).
|
|
# Its rule gives the category back without a link; the catalog entry gives the site's sentence.
|
|
POST_WITHOUT_VIDEO_MESSAGE = "No single video was found in this post."
|
|
|
|
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
|
|
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
|
|
_RULES: tuple[tuple[str, str], ...] = (
|
|
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
|
|
# first — it is our wording, so it can never be a false match on an upstream message.
|
|
("download storage is full", "quota_exceeded"),
|
|
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
|
|
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
|
|
# Our own verdict on a post link whose page holds no single video — our wording too.
|
|
(POST_WITHOUT_VIDEO_MESSAGE.lower(), "format_unavailable"),
|
|
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
|
|
("sign in to confirm your age", "login_required"),
|
|
("confirm your age", "login_required"),
|
|
("sign in to confirm you're not a bot", "login_required"),
|
|
("not a bot", "login_required"),
|
|
("sign in to confirm", "login_required"),
|
|
("members-only", "login_required"),
|
|
("available to this channel's members", "login_required"),
|
|
("join this channel", "login_required"),
|
|
("this video is only available to", "login_required"),
|
|
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
|
|
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
|
|
("authentication is required", "login_required"),
|
|
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
|
|
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
|
|
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
|
|
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
|
|
("log in to", "login_required"),
|
|
("login required", "login_required"),
|
|
("you must be logged in", "login_required"),
|
|
("only available for registered users", "login_required"),
|
|
("use --cookies", "login_required"),
|
|
("--cookies-from-browser", "login_required"),
|
|
# Geo-restricted.
|
|
("in your country", "geo_blocked"),
|
|
("available in your location", "geo_blocked"),
|
|
("not available from your location", "geo_blocked"),
|
|
("geo restricted", "geo_blocked"),
|
|
# An edit's source download went away — our own wording, and MORE specific than the generic
|
|
# "no longer available" below, so it must be matched first.
|
|
("source download is no longer available", "source_missing"),
|
|
("source file is missing", "source_missing"),
|
|
# Private / deleted / removed / terminated.
|
|
("private video", "unavailable"),
|
|
("video unavailable", "unavailable"),
|
|
("no longer available", "unavailable"),
|
|
("has been removed", "unavailable"),
|
|
("removed by the uploader", "unavailable"),
|
|
("account associated with this video has been terminated", "unavailable"),
|
|
("this video is not available", "unavailable"),
|
|
("this video has been removed", "unavailable"),
|
|
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
|
|
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
|
|
("unsupported url", "unsupported_host"),
|
|
("no video could be found in this url", "unsupported_host"),
|
|
# DRM.
|
|
("drm protected", "drm"),
|
|
("drm-protected", "drm"),
|
|
# No usable media format.
|
|
("requested format is not available", "format_unavailable"),
|
|
("only images are available", "format_unavailable"),
|
|
("no video formats found", "format_unavailable"),
|
|
("unable to extract", "format_unavailable"),
|
|
# Our own post-processing (ffmpeg) step failed.
|
|
("ffmpeg failed", "postprocess"),
|
|
("ffmpeg stalled", "postprocess"),
|
|
("postprocessing", "postprocess"),
|
|
("produced no output file", "postprocess"),
|
|
("no output file", "postprocess"),
|
|
# Network / transport.
|
|
("unable to download", "network"),
|
|
("http error", "network"),
|
|
("timed out", "network"),
|
|
("read timed out", "network"),
|
|
("temporary failure in name resolution", "network"),
|
|
("connection reset", "network"),
|
|
("connection refused", "network"),
|
|
("failed to resolve", "network"),
|
|
)
|
|
|
|
|
|
class Diagnosis(NamedTuple):
|
|
"""What a failure was: the generic `code` (category) and, when a catalog entry matched, its
|
|
host-specific `detail`."""
|
|
|
|
code: str
|
|
detail: str | None = None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class _Entry:
|
|
"""One measured failure of one site. It matches when the download URL belongs to `host`
|
|
(`credentials.SUPPORTED_HOSTS`), every one of `markers` is in the message, the cookie state is
|
|
`cookie` (None: either), and — when `link` is given — one of those fragments is in the URL path."""
|
|
|
|
host: str
|
|
markers: tuple[str, ...]
|
|
code: str
|
|
detail: str
|
|
cookie: bool | None = None
|
|
link: tuple[str, ...] = ()
|
|
|
|
|
|
_CATALOG: tuple[_Entry, ...] = (
|
|
# Facebook answers a signed-out client with a page yt-dlp cannot parse — an 18+ reel, a closed
|
|
# group's post (measured 2026-09-13: prod jobs 145, 156, 256 download once the owner's cookie is
|
|
# attached). "Cannot parse data" is yt-dlp's generic wording for a broken extractor on EVERY site,
|
|
# which is why the entry is scoped to Facebook and never becomes a general rule.
|
|
_Entry(
|
|
host="facebook",
|
|
markers=("[facebook]", "cannot parse data"),
|
|
code="login_required",
|
|
detail="facebook.login_wall",
|
|
cookie=False,
|
|
),
|
|
# The same text WITH a cookie: an expired / signed-out cookie, or a page that also needs
|
|
# impersonation (prod job 257). Telling this user to add cookies would be wrong — they have.
|
|
_Entry(
|
|
host="facebook",
|
|
markers=("[facebook]", "cannot parse data"),
|
|
code="login_required",
|
|
detail="facebook.cookie_rejected",
|
|
cookie=True,
|
|
),
|
|
# A group post with no single video of its own. With a cookie the adapter reads the post's page
|
|
# and says so itself (POST_WITHOUT_VIDEO_MESSAGE); without one yt-dlp is given the post link and
|
|
# answers "No video formats found" (measured on prod jobs 66 and 145, whose videos are reshared
|
|
# clips). Only for group links: a reel with no formats is a different failure.
|
|
_Entry(
|
|
host="facebook",
|
|
markers=(POST_WITHOUT_VIDEO_MESSAGE.lower(),),
|
|
code="format_unavailable",
|
|
detail="facebook.group_post_video",
|
|
link=("/groups/",),
|
|
),
|
|
_Entry(
|
|
host="facebook",
|
|
markers=("[facebook]", "no video formats found"),
|
|
code="format_unavailable",
|
|
detail="facebook.group_post_video",
|
|
link=("/groups/",),
|
|
),
|
|
)
|
|
|
|
|
|
def classify(msg: str | None) -> str:
|
|
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
|
|
if not msg:
|
|
return "unknown"
|
|
low = msg.lower()
|
|
for marker, category in _RULES:
|
|
if marker in low:
|
|
return category
|
|
return "unknown"
|
|
|
|
|
|
def diagnose(
|
|
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
|
|
) -> Diagnosis:
|
|
"""The catalog entry for this failure of this link, else just its category (see the module doc).
|
|
|
|
`url` is the address yt-dlp was given; `cookie_attached` whether the acting user's cookie went
|
|
with it. Without a URL only the category can be known."""
|
|
host = host_key_for_url(url) if msg and url else None
|
|
if host is not None:
|
|
low = msg.lower() # type: ignore[union-attr] — `host` is only set when `msg` is truthy
|
|
path = urlparse(url).path.lower()
|
|
for e in _CATALOG:
|
|
if (
|
|
e.host == host
|
|
and all(m in low for m in e.markers)
|
|
and (e.cookie is None or e.cookie == cookie_attached)
|
|
and (not e.link or any(fragment in path for fragment in e.link))
|
|
):
|
|
return Diagnosis(e.code, e.detail)
|
|
return Diagnosis(classify(msg))
|
|
|
|
|
|
def user_error_text(source_kind: str, msg: str, code: str) -> str:
|
|
"""The failure text to STORE on the job, given the raw message and its category.
|
|
|
|
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
|
|
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
|
|
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
|
|
stops the fetch aiming at our own network; this stops the remaining readout.
|
|
|
|
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
|
|
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
|
|
function rather than an inline branch in the worker so it can be pinned by a test — the raw
|
|
text leaking back is a security regression that would otherwise pass a green suite."""
|
|
if source_kind == "url" and code == "network":
|
|
return NETWORK_REDACTED_MESSAGE
|
|
return msg
|