fix(downloads): E11.5 epic-end review round 1

A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.

Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.

UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.

Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
This commit is contained in:
2026-09-14 00:50:05 +02:00
parent b17d92d7c2
commit e887233210
20 changed files with 610 additions and 112 deletions
+31 -11
View File
@@ -19,6 +19,14 @@ from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
# below and the failure catalog (errors.py) decide on it, so the wording exists once.
FACEBOOK_PARSE_WALL = "cannot parse data"
# A source's own verdicts on a link, raised from `resolve_url`. errors.py classifies them; it imports
# them from here because this module must not import errors (errors → credentials → adapters).
POST_WITHOUT_VIDEO_MESSAGE = "No single video was found in this post."
POST_NOT_SHOWN_MESSAGE = "Facebook did not show this post to the signed-in session."
class SourceAdapter:
"""Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
@@ -26,10 +34,11 @@ class SourceAdapter:
key = "generic"
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
def resolve_url(self, url: str, ydl_opts: dict) -> str | None:
"""The link to hand yt-dlp for `url`, resolved BEFORE the first attempt, or None when the source
itself shows there is no single video to download there. `ydl_opts` are the download's own
options (cookie, timeouts, address family) for any page this needs to read. Generic: `url`."""
def resolve_url(self, url: str, ydl_opts: dict) -> str:
"""The link to hand yt-dlp for `url`, resolved BEFORE the first attempt. Raises a RuntimeError
carrying one of this module's verdicts when the source itself shows there is no single video to
download there. `ydl_opts` are the download's own options (cookie, timeouts, address family) for
any page this needs to read. Generic: `url`."""
return url
def client_sets(self, authed: bool = False) -> list:
@@ -110,11 +119,10 @@ class FacebookAdapter(SourceAdapter):
# cookie keep their proven path, and only the ones stuck at the wall go through curl_cffi. Without
# a cookie the same error is the login wall, which no disguise gets past.
_IMPERSONATE = "impersonate"
_WALL = "cannot parse data"
# A post page is ~4–5 MB of inline JSON; anything far past that is not a post page.
_POST_PAGE_MAX_BYTES = 16 * 1024 * 1024
def resolve_url(self, url: str, ydl_opts: dict) -> str | None:
def resolve_url(self, url: str, ydl_opts: dict) -> str:
# Measured (E11.5 S4): given a group post's link, yt-dlp either finds no formats or — worse —
# downloads ANOTHER video from the page's feed data. The post's own video is the one Video node
# in the page scripts that carry the post id; its watch link downloads (plain first, then the
@@ -124,9 +132,16 @@ class FacebookAdapter(SourceAdapter):
return url
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
video_id = facebook_post_video_id(raw.decode("utf-8", "replace"), post_id)
# Never fall back to the post link: that is exactly where the wrong video comes from.
return f"https://www.facebook.com/watch/?v={video_id}" if video_id else None
page = raw.decode("utf-8", "replace")
# Measured (E11.5 review): a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise RuntimeError(POST_NOT_SHOWN_MESSAGE)
video_id = facebook_post_video_id(page, post_id)
if video_id is None:
# Never fall back to the post link: that is exactly where the wrong video comes from.
raise RuntimeError(POST_WITHOUT_VIDEO_MESSAGE)
return f"https://www.facebook.com/watch/?v={video_id}"
def client_sets(self, authed: bool = False) -> list:
return [None, self._IMPERSONATE] if authed else [None]
@@ -137,7 +152,7 @@ class FacebookAdapter(SourceAdapter):
return {}
def retriable(self, error_text: str) -> bool:
return self._WALL in error_text.lower()
return FACEBOOK_PARSE_WALL in error_text.lower()
def uploader_url(self, info: dict) -> str | None:
# Facebook gives neither channel_url nor uploader_url, but exposes a numeric `uploader_id`
@@ -157,6 +172,11 @@ _SCRIPT_BLOCK = re.compile(r"<script[^>]*>(.*?)</script>", re.S)
_VIDEO_NODE = re.compile(r'"__typename":"Video","id":"(\d+)"')
def _quoted(post_id: str) -> str:
# How the page JSON carries the id: quoted, so a longer id that merely starts with it never counts.
return f'"{post_id}"'
def facebook_group_post_id(url: str) -> str | None:
"""The post id of a Facebook group post link (`/groups/<group>/permalink|posts/<id>/`), else None."""
parsed = urlparse(url)
@@ -170,7 +190,7 @@ def facebook_post_video_id(page: str, post_id: str) -> str | None:
"""The id of the one video attached to `post_id`, read from its page: the Video nodes of the script
blocks that carry the post id (the page's other ~10 videos — feed, reels — sit in other blocks).
None for no video AND for more than one: guessing between two would be the wrong-video bug again."""
quoted = f'"{post_id}"'
quoted = _quoted(post_id)
ids = {
vid
for block in _SCRIPT_BLOCK.findall(page)