Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was
answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN
then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden".
Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in
full. So the wall moves between the player response and the stream, and the text it wears there was
costing the download twice.
retriable: the format URLs belong to the client that minted them, so the next client set is a real
second chance — but this text classifies as `network` and nothing in the YouTube marker list matched
it, so the ladder ended the job on attempt 1 and never tried the fallback.
catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network
error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly
as narrowly as the measurement: another host keeps the generic `network` category, and so does a
download that already carried a cookie — telling a signed-in user to sign in is the failure the
catalog exists to avoid. EN + HU copy.
A YouTube link can fail with "Sign in to confirm you're not a bot" while every
other YouTube download works — YouTube gates INDIVIDUAL videos behind a
signed-in session. Measured on prod job 273: all twelve of yt-dlp's player
clients fail on that one video while its neighbours download fine, the bgutil
PO-token sidecar is live and answering, and the newest yt-dlp fails identically.
So the download is not broken by our stack or a stale yt-dlp, and no retry,
client switch or upgrade gets past it. A cookie is the only way in.
The failure copy already sends the user to Settings -> Account -> Download
authentication, but YouTube was not in the cookie host registry, so that
dropdown could not help them: the app asked for something it did not offer.
credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be,
youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the
private-asset cache salt all derive from the row, so that is the whole wiring.
adapters: a signed-in YouTube download runs only cookie-capable player clients.
Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies
attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set
it empties that way raises "No player clients have been requested" — which is
not a per-client flake it retries past. Our primary set is android_vr, one of
those, so without this the cookie would have failed the download on attempt 1
and never reached the POT-backed web fallback it exists for. A test pins the
client list against the installed yt-dlp so a version bump turns the gate red
instead of silently un-authing every YouTube download, and the empty-set message
joins the retriable markers as the second line of defence.
errors: catalog entries for the wall with and without a cookie, so a user who
has already added cookies is told they were rejected (YouTube rotates exported
ones) rather than to add some. The marker is quote-agnostic: YouTube writes the
apostrophe as U+2019, so "you're not a bot" with a straight quote never matches
the real text. EN + HU copy.
Since a Facebook cookie was added on prod, every reel came down titled
"Video": measured on dev, a signed-in read gives yt-dlp the placeholder
"Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text
is still in the description. A new SourceAdapter.title hook takes the whole
first line of the description in that case, and both title writes (the early
fill and the finished download) go through it, so the row, the file name and
the share preview carry the post's text again. The line is not cut with an
ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to
".", which read as a finished sentence (seen in dev UAT).
The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the
browser-safe step (the audio spec is clamped to container mp4 too) and served
as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now
skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4.
Audio downloads version their cache identity on their own (v5), so a cached
remuxed audio re-derives without re-downloading every cached video.
Measured on dev: with the user's cookie some reels still answer "Cannot parse
data" and only extract when the request impersonates Chrome. Impersonation is
a second attempt behind that one error, only when a cookie is attached — links
that already download with the cookie keep their plain path, and without a
cookie the same error is the login wall.
curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it
every target silently reports "unavailable"); the lock only gains the new
packages. A test fails the gate if the locked pair can no longer impersonate.
resolve_adapter matched sources by loose substring over the netloc, so a
look-alike host ('myyoutube.com', 'youtube.com.evil') misresolved. Match the
host exactly or as a subdomain via a _host_is() helper, and identify from the
netloc of BOTH source_ref and yt-dlp's webpage_url — the latter by host only,
never a substring over its path. This also restores a host signal to the
early-fill path (resolve_adapter(None, None, info)), which has no source_ref,
so a Facebook asset's uploader_url no longer depends on extractor_key being
present in the progress-hook info_dict.
Tests: look-alike hosts stay generic while real subdomains resolve; a
webpage_url netloc identifies the source when extractor_key is absent.
- resolve_adapter: identify from the yt-dlp extractor_key only, not the full
webpage_url (a sibling source's name in a URL path no longer misresolves);
add youtube-nocookie.com to the YouTube host match so nocookie embeds keep
the POT/player-client extraction strategy.
- _download: re-resolve the adapter with the post-extraction info dict so the
.nfo uniqueid type and uploader_url agree on one source (dedups the inline
resolve too).
- MediaMeta.uniqueid_type defaults to source-neutral "generic" instead of
"youtube", so an un-updated caller can never falsely label a non-YouTube
download as youtube.
- Tests: lock extractor-key-only resolution + nocookie host; default nfo type.
Consolidates source-specific policy (youtube / facebook / reddit / generic) that was scattered
across worker.py, backfill.py, service.py, formats.py and storage.py with no shared source
identity. app/downloads/adapters.py holds the base (= generic yt-dlp behaviour) + per-source
overrides; resolve_adapter(source_kind, source_ref, info) picks one by source_kind, the yt-dlp
extractor_key, or the URL host, falling back to generic. yt-dlp stays the engine — the adapter is
policy only.
Moved into adapters (behaviour-equivalent):
- uploader_url derivation (YouTube channel_url / Facebook uploader_id / reddit None) — removes the
worker/backfill duplication (build_uploader_url + _should_refetch gone).
- the player-client fallback sets + retriable markers + YouTube extractor_args/POT (build_ydl_opts
now takes source-agnostic opts plus an `extra` dict from the adapter).
- source_url reconstruction (service.source_url delegates to the adapter).
Two intentional (approved) behaviour changes:
- the player-client fallback is now YouTube-only; non-YouTube sources make a single extraction
attempt instead of a pointless double attempt with youtube-only client args.
- the .nfo <uniqueid type=...> follows the source (adapter.nfo_uniqueid_type) instead of the
hardcoded "youtube" that mislabelled every url-source download.
Deliberately left generic (documented non-moves): title cleaning (normalize_title is harmless
across sources), the classify marker table, thumbnail_to_jpg, the format selectors, and og.py's
YouTube-hqdefault og:image preference (a self-contained thumbnail heuristic on the share path).
Verified end-to-end on localdev across all four adapters: downloads succeed and uploader_url + the
.nfo uniqueid type are correct per source.
peter