release: 0.118.0 (improvement/e59-small-residuals)

`draw.recency` weighs every watch of both affinities: an undated watch
and a future one now have a test (deleting either branch fails it). The
over-the-picture contrast test moves below the form-on-chrome test, so
each sits under the comment that explains it.

.gitleaksignore

@@ -0,0 +1,8 @@
+# gitleaks false positives, one fingerprint per finding (commit:file:rule:line). Each says why it is no secret.
+#
+# YouTube's public WEB InnerTube key: embedded in every youtube.com results page, read live and cached;
+# this constant is only the fallback (backend/app/youtube/innertube.py).
+8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/app/youtube/innertube.py:gcp-api-key:23
+# A recorded watch-next page's continuation cursors (continuationCommand.token): page pointers, not credentials.
+8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:768
+8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:792

CHANGELOG.md

@@ -7,6 +7,192 @@ edit the release notes and re-run the generator.
 
 Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
 
+## 0.118.0 — 2026-11-08
+
+### New
+
+- Surprise me now leans on the channels and genres you watch, and one pick in three stays a blind draw, so it remains a surprise.
+- On a connection too slow for a Plex video, playback waits and goes on where it was, and says how fast the video needs and how fast it arrives.
+
+### Fixed
+
+- A 24/7 stream's archive no longer adds years to a list's total: a total names the videos it could not count as "+N without length". The Feed sorted by Surprise me opens faster.
+- Plex films the server's GPU cannot decode (VC-1, older DivX or XviD) start in a few seconds instead of about 20. Text over the Plex picture is readable on any frame.
+
+## 0.117.0 — 2026-11-06
+
+### New
+
+- A share link's address now says what it opens: /file/ for a file, /folder/ for a folder, /watch/ for a video. Links you already sent keep working.
+- The upload dialog shows which folder the file goes to, and you can pick another one or make a new folder there. Opening Share copies a link at once.
+- The Channels tag chips show how many followed channels each holds. Continue watching in Plex has a See all, and a Plex item's Info shows the file size.
+
+### Fixed
+
+- An expired share link is marked as expired and is never copied. A crop of a video with wide pixels keeps exactly what its frame showed.
+- A Plex auto-skip waits 3 seconds into the marker before its countdown starts.
+
+## 0.116.3 — 2026-11-04
+
+### Fixed
+
+- If a stylesheet is lost on the way, the app now asks for it again, so a poor connection no longer leaves the page unstyled until a reload.
+
+## 0.116.2 — 2026-11-03
+
+### Fixed
+
+- A Plex video paused or muted in a background tab no longer loses its session after a minute, and if a session is lost anyway, playback comes back at once instead of after half a minute.
+
+## 0.116.1 — 2026-11-03
+
+### Fixed
+
+- A playlist now shows the order the server holds after a revert to YouTube, a sync or a change on another device, and the next drag no longer brings back the old one.
+- An Undo on a removed or moved row puts it back in the right place while another change is still saving.
+
+## 0.116.0 — 2026-11-03
+
+### New
+
+- In a playlist, tick rows with their box, Shift+click or x, then move or remove them together. An Undo puts them back at their places, and a single row's move has one too.
+- Drag rows onto another list in the rail: one row, or all the selected ones, also under a sort or a filter.
+
+### Fixed
+
+- Removing a video from inside the player no longer loses your place in the list when the player closes.
+
+## 0.115.0 — 2026-11-01
+
+### New
+
+- Clear watch progress is on every YouTube video's menu, on Home, in playlists and in the player's related videos. It forgets where you left a started video and changes nothing else.
+- Channels have an Added column, and an About card opens only when the pointer rests on it.
+- In Files, a press anywhere on a row plays the file.
+- The feed's toolbar pills are named by a glyph, with Sort and View on the filters' row.
+
+### Fixed
+
+- In a play queue, Previous and Next pass over a video that can't play, in the direction you were going, and say why when only such videos are left.
+- A Facebook download titled only “Facebook” now takes the post's text as its name.
+- The player's big view keeps its way out in sight on a tablet, and Tab no longer lands on controls hidden under it.
+
+## 0.114.3 — 2026-11-01
+
+### Fixed
+
+- Editing a file that is neither video nor audio now says why it can't be done, instead of a generic error.
+
+## 0.114.2 — 2026-10-30
+
+### Fixed
+
+- Security: a library the sign-in relies on is updated, closing two published vulnerabilities.
+
+## 0.114.1 — 2026-10-30
+
+### Fixed
+
+- Notifications follow a language switch: a notice on screen and every row in the notification center are said again in the new language, their buttons too.
+
+## 0.114.0 — 2026-10-29
+
+### New
+
+- Notifications: what asks for you comes first, then everything else as one stream, newest first. Read notices older than a week fold under Earlier, and Back returns to the row you left.
+
+### Fixed
+
+- Plex: a progress save sent just before an episode ends no longer leaves it unwatched, and a refused finish no longer sticks.
+- After a dialog closes, the keyboard focus returns to where it was, also inside a menu that stays open.
+
+## 0.113.0 — 2026-10-28
+
+### New
+
+- The address names what you see: tabs, list views, sorts and searches (the live YouTube search too), the YouTube player and the file viewer. A reload, a bookmark or a shared link opens the same view.
+- A link to a video reopens the player over its page once; Back closes it. A video the page no longer lists plays on its own and says so.
+
+### Fixed
+
+- Back and Forward step through the views as you saw them, across dialogs, players and reloads: a reload under an open player no longer leaves a Back step that does nothing.
+- Signed out on a Plex page, the sign-in card could show a button with no text.
+- The channel page's address names the channel and its own view, not the page under it.
+- A reload during a Plex binge reopens the episode playing now, not the one it started from.
+
+## 0.112.1 — 2026-10-27
+
+### Fixed
+
+- Security: the sign-in library is updated, closing two published vulnerabilities.
+
+## 0.112.0 — 2026-10-25
+
+### New
+
+- Player: a Related tab lists videos related to the one playing, leaving out what you have already watched (your uploaded YouTube history included). Channels you follow are marked, and you can switch between On topic and Trending.
+- Play a related video in place; Back returns to the one you opened. Admins set a daily per-user limit in Config › Quota (default 200, 0 turns it off).
+- Side panel: the list controls and the watched toggle are glyphs on one line, and Auto and Loop show when they are on, also under the pointer.
+
+### Fixed
+
+- Live search: a search that never reached YouTube no longer uses up a daily slot, a busy moment says so after a few seconds instead of waiting on, and a first page that failed offers Retry.
+- Menus: Play from the beginning wears the same ↺ glyph everywhere, and Reset its own eraser glyph.
+- On the day of this update, API searches made earlier that day stop counting toward the daily search limit, until Pacific midnight.
+
+## 0.111.0 — 2026-10-24
+
+### New
+
+- Channels › Discover from your history: the channels your uploaded YouTube history shows you watching often (five videos of five minutes or more) but not following.
+- Follow one, put it out of the list (with Undo), or open the videos you watched there and play them one by one or all in a row.
+
+### Fixed
+
+- A channel someone explored and then followed on YouTube is now part of the shared Library like any followed channel, and so are its videos; before, they could stay hidden and their watched marks never applied.
+
+## 0.110.0 — 2026-10-23
+
+### New
+
+- YouTube history: upload your Google Takeout export in Settings › Account. What you watched on YouTube is marked watched here (never a Short, never over a state you set), and your YouTube Watch later list joins yours.
+- A video the catalog does not know yet is remembered and marked once it arrives. The import's outcome reaches your inbox, and admins see each run on the Scheduler page.
+
+### Fixed
+
+- Scheduler: a channel's full-history backfill asked for again while one is running now runs after it, instead of being silently dropped.
+
+## 0.109.2 — 2026-10-23
+
+### Fixed
+
+- Plex player: when the server restarts mid-episode (an update), playback carries on from the buffer and picks up where you are once it is back, instead of showing "Playback couldn't start" and stopping where the buffer ended.
+
+## 0.109.1 — 2026-10-23
+
+### New
+
+- Development: the frontend is one folder per page (its page, rail, store and definition), with the layers held by lint and the shared /watch page held apart by a contract test; the header knows no page, and the feed and Plex each keep their own saved views.
+
+### Fixed
+
+- Saved views: a view you applied and then changed is still active after a visit to another page, in the feed and in Plex.
+
+## 0.109.0 — 2026-10-21
+
+### New
+
+- Accounts: a browser is signed in to one account at a time. The account menu's "Sign in as someone else" replaces it; the account list and switcher are gone. After this update everyone signs in once more.
+- Settings follow you: the Home start page, the channel manager and Plex playlist layouts, the discovery sorts and a dismissed onboarding or release banner are saved to your account, on every device. Performance mode stays per device.
+- Development: every key the browser stores declares its kind, held by a contract test; one Google connect route serves the sign-in and the YouTube access grants.
+
+### Fixed
+
+- Accounts: a tab still showing another account than the one signed in now reloads as that one and says why; a failed "Sign in as someone else" says what happened.
+- A link whose path starts with // no longer opens the error screen.
+- A member following a link to an admin page lands on the feed, with the rail and the address bar in step.
+- Google: an unknown access request is refused instead of quietly asking for read access, and a YouTube grant on an instance without Google says so.
+
 ## 0.108.1 — 2026-10-20
 
 ### New

CLAUDE.md

@@ -65,7 +65,7 @@ how it gets built. Concretely:
 `routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
 `downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
 `sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
-`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
+`security`, `logs`, `logsink`, `diagnostics`, `debugsession`, `draw`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
 The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
 sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
 the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
@@ -73,16 +73,24 @@ only along a declared edge and writes them only through that unit's service; cor
 `queries`. **Fence-held, with no exemptions** (`backend/tests/test_unit_fences.py`) — as far as an AST sees: what it
 cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
 [`backend/app/README.md`](backend/app/README.md) with the conventions.
-Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
-entry, `watch.html` → `watchMain.tsx`, held app-free by a contract test) → `moduleRegistry` route-chunks → module
-components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
-its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
-boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
-feature component or a domain API slice, and one layer down `lib/` may not import from `components/` at
-all, not even a type (`no-restricted-imports` in `eslint.config.js`, with its own test). A new generic
-component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
-Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
-`lib/api/*` slices over `core.req<T>`.
+Frontend, by folder under `src/`: `main.tsx` provider tree → `shell/` (`App`, header, nav rail, palette,
+boot; a share link (`/watch`, `/file`, `/folder`) boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
+app-free by a contract test) → **`modules/<page>/`** (one folder per page: the page, its rail, its store, its tests,
+and its `module.ts` definition — label, icon, gate, the lazy page and rail, its store — collected in nav order by
+`features/navigation/moduleRegistry`, which also mounts every store at the root; a new page is its folder, its id in
+`lib/urlState` and its entry there, plus its panel in `lib/panelLayout` if it has a rail) →
+**`features/<name>/`** (what several pages share: video cards, filters, downloads, the player, navigation) → the
+**platform layer `components/ui/`** (domain-free primitives: the popover engine and its menus, `Modal`/`Overlay`,
+`Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`,
+…) → `lib/`. The direction is one-way and **lint-enforced**, not just described (`local/layers`, with
+its own test): an import only goes down; another module is reached only through its **`index.ts`** (its
+public face — the stores, mounted at the root and read app-wide, which `features/` may read too) or
+lazily by `import()`; a platform component may not import a domain API slice, and `lib/` nothing above
+it, not even a type. A lazily imported page, rail or dialog that a static import drags onto the boot
+path fails the build (`chunkGuard.ts`). A new generic component goes in `ui/`; see
+[`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
+Data via react-query (`qk` factory), the signed-in account's `localStorage` (`storage.ts`), and server
+prefs; API through `lib/api/*` slices over `core.req<T>`.
 
 **Not wanted** (reviewed and rejected as over-engineering for this size): microservices, event bus,
 CQRS, a repository/DI framework, a router library, a generic CRUD-router factory or base serializer.
@@ -97,16 +105,20 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
 | Ownership check                      | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits                 |
 | Admin gate import                    | `app.auth`                                                                                                                                                                                                                                    | the `app.routes.admin` re-export                                                                                                                                                            |
 | Upstream failure status              | `422` with JSON detail                                                                                                                                                                                                                        | `502` (reserve 502 for raw byte-proxies)                                                                                                                                                    |
-| User-facing error                    | structured `detail` (`{code, reason}`)                                                                                                                                                                                                        | a bare English string under a bilingual UI                                                                                                                                                  |
+| User-facing error                    | structured `detail` (`{code, reason}`), its sentence named in `lib/api/refusalCodes.ts` (contract-held)                                                                                                                                       | a bare English string under a bilingual UI                                                                                                                                                  |
 | Mutation response                    | `{"ok": true, …}` (`{"deleted": id}`), test-held                                                                                                                                                                                              | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok`                                                                                                                                         |
 | Transaction commit                   | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module)                                                                                                                                                   | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
 | Admin knob storage                   | `sysconfig` (values)                                                                                                                                                                                                                          | env / `app_state` / a table of its own (the job intervals had one)                                                                                                                          |
 | Job status word                      | the `downloads/status.py` StrEnum                                                                                                                                                                                                             | loose `"canceled"`/`"cancelled"` literals                                                                                                                                                   |
 | Current time                         | inline `datetime.now(timezone.utc)`                                                                                                                                                                                                           | new `now_utc`/`_now` helpers                                                                                                                                                                |
-| FE per-account state                 | `usePref`                                                                                                                                                                                                                                     | raw `readAccount`/`writeAccount` or bespoke hooks                                                                                                                                           |
+| FE browser storage                   | a key in `storage.ts` with its kind (contract-held): a `pref` through `usePref` (the server's, follows the account); `device` / `view` / `cache` this browser's; `public` kept for the /watch page                                            | a `"siftlode.x"` literal; a preference kept only in `localStorage` (it stays on one device)                                                                                                 |
 | FE keyboard shortcut                 | a row in `lib/keymap.ts` + `dispatch()`                                                                                                                                                                                                       | a private `keydown` listener with its own key literals                                                                                                                                      |
 | FE list/grid keys                    | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open`                                                                                                                                                                                    | a private arrow-key / roving-tabindex handler                                                                                                                                               |
 | FE view push (history)               | `rememberNavPosition()` right before `history.pushState`                                                                                                                                                                                      | a push that forgets: Back strands focus on `<body>`                                                                                                                                         |
+| FE view state                        | the module's URL codec: its keys in `urlKeys` (`module.ts`), read and written by `lib/<id>UrlState.ts` through `useUrlMirror` (contract-held, `e2e/bookmark.spec.ts`)                                                                         | `useState` / sessionStorage alone: a reload, a bookmark or a shared link loses the view                                                                                                     |
+| FE a page's header content           | the page renders it into the floating header itself (`PageHeader`, `ui/PageShell`)                                                                                                                                                            | a `page === …` branch in the shell's header, or a bridge carrying a page's state up to it                                                                                                   |
+| FE module store                      | its definition's `Store` (`module.ts`), mounted at the root by `ModuleStores`; it reads no other module's store (contract-held)                                                                                                               | a provider hand-nested in `main.tsx`                                                                                                                                                        |
+| FE saved views                       | the module's own store from `features/filters` (`createSavedViewsStore` + `useSavedViewsStore`)                                                                                                                                               | one store switched by the active page — every visit elsewhere reset it                                                                                                                      |
 | FE static label / button-like link   | `Badge` (flat, no edge) · `LinkButton` / `IconLink`                                                                                                                                                                                           | a glass pill that reads as a chip; an `<a>` hand-painted like a button                                                                                                                      |
 | FE loading/empty/error               | `QueryState` / `StateMessage`                                                                                                                                                                                                                 | ad-hoc conditionals                                                                                                                                                                         |
 | FE failed write                      | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect                                                                                        | a bare `catch {}` or `notify()` with no error attached                                                                                                                                      |

VERSION

@@ -1 +1 @@
-0.108.1
+0.118.0

backend/.basedpyright/baseline.json

@@ -185,118 +185,6 @@
                     "lineCount": 1
                 }
             },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 50,
-                    "endColumn": 56,
-                    "lineCount": 1
-                }
-            },
             {
                 "code": "reportArgumentType",
                 "range": {
@@ -364,25 +252,7 @@
                 }
             }
         ],
-        "./app/downloads/adapters.py": [
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 30,
-                    "endColumn": 38,
-                    "lineCount": 1
-                }
-            }
-        ],
         "./app/downloads/backfill.py": [
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 34,
-                    "endColumn": 38,
-                    "lineCount": 1
-                }
-            },
             {
                 "code": "reportReturnType",
                 "range": {
@@ -413,14 +283,6 @@
             }
         ],
         "./app/downloads/pipeline.py": [
-            {
-                "code": "reportArgumentType",
-                "range": {
-                    "startColumn": 34,
-                    "endColumn": 38,
-                    "lineCount": 1
-                }
-            },
             {
                 "code": "reportAttributeAccessIssue",
                 "range": {

backend/alembic/versions/0113_takeout_items.py

@@ -0,0 +1,51 @@
+"""A user's Google Takeout export: what they watched on YouTube, and their Watch later list
+
+Revision ID: 0113_takeout_items
+Revises: 0112_one_watch_later
+Create Date: 2026-10-23
+
+One row per video and kind. A video the catalog does not know yet waits with no outcome, so the id
+is not a foreign key; the partial index is what a newly known video looks its waiting rows up by.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0113_takeout_items"
+down_revision: str | None = "0112_one_watch_later"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "takeout_items",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "user_id",
+            sa.Integer(),
+            sa.ForeignKey("users.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column("kind", sa.String(12), nullable=False),
+        sa.Column("video_id", sa.String(16), nullable=False),
+        sa.Column("channel_id", sa.String(32), nullable=True),
+        sa.Column("at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("outcome", sa.String(12), nullable=True),
+        sa.Column(
+            "created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
+        ),
+        sa.UniqueConstraint("user_id", "kind", "video_id", name="uq_takeout_item"),
+        sa.CheckConstraint("kind IN ('watched', 'watch_later')", name="ck_takeout_item_kind"),
+    )
+    op.create_index(
+        "ix_takeout_items_waiting",
+        "takeout_items",
+        ["video_id"],
+        postgresql_where=sa.text("outcome IS NULL"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_takeout_items_waiting", table_name="takeout_items")
+    op.drop_table("takeout_items")

backend/alembic/versions/0114_video_unmarks.py

@@ -0,0 +1,44 @@
+"""When a user last took a video back to "new"
+
+Revision ID: 0114_video_unmarks
+Revises: 0113_takeout_items
+Create Date: 2026-10-23
+
+Un-marking or resetting a video leaves no `video_states` row, so this is what tells an import of the
+user's YouTube history a video they un-marked here from one never touched.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0114_video_unmarks"
+down_revision: str | None = "0113_takeout_items"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "video_unmarks",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "user_id",
+            sa.Integer(),
+            sa.ForeignKey("users.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column(
+            "video_id",
+            sa.String(16),
+            sa.ForeignKey("videos.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column("at", sa.DateTime(timezone=True), nullable=False),
+        sa.UniqueConstraint("user_id", "video_id", name="uq_video_unmark"),
+    )
+    op.create_index("ix_video_unmarks_video_id", "video_unmarks", ["video_id"])
+
+
+def downgrade() -> None:
+    op.drop_index("ix_video_unmarks_video_id", table_name="video_unmarks")
+    op.drop_table("video_unmarks")

backend/alembic/versions/0115_takeout_channels.py

@@ -0,0 +1,49 @@
+"""A watched Takeout video's length, and the channels a user put out of their list
+
+Revision ID: 0115_takeout_channels
+Revises: 0114_video_unmarks
+Create Date: 2026-10-24
+
+The "channels you watch but do not follow" list counts a user's watched videos over five minutes
+by channel. The length is asked of YouTube for that list and kept on the Takeout row; a channel the
+user does not want on the list is kept apart.
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0115_takeout_channels"
+down_revision: str | None = "0114_video_unmarks"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "takeout_items", sa.Column("looked_up_at", sa.DateTime(timezone=True), nullable=True)
+    )
+    op.add_column("takeout_items", sa.Column("duration_seconds", sa.Integer(), nullable=True))
+    op.create_table(
+        "takeout_channel_dismissals",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "user_id",
+            sa.Integer(),
+            sa.ForeignKey("users.id", ondelete="CASCADE"),
+            nullable=False,
+        ),
+        sa.Column("channel_id", sa.String(32), nullable=False),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.func.now(),
+            nullable=False,
+        ),
+        sa.UniqueConstraint("user_id", "channel_id", name="uq_takeout_channel_dismissal"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("takeout_channel_dismissals")
+    op.drop_column("takeout_items", "duration_seconds")
+    op.drop_column("takeout_items", "looked_up_at")

backend/alembic/versions/0116_followed_explored_channels.py

@@ -0,0 +1,34 @@
+"""An explored channel someone follows is the catalog's own, and so are its videos
+
+Revision ID: 0116_followed_explored_channels
+Revises: 0115_takeout_channels
+Create Date: 2026-10-24
+
+A follow pulled in from YouTube (the subscription import) left an explored channel flagged as an
+exploration's, and a subscribe landing while an exploration's videos came in left those videos
+flagged on a channel that no longer was one. Both are held in the code now; this clears what they
+left: the channel of a subscription is not an exploration's, and no video is an exploration's on a
+channel that is not one. Data only, so the downgrade has nothing to undo.
+"""
+
+from alembic import op
+
+revision: str = "0116_followed_explored_channels"
+down_revision: str | None = "0115_takeout_channels"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.execute(
+        "UPDATE channels SET from_explore = false WHERE from_explore"
+        " AND EXISTS (SELECT 1 FROM subscriptions s WHERE s.channel_id = channels.id)"
+    )
+    op.execute(
+        "UPDATE videos SET via_explore = false WHERE via_explore"
+        " AND channel_id IN (SELECT id FROM channels WHERE NOT from_explore)"
+    )
+
+
+def downgrade() -> None:
+    pass

backend/alembic/versions/0117_videos_length_sort_indexes.py

@@ -0,0 +1,76 @@
+"""videos: the duration sorts ride an index on the video's length, not on the raw column
+
+Revision ID: 0117_videos_length_sort_indexes
+Revises: 0116_followed_explored_channels
+Create Date: 2026-11-07
+
+A YouTube length past a day is not a length (`catalog.models.LENGTH_MAX_S`): a 24/7 stream's archive
+reports the years it has run. The feed's duration sorts now order by `Video.length_seconds`, which
+is NULL for such a video, so "longest first" no longer opens on a stream archive. Migration 0096's
+two duration indexes are on the raw column and no longer match that ORDER BY; they are replaced by
+the same indexes over the length expression. The expression is spelled out here as SQLAlchemy emits
+it, with the limit as a literal (an index matches a query's constant, never a bound parameter);
+tests/test_feed_sort_indexes.py fails if the two drift apart.
+
+CONCURRENTLY (autocommit_block), re-runnable, as in 0096.
+"""
+
+from alembic import op
+from sqlalchemy import text
+
+revision: str = "0117_videos_length_sort_indexes"
+down_revision: str | None = "0116_followed_explored_channels"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+# Frozen with this migration: `catalog.models.LENGTH_MAX_S` is 24 h.
+LENGTH = "(CASE WHEN (duration_seconds > 86400) THEN NULL ELSE duration_seconds END)"
+
+NEW = {
+    "ix_videos_length_desc": f"{LENGTH} DESC NULLS LAST, id",
+    "ix_videos_length_asc": f"{LENGTH} ASC NULLS LAST, id",
+}
+OLD = {
+    "ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
+    "ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
+}
+
+
+def _valid(name: str) -> bool | None:
+    """The index's `indisvalid`, or None when there is no index of that name."""
+    row = (
+        op.get_bind()
+        .execute(
+            text(
+                "SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
+                "WHERE c.relname = :name"
+            ),
+            {"name": name},
+        )
+        .first()
+    )
+    return None if row is None else bool(row[0])
+
+
+def _build(name: str, columns: str) -> None:
+    """CREATE INDEX CONCURRENTLY, re-runnable: an INVALID leftover of a failed build is dropped first
+    (see 0096 `_build`)."""
+    if _valid(name) is False:
+        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
+    op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} ON videos ({columns})")
+
+
+def upgrade() -> None:
+    with op.get_context().autocommit_block():
+        for name, columns in NEW.items():
+            _build(name, columns)
+        for name in OLD:
+            op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
+
+
+def downgrade() -> None:
+    with op.get_context().autocommit_block():
+        for name, columns in OLD.items():
+            _build(name, columns)
+        for name in NEW:
+            op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")

backend/alembic/versions/0118_videos_shuffle_key.py

@@ -0,0 +1,40 @@
+"""videos: a random key per video, so a seeded shuffle walks an index instead of sorting the scope
+
+Revision ID: 0118_videos_shuffle_key
+Revises: 0117_videos_length_sort_indexes
+Create Date: 2026-11-07
+
+The feed's Shuffle sort and the Home surprise ordered the whole feed scope by md5(id || seed) on every
+call: ~350 ms on prod for a scope of ~213k videos, and a Shuffle page deeper in the list cost more
+(767 ms on page 7 on localdev, against 4 ms for Newest). Each video now carries a random number drawn
+once (`Video.shuffle_key`); a seed picks a point on that ring and the draw reads the index from there
+(`feedquery.shuffle_walk`).
+
+One ADD COLUMN with a volatile default rewrites the table and fills every row with its own value. On
+localdev (278k videos) that took 28 s, against 84 s for an UPDATE of a nullable column, so the rewrite
+it is: the table is locked for that time, and a deploy has the app down while it migrates anyway.
+The index is (shuffle_key, id): the walk's keyset cursor compares the pair.
+"""
+
+from alembic import op
+
+revision: str = "0118_videos_shuffle_key"
+down_revision: str | None = "0117_videos_length_sort_indexes"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+def upgrade() -> None:
+    op.execute(
+        "ALTER TABLE videos ADD COLUMN IF NOT EXISTS shuffle_key double precision "
+        "NOT NULL DEFAULT random()"
+    )
+    with op.get_context().autocommit_block():
+        op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
+        op.execute("CREATE INDEX CONCURRENTLY ix_videos_shuffle_key ON videos (shuffle_key, id)")
+
+
+def downgrade() -> None:
+    with op.get_context().autocommit_block():
+        op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
+    op.execute("ALTER TABLE videos DROP COLUMN IF EXISTS shuffle_key")

backend/alembic/versions/0119_plex_transcode_gpu.py

@@ -0,0 +1,65 @@
+"""plex_items.transcode_hw: the facet says whether the GPU ENCODES, under the measured codec sets
+
+Measured on prod (2026-11-07): the iGPU has no VC-1 Advanced decode, so a VC-1 film never ran on the
+GPU although the facet said it did; VC-1, XviD and DivX now take the CPU-decode → GPU-encode path; and an
+HDR film runs on the GPU through the OpenCL tonemap although the facet said software. The rule is
+`plex.sync.transcode_hw_capable`; the two codec sets below are pinned to `plex.sync` by
+tests/test_plex_playback_class.py.
+
+Rewritten from the stored codec and bit depth. HDR is not stored, so a 10-bit HW-decodable item is set to
+software here and its probe watermark cleared: the bit-depth backfill (`plex.sync._probe_media_batch`,
+a bounded batch per Plex sync) reads its HDR flag again and moves an HDR one back to the GPU.
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+
+revision: str = "0119_plex_transcode_gpu"
+down_revision: Union[str, None] = "0118_videos_shuffle_key"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+HW_DECODABLE_CODECS = ("h264", "hevc", "h265", "vp9", "mpeg2video")
+CPU_DECODE_CODECS = ("vc1", "mpeg4", "msmpeg4v3")
+
+
+def _in(codecs: tuple[str, ...]) -> str:
+    return ", ".join(f"'{c}'" for c in codecs)
+
+
+def upgrade() -> None:
+    op.execute(
+        f"""
+        UPDATE plex_items
+           SET transcode_hw = CASE
+               WHEN lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
+               THEN coalesce(video_bit_depth, 8) <= 8
+               WHEN lower(codec_video) IN ({_in(CPU_DECODE_CODECS)})
+               THEN coalesce(video_bit_depth, 8) <= 8
+               ELSE false
+           END
+         WHERE playable = 'transcode'
+        """
+    )
+    op.execute(
+        f"""
+        UPDATE plex_items
+           SET media_probed_at = NULL
+         WHERE playable = 'transcode'
+           AND lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
+           AND video_bit_depth > 8
+        """
+    )
+
+
+def downgrade() -> None:
+    # The old rule: a decodable codec (VC-1 counted) at 8 bits; HDR and every other codec software.
+    op.execute(
+        f"""
+        UPDATE plex_items
+           SET transcode_hw = lower(codec_video) IN ({_in(HW_DECODABLE_CODECS + ("vc1",))})
+                              AND coalesce(video_bit_depth, 8) <= 8
+         WHERE playable = 'transcode'
+        """
+    )

backend/app/README.md

@@ -25,8 +25,8 @@ Three layers, by *who* runs the code — not by feature:
    background run's durable row, its words in `runstatus`),
    `notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
    configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
-   level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
-   reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
+   level), `debugsession` (a recording that tags its user's lines, E16), `collation` (A→Z in the
+   reader's alphabet, E30) and `draw` (the weighted, seeded draw "Surprise me" deals from), … are single-file primitives many layers import. They depend on nothing above them (a core
    module must never import a `routes/…` module).
 
 **Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a

backend/app/auth.py

@@ -25,8 +25,6 @@ def operator_contact() -> str | None:
     return next(iter(sorted(settings.admin_email_set)), None)
 
 
-MAX_SESSION_ACCOUNTS = 6
-
 # OAuth scope vocabulary — the atoms + the request bundles, kept together as core. has_*_scope check
 # the atoms against a user's stored grant; the *_SCOPES bundles are what the login/callback endpoints
 # request (imported by app/routes/auth.py).
@@ -253,94 +251,63 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
     background.add_task(email_mod.send_account_deleted, email, lang, operator_contact())
 
 
-ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
+def session_user_id(conn: HTTPConnection) -> int | None:
+    """The account this browser session signed in as, read from the signed cookie alone (no DB).
+    One account per session: signing in as another replaces it (`routes.auth._establish_session`)."""
+    return conn.session.get("user_id")
 
 
-def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int | None, bool]:
-    """Which account this connection acts as, and whether that's the session's default account.
-    Takes an HTTPConnection rather than a Request: it reads only the session and the headers.
-
-    The signed cookie holds the *wallet* (`account_ids`, every account signed into this browser)
-    plus a default `user_id`. A caller may name its account with the X-Siftlode-Account header (or
-    `?account=`), but ONLY an account in the wallet — so a tab can't impersonate an account that
-    never authenticated here. A connection that names no account acts as the cookie default.
-
-    A connection naming an account NOT in the wallet acts as nobody (`None`) — typically a tab still
-    pinned to an account that was signed out in another tab. It must not fall back to the default:
-    that tab would go on acting as an account it never showed, its player checkpoints and pref saves
-    landing there and its Sign out signing THAT account out. Only the bootstrap /api/me probe passes
-    `strict=False` and gets the default, which is how such a tab learns who it now is (the frontend
-    re-pins it and reloads). An account whose session ended while it stayed in the wallet (deleted,
-    suspended, revoked) leaves the wallet when it is refused (`_session_user`), so a tab pinned to it
-    takes the same way back.
-    """
-    default_id = conn.session.get("user_id")
-    wallet = conn.session.get("account_ids") or []
-    # Header for normal XHR; ?account= for a context that can't set headers (a plain <a> file
-    # download). Both are wallet-gated below.
-    hdr = conn.headers.get(ACTIVE_ACCOUNT_HEADER) or conn.query_params.get("account")
-    if hdr:
-        try:
-            hid = int(hdr)
-        except ValueError:
-            # Under `strict` this becomes a 401 and the tab authenticates as nobody, which reads
-            # in the browser exactly like a session that expired. The value itself stays out —
-            # it came from a request, and `wallet` says which accounts were actually on offer.
-            log.debug(
-                "the active-account header could not be read as an id",
-                extra=logs.event("auth.account_header.invalid", accounts=len(wallet)),
-            )
-            hid = None
-        if hid is not None and hid in wallet:
-            return hid, hid == default_id
-        if strict:
-            return None, False
-    return default_id, True
+def stamp_session(request: Request, user: User) -> None:
+    """Record the account's current `session_epoch` in the signed cookie, so `current_user` can later
+    reject the cookie once the epoch moves (password reset/change, or "log out everywhere")."""
+    request.session["epoch"] = user.session_epoch or 0
 
 
-def drop_from_wallet(session: dict, user_id: int) -> None:
-    """Take one account out of this browser's wallet (`account_ids`); the session default is untouched."""
-    session["account_ids"] = [a for a in (session.get("account_ids") or []) if a != user_id]
+# The account a tab was built for, sent with every request it makes (`identityHeaders`, the
+# frontend). The cookie alone says who is asking, and one browser holds one account: a sign-in as
+# someone else in another tab replaces it under every open tab. A tab still showing the previous
+# account would otherwise act as the new one — its player checkpoints, pref saves, Sign out and
+# Delete account landing there — until it noticed. It never selects an account; a mismatch is
+# refused, and the tab reloads as whoever is signed in now.
+EXPECTED_ACCOUNT_HEADER = "x-siftlode-expect-account"
 
 
-def _end_account_session(request: Request, user_id: int, is_default: bool) -> None:
-    """An account whose session here is no longer valid. Only nuke the whole session when it is the
-    SESSION DEFAULT; a per-tab account just leaves the wallet (other tabs' default may still be valid).
-    Leaving is what lets a tab pinned to it recover: its next /api/me names an account no longer signed
-    in here and gets the default. Left in, it answered "signed out" — the landing page, with the default
-    still signed in, and no way off it but closing the tab."""
-    if is_default:
-        request.session.clear()
-    else:
-        drop_from_wallet(request.session, user_id)
+def built_for_another_account(user_id: int, expected: str | None) -> bool:
+    """Whether a tab naming `expected` is showing another account than the one signed in. No name is
+    no claim (a caller from before this check, a script): only a mismatch is refused, and the
+    session stays — it is the other tab's sign-in, and it holds. The one comparison for every way a
+    tab names its account: this header on a request, `expect` on a navigation."""
+    if expected is None or expected == str(user_id):
+        return False
+    log.info(
+        "a request from a tab built for another account was refused",
+        extra=logs.event("auth.session.account_changed", user_id=user_id),
+    )
+    return True
 
 
 def current_user(request: Request, db: Session = Depends(get_db)) -> User:
-    return _session_user(request, db, *resolved_user_id(request))
+    user = _session_user(request, db, session_user_id(request))
+    if built_for_another_account(user.id, request.headers.get(EXPECTED_ACCOUNT_HEADER)):
+        raise refused(401, "account_changed")
+    return user
 
 
-def _session_user(request: Request, db: Session, user_id: int | None, is_default: bool) -> User:
+def _session_user(request: Request, db: Session, user_id: int | None) -> User:
     if not user_id:
         raise refused(401, "not_authenticated")
     user = db.get(User, user_id)
     if user is None or not user.is_active or user.is_suspended:
         # Suspended/deactivated mid-session → end it so the block takes effect at once.
-        _end_account_session(request, user_id, is_default)
+        request.session.clear()
         raise refused(401, "not_authenticated")
-    # SA4: reject a cookie whose recorded epoch is behind the account's current one (revoked by a
-    # password reset/change or "log out everywhere"). A cookie with no recorded epoch (pre-SA4, or an
-    # account signed in before this shipped) is treated as epoch 0 — so the FIRST bump revokes it too.
-    epochs = request.session.get("epochs") or {}
-    if int(epochs.get(str(user_id), 0)) != (user.session_epoch or 0):
-        _end_account_session(request, user_id, is_default)
+    # A cookie whose epoch is behind the account's (revoked by a password reset/change or "log out
+    # everywhere") is refused — and so is one with no epoch at all, which only a build before the
+    # one-account session wrote: its holder signs in once more.
+    epoch = request.session.get("epoch")
+    if epoch is None or int(epoch) != (user.session_epoch or 0):
+        request.session.clear()
         raise refused(401, "not_authenticated")
-    # Always keep the session default in the switchable list — covers sessions created before
-    # multi-session existed (their account_ids was never seeded), so the switcher isn't empty.
-    default_id = request.session.get("user_id")
-    accounts = request.session.get("account_ids") or []
-    if default_id and default_id not in accounts:
-        accounts.append(default_id)
-        request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
     # Attribute this request's log lines to the account. Here, not in the request middleware:
     # this is the one place a session becomes an account, and the middleware runs outside both the
     # session middleware and this task's context. `request.state` is shared across middleware layers
@@ -358,25 +325,17 @@ def _session_user(request: Request, db: Session, user_id: int | None, is_default
 def optional_current_user(request: Request, db: Session = Depends(get_db)) -> User | None:
     """Like `current_user` but returns None instead of raising 401 when there's no valid session.
     Used by the bootstrap /api/me probe so the logged-out landing doesn't log a failed request to
-    the browser console (a non-2xx fetch is a console error no matter how the app handles it).
-    The one lenient resolution (`strict=False`): a tab naming an account no longer signed in here gets
-    the default account, which is how it learns who it now is — see `resolved_user_id`. That includes an
-    account refused right here, which `_session_user` has just taken out of the wallet."""
-    user_id, is_default = resolved_user_id(request, strict=False)
+    the browser console (a non-2xx fetch is a console error no matter how the app handles it)."""
+    user_id = session_user_id(request)
+    if user_id is None:
+        return None
     try:
-        return _session_user(request, db, user_id, is_default)
-    # silent-ok: this is the anonymous case as much as the refused one — every landing page hit
-    # comes through here. The second attempt below is the handling, and it reports.
+        return _session_user(request, db, user_id)
     except HTTPException:
-        if is_default:
-            return None
-    try:
-        return _session_user(request, db, *resolved_user_id(request, strict=False))
-    except HTTPException:
-        # Both the pinned account AND the default one were refused, so the tab is signed out
-        # without being told why — and it got here from a session that named a real account.
+        # A session that named an account, refused: the browser is signed out without being told
+        # why, so say it here.
         log.info(
-            "a session was resolved to nobody: neither the named nor the default account held",
+            "a session was refused: its account no longer holds",
             extra=logs.event("auth.session.probe_rejected", user_id=user_id),
         )
         return None

backend/app/bodylimit.py

@@ -72,6 +72,13 @@ DEFAULT_MAX_BODY_BYTES = 1024 * 1024
 # A route belongs here ONLY IF ITS OWN CAP IS LARGER THAN THE DEFAULT. That is why
 # `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
 # default, so listing it could only widen what it accepts.
+# A Google Takeout file (`reader.takeout`). A real watch history measured 3.7 MB for 9500 views, ~390
+# bytes a view, so 32 MiB holds ~86k views. YouTube keeps the history 36 months by default on an
+# account made since mid-2020 and for good on an older one, so a heavy viewer's can pass 16 MiB. The
+# file's text is decoded a window at a time and read an entry at a time (`takeout._json_list`), so
+# it costs its bytes and a window, not a tree of it; the route takes two at once across every user.
+TAKEOUT_MAX_BYTES = 32 * 1024 * 1024
+
 ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
     # Ingesting a local file. The body IS the file now — no multipart envelope to allow for —
     # so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
@@ -95,6 +102,7 @@ ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
     # sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
     ("POST", "/api/files/upload"): settings.download_upload_max_bytes,
     ("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
+    ("POST", "/api/takeout"): TAKEOUT_MAX_BYTES,
 }
 
 # The refusal, byte-for-byte what `capped_body`'s `HTTPException(413, {"code": "too_large"})` renders

backend/app/catalog/models.py

@@ -10,16 +10,22 @@ from sqlalchemy import (
     Boolean,
     Computed,
     DateTime,
+    Float,
     ForeignKey,
     Integer,
     JSON,
     String,
     Text,
     UniqueConstraint,
+    case,
     func,
+    literal_column,
+    null,
 )
 from sqlalchemy.dialects.postgresql import TSVECTOR
+from sqlalchemy.ext.hybrid import hybrid_property
 from sqlalchemy.orm import Mapped, mapped_column, relationship
+from sqlalchemy.sql.elements import ColumnElement
 
 from app.db import Base
 from app.models import TimestampMixin
@@ -71,10 +77,12 @@ class Channel(Base, TimestampMixin):
         Boolean, default=False, server_default="false", index=True
     )
     # True when this channel exists only because a user EXPLORED it (opened its page without
-    # subscribing) — an ephemeral, per-explorer addition. The cleanup job hard-deletes such a
-    # channel (and its via_explore videos) once nobody keeps it (no subscription, no live
-    # ExploredChannel row, no per-user state) after a grace period. Cleared when someone
-    # subscribes (the "keep" signal promotes it to organic).
+    # subscribing) — an ephemeral, per-explorer addition. After a grace period the cleanup job
+    # hard-deletes its via_explore videos nobody keeps (no subscription, no live ExploredChannel
+    # row, no per-user state), and then the channel row itself once it holds no videos — unless a
+    # Takeout watch history shows it watched: the "watched but not followed" list names it from
+    # that row. Cleared when someone subscribes, here or on YouTube (the "keep" signal promotes it
+    # to organic).
     from_explore: Mapped[bool] = mapped_column(
         Boolean, default=False, server_default="false", index=True
     )
@@ -112,6 +120,20 @@ class Subscription(Base, TimestampMixin):
 # hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
 LIVE_OR_UPCOMING = ("live", "upcoming")
 
+# A YouTube length past a day is not a length. A 24/7 stream's archive (`was_live`) reports the years
+# the stream has run (measured on prod, 2026-11-06: 122,962,914 s), and one such video made a Watch
+# later total read 44,073 hours. YouTube caps an upload at 12 hours, so nothing real is lost. Such a
+# length is unknown wherever a length is used: the API sends none (`Video.length_seconds`), sums leave
+# it out (a playlist's counts it as `without_length`), and the duration filter, sort and Home budget never
+# match it. The raw `duration_seconds` stays for what it does prove — that the video is not a Short —
+# and for the jobs that ask whether YouTube has answered yet (test_video_length holds who reads it).
+LENGTH_MAX_S = 24 * 3600
+
+
+def known_length(seconds: int | None) -> int | None:
+    """`seconds` as a video's length, or None when it is not one (see LENGTH_MAX_S)."""
+    return seconds if seconds is not None and seconds <= LENGTH_MAX_S else None
+
 
 class Video(Base, TimestampMixin):
     """A YouTube video. Shared across users; per-user state lives elsewhere."""
@@ -139,8 +161,8 @@ class Video(Base, TimestampMixin):
     )
     thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
     # The feed's views / duration / title sorts each ride a migration-defined index in their exact
-    # ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
-    # `feedquery.order_by` by test_feed_sort_indexes.
+    # ORDER BY, one per direction (and per ICU collation for title): migration 0096 (the duration
+    # pair since 0117, on `length_seconds`), held to `feedquery.order_by` by test_feed_sort_indexes.
     duration_seconds: Mapped[int | None] = mapped_column(Integer)
     view_count: Mapped[int | None] = mapped_column(BigInteger)
     like_count: Mapped[int | None] = mapped_column(BigInteger)
@@ -195,6 +217,37 @@ class Video(Base, TimestampMixin):
     last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     unavailable_reason: Mapped[str | None] = mapped_column(String(24))
+    # A random number in [0, 1), drawn once when the row is written: the ring a seeded shuffle walks
+    # (`feedquery.shuffle_walk`, index ix_videos_shuffle_key on (shuffle_key, id), migration 0118).
+    shuffle_key: Mapped[float] = mapped_column(Float, nullable=False, server_default=func.random())
+
+    @hybrid_property
+    def length_seconds(self) -> int | None:
+        """The video's length, or None when it is unknown — not fetched yet, or not a length at all
+        (`LENGTH_MAX_S`). What every reader of a length reads."""
+        return known_length(self.duration_seconds)
+
+    @length_seconds.inplace.expression
+    @classmethod
+    def _length_seconds_expression(cls) -> ColumnElement[int | None]:
+        # The limit is a literal, not a bound parameter: the duration sort indexes (migration 0117)
+        # are built on this very expression, and Postgres matches an index to a query's constant only.
+        return case(
+            (cls.duration_seconds > literal_column(str(LENGTH_MAX_S)), null()),
+            else_=cls.duration_seconds,
+        )
+
+    @hybrid_property
+    def without_length(self) -> bool:
+        """A video whose length is unknown — live, an archive past a day, or not fetched yet: what a
+        sum of lengths leaves out and names as "+N without length". Every such video, so the count
+        and the sum never disagree about what was left out."""
+        return self.length_seconds is None
+
+    @without_length.inplace.expression
+    @classmethod
+    def _without_length_expression(cls) -> ColumnElement[bool]:
+        return cls.length_seconds.is_(None)
 
     # Weighted full-text search document (DB-generated, never written by the app): title (A) >
     # creator keywords + search queries (B) > truncated description (C). ts_rank honours the
@@ -235,8 +288,9 @@ class ExploredChannel(Base, TimestampMixin):
     Exploring an un-subscribed channel ingests its recent uploads (flagged via_explore) so the
     user can browse them, but keeps them out of everyone else's catalog. This row (a) gates the
     explorer's access to those ephemeral videos in the feed query and (b) drives the grace-period
-    purge: a from_explore channel with no live ExploredChannel row (and no subscription / no
-    per-user state on its videos) is hard-deleted by the cleanup job. Subscribing is the "keep"
+    purge: of a from_explore channel with no live ExploredChannel row (and no subscription) the
+    cleanup job hard-deletes the videos without per-user state, then the channel once it holds none
+    and no Takeout watch history shows it watched. Subscribing is the "keep"
     signal — it promotes the channel to organic and these rows become irrelevant. `created_at`
     (from TimestampMixin) is the grace clock; a re-visit refreshes it."""
 

backend/app/catalog/queries.py

@@ -44,7 +44,8 @@ def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
 
 
 def channel_aggregates(channel_ids: list[str]) -> Select:
-    """Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
+    """Per-channel stored count, last upload and total duration (the lengths it knows: a video with
+    none is left out, `Video.length_seconds`), in one grouped pass. Every column it
     reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
     alone; a test holds that (test_channel_list_index_only)."""
     return (
@@ -52,7 +53,7 @@ def channel_aggregates(channel_ids: list[str]) -> Select:
             Video.channel_id,
             func.count(Video.id),
             func.max(Video.published_at),
-            func.coalesce(func.sum(Video.duration_seconds), 0),
+            func.coalesce(func.sum(Video.length_seconds), 0),
         )
         .where(Video.channel_id.in_(channel_ids))
         .group_by(Video.channel_id)
@@ -158,6 +159,19 @@ def blocked_ids(db: Session, user: User) -> set[str]:
     )
 
 
+def followed_among(db: Session, user: User, channel_ids: set[str]) -> set[str]:
+    """Which of these channels the user subscribes to."""
+    if not channel_ids:
+        return set()
+    return set(
+        db.scalars(
+            select(Subscription.channel_id).where(
+                owned_by(Subscription.user_id, user), Subscription.channel_id.in_(channel_ids)
+            )
+        )
+    )
+
+
 def blocked_channels(db: Session, user: User, lang: str) -> list[Channel]:
     return list(
         db.execute(

backend/app/catalog/search.py

@@ -75,7 +75,8 @@ def ingest_candidates(
 ) -> list[str]:
     """Materialise one page of search candidates into the catalog (channel stubs + enrich,
     video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
-    video ids — non-Short, non-live — in the input relevance order."""
+    video ids — non-Short, non-live — in the input relevance order. A videos.list that fails
+    raises its YouTubeError: with nothing enriched, nothing could be kept."""
     ids = [it["id"] for it in candidates]
     if not ids:
         return []
@@ -132,13 +133,16 @@ def ingest_candidates(
         with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
             detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
     except YouTubeError:
-        # Every candidate then stays unenriched, and an unenriched video cannot be judged live or
-        # Short — so it is dropped. A working search returns nothing and reads as "no results".
+        # An unenriched video cannot be judged live or Short, so none of this page could be kept:
+        # swallowed, the failure answered as a page with nothing on it — "no results" for a search
+        # that never ran (the shared budget spent, no API key). The stubs stay (`insert_stubs`
+        # committed them), as the channels' do above; the caller answers the failure (its refusal
+        # says which).
         log.warning(
             "the videos found by a search could not be enriched",
             extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
         )
-        detail_by_id = {}
+        raise
     enriched_at = datetime.now(
         timezone.utc
     )  # when THIS enrich ran (distinct from the stub placeholder)
@@ -212,17 +216,25 @@ def clear(db: Session, user: User, video_ids: list[str]) -> int:
     """Discard these results "as if never added": drop this user's finds of them, then delete the
     ones nobody kept — a `via_search` video no reader holds (`reader.queries.kept_by_a_reader`),
     whose channel nobody subscribes to, and that no other user still has as a find. Returns how
-    many videos went."""
-    db.execute(
-        delete(SearchFind).where(
-            owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids)
+    many videos went.
+
+    Only the ones this user found: a video in the catalog with no find at all (a related list's,
+    which records none) is in someone's list right now, and an id any user could name would let
+    them delete it from under that list."""
+    mine = list(
+        db.scalars(
+            delete(SearchFind)
+            .where(owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids))
+            .returning(SearchFind.video_id)
         )
     )
+    if not mine:
+        return 0
     subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
     other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
     result = db.execute(
         delete(Video).where(
-            Video.id.in_(video_ids),
+            Video.id.in_(mine),
             Video.via_search.is_(True),
             ~reader_queries.kept_by_a_reader(Video.id),
             ~subscribed,

backend/app/catalog/service.py

@@ -96,13 +96,24 @@ def record_subscription(db: Session, user: User, channel: Channel, yt_subscripti
             subscribed_at=datetime.now(timezone.utc),
         )
     )
-    if channel.from_explore:
-        channel.from_explore = False
-        db.execute(
-            update(Video)
-            .where(Video.channel_id == channel.id, Video.via_explore.is_(True))
-            .values(via_explore=False)
-        )
+    db.flush()
+    settle_explored(db, channel)
+
+
+def settle_explored(db: Session, channel: Channel) -> None:
+    """Hold the rule that a channel's videos are an exploration's only while nobody follows it,
+    decided by who follows it now, not by the flag held in memory: a subscribe in another request
+    may have cleared that while YouTube answered this one, and the videos that came in meanwhile
+    would stay flagged. A channel someone follows, however the follow came (here or pulled in from
+    YouTube), is the catalog's own, and so are its videos."""
+    if channel.from_explore and not queries.followed(db, channel.id):
+        return
+    channel.from_explore = False
+    db.execute(
+        update(Video)
+        .where(Video.channel_id == channel.id, Video.via_explore.is_(True))
+        .values(via_explore=False)
+    )
     db.flush()
 
 

backend/app/config.py

@@ -113,6 +113,11 @@ class Settings(BaseSettings):
     # each search.list costs 100 units, so the shared budget only affords ~80-90/day total; with
     # the zero-quota scrape source it's a pure anti-abuse rate limit. Admin-tunable.
     search_daily_limit_per_user: int = 70
+    # Per-user cap on related-video requests per day — a list opened and each "Show more" (the
+    # InnerTube watch page, no quota; its enrich is up to ~8 units: up to 4 pages, each a videos.list
+    # and a channels.list). A failed one gives its slot back. 0 turns related videos off.
+    # Admin-tunable.
+    related_daily_limit_per_user: int = 200
     # Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
     # 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
     search_source: SearchSource = "scrape"

backend/app/discover/queries.py

@@ -10,16 +10,25 @@ explore-visible and in-progress for free, and cannot drift from them. The Plex h
 narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
 sidebar and wrong for a landing page that does not.
 
-**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
-stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
-un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
-otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
-here takes the feed's seeded-md5 idiom instead of `random()`.
+**One seed drives both halves.** The YouTube half starts on the ring of stored random keys at the
+seed's point and shuffles that window by `md5(id || seed)`; the Plex surprise rail uses
+`ORDER BY random()`, which is fine for a single un-paged draw but cannot be reproduced. A
+cross-source reroll needs both halves to move together — otherwise "Mást kérek" re-draws the Plex
+picks while the YouTube ones stand still — so the Plex side here orders by the same seeded md5
+instead of `random()`.
 """
 
-from sqlalchemy import and_, func, or_, select
+from datetime import datetime, timezone
+
+from sqlalchemy import (
+    and_,
+    func,
+    or_,
+    select,
+)
 from sqlalchemy.orm import Session, aliased
 
+from app import draw
 from app.discover.budget import band
 from app.feedquery import (
     feed_columns,
@@ -27,13 +36,14 @@ from app.feedquery import (
     not_blocked_cond,
     order_by,
     serialize,
+    shuffle_walk,
     sort_keys,
 )
 from app.models import User
 from app.plex.models import PlexItem, PlexShow, PlexState
 from app.catalog.models import Channel, Video
 from app.reader.models import PlaylistItem, VideoState
-from app.plex import rollup
+from app.plex import affinity, rollup
 from app.plex.cards import display_title, leaf_card, movie_card, show_card
 from app.plex.query import apply_show_state_filter, content_libs
 from app.userscope import owned_by
@@ -67,7 +77,8 @@ _FEED_BASE: dict = {
 
 
 def _seeded(col, seed: int):
-    """The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
+    """The Home surprise's seeded order (md5 of the key and the seed), the same for Plex so one
+    seed moves both halves."""
     return func.md5(func.concat(col, str(seed)))
 
 
@@ -112,17 +123,75 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
     Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
     actually a good response to, and the feed's own default keeps them out otherwise."""
     lo, hi = band(budget)
-    params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
-    query, rank_expr = filtered_query(db, user, **params)
-    keys = sort_keys("shuffle", seed, rank_expr)
-    # Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
-    # hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
-    # go: the resume tier has already had its chance at them, and it may have passed precisely because
-    # the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
-    # runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
-    rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
-    out = [serialize(r) for r in rows]
-    return [r for r in out if not r["position_seconds"]][:limit]
+    params: dict = {
+        **_FEED_BASE,
+        "min_duration": lo,
+        "max_duration": hi,
+        "include_shorts": budget == "short",
+    }
+    query, _rank = filtered_query(db, user, **params)
+
+    # A window of the ring, not the whole scope: the seed's SURPRISE_WINDOW videos read along the
+    # stored random key (`shuffle_walk`, a millisecond), then shuffled by the seed. Dealing straight
+    # from the ring would favour for good every video that sits after a wide gap of it (the first
+    # video past a random point is the one whose gap the point fell into); shuffling the window makes
+    # each video's chance the window's share, even to within a few percent. A scope no larger than
+    # the window is the whole scope, in exactly the md5 order the draw always had.
+    #
+    # Anything already started is dropped. The feed's `unwatched` scope keeps watched and hidden out
+    # but deliberately keeps in-progress rows in (they belong in the feed). Here they must go: the
+    # resume tier has already had its chance at them, and it may have passed precisely because the
+    # REMAINING time did not fit the budget — offering the same item back as a surprise, whose runtime
+    # does fit, would label a half-finished thing "Meglepetés" and contradict the question.
+    window = [
+        r
+        for r in map(serialize, shuffle_walk(db, query, seed, SURPRISE_WINDOW))
+        if not r["position_seconds"]
+    ]
+    blind = draw.shuffled(window, seed, key=_video_id)
+    channels = _channel_affinity(db, user)
+    if not channels:
+        return blind[:limit]
+
+    def weight(row: dict) -> float:
+        return 1.0 + CHANNEL_WEIGHT * channels.get(row["channel_id"], 0.0)
+
+    leaning = draw.leaning(window, seed, key=_video_id, weight=weight)
+    return draw.blend(leaning, blind, seed, limit, key=_video_id)
+
+
+# How many videos of the ring a surprise draws from: wide enough that a channel the user watches has
+# a few in it, small enough to read in a millisecond or two.
+SURPRISE_WINDOW = 500
+
+
+def _video_id(row: dict) -> str:
+    return row["id"]
+
+
+# How much a channel the user watches most outweighs one they never watch (1 + this), and how far back
+# the watched history counts: the latest WATCHED_RECENT watches, the recent ones weighing more.
+CHANNEL_WEIGHT = 3.0
+WATCHED_RECENT = 300
+WATCHED_HALF_LIFE_DAYS = 90
+
+
+def _channel_affinity(db: Session, user: User) -> dict[str, float]:
+    """Each channel the user watched lately, scored 0..1 against the most watched: the sum of its
+    watches, each weighing half as much every WATCHED_HALF_LIFE_DAYS. Empty with no history."""
+    rows = db.execute(
+        select(Video.channel_id, VideoState.watched_at)
+        .join(Video, Video.id == VideoState.video_id)
+        .where(owned_by(VideoState.user_id, user), VideoState.status == "watched")
+        .order_by(VideoState.watched_at.desc().nullslast())
+        .limit(WATCHED_RECENT)
+    ).all()
+    now = datetime.now(timezone.utc)
+    scores: dict[str, float] = {}
+    for channel_id, watched_at in rows:
+        r = draw.recency(watched_at, now, WATCHED_HALF_LIFE_DAYS)
+        scores[channel_id] = scores.get(channel_id, 0.0) + r
+    return draw.scaled(scores)
 
 
 def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
@@ -289,21 +358,24 @@ def plex_surprise(
     if not movie_libs:
         return []
     lo, hi = band(budget)
-    q = (
-        select(PlexItem)
-        .where(
-            PlexItem.kind == "movie",
-            PlexItem.library_id.in_(movie_libs),
-            _unseen_movie_conds(uid),
-        )
-        .order_by(_seeded(PlexItem.rating_key, seed))
-        .limit(limit)
+    q = select(PlexItem).where(
+        PlexItem.kind == "movie",
+        PlexItem.library_id.in_(movie_libs),
+        _unseen_movie_conds(uid),
     )
     if lo is not None:
         q = q.where(PlexItem.duration_s >= lo)
     if hi is not None:
         q = q.where(PlexItem.duration_s <= hi)
-    items = db.execute(q).scalars().all()
+    shuffled = _seeded(PlexItem.rating_key, seed)
+    blind = db.execute(q.order_by(shuffled, PlexItem.id).limit(limit)).scalars().all()
+    leaning = affinity.for_user(db, uid)
+    if leaning:
+        key = draw.weighted_key(draw.seeded_u(PlexItem.rating_key, seed), leaning.weight(PlexItem))
+        weighted = db.execute(q.order_by(key, shuffled, PlexItem.id).limit(limit)).scalars().all()
+        items = draw.blend(list(weighted), list(blind), seed, limit, key=lambda it: it.id)
+    else:
+        items = list(blind)
     states = rollup.item_state_map(db, uid, [it.id for it in items])
     return [movie_card(it, states.get(it.id), lang=lang) for it in items]
 

backend/app/downloads/adapters.py

@@ -14,10 +14,10 @@ Resolution works both before extraction (from source_kind / URL host) and after
 import re
 from urllib.parse import urlparse
 
-import yt_dlp
 from yt_dlp.networking.impersonate import ImpersonateTarget
 
 from app.config import settings
+from app.downloads import egress
 from app.downloads.stated import StatedFailure
 
 # Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
@@ -33,6 +33,9 @@ class SourceAdapter:
     """Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
 
     key = "generic"
+    # SEC-17: every fetch of this source's download goes through the egress gate (`egress`). A
+    # source's own host list is no guarantee — its pages, redirects and formats can point anywhere.
+    egress_guarded = True
 
     # --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
     def resolve_url(self, url: str, ydl_opts: dict) -> str:
@@ -66,6 +69,11 @@ class SourceAdapter:
         """The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
         return info.get("title")
 
+    def is_placeholder_title(self, title: str | None) -> bool:
+        """Whether `title` is a stand-in the source gives in place of the real one (worth one more,
+        signed-out read of the page). Generic: never."""
+        return False
+
     def uploader_url(self, info: dict) -> str | None:
         """The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
         `channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
@@ -87,6 +95,9 @@ class SourceAdapter:
 
 class YouTubeAdapter(SourceAdapter):
     key = "youtube"
+    # Not through the gate: YouTube's fetches go to Google's fixed hosts, and the PO-token provider
+    # it needs is the sidecar on our internal network, which the gate exists to refuse.
+    egress_guarded = False
     # Errors worth retrying with a different player-client set (YouTube per-client flakiness).
     _RETRIABLE = (
         "not a bot",
@@ -174,7 +185,7 @@ class FacebookAdapter(SourceAdapter):
         post_id = facebook_group_post_id(url)
         if post_id is None or not ydl_opts.get("cookiefile"):
             return url
-        with yt_dlp.YoutubeDL(ydl_opts) as ydl:
+        with egress.open_ydl(ydl_opts) as ydl:
             raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
         page = raw.decode("utf-8", "replace")
         # Measured: a refused or expired cookie gets the login page (~450 KB, a login
@@ -201,11 +212,17 @@ class FacebookAdapter(SourceAdapter):
     # Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
     # 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
     # signed-out read titles the page with that same text behind the engagement counts, cut short.
+    # Measured (2026-11-01, yt-dlp 2026.08.19): since 2026-10-28 every signed-in reel read gives
+    # "Facebook" and NO description, while a signed-out read of the same reel still carries the text —
+    # so a placeholder left with nothing behind it gets one signed-out read (`backfill.public_title_info`).
     _PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
 
+    def is_placeholder_title(self, title: str | None) -> bool:
+        return (title or "").strip().lower() in self._PLACEHOLDER_TITLES
+
     def title(self, info: dict) -> str | None:
         raw = info.get("title")
-        if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
+        if not self.is_placeholder_title(raw):
             return raw
         line = next(
             (ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""

backend/app/downloads/backfill.py

@@ -5,7 +5,10 @@ auto-detected channel becomes a clickable link) and `poster_path` (an ffmpeg fra
 thumbnail-less sources, so the card / watch page / link preview still shows an image). Best-effort
 and safe to re-run: it only touches assets still missing the field, and only re-fetches YouTube /
 Facebook sources (a direct media URL like a reddit HLS manifest carries no channel and its signed
-URL may have expired — those just get a locally-generated poster instead).
+URL may have expired — those just get a locally-generated poster instead). A Facebook download
+titled with the source's placeholder ("Facebook", "Video") is retitled from a signed-out read of its
+page, and so is each download name that only repeated the placeholder. That page read
+(`public_title_info`) also titles every new download, from the worker.
 
 Run once per environment:
     docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
@@ -13,21 +16,23 @@ Run once per environment:
 
 from urllib.parse import urlsplit
 
-import yt_dlp
-from sqlalchemy import select
+from sqlalchemy import select, update
+from sqlalchemy.orm import Session
 
 from app import logs
 from app.config import settings
 from app.db import SessionLocal
 from app.downloads import adapters
 from app.downloads import edit as editmod
-from app.downloads import formats, service
-from app.downloads.models import MediaAsset
+from app.downloads import egress, formats, service
+from app.downloads.models import DownloadJob, MediaAsset
+from app.models import DISPLAY_TEXT_MAX
+from app.titles import normalize_title
 
 log = logs.get_logger(__name__)
 
 
-def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
+def extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
     """Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
     needs the POT/player-client strategy even just to extract)."""
     with formats.hiding(url):
@@ -37,11 +42,15 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> d
 def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
     for clients in adapter.client_sets():
         opts = formats.build_ydl_opts(
-            {}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
+            {},
+            "%(id)s.%(ext)s",
+            lambda d: None,
+            extra=adapter.ydl_opts_extra(clients),
+            egress=adapter.egress_guarded,
         )
         opts["skip_download"] = True
         try:
-            with yt_dlp.YoutubeDL(opts) as ydl:
+            with egress.open_ydl(opts) as ydl:
                 info = ydl.extract_info(url, download=False)
             if info:
                 return info
@@ -62,8 +71,43 @@ def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: in
     return None
 
 
+def public_title_info(info: dict, url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict:
+    """`info` to title a download from: itself, unless its title is a source's placeholder with
+    nothing behind it (a signed-in Facebook read: "Facebook", no description). Then one signed-out,
+    metadata-only read of the page lends its text — the description, else its title — as the
+    description the adapter takes a placeholder's title from. Nothing better there: `info` as it was."""
+    if not adapter.is_placeholder_title(adapter.title(info)):
+        return info
+    public = extract_meta(url, adapter, asset_id)
+    text = public and (public.get("description") or public.get("title"))
+    return {**info, "description": text} if text else info
+
+
+def _retitle(db: Session, a: MediaAsset, adapter: adapters.SourceAdapter, url: str) -> bool:
+    """Swap a placeholder title kept from an earlier download for the page's text, and the download
+    names that only repeated it. A name its owner gave (anything else) stays."""
+    if not adapter.is_placeholder_title(a.title):
+        return False
+    info = public_title_info({"title": a.title, "description": None}, url, adapter, a.id)
+    named = normalize_title(adapter.title(info))
+    if not named or adapter.is_placeholder_title(named):
+        return False
+    # One conditional write, not a read and a compare: a name its owner gives while the page is read
+    # is not overwritten by what was read before it. The asset first, as every other writer of the
+    # two locks them (`service.lock_own_job`): the session does not flush by itself.
+    placeholder = a.title
+    a.title = named
+    db.flush()
+    db.execute(
+        update(DownloadJob)
+        .where(DownloadJob.asset_id == a.id, DownloadJob.display_name == placeholder)
+        .values(display_name=named[:DISPLAY_TEXT_MAX])
+    )
+    return True
+
+
 def run() -> dict:
-    filled_url = filled_poster = 0
+    filled_url = filled_poster = retitled = 0
     with SessionLocal() as db:
         assets = (
             db.execute(
@@ -78,7 +122,7 @@ def run() -> dict:
             adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
             if a.uploader_url is None and adapter.wants_metadata_backfill():
                 url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
-                info = _extract_meta(url, adapter, a.id)
+                info = extract_meta(url, adapter, a.id)
                 uu = adapter.uploader_url(info) if info else None
                 if uu:
                     a.uploader_url = uu
@@ -86,6 +130,14 @@ def run() -> dict:
                     log.info(
                         "asset %s uploader_url filled", a.id
                     )  # the URL names what they downloaded
+            if adapter.wants_metadata_backfill():
+                url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
+                if _retitle(db, a, adapter, url):
+                    # Before the poster step: it can cut a frame for 30 s, and these locks held
+                    # across it block a rename and deadlock a delete.
+                    db.commit()
+                    retitled += 1
+                    log.info("asset %s retitled", a.id)  # the title names what they downloaded
             if a.poster_path is None:
                 # Record a self-hosted poster for every download (the reliable og:image source):
                 # ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
@@ -99,6 +151,7 @@ def run() -> dict:
         "assets": len(assets),
         "filled_uploader_url": filled_url,
         "filled_poster": filled_poster,
+        "retitled": retitled,
     }
     log.info("backfill done: %s", result)
     return result

backend/app/downloads/clip.py

@@ -22,6 +22,7 @@ import json
 import math
 import re
 from pathlib import Path
+from typing import Literal
 
 from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
 from app.plex.sync import HDR_TRANSFERS
@@ -57,12 +58,27 @@ _CROP_ASPECTS = (0.25, 4.0)
 _EDGE_SLACK = 1e-3
 
 
+# Every reason a clip is refused; the UI names each one (`clip.errors.<reason>`, held by the frontend's
+# backendIds contract, which reads this Literal).
+ClipReason = Literal[
+    "bad_source",
+    "bad_range",
+    "too_short",
+    "too_long",
+    "bad_audio",
+    "bad_crop",
+    "not_found",
+    "media_missing",
+    "past_end",
+]
+
+
 class ClipError(Exception):
     """A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
     message names (the cap itself, so the client never says a stale one)."""
 
-    def __init__(self, reason: str, **params: float):
-        self.reason = reason
+    def __init__(self, reason: ClipReason, **params: float):
+        self.reason: ClipReason = reason
         self.params = params
         super().__init__(reason)
 
@@ -185,6 +201,16 @@ def _ratio(v: object) -> float | None:
     return n / d if n > 0 and d > 0 else None
 
 
+def quarter_turned(probe: dict) -> bool | None:
+    """Whether the first video stream of an ffprobe `-show_streams` JSON is stored turned a quarter
+    (ffmpeg stands it upright before any filter, so its filters see width and height swapped). None
+    when there is no video stream."""
+    for s in probe.get("streams") or []:
+        if s.get("codec_type") == "video":
+            return _quarter_turned(s)
+    return None
+
+
 def _quarter_turned(stream: dict) -> bool:
     rotation: object = (stream.get("tags") or {}).get("rotate")
     for sd in stream.get("side_data_list") or []:

backend/app/downloads/credentials.py

@@ -25,6 +25,7 @@ Security controls enforced here:
 
 import os
 from pathlib import Path
+from typing import Literal
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
@@ -163,9 +164,18 @@ def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredenti
     ).scalar_one_or_none()
 
 
+# Every reason a pasted cookies.txt is refused; the UI names each one (`downloads.errors.cookies.<reason>`,
+# held by the frontend's backendIds contract, which reads this Literal).
+CookieReason = Literal["empty", "format"]
+
+
 class InvalidCookies(ValueError):
     """The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
 
+    def __init__(self, reason: CookieReason):
+        super().__init__(reason)
+        self.reason: CookieReason = reason
+
 
 def validate_cookies(raw: str) -> str:
     """Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads

backend/app/downloads/edit.py

@@ -11,7 +11,9 @@ with the worker branching on the asset's `source_kind` to run ffmpeg instead of
                        (the `-ss` seeks first), so a short clip is cheap even from a long source
   * crop (± trim)    → always re-encode (a filter can't be stream-copied), frame-accurate
 
-`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h}, accurate?: bool}`. `accurate` is the
+`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h, frame_w?, frame_h?}, accurate?: bool}`. A
+crop with `frame_w`/`frame_h` counts the picture as it is SHOWN, measured on a frame of that size
+(`crop_filter`); one without them is in the file's stored pixels (its older form). `accurate` is the
 per-edit user choice for a trim-only cut (ignored when a crop forces a re-encode anyway). `split`
 is a frontend concept: the
 UI fans a split out into N trim jobs, so there is one output file per job here.
@@ -60,6 +62,46 @@ class EditAborted(Exception):
 _MAX_SEGMENTS = 100
 
 
+def _crop_frame(crop: dict) -> dict | None:
+    """The picture size a crop was measured on, or None when it has none it could be scaled by (a
+    spec from before it was sent, which `crop_filter` cuts in the file's own pixels)."""
+    try:
+        w, h = int(crop["frame_w"]), int(crop["frame_h"])
+    # silent-ok: no frame is a crop's older form, and it is still the crop that was asked for. An
+    # infinite number (JSON `1e400`) is OverflowError, and it is no more a frame than "x" is.
+    except (KeyError, TypeError, ValueError, OverflowError):
+        return None
+    return {"frame_w": w, "frame_h": h} if w > 0 and h > 0 else None
+
+
+def crop_filter(crop: dict) -> str:
+    """The ffmpeg `crop` of an edit. Its numbers count the picture as the browser reports it, which
+    is its SHOWN size: a video with wide pixels (320 stored, 640 shown) would be cut off to one side
+    in ffmpeg's stored ones, so they are scaled to `iw` / `ih` by the size they were measured on. A
+    rotated file needs nothing more: both the browser and ffmpeg's filters see it upright."""
+    frame_w, frame_h = crop.get("frame_w"), crop.get("frame_h")
+    if not frame_w or not frame_h:
+        return f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"
+    return (
+        f"crop=iw*{crop['w']}/{frame_w}:ih*{crop['h']}/{frame_h}"
+        f":iw*{crop['x']}/{frame_w}:ih*{crop['y']}/{frame_h}"
+    )
+
+
+def crop_stored_size(crop: dict, src_w: int | None, src_h: int | None) -> tuple[int, int] | None:
+    """The size `crop_filter` cuts out of a `src_w` x `src_h` frame, evened as `EVEN_FRAME` evens it —
+    the clip's size when ffprobe cannot tell. The frame is the one ffmpeg's filters see: a rotated
+    file's stood upright, its stored size turned (the caller's to turn). A crop counted in the shown
+    picture is a fraction of that frame, not its own pixel count; None when the size is unknown."""
+    if crop.get("frame_w") and crop.get("frame_h"):
+        if not src_w or not src_h:
+            return None
+        w, h = src_w * crop["w"] // crop["frame_w"], src_h * crop["h"] // crop["frame_h"]
+    else:
+        w, h = crop["w"], crop["h"]
+    return w // 2 * 2, h // 2 * 2
+
+
 def normalize_edit_spec(spec: dict | None) -> dict:
     """Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
 
@@ -72,7 +114,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
     if crop:
         try:
             c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
-        except (KeyError, TypeError, ValueError) as exc:
+        except (KeyError, TypeError, ValueError, OverflowError) as exc:
             # The clip still renders — uncropped. The user asked for a crop, got a whole frame,
             # and nothing anywhere called it a failure; this line is the only trace there is.
             log.warning(
@@ -81,7 +123,8 @@ def normalize_edit_spec(spec: dict | None) -> dict:
             )
             c = None
         if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
-            out["crop"] = c
+            frame = _crop_frame(crop)
+            out["crop"] = {**c, **frame} if frame else c
 
     # Multi-segment JOIN (cut-list → one concatenated file). Takes precedence over `trim`.
     segs = spec.get("segments")
@@ -268,7 +311,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
     if reencode:
         # Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
         # the exact cut). A crop adds the filter; an accurate trim only evens the frame.
-        cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
+        cropf = f"{crop_filter(crop)}," if crop else ""
         cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
     else:
         cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -313,7 +356,7 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
     ]
 
     if needs_reencode(spec):
-        cropf = f",crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}" if crop else ""
+        cropf = f",{crop_filter(crop)}" if crop else ""
         parts, labels = [], []
         for i, s in enumerate(segs):
             st, en = s["start_s"], s["end_s"]

backend/app/downloads/egress.py

@@ -0,0 +1,297 @@
+"""The egress gate: every fetch of a user-supplied download goes through here, and this is where the
+address is judged — at the moment of the connection, not when the URL was typed (SEC-17).
+
+`urlguard` judges the URL the user typed, at enqueue and again in the worker. What that cannot see is
+every request yt-dlp makes after it: a public host answering 3xx with an internal `Location`, a page
+whose formats point inside, a name that resolves public for the check and private for the fetch (DNS
+rebinding). So a guarded download runs with yt-dlp's `proxy` set to this gate, a small HTTP proxy on
+the worker's loopback. For each request it resolves the host ONCE, refuses the connection if any
+address the name has is one the server must not fetch from (`urlguard.is_forbidden`), and otherwise
+connects to the very address it vetted — there is no second lookup for a rebinding answer to slip into.
+
+Measured (yt-dlp 2026.8.19, 2026-11-06) on the worker image, with a logging proxy: the `proxy` option
+reaches every path a download takes — the Requests handler, the CurlCFFI one an impersonated request
+uses, the native HLS downloader and an ffmpeg HLS download (segments included). ffmpeg itself also
+speaks rtmp, rtsp, ftp, tcp and more, which no HTTP proxy covers, so a guarded download keeps only the
+formats it fetches over http(s) (`HttpOnlyFormats`, added by `open_ydl`) and fetches HLS natively.
+
+A refused connection is answered `403` before anything is dialled, so a live internal port and a dead
+one read the same; `errors.user_error_text` then stores the one redacted sentence for both.
+
+Not used for YouTube: those downloads talk to Google's fixed hosts and to the PO-token sidecar, which
+lives on the internal network on purpose (`adapters.YouTubeAdapter.egress_guarded`).
+"""
+
+import ipaddress
+import select
+import socket
+import socketserver
+import threading
+from collections.abc import Callable
+from typing import Any, cast
+from urllib.parse import urlsplit
+
+import yt_dlp
+from yt_dlp.postprocessor.common import PostProcessor
+
+from app import logs
+from app.downloads import urlguard
+
+log = logs.get_logger(__name__)
+
+Resolver = Callable[..., list]
+Forbidden = Callable[[ipaddress.IPv4Address | ipaddress.IPv6Address], bool]
+
+_BLOCKED = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
+_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
+_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
+_MAX_HEAD = 64 * 1024
+# A tunnel with no traffic either way for this long is closed. Above yt-dlp's own socket_timeout
+# (30 s, `formats.build_ydl_opts`), so the gate never ends a transfer yt-dlp would still wait for.
+_IDLE_S = 120
+# Hop-by-hop headers a proxy must not pass on; `Connection: close` replaces them, so one connection
+# carries one request and a reused client connection can never be steered to a second host unvetted.
+_HOP = (b"proxy-", b"connection:", b"keep-alive:")
+
+
+def vetted_address(
+    host: str, port: int, *, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden
+) -> tuple[str, int] | None:
+    """The address to connect to for `host:port`, or None when it must not be fetched: the name has
+    no answer, or ANY of its addresses is forbidden (a round-robin answer is not a coin flip). IPv4
+    only, as every download is (`source_address` 0.0.0.0 in `formats.build_ydl_opts`)."""
+    try:
+        infos = resolve(host, port, socket.AF_INET, socket.SOCK_STREAM)
+    except (socket.gaierror, UnicodeError, OSError):
+        # silent-ok: "no answer" is a refusal, and the caller's refusal is the line that is logged.
+        return None
+    addrs = [info[4] for info in infos]
+    if not addrs:
+        return None
+    for addr in addrs:
+        try:
+            ip = ipaddress.ip_address(addr[0])
+        except ValueError:
+            # silent-ok: an answer that is not an address is refused like a forbidden one, and the
+            # caller logs the refusal.
+            return None
+        if forbidden(ip):
+            return None
+    return addrs[0][0], addrs[0][1]
+
+
+def _read_head(sock: socket.socket) -> bytes | None:
+    buf = b""
+    while b"\r\n\r\n" not in buf:
+        if len(buf) > _MAX_HEAD:
+            return None
+        chunk = sock.recv(8192)
+        if not chunk:
+            return None
+        buf += chunk
+    return buf
+
+
+def _pipe(a: socket.socket, b: socket.socket) -> None:
+    try:
+        _relay(a, b)
+    except OSError:
+        # silent-ok: one side hung up mid-transfer (yt-dlp closes a connection once it has read a
+        # file's size, or gives up on a slow host); whichever side cared reports it itself.
+        return
+
+
+def _relay(a: socket.socket, b: socket.socket) -> None:
+    socks = [a, b]
+    while True:
+        ready, _, _ = select.select(socks, [], [], _IDLE_S)
+        if not ready:
+            return
+        for s in ready:
+            data = s.recv(65536)
+            if not data:
+                return
+            (b if s is a else a).sendall(data)
+
+
+class _Handler(socketserver.BaseRequestHandler):
+    def handle(self) -> None:
+        client: socket.socket = self.request
+        head = _read_head(client)
+        if head is None:
+            return
+        head_part, _, body = head.partition(b"\r\n\r\n")
+        lines = head_part.split(b"\r\n")
+        try:
+            method, target, version = lines[0].decode("latin1").split(" ", 2)
+        except ValueError:
+            # silent-ok: the only client is yt-dlp on this loopback; a malformed request is answered
+            # 400 and reaches its own error, which the job records.
+            client.sendall(_BAD_REQUEST)
+            return
+        if method == "CONNECT":
+            host, _, port_s = target.rpartition(":")
+            host = host.strip("[]")
+            upstream = self._dial(host, port_s)
+            if upstream is None:
+                return
+            with upstream:
+                client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
+                if body:
+                    upstream.sendall(body)
+                _pipe(client, upstream)
+            return
+        u = urlsplit(target)
+        if u.scheme != "http" or not u.hostname:
+            client.sendall(_BLOCKED)
+            return
+        try:
+            port = u.port or 80
+        except ValueError:
+            # silent-ok: as for a malformed request line above — answered 400, recorded by the job.
+            client.sendall(_BAD_REQUEST)
+            return
+        upstream = self._dial(u.hostname, str(port))
+        if upstream is None:
+            return
+        path = (u.path or "/") + (f"?{u.query}" if u.query else "")
+        headers = [h for h in lines[1:] if not h.lower().startswith(_HOP)]
+        request = b"\r\n".join(
+            [f"{method} {path} {version}".encode("latin1"), *headers, b"Connection: close"]
+        )
+        with upstream:
+            upstream.sendall(request + b"\r\n\r\n" + body)
+            _pipe(client, upstream)
+
+    def _dial(self, host: str, port_s: str) -> socket.socket | None:
+        """Connect to the address `host` was vetted at, or answer the client and return None."""
+        client: socket.socket = self.request
+        try:
+            port = int(port_s)
+        except ValueError:
+            # silent-ok: as for a malformed request line above — answered 400, recorded by the job.
+            client.sendall(_BAD_REQUEST)
+            return None
+        gate = cast("_Gate", self.server)
+        addr = vetted_address(host, port, resolve=gate.resolve, forbidden=gate.forbidden)
+        if addr is None:
+            log.warning(
+                "egress refused a connection to %s:%s",
+                host,  # log-pii-ok: the host a download tried to reach, never its path or query
+                port,
+                extra=logs.event("download.egress.refused", port=port),
+            )
+            client.sendall(_BLOCKED)
+            return None
+        try:
+            return socket.create_connection(addr, timeout=_IDLE_S)
+        except OSError:
+            # silent-ok: a public host that does not answer; yt-dlp reports it as the job's own
+            # network failure, which the job records.
+            client.sendall(_BAD_GATEWAY)
+            return None
+
+
+class _Gate(socketserver.ThreadingTCPServer):
+    daemon_threads = True
+    allow_reuse_address = True
+
+    def __init__(self, resolve: Resolver, forbidden: Forbidden):
+        self.resolve = resolve
+        self.forbidden = forbidden
+        super().__init__(("127.0.0.1", 0), _Handler)
+
+    @property
+    def url(self) -> str:
+        return f"http://127.0.0.1:{self.server_address[1]}"
+
+    def handle_error(self, request, client_address) -> None:
+        # socketserver's default prints a bare traceback to stderr, outside the log; a connection the
+        # gate could not finish is a line with its code instead.
+        log.exception("egress gate dropped a connection", extra=logs.event("download.egress.error"))
+
+
+def start_gate(
+    *, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden = urlguard.is_forbidden
+) -> _Gate:
+    """A gate serving on its own thread. The worker uses the one `proxy_url` starts; a test starts
+    its own, with a resolver and a judgement it controls."""
+    gate = _Gate(resolve, forbidden)
+    threading.Thread(target=gate.serve_forever, name="egress-gate", daemon=True).start()
+    return gate
+
+
+# The options key `formats.build_ydl_opts` marks a guarded download with; `open_ydl` reads it. yt-dlp
+# keeps unknown keys in its params and never acts on them.
+GUARDED = "siftlode_egress_guarded"
+
+
+def open_ydl(opts: dict) -> yt_dlp.YoutubeDL:
+    """The one way this package opens yt-dlp (a contract test holds it): a guarded download also
+    gets `HttpOnlyFormats`, which yt-dlp takes only as an instance, not through the options dict."""
+    # yt-dlp's stub types its params as a TypedDict; these are the plain dict `formats.build_ydl_opts`
+    # assembles, the same every caller handed yt-dlp before.
+    ydl = yt_dlp.YoutubeDL(opts)  # type: ignore[arg-type]
+    if opts.get(GUARDED):
+        ydl.add_post_processor(HttpOnlyFormats(), when="pre_process")
+    return ydl
+
+
+_gate: _Gate | None = None
+_gate_lock = threading.Lock()
+
+
+def proxy_url() -> str:
+    """This process's gate, started on first use (the worker's jobs share it)."""
+    global _gate
+    with _gate_lock:
+        if _gate is None:
+            _gate = start_gate()
+        return _gate.url
+
+
+# The protocols fetched over http(s) through the proxy option — the only ones the gate sees: yt-dlp's
+# own fetchers (http, f4m, ism, the DASH segments and their generator, native HLS — which a bare m3u8
+# also gets, as a guarded download sets hls_prefer_native) and ffmpeg's http input (a live stream or
+# a clip), which takes the proxy too. Everything else (rtmp, rtsp, mms, ws) ffmpeg or another tool
+# would dial directly.
+_HTTP_PROTOCOLS = frozenset(
+    {
+        "http",
+        "https",
+        "f4m",
+        "ism",
+        "m3u8",
+        "m3u8_native",
+        "http_dash_segments",
+        "http_dash_segments_generator",
+    }
+)
+
+
+class HttpOnlyFormats(PostProcessor):
+    """Drops every format a guarded download would fetch past the gate. Runs as `pre_process`, which
+    yt-dlp re-reads its formats after ("The pre-processors may have modified the formats"), so the
+    selector never sees them. With nothing left, yt-dlp reports no formats — a refusal, not a leak."""
+
+    def set_downloader(self, downloader):
+        # Not the job's postprocessor hooks: they would report this filter as a "processing" step
+        # before a byte is fetched (yt-dlp attaches them here, once per call).
+        self._downloader = downloader
+
+    def run(self, information):
+        # yt-dlp's stub types the info dict as a TypedDict with optional keys; it is a dict here.
+        info = cast("dict[str, Any]", information)
+        if info.get("formats"):
+            info["formats"] = [f for f in info["formats"] if _over_http(f)]
+        elif info.get("url") and not _over_http(info):
+            info["url"] = None
+        return [], information
+
+
+def _over_http(fmt: dict) -> bool:
+    url = fmt.get("manifest_url") or fmt.get("url") or ""
+    if urlsplit(url).scheme not in ("http", "https"):
+        return False
+    protocol = fmt.get("protocol") or urlsplit(fmt.get("url") or "").scheme
+    return all(p in _HTTP_PROTOCOLS for p in protocol.split("+"))

backend/app/downloads/folders.py

@@ -24,6 +24,7 @@ kept it — and its `ready_at`, the moment it finished (`stamp_done`).
 
 from datetime import datetime, timedelta, timezone
 from enum import StrEnum
+from typing import Literal
 from urllib.parse import urlsplit
 
 from sqlalchemy import select
@@ -181,12 +182,22 @@ def stamp_done(db: Session, job: DownloadJob) -> None:
     job.expiry_warned = False
 
 
+# Every folder change the rules refuse, answered as a generic refusal; the UI names each one
+# (`errors.refused.<reason>`, held by the frontend's backendIds contract, which reads this Literal).
+FolderReason = Literal[
+    "file_folder_gone",
+    "file_folder_name_taken",
+    "file_folder_not_its_source",
+    "file_folder_not_own",
+]
+
+
 class FolderError(Exception):
     """A folder change the rules refuse; `reason` is the refusal the route answers with."""
 
-    def __init__(self, reason: str) -> None:
+    def __init__(self, reason: FolderReason) -> None:
         super().__init__(reason)
-        self.reason = reason
+        self.reason: FolderReason = reason
 
 
 def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:

backend/app/downloads/formats.py

@@ -28,6 +28,7 @@ import urllib.parse
 from collections.abc import Iterator
 
 from app import logs
+from app.downloads import egress as egress_gate
 
 log = logs.get_logger(__name__)
 
@@ -294,6 +295,8 @@ def build_ydl_opts(
     extra: dict | None = None,
     cookiefile: str | None = None,
     max_bytes: int | None = None,
+    *,
+    egress: bool,
 ) -> dict:
     """Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
 
@@ -309,7 +312,11 @@ def build_ydl_opts(
     a file announced larger than it rather than filling the disk and only then being caught by the
     quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
     a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
-    pipeline also holds every produced file to what is left by its real size."""
+    pipeline also holds every produced file to what is left by its real size.
+
+    `egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
+    into our own network, and keeps only the formats fetched over http(s). Required, with no default:
+    a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
     s = normalize(spec)
     h = s["max_height"]
     opts: dict = {
@@ -364,6 +371,13 @@ def build_ydl_opts(
     if cookiefile:
         opts["cookiefile"] = cookiefile
 
+    # SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
+    # guarded download past the gate.
+    if egress:
+        opts["proxy"] = egress_gate.proxy_url()
+        opts["hls_prefer_native"] = True
+        opts[egress_gate.GUARDED] = True
+
     if s["mode"] == "a":
         opts["format"] = "bestaudio/best"
         opts["postprocessors"].append(

backend/app/downloads/links.py

@@ -127,14 +127,35 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
 # --- serialization -------------------------------------------------------------------------
 
 
-def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
+def opens(db: Session, link: DownloadLink) -> str:
+    """What a link opens, in the words its address and its preview use: "folder", "video" or
+    "file". A file still being made reads as the video it mostly is."""
+    from app.downloads import mediakind
+
+    if link.folder_id is not None:
+        return "folder"
+    job = db.get(DownloadJob, link.job_id)
+    asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
+    kind = asset.media_kind if asset else None
+    return "file" if kind and kind != mediakind.VIDEO else "video"
+
+
+def share_path(db: Session, link: DownloadLink, token: str) -> str:
+    """The address a link is given out on: its path says what it opens (`opens`). Only a word for the
+    reader, the token is the whole key: every prefix of `main.SHARE_PREFIXES` opens every link, so
+    the links already sent on /watch/ work for good."""
+    prefix = {"folder": "folder", "file": "file", "video": "watch"}[opens(db, link)]
+    return f"/{prefix}/{token}"
+
+
+def owner_view(db: Session, link: DownloadLink, token: str | None = None) -> dict:
     """What the file owner sees when managing their links. `token` is the one just made (it is in
     hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
     token = token or recover(link)
     return {
         "id": link.id,
         "token": token,
-        "url": f"{base_url}/watch/{token}" if token else None,
+        "url": share_path(db, link, token) if token else None,
         "allow_download": link.allow_download,
         "has_password": link.password_hash is not None,
         "expires_at": link.expires_at.isoformat() if link.expires_at else None,

backend/app/downloads/models.py

@@ -234,7 +234,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
     source_asset_id: Mapped[int | None] = mapped_column(
         ForeignKey("media_assets.id", ondelete="SET NULL")
     )
-    # For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
+    # For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h,frame_w?,frame_h?}}, its
+    # crop in the SHOWN picture's pixels when it carries the frame it was measured on (`edit.crop_filter`),
+    # else in the file's stored ones. For clip
     # jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
     # in fractions of the box it was drawn in (`downloads.clip.normalize`).
     edit_spec: Mapped[dict | None] = mapped_column(JSON)

backend/app/downloads/og.py

@@ -1,4 +1,4 @@
-"""Server-rendered Open Graph tags for the public /watch/{token} page.
+"""Server-rendered Open Graph tags for the public page of a share link (/watch/, /file/, /folder/).
 
 The watch page is a client-side SPA, so a social crawler (facebookexternalhit, Twitterbot, …) that
 fetches /watch/{token} would otherwise only see the generic index.html — a blank/greyed link card.
@@ -76,16 +76,21 @@ def _watch_tags(token: str, db: Session) -> str | None:
     link = linksmod.find(db, token)
     if link is None or linksmod.is_expired(link):
         return None
-    url = f"{settings.app_base}/watch/{token}"
-    folder = link.folder_id is not None
+    url = f"{settings.app_base}{linksmod.share_path(db, link, token)}"
+    # What the card calls it, as its address does.
+    kind = linksmod.opens(db, link)
+    folder, video = kind == "folder", kind == "video"
     if link.password_hash:
         # Password-gated: don't leak the title/thumbnail in the unfurl.
         return "\n    ".join(
             [
                 "<title>Siftlode</title>",
-                _tag("og:title", "Password-protected " + ("folder" if folder else "video")),
+                _tag(
+                    "og:title",
+                    "Password-protected " + ("folder" if folder else "video" if video else "file"),
+                ),
                 _tag("og:description", "Shared via Siftlode"),
-                _tag("og:type", "website" if folder else "video.other"),
+                _tag("og:type", "video.other" if video else "website"),
                 _tag("og:url", url),
                 _tag("og:site_name", "Siftlode"),
             ]
@@ -96,7 +101,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
     asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
     if job is None or asset is None or asset.status != "ready":
         return None
-    title = (job.display_name or asset.title or "Video").strip()
+    title = (job.display_name or asset.title or ("Video" if video else "File")).strip()
     channel = (job.display_uploader or asset.uploader or "").strip()
     # Pick the link-preview image. Facebook's scraper REJECTS our own poster ("Corrupted Image" in the
     # Sharing Debugger): the poster is an ffmpeg-produced JPEG whose non-standard marker order FB's
@@ -118,7 +123,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
         f"<title>{html.escape(title)}</title>",
         _tag("og:title", title),
         _tag("og:description", channel or "Shared via Siftlode"),
-        _tag("og:type", "video.other"),
+        _tag("og:type", "video.other" if video else "website"),
         _tag("og:url", url),
         _tag("og:site_name", "Siftlode"),
         _tag("twitter:title", title, attr="name"),

backend/app/downloads/pipeline.py

@@ -7,6 +7,7 @@ from datetime import date, datetime, timezone
 from pathlib import Path
 
 import yt_dlp
+from sqlalchemy import update
 from yt_dlp.utils import download_range_func
 
 from app import logs, sysconfig
@@ -16,6 +17,7 @@ from app.downloads import clip as clipmod
 from app.downloads import edit as editmod
 from app.downloads import errors as dlerrors
 from app.downloads import adapters, credentials, formats, mediakind, playable, probe, quota
+from app.downloads import backfill, egress
 from app.downloads import service, storage
 from app.downloads import urlguard
 from app.downloads.jobdb import (
@@ -392,6 +394,7 @@ def _extract_with_fallback(
             _make_progress_hook(job_id, asset_id),
             cookiefile=cookiefile,
             max_bytes=max_bytes,
+            egress=adapter.egress_guarded,
         ),
     )
     last_exc: Exception | None = None
@@ -403,6 +406,7 @@ def _extract_with_fallback(
             extra=adapter.ydl_opts_extra(clients),
             cookiefile=cookiefile,
             max_bytes=max_bytes,
+            egress=adapter.egress_guarded,
         )
         opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
         if section is not None:
@@ -413,7 +417,7 @@ def _extract_with_fallback(
         # (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
         # killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
         try:
-            with yt_dlp.YoutubeDL(opts) as ydl:
+            with egress.open_ydl(opts) as ydl:
                 return ydl.extract_info(target, download=True)
         except yt_dlp.utils.DownloadError as exc:
             last_exc = exc
@@ -585,9 +589,14 @@ def _download(
         # missing duration hid the trim action. Probed before the sidecars so the .nfo runtime has it
         # too. Same probe the upload ingest uses.
         vcodec, acodec, width, height, duration_s = probe.resolve_media_facts(produced, info)
+        # A placeholder title with nothing behind it ("Facebook" from a signed-in read) borrows the
+        # page's text from one signed-out read; the download itself stays the signed-in one. The page
+        # yt-dlp read, not the link queued: a group post's link is rewritten to its video, and
+        # signed out the post itself is the login wall.
+        titled = backfill.public_title_info(info, info.get("webpage_url") or url, adapter, asset_id)
         meta = storage.MediaMeta(
             video_id=info.get("id") or source_ref,
-            title=normalize_title(adapter.title(info)) or source_ref,
+            title=normalize_title(adapter.title(titled)) or source_ref,
             uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
             upload_date=_parse_upload_date(info.get("upload_date")),
             uniqueid_type=adapter.nfo_uniqueid_type(),
@@ -830,6 +839,18 @@ def _fail_edit(
     )
 
 
+def _filter_frame(
+    path: Path, width: int | None, height: int | None
+) -> tuple[int | None, int | None]:
+    """A file's frame as ffmpeg's filters see it: the stored size, turned when the stream is stored
+    turned a quarter (ffmpeg stands it upright first). Unknown when the file cannot be read."""
+    data = probe.probe_json(path)
+    turned = clipmod.quarter_turned(data) if data else None
+    if turned is None:
+        return None, None
+    return (height, width) if turned else (width, height)
+
+
 def _process_edit(job_id: int, asset_id: int) -> None:
     """Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
     with SessionLocal() as db:
@@ -898,6 +919,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
         size = final.stat().st_size if final.exists() else None
         # The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
         _, _, out_w, out_h, _ = probe.probe_media(final)
+        # Unsized, a crop's own size stands in, cut from the frame the filters saw.
+        cut = (
+            editmod.crop_stored_size(edit_spec["crop"], *_filter_frame(src_path, src_w, src_h))
+            if crop and not (out_w and out_h)
+            else None
+        )
 
         with SessionLocal() as db:
             asset = db.get(MediaAsset, asset_id)
@@ -909,11 +936,10 @@ def _process_edit(job_id: int, asset_id: int) -> None:
                 asset.duration_s = total_s or asset.duration_s
                 if out_w and out_h:
                     asset.width, asset.height = out_w, out_h
-                elif crop:
-                    asset.width = edit_spec["crop"]["w"]
-                    asset.height = edit_spec["crop"]["h"]
+                elif cut:
+                    asset.width, asset.height = cut
                 else:
-                    asset.width, asset.height = src_w, src_h
+                    asset.width, asset.height = (None, None) if crop else (src_w, src_h)
                 asset.error = None
                 asset.last_access_at = datetime.now(timezone.utc)
             job = db.get(DownloadJob, job_id)
@@ -1116,14 +1142,20 @@ def _process_clip(job_id: int, asset_id: int) -> None:
             fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
             # A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
             placeholder = service.clip_title(None, None)
+            named = None
             if asset and fetched_title and asset.title == placeholder:
                 named = service.clip_title(fetched_title, None)
                 asset.title = named
-                if job_row := db.get(DownloadJob, job_id):
-                    if job_row.display_name == placeholder:
-                        fields["display_name"] = named[:DISPLAY_TEXT_MAX]
             title = asset.title if asset else None
             finalized = service.finalize_if_running(db, job_id, **fields)
+            if finalized and named:
+                # Only a name that still repeats the placeholder, in one conditional write: one its
+                # owner gave meanwhile stays.
+                db.execute(
+                    update(DownloadJob)
+                    .where(DownloadJob.id == job_id, DownloadJob.display_name == placeholder)
+                    .values(display_name=named[:DISPLAY_TEXT_MAX])
+                )
             db.commit()
             if finalized:
                 _notify_download_done(db, user_id, title, job_id, kind="edit")

backend/app/downloads/quota.py

@@ -16,10 +16,12 @@ That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user
 remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
 max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
 is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
-their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
+their output's real size before it is placed. Edit jobs consult it with their output's real size
+before it is placed (`test_edit_cap.py`).
 """
 
 from dataclasses import dataclass
+from typing import Literal
 
 from sqlalchemy import and_, func, or_, select
 from sqlalchemy.orm import Session
@@ -50,11 +52,16 @@ class QuotaLimits:
     unlimited: bool
 
 
-class QuotaExceeded(Exception):
-    """Raised by check_enqueue; `reason` is one of max_jobs|max_bytes (i18n key on the client)."""
+# Every cap a download can hit; the UI names each one (`downloads.errors.quota.<reason>`, held by the
+# frontend's backendIds contract, which reads this Literal).
+QuotaReason = Literal["max_jobs", "max_bytes"]
 
-    def __init__(self, reason: str, limit: int, current: int):
-        self.reason = reason
+
+class QuotaExceeded(Exception):
+    """Raised by check_enqueue; `reason` names the cap it hit (an i18n key on the client)."""
+
+    def __init__(self, reason: QuotaReason, limit: int, current: int):
+        self.reason: QuotaReason = reason
         self.limit = limit
         self.current = current
         super().__init__(f"download quota exceeded: {reason} ({current}/{limit})")

backend/app/downloads/service.py

@@ -10,6 +10,7 @@ handler); the folder, link and quota writes are `folders`, `links` and `quota`.
 
 import uuid
 from datetime import datetime, timezone
+from typing import Literal
 
 from sqlalchemy import delete as delete_rows
 from sqlalchemy import func, select, update
@@ -39,11 +40,16 @@ from app.plex import paths as plex_paths
 from app.userscope import owned_by, owned_row
 
 
-class EditError(Exception):
-    """Bad edit request (empty spec / source not ready); `reason` is an i18n-mappable key."""
+# Every reason an edit is refused; the UI names each one (`downloads.errors.edit.<reason>`, held by
+# the frontend's backendIds contract, which reads this Literal).
+EditReason = Literal["source_not_ready", "not_media", "empty_edit"]
 
-    def __init__(self, reason: str):
-        self.reason = reason
+
+class EditError(Exception):
+    """Bad edit request (empty spec / source not ready / not media); `reason` names the sentence."""
+
+    def __init__(self, reason: EditReason):
+        self.reason: EditReason = reason
         super().__init__(reason)
 
 
@@ -147,7 +153,7 @@ def populate_from_catalog(db: Session, asset: MediaAsset) -> bool:
         return False
     asset.title = video.title
     asset.thumbnail_url = video.thumbnail_url
-    asset.duration_s = video.duration_seconds
+    asset.duration_s = video.length_seconds
     if video.published_at:
         asset.upload_date = video.published_at.date()
     channel = db.get(Channel, video.channel_id)
@@ -331,7 +337,7 @@ def _clip_source_facts(db: Session, spec: dict) -> dict:
             "title": video.title,
             "uploader": channel.title if channel else None,
             "thumbnail_url": video.thumbnail_url,
-            "length_s": video.duration_seconds,
+            "length_s": video.length_seconds,
         }
     item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
     if item is None:

backend/app/downloads/urlguard.py

@@ -10,22 +10,12 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
 is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
 differently between the two moments, and the worker is where the socket is actually opened.
 
-What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
-the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
-
-* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
-  with an internal address.
-* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
-  and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
-  open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
-  response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
-  the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
-  class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
-  one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
-  kept). Tracked as SEC-17.
-
-The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
-hostname into the box), which is the reachable one.
+What this alone cannot see is every request yt-dlp makes AFTER the typed URL: a public host
+answering 3xx with an internal ``Location``, and DNS rebinding (a name that passes this check and
+then resolves inside for yt-dlp's own lookup). Those are closed one layer down, by `egress`: a guarded
+download's every connection goes through a gate that judges the address it actually dials, with
+this module's `is_forbidden`. The checks here stay as the early, structured refusal — a user typing
+``http://127.0.0.1:6379`` hears why at once instead of from a failed job.
 
 Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
 user-controlled destination to guard.
@@ -33,27 +23,36 @@ user-controlled destination to guard.
 
 import ipaddress
 import socket
+from typing import Literal
 from urllib.parse import urlparse
 
 # Reason codes travel to the client inside the structured 422 detail, so they are part of the API
-# vocabulary (frontend `localizeDetail`) — keep them stable.
-REASON_SCHEME = "scheme"  # not http(s)
-REASON_NO_HOST = "no_host"  # URL carries no hostname
-REASON_MALFORMED = "malformed"  # unparseable authority or port; urlparse raises late, so guard it
-REASON_UNRESOLVABLE = "unresolvable"  # DNS said nothing (honest: not the same as "blocked")
-REASON_PRIVATE = "private"  # resolves into a network the server must not fetch from
+# vocabulary — keep them stable. The UI names each one (`downloads.errors.urlBlocked.<reason>`, held by
+# the frontend's backendIds contract, which reads this Literal).
+UrlBlockedReason = Literal["scheme", "no_host", "malformed", "unresolvable", "private"]
+REASON_SCHEME: UrlBlockedReason = "scheme"  # not http(s)
+REASON_NO_HOST: UrlBlockedReason = "no_host"  # URL carries no hostname
+REASON_MALFORMED: UrlBlockedReason = (
+    "malformed"  # unparseable authority or port; urlparse raises late
+)
+REASON_UNRESOLVABLE: UrlBlockedReason = (
+    "unresolvable"  # DNS said nothing (not the same as "blocked")
+)
+REASON_PRIVATE: UrlBlockedReason = (
+    "private"  # resolves into a network the server must not fetch from
+)
 
 
 class BlockedUrl(Exception):
     """A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
     REASON_* codes above."""
 
-    def __init__(self, reason: str):
+    def __init__(self, reason: UrlBlockedReason):
         super().__init__(reason)
-        self.reason = reason
+        self.reason: UrlBlockedReason = reason
 
 
-def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
+def is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
     """Every address family we must not open a connection to. Checked explicitly rather than
     leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
     # ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
@@ -104,5 +103,5 @@ def check_url(raw: str) -> None:
             ip = ipaddress.ip_address(addr)
         except ValueError:  # getaddrinfo returned something unparseable; treat as unusable
             raise BlockedUrl(REASON_PRIVATE) from None
-        if _is_forbidden(ip):
+        if is_forbidden(ip):
             raise BlockedUrl(REASON_PRIVATE)

backend/app/draw.py

@@ -0,0 +1,118 @@
+"""A weighted draw that stays a surprise: what "Surprise me" deals from.
+
+A title's chance to come first is its weight over the pool's (Efraimidis–Spirakis: order by
+-ln(1-u)/w, u uniform per row). The weight is 1 + the title's affinity to what the user watches
+(`plex.affinity`, `discover.queries`), so a title nobody leans towards still comes up — only less
+often. Two things keep it a surprise rather than a ranking:
+
+- **u is the seed's md5 of the row's key**, the Home surprise's seeded order. With every weight 1
+  the order is exactly that md5 order, so a user with no history sees the unweighted draw, and one
+  seed still moves both sources of the Home hero together. A draw that is not seeded (the Plex
+  Discover rail) takes u from random().
+- **One slot of three is blind** (`blend`): dealt from the unweighted order, so the same few
+  favourites cannot fill the rail.
+
+The YouTube surprise draws over a window of the feed scope rather than all of it (`yt_surprise`):
+the window is read in SQL, and `shuffled` / `leaning` order it here with the same u.
+
+Both affinities weigh a watch by `recency` and score against the strongest with `scaled`, so the two
+sources lean by one rule.
+"""
+
+import hashlib
+import math
+from collections.abc import Callable, Hashable
+from datetime import datetime
+from typing import Any
+
+from sqlalchemy import BigInteger, Float, cast, func, literal
+from sqlalchemy.dialects.postgresql import BIT
+from sqlalchemy.orm import InstrumentedAttribute
+from sqlalchemy.sql.elements import ColumnElement
+
+# 2**32: the first 8 hex digits of an md5 as a fraction of one.
+_SPAN = 4294967296.0
+
+# A watch with no date (an old import) counts as one this many half-lives ago.
+UNDATED_HALF_LIVES = 2
+
+
+def recency(watched_at: datetime | None, now: datetime, half_life_days: float) -> float:
+    """A watch's weight in an affinity: 1 now, half as much every `half_life_days`."""
+    if watched_at is None:
+        return 0.5**UNDATED_HALF_LIVES
+    return 0.5 ** (max(0.0, (now - watched_at).total_seconds() / 86400) / half_life_days)
+
+
+def scaled(scores: dict[str, float]) -> dict[str, float]:
+    """Each score over the strongest, so the strongest is 1; empty when nothing scored."""
+    top = max(scores.values(), default=0.0)
+    return {k: v / top for k, v in scores.items()} if top else {}
+
+
+def seeded_u(
+    key: ColumnElement[Any] | InstrumentedAttribute[Any], seed: int
+) -> ColumnElement[float]:
+    """A uniform u in [0, 1) per row, from md5(key || seed) — the order `md5(...)` ascending gives."""
+    digest = func.md5(func.concat(key, str(seed)))
+    head = cast(cast(literal("x") + func.substr(digest, 1, 8), BIT(32)), BigInteger)
+    return cast(head, Float) / _SPAN
+
+
+def weighted_key(u: ColumnElement[float], weight: ColumnElement[float]) -> ColumnElement[float]:
+    """The row's place in a weighted draw, ascending: -ln(1-u)/w. Monotonic in u when every w is 1,
+    so the unweighted order is unchanged."""
+    return -func.ln(1.0 - u) / weight
+
+
+def random_u() -> ColumnElement[float]:
+    return func.random()
+
+
+def u_of(key: str, seed: int) -> float:
+    """`seeded_u` in Python: the same u for the same key and seed."""
+    return int(hashlib.md5(f"{key}{seed}".encode()).hexdigest()[:8], 16) / _SPAN
+
+
+def shuffled(items: list, seed: int, key: Callable[[Any], str]) -> list:
+    """`items` in the seed's md5 order — the order `ORDER BY md5(key || seed)` gives."""
+    return sorted(items, key=lambda it: hashlib.md5(f"{key(it)}{seed}".encode()).hexdigest())
+
+
+def leaning(
+    items: list, seed: int, key: Callable[[Any], str], weight: Callable[[Any], float]
+) -> list:
+    """`items` in the weighted draw's order (`weighted_key` in Python), ties in the md5 order."""
+    order = {key(it): i for i, it in enumerate(shuffled(items, seed, key))}
+    return sorted(
+        items,
+        key=lambda it: (-math.log(1.0 - u_of(key(it), seed)) / weight(it), order[key(it)]),
+    )
+
+
+def blend(
+    weighted: list, blind: list, offset: int, n: int, key: Callable[[Any], Hashable] = lambda x: x
+) -> list:
+    """`n` picks: slot i comes from `blind` when (i + offset) % 3 == 0, else from `weighted`, each
+    skipping what an earlier slot took (`key` tells two picks apart). A side that runs out leaves the
+    rest to the other."""
+    out: list = []
+    taken: set = set()
+    sources = (iter(weighted), iter(blind))
+
+    def nxt(source):
+        for item in source:
+            if key(item) not in taken:
+                return item
+        return None
+
+    for i in range(n):
+        first, second = (sources[1], sources[0]) if (i + offset) % 3 == 0 else sources
+        item = nxt(first)
+        if item is None:
+            item = nxt(second)
+        if item is None:
+            break
+        taken.add(key(item))
+        out.append(item)
+    return out

backend/app/feedquery.py

@@ -13,7 +13,19 @@ import binascii
 import json
 from datetime import date, datetime, timedelta, timezone
 
-from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, select
+from sqlalchemy import (
+    BigInteger,
+    Select,
+    and_,
+    case,
+    cast,
+    false,
+    func,
+    literal,
+    or_,
+    select,
+    tuple_,
+)
 from sqlalchemy.orm import Session, aliased
 
 from app.collation import collated
@@ -68,7 +80,7 @@ def feed_columns(user: User, state) -> list:
         Channel.handle.label("channel_handle"),
         Video.published_at,
         Video.thumbnail_url,
-        Video.duration_seconds,
+        Video.length_seconds.label("length_seconds"),
         Video.view_count,
         Video.is_short,
         Video.live_status,
@@ -88,7 +100,7 @@ def serialize(row) -> dict:
         "channel_url": _channel_url(row.channel_id, row.channel_handle),
         "published_at": iso(row.published_at),
         "thumbnail_url": row.thumbnail_url,
-        "duration_seconds": row.duration_seconds,
+        "duration_seconds": row.length_seconds,
         "view_count": row.view_count,
         "is_short": row.is_short,
         "live_status": row.live_status,
@@ -260,9 +272,9 @@ def filtered_query(
         # OR across the picked channels; every other filter still applies on top.
         query = query.where(Video.channel_id.in_(channel_ids))
     if min_duration is not None:
-        query = query.where(Video.duration_seconds >= min_duration)
+        query = query.where(Video.length_seconds >= min_duration)
     if max_duration is not None:
-        query = query.where(Video.duration_seconds <= max_duration)
+        query = query.where(Video.length_seconds <= max_duration)
     if max_age_days is not None:
         cutoff = datetime.now(timezone.utc).timestamp() - max_age_days * 86400
         query = query.where(Video.published_at >= datetime.fromtimestamp(cutoff, tz=timezone.utc))
@@ -503,10 +515,10 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
         "views": [{"expr": Video.view_count, "asc": False, "nullable": True, "kind": "num"}],
         "views_asc": [{"expr": Video.view_count, "asc": True, "nullable": True, "kind": "num"}],
         "duration_desc": [
-            {"expr": Video.duration_seconds, "asc": False, "nullable": True, "kind": "num"}
+            {"expr": Video.length_seconds, "asc": False, "nullable": True, "kind": "num"}
         ],
         "duration_asc": [
-            {"expr": Video.duration_seconds, "asc": True, "nullable": True, "kind": "num"}
+            {"expr": Video.length_seconds, "asc": True, "nullable": True, "kind": "num"}
         ],
         "title": [{"expr": title, "asc": True, "nullable": True, "kind": "str"}],
         "title_desc": [{"expr": title, "asc": False, "nullable": True, "kind": "str"}],
@@ -536,15 +548,17 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
             },
             pub,
         ],
-        # Deterministic per-seed shuffle: the same seed yields the same total order, so
-        # the cursor stays valid across pages of one shuffled run.
+        # The ring of stored random keys from the seed's point (`shuffle_walk`): the keys at or after
+        # it, then the ones before. The same seed yields the same total order, so the cursor stays
+        # valid across pages; `feed_page` reads it as two index ranges, never as this sort.
         "shuffle": [
             {
-                "expr": func.md5(func.concat(Video.id, str(seed))),
+                "expr": case((Video.shuffle_key < shuffle_start(seed), 1), else_=0),
                 "asc": True,
                 "nullable": False,
-                "kind": "str",
-            }
+                "kind": "num",
+            },
+            {"expr": Video.shuffle_key, "asc": True, "nullable": False, "kind": "num"},
         ],
     }
     # Relevance (search-result ranking) only exists when there's a query to rank against;
@@ -700,6 +714,43 @@ class BadCursor(ValueError):
     """A `cursor` this module did not write (or one from another sort): the caller's bad input."""
 
 
+# The golden ratio's fraction: consecutive seeds land far apart on the ring, never next to each other.
+_GOLDEN = 0.6180339887498949
+
+
+def shuffle_start(seed: int) -> float:
+    """The point in [0, 1) where a seed's shuffle starts on the ring of `Video.shuffle_key`.
+
+    Folded to 32 bits first: an int past a float's range overflowed the product, and one past 2**53
+    left the product no fraction to land on. A seed the client draws (under a billion) is unchanged."""
+    return ((seed % 2**32) * _GOLDEN) % 1.0
+
+
+def shuffle_walk(
+    db: Session, query: Select, seed: int, n: int, after: tuple[int, float, str] | None = None
+) -> list:
+    """Up to `n` rows of `query` in the seed's ring order: shuffle_key from the seed's point upwards,
+    then from 0 up to that point. `after` is a cursor's (phase, key, id): the walk resumes past it.
+
+    Each arc is an ascending range read along ix_videos_shuffle_key with the scope's filters checked
+    row by row, so a draw costs the rows it reads, not a sort of the scope — the md5(id || seed) order
+    it replaced read and sorted every row the filters let through (~350 ms on prod for ~213k)."""
+    start = shuffle_start(seed)
+    phase_from, key, last_id = after if after is not None else (0, None, None)
+    rows: list = []
+    for phase in (0, 1):
+        if phase < phase_from or len(rows) >= n:
+            continue
+        arc = Video.shuffle_key >= start if phase == 0 else Video.shuffle_key < start
+        q = query.where(arc)
+        if key is not None and phase == phase_from:
+            q = q.where(
+                tuple_(Video.shuffle_key, Video.id) > tuple_(literal(key), literal(last_id))
+            )
+        rows += db.execute(q.order_by(Video.shuffle_key, Video.id).limit(n - len(rows))).all()
+    return rows
+
+
 def feed_page(
     db: Session,
     user: User,
@@ -734,6 +785,14 @@ def feed_page(
             k["value"] = v
     if ranked and window is not None:
         rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
+    elif sort == "shuffle":
+        after = None
+        if vals is not None and cursor_id is not None:
+            try:
+                after = (int(vals[0]), float(vals[1]), str(cursor_id))
+            except (TypeError, ValueError, OverflowError) as e:  # Infinity, a 400-digit key
+                raise BadCursor("Invalid feed cursor") from e
+        rows = shuffle_walk(db, query, seed, limit + 1, after)
     else:
         query = query.order_by(*order_by(keys))
         if cursor_id is not None:

backend/app/jobruns.py

@@ -119,6 +119,16 @@ def recent_runs(db: Session, jobs: list[str], per_job: int) -> dict[str, list[Jo
     return out
 
 
+def last_run(db: Session, job: str, target: str) -> JobRun | None:
+    """The newest run of an ad-hoc `job` on `target` (a user's own import), or None."""
+    return db.execute(
+        select(JobRun)
+        .where(JobRun.job == job, JobRun.target == target)
+        .order_by(JobRun.started_at.desc(), JobRun.id.desc())
+        .limit(1)
+    ).scalar_one_or_none()
+
+
 def gc(db: Session, retention_days: int) -> int:
     """Delete finished runs older than `retention_days` (0 = keep forever). A run still going is
     never reaped. The caller commits."""

backend/app/main.py

@@ -151,7 +151,7 @@ async def _reap_hls_sessions() -> None:
     somebody pressed play again or the process restarted.
 
     A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
-    player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
+    player keeps checkpointing and that route touches the session, so a paused viewer is no longer
     invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
     held one whole segmented copy of a film per item on the download volume."""
     while True:
@@ -368,6 +368,10 @@ INDEX_HTML = STATIC_DIR / "index.html"
 # A public share link's own page: the player alone, without the app's bundle (frontend/watch.html).
 WATCH_HTML = STATIC_DIR / "watch.html"
 INDEX_HEADERS = {"Cache-Control": "no-cache"}
+# The paths a share link opens on, by what it opens (`links.share_path`). Any of them opens any link.
+# A new one also goes in frontend/public/robots.txt, the Vite dev entry and WatchPage's token parsing
+# (watchEntry.contract.test.ts).
+SHARE_PREFIXES = ("watch/", "file/", "folder/")
 app.mount(
     "/assets",
     StaticFiles(directory=STATIC_DIR / "assets", check_dir=False),
@@ -381,8 +385,10 @@ async def index() -> FileResponse:
 
 
 @app.get("/watch/{token}")
-async def watch_page(token: str):
-    """Serve the public player page for a share link, with per-video Open Graph tags injected so the
+@app.get("/file/{token}")
+@app.get("/folder/{token}")
+async def share_page(token: str):
+    """Serve the public page of a share link, with per-link Open Graph tags injected so the
     link unfurls richly (title/channel/thumbnail) in Messenger/social crawlers. A real browser
     ignores the extra tags and boots WatchPage; invalid/expired/password links fall back to the
     generic card."""
@@ -409,8 +415,8 @@ async def spa_fallback(full_path: str) -> FileResponse:
             # Real SPA-root assets (welcome images, favicon, robots.txt) rarely change and have
             # stable names, so a moderate cache is safe and satisfies the cache-lifetime audit.
             return FileResponse(candidate, headers={"Cache-Control": "public, max-age=604800"})
-    # A /watch/ path the route above does not match (a trailing segment, no token) is still the
+    # A share path the route above does not match (a trailing segment, no token) is still the
     # player's: WatchPage reads the token off the path and says when a link is gone.
-    if full_path.startswith("watch/"):
+    if full_path.startswith(SHARE_PREFIXES):
         return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
     return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)

backend/app/plex/affinity.py

@@ -0,0 +1,110 @@
+"""What a user's Plex history leans towards, as a weight for a surprise draw (`app.draw`).
+
+Two signals, both already in the database (no TMDB call):
+- **genres** of the titles the user watched, the recent ones weighing more (a half-life of
+  HALF_LIFE_DAYS), each scored 0..1 against the strongest;
+- the **TMDB recommendations** stored on those titles (`recommendations`), each scored by the most
+  recent title that recommends it.
+
+A title counts once however many of its episodes were watched — a binge of one show would otherwise
+outvote every film — and only the RECENT_TITLES latest titles count. A candidate's weight is
+1 + GENRE_WEIGHT × (the mean score of its genres) + REC_WEIGHT × (its recommendation score), so a
+title with no affinity keeps weight 1 and still comes up.
+"""
+
+import json
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+
+from sqlalchemy import Float, cast, func, literal, literal_column, select
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Session
+from sqlalchemy.sql.elements import ColumnElement
+
+from app import draw
+from app.plex.models import PlexItem, PlexShow, PlexState
+from app.userscope import owned_by
+
+HALF_LIFE_DAYS = 180
+RECENT_TITLES = 200
+GENRE_WEIGHT = 2.0
+REC_WEIGHT = 3.0
+
+
+@dataclass(frozen=True)
+class Affinity:
+    genres: dict[str, float] = field(default_factory=dict)
+    movie_recs: dict[str, float] = field(default_factory=dict)
+    show_recs: dict[str, float] = field(default_factory=dict)
+
+    def __bool__(self) -> bool:
+        return bool(self.genres or self.movie_recs or self.show_recs)
+
+    def weight(self, model: type[PlexItem] | type[PlexShow]) -> ColumnElement[float]:
+        """The draw weight of a `model` row (a movie or a show)."""
+        scores = cast(literal(json.dumps(self.genres)), JSONB)
+        recs = self.show_recs if model is PlexShow else self.movie_recs
+        genre = func.jsonb_array_elements_text(model.genres).table_valued("value").alias("g")
+        genre_aff = (
+            select(
+                func.coalesce(
+                    func.avg(func.coalesce(cast(scores.op("->>")(genre.c.value), Float), 0.0)),
+                    0.0,
+                )
+            )
+            .select_from(genre)
+            .scalar_subquery()
+        )
+        rec_aff = func.coalesce(
+            cast(cast(literal(json.dumps(recs)), JSONB).op("->>")(model.guid), Float), 0.0
+        )
+        return literal_column("1.0") + GENRE_WEIGHT * genre_aff + REC_WEIGHT * rec_aff
+
+
+def for_user(db: Session, uid: int) -> Affinity:
+    """The user's affinity from their latest watched titles; empty (`not aff`) with no history."""
+    # One key per title: a show for an episode, the item for a film. concat() of a NULL show_id is
+    # just "s", which nullif turns back into NULL so the film's key takes over.
+    title = func.coalesce(
+        func.nullif(func.concat("s", PlexItem.show_id), "s"), func.concat("m", PlexItem.id)
+    ).label("title")
+    latest = (
+        select(title, PlexState.watched_at, PlexItem.id.label("item_id"), PlexItem.show_id)
+        .join(PlexItem, PlexItem.id == PlexState.item_id)
+        .where(owned_by(PlexState.user_id, uid), PlexState.status == "watched")
+        .distinct(title)
+        .order_by(title, PlexState.watched_at.desc().nullslast())
+        .subquery()
+    )
+    rows = db.execute(
+        select(
+            latest.c.watched_at,
+            PlexItem.genres,
+            PlexItem.recommendations,
+            PlexShow.genres,
+            PlexShow.recommendations,
+        )
+        .select_from(latest)
+        .join(PlexItem, PlexItem.id == latest.c.item_id)
+        .outerjoin(PlexShow, PlexShow.id == latest.c.show_id)
+        .order_by(latest.c.watched_at.desc().nullslast())
+        .limit(RECENT_TITLES)
+    ).all()
+    if not rows:
+        return Affinity()
+
+    now = datetime.now(timezone.utc)
+    genres: dict[str, float] = {}
+    movie_recs: dict[str, float] = {}
+    show_recs: dict[str, float] = {}
+    for watched_at, item_genres, item_recs, show_genres, show_recs_json in rows:
+        r = draw.recency(watched_at, now, HALF_LIFE_DAYS)
+        for g in (show_genres if show_genres is not None else item_genres) or []:
+            genres[g] = genres.get(g, 0.0) + r
+        for rec in (show_recs_json if show_recs_json is not None else item_recs) or []:
+            if not rec.get("tmdb_id"):
+                continue
+            bucket = show_recs if rec.get("media_type") == "tv" else movie_recs
+            guid = f"tmdb:{rec['tmdb_id']}"
+            bucket[guid] = max(bucket.get(guid, 0.0), r)
+    return Affinity(draw.scaled(genres), draw.scaled(movie_recs), draw.scaled(show_recs))

backend/app/plex/discover.py

@@ -22,7 +22,8 @@ from sqlalchemy.orm import Session
 from app.collation import sorted_words
 from app.discover.queries import RAIL_LIMIT
 from app.models import User
-from app.plex import rollup
+from app import draw
+from app.plex import affinity, rollup
 from app.plex.cards import SHOW_NEW, display_title, leaf_card, movie_card, show_card
 from app.plex.models import PlexItem, PlexShow, PlexState
 from app.plex.query import (
@@ -399,22 +400,25 @@ def discover_page(db: Session, user: User, scope: str, lang: str, f: LibraryFilt
     }
 
 
-def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
+def _draw(db: Session, stmt, key_col, per: int, exclude: list[str], weight=None) -> list:
     """`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
     reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
-    short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
-    fresh = (
-        db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
-        .scalars()
-        .all()
-        if exclude
-        else []
-    )
+    short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail.
+    With a `weight` (`plex.affinity`) the draw leans on it, every third slot left blind (`app.draw`)."""
+
+    def pick(q, n: int) -> list:
+        blind = db.execute(q.order_by(func.random()).limit(n)).scalars().all()
+        if weight is None:
+            return list(blind)
+        key = draw.weighted_key(draw.random_u(), weight)
+        leaning = db.execute(q.order_by(key).limit(n)).scalars().all()
+        return draw.blend(list(leaning), list(blind), 1, n, key=lambda r: r.rating_key)
+
+    fresh = pick(stmt.where(key_col.not_in(exclude)), per) if exclude else []
     if len(fresh) == per:
-        return list(fresh)
+        return fresh
     taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
-    rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
-    return [*fresh, *rest.all()]
+    return [*fresh, *pick(stmt.where(*taken), per - len(fresh))]
 
 
 def _kept(
@@ -474,6 +478,8 @@ def surprise(
     if keep:
         return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
     per = min(limit, SURPRISE_PER_KIND)
+    # What the user's history leans towards; with none, every title weighs the same (a plain draw).
+    leaning = affinity.for_user(db, user.id)
     movies = []
     if movie_ids:
         # Exclude movies this user has watched.
@@ -501,6 +507,7 @@ def surprise(
             PlexItem.rating_key,
             per,
             exclude,
+            leaning.weight(PlexItem) if leaning else None,
         )
     shows = []
     if show_ids:
@@ -523,6 +530,7 @@ def surprise(
             PlexShow.rating_key,
             per,
             exclude,
+            leaning.weight(PlexShow) if leaning else None,
         )
     mcards = _movie_cards(db, user.id, movies, lang)
     scards = _show_cards(db, user.id, shows, lang)

backend/app/plex/models.py

@@ -227,16 +227,17 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
     playable: Mapped[str | None] = mapped_column(String(12), index=True)
     probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     # Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
-    # `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
-    # AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
-    # breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
-    # set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
-    # backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
-    # home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
-    # classifier so the catalog facet and the runtime encoder choice never disagree by codec.
+    # `playable="transcode"` item: True = the prod iGPU ENCODES it (a HW-decodable 8-bit SDR source, a
+    # true-HDR one through the OpenCL tonemap, or an 8-bit source in a codec the CPU decodes for it —
+    # VC-1, XviD, DivX), False = software libx264 (AV1, or a 10-bit SDR source — see plex/stream.py).
+    # Derived by `plex.sync.transcode_hw_capable`: set codec-only at sync (bit depth unknown from the
+    # cheap listing), then refined by the bit-depth backfill (`_probe_media_batch`), which downgrades a
+    # 10-bit SDR HEVC to False. The filterable twin of the rule `_select_encode` applies at play time;
+    # the two share their codec sets, not their code, so a change to one tree is a change to both.
     transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
     # Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
-    # not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
+    # not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw` (a
+    # 10-bit SDR source drops to software; an HDR one keeps the GPU through the tonemap).
     video_bit_depth: Mapped[int | None] = mapped_column(Integer)
     # Watermark for the bit-depth backfill: when this item's media Stream was last probed for
     # bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are

backend/app/plex/service.py

@@ -6,9 +6,9 @@ The rest of the mirror's writes stay with the sync (`plex.sync`): a collection's
 after a write-back, its local cleanup after a delete.
 """
 
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 
-from sqlalchemy import CursorResult, delete, func, select, update
+from sqlalchemy import CursorResult, delete, func, or_, select, update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.orm import Session
 
@@ -271,6 +271,10 @@ def reorder_playlist(db: Session, pl: PlexPlaylist, rating_keys: list[str]) -> N
 
 # --- a user's watch state --------------------------------------------------------------------
 
+# How long after a watched mark a mid-item position is read as the finishing player's late tick
+# (`checkpoint`): the tick is every 10 s, plus the request's own latency and a retry.
+_LATE_CHECKPOINT_S = 30
+
 
 def mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
     """Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
@@ -352,16 +356,33 @@ def checkpoint(
     now: datetime,
     *,
     dirty: bool,
-) -> PlexState:
-    """Record a resume position. `dirty` marks it not yet mirrored to Plex — set before a push, so
-    the reconcile re-pushes it if the push fails and the push can settle it once Plex confirms."""
+) -> bool:
+    """Record a resume position, and say whether it was recorded. `dirty` marks it not yet mirrored
+    to Plex — set before a push, so the reconcile re-pushes it if the push fails and the push can
+    settle it once Plex confirms.
+
+    Not onto a row marked watched in the last `_LATE_CHECKPOINT_S`: that position is the player's
+    tick sent just before its own finish (or a mark from another tab), and writing it made a watched
+    item in progress again — on Plex too, through the resume push it carried. Decided by the write
+    itself, since the mark can commit between any read here and the write. A rewatch is checkpointed
+    as before once the mark is that far behind it."""
     st = state_row(db, user, item_id, st)
-    st.position_seconds = position
-    st.progress_updated_at = now
+    values: dict[str, object] = {"position_seconds": position, "progress_updated_at": now}
     if dirty:
-        st.synced_to_plex = False
+        values["synced_to_plex"] = False
+    not_just_finished = or_(
+        PlexState.status != "watched",
+        PlexState.watched_at.is_(None),
+        PlexState.watched_at <= now - timedelta(seconds=_LATE_CHECKPOINT_S),
+    )
+    result = db.execute(
+        update(PlexState)
+        .where(PlexState.id == st.id, not_just_finished)
+        .values(values)
+        .execution_options(synchronize_session="fetch")
+    )
     db.flush()
-    return st
+    return isinstance(result, CursorResult) and result.rowcount > 0
 
 
 def forget_state(db: Session, st: PlexState) -> None:

backend/app/plex/stream.py

@@ -4,9 +4,10 @@ Playback is from the LOCAL physical file. Browser-compatible files (playable="di
 raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
 the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
 when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
-to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
-software libx264 as the automatic fallback where the render node is absent or the source codec isn't
-hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
+to H.264: on the prod host that is encoded on the Intel iGPU via VAAPI (`h264_vaapi`) — decoded there
+too when it can, else decoded on the CPU and uploaded (VC-1, XviD, DivX) — with software libx264 as
+the fallback where the render node is absent, for a source neither GPU route takes (AV1, 10-bit SDR),
+and when a GPU session produces nothing. Same HLS session/seek machinery either way.
 
 Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
 (fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
@@ -35,7 +36,7 @@ from app.config import settings
 from app.downloads.edit import encode_thread_cap
 from app.plex.models import PlexItem
 from app.plex import paths
-from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
+from app.plex.sync import CPU_DECODE_CODECS, HDR_TRANSFERS, HW_DECODABLE_CODECS
 
 log = logs.get_logger(__name__)
 _HLS_ROOT = (
@@ -66,7 +67,9 @@ _SESSION_IDLE_S = 600  # reap a session with no access for this long
 # restart leaves them behind like any other. `sweep_orphans` deletes ONLY names
 # matching one of those two shapes: `_HLS_ROOT` is admin-configurable (PLEX_HLS_DIR), so it may
 # legitimately point at a shared scratch path we do not own, and a blanket "delete every directory
-# under the root" would take the neighbours with it.
+# under the root" would take the neighbours with it. A torn-down session's directory is renamed to
+# its own name + `_TOMB` + 8 hex digits before it is deleted (see `_detach`), so a crash between the
+# two leaves a name this sweep still recognises.
 #
 # The KEY is anchored to digits (a Plex rating_key is a number) — that is what keeps foreign names
 # out. A `.+` there let `snapshot_2024_backup_+1.00` through, and pairing `.+` with an optional tail
@@ -74,8 +77,10 @@ _SESSION_IDLE_S = 600  # reap a session with no access for this long
 # non-numeric rating_key would merely leave its directory unswept, which is the safe direction to
 # fail. Covered by test_plex_stream.py — this regex is the only thing standing between the sweep
 # and someone else's data.
+_TOMB = ".dead-"
 _SESSION_DIR_RE = re.compile(
-    r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{2}$"
+    r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$"
+    rf"|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{{2}}({re.escape(_TOMB)}[0-9a-f]{{8}})?$"
 )
 # When this process started. `sweep_orphans` only deletes directories older than this: everything
 # newer belongs to a session THIS run created, and since `start_session` reuses a directory path
@@ -93,7 +98,7 @@ _MTIME_GRACE_S = 2.0
 
 _lock = threading.Lock()
 # Keyed by VIEWER and item, (user_id, rating_key). Keyed by the item alone, a second account opening
-# the same film `_kill`ed the first one's session: the displaced viewer got a 404, reclaimed at its
+# the same film tore down the first one's session: the displaced viewer got a 404, reclaimed at its
 # playhead, and so evicted the other in turn — two people could not watch one film at once. (One
 # account on two devices still shares a session; that case ends in the player's bounded reclaim.)
 _sessions: dict[tuple[int, str], "HlsSession"] = {}
@@ -116,12 +121,13 @@ class HlsSession:
         hw: bool = False,
         tonemap: bool = False,
         user_id: int = 0,
+        cpu_decode: bool = False,
     ):
         # Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
         # replaces the viewer's registered session for the item — and the old player's next request
         # for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
         # under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it.
-        # A request naming a session that is no longer registered is a 404,
+        # A request naming a session that is no longer registered is a 410,
         # which the player recovers from by restarting at the playhead.
         self.id = secrets.token_urlsafe(9)
         self.key = key  # the item's rating_key (what the logs name)
@@ -131,6 +137,7 @@ class HlsSession:
         self.transcode = transcode  # video re-encoded (non-h264 source), not stream-copied
         self.hw = hw  # transcode ran on the VAAPI hardware encoder (else libx264 software)
         self.tonemap = tonemap  # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
+        self.cpu_decode = cpu_decode  # HW encode of frames the CPU decoded (a codec the GPU can't)
         self.start_s = start_s  # the REQUESTED seek offset
         # The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
         # keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
@@ -184,8 +191,11 @@ _HLS_TAIL = [
 #   * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
 #     (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
 #     1080p transcodes are fine and the API stays responsive.
+#   * CPU decode → VAAPI encode (`format=nv12,hwupload` + `h264_vaapi`) — a codec the GPU cannot decode
+#     but whose 8-bit picture it encodes (`CPU_DECODE_CODECS`: VC-1, XviD, DivX). The encode is the
+#     expensive half, so this still takes most of the load off the CPU.
 #   * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
-#     iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
+#     iGPU enablement) or the source fits neither GPU route (AV1, a 10-bit SDR source). CPU-bound, so
 #     it reserves a core via `-threads` (see encode_thread_cap).
 # Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
 # unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
@@ -194,10 +204,10 @@ _VAAPI_DEVICE = "/dev/dri/renderD128"
 # here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
 _OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
 # Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
-# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
-# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
-# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
-# codec; aliased to the old name to keep the call sites below unchanged.
+# full-GPU VAAPI pipeline. A source outside it is decoded on the CPU when its codec is in
+# `CPU_DECODE_CODECS` (the GPU still encodes), else transcoded in software. THE single source of both
+# sets is plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't
+# disagree by codec; aliased to the old name to keep the call sites below unchanged.
 _HW_DECODABLE = HW_DECODABLE_CODECS
 # Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
 # plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
@@ -298,16 +308,22 @@ def _opencl_present() -> bool:
         return False
 
 
-def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
+def _hwaccel_in_opts(hw: bool, tonemap: bool = False, cpu_decode: bool = False) -> list[str]:
     """VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
     `-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
     round-trip to system memory. Empty for the software path.
 
     The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
     tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
-    uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
+    uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`.
+
+    The CPU-decode variant opens the VAAPI device for the FILTER graph only (`hwupload` needs one) and
+    asks for no `-hwaccel`: the codec is one the GPU refuses to decode. This exact form is the one
+    measured on prod with the HLS muxer and `-copyts` (and to the end of the file)."""
     if not hw:
         return []
+    if cpu_decode:
+        return ["-init_hw_device", f"vaapi=va:{_VAAPI_DEVICE}", "-filter_hw_device", "va"]
     if tonemap:
         return [
             "-init_hw_device",
@@ -355,6 +371,7 @@ def _venc_opts(
     preset: str = "veryfast",
     crf: int = 20,
     max_height: int = 0,
+    cpu_decode: bool = False,
 ) -> list[str]:
     """The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
     when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
@@ -381,6 +398,10 @@ def _venc_opts(
             "h264_vaapi",
             *force_kf,
         ]
+    if hw and cpu_decode:
+        # The CPU decoded the frames: convert them to the encoder's 8-bit nv12 and upload them to the
+        # device `-filter_hw_device` named (see _hwaccel_in_opts).
+        return ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", *force_kf]
     if hw:
         # scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
         # h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
@@ -418,13 +439,18 @@ def _render_node_present() -> bool:
     return Path(_VAAPI_DEVICE).exists()
 
 
-def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
-    """Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
-      * `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
-        10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
-      * `(True, False)`  → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
-      * `(True, True)`   → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
-        the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
+def _select_encode(
+    db: DbSession, item: PlexItem, transcode: bool, src: Path
+) -> tuple[bool, bool, bool]:
+    """Pick the transcode video pipeline for this session — returns `(hw, tonemap, cpu_decode)`:
+      * `(False, False, False)` → software libx264 (the fallback: no render node, a source neither GPU
+        route takes, or a 10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
+      * `(True, False, False)`  → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR
+        source the GPU decodes.
+      * `(True, True, False)`   → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG)
+        source, when the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
+      * `(True, False, True)`   → CPU decode → `hwupload` → VAAPI encode — an 8-bit SDR source in a codec
+        the GPU cannot decode (`CPU_DECODE_CODECS`).
 
     A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
     not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
@@ -434,31 +460,40 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
     backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
     graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
     present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
-    source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
+    source that has an OpenCL runtime, while a 10-bit SDR source drops to software. In both "on" and
+    "auto" a codec the GPU is known not to decode goes the CPU-decode route: asking the GPU to decode it
+    only bought the fallback's wasted second before libx264."""
+    sw = (False, False, False)
     if not transcode:
-        return (False, False)
+        return sw
     mode = sysconfig.get_str(db, "plex_transcode_hw")
     if mode == "off" or not _render_node_present():
-        return (False, False)
+        return sw
+    codec = (item.codec_video or "").lower()
     if mode == "on":
+        if codec in CPU_DECODE_CODECS:
+            return (True, False, True)
         # Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
         # backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
         # source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
         # (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
         tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
-        return (True, tonemap)
+        return (True, tonemap, False)
     # auto
-    if (item.codec_video or "").lower() not in _HW_DECODABLE:
-        return (False, False)
+    if codec in CPU_DECODE_CODECS:
+        # The upload converts to 8-bit nv12: only an 8-bit SDR picture comes through it unchanged.
+        return (True, False, True) if _probe_video_kind(src) == _VK_HWSAFE else sw
+    if codec not in _HW_DECODABLE:
+        return sw
     kind = _probe_video_kind(src)
     if kind == _VK_HWSAFE:
-        return (True, False)  # 8-bit SDR → full-GPU scale path
+        return (True, False, False)  # 8-bit SDR → full-GPU scale path
     if kind == _VK_HDR:
         # PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
-        return (True, True) if _opencl_present() else (False, False)
+        return (True, True, False) if _opencl_present() else sw
     # _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
     # curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
-    return (False, False)
+    return sw
 
 
 def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
@@ -467,16 +502,17 @@ def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
     return _select_encode(db, item, transcode, src)[0]
 
 
-# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
-# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
-# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
-# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
+# CPU-nice increment for a transcode child that works the CPU: a software encode, or a CPU decode
+# feeding the GPU encoder. The Plex ffmpeg is a Popen in the long-lived API process (which must NOT
+# itself be niced), so — unlike the download worker's _deprioritize (run once in its own main()) — we
+# lower priority per-child via preexec_fn in the fork. The full-GPU path needs none of this. Reuses
+# the download child's nice default so all co-located re-encoders share one handicap.
 _SW_TRANSCODE_NICE = settings.download_child_nice
 
 
 def _sw_child_preexec() -> None:
-    """Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
-    can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
+    """Runs in the forked child (POSIX) before exec: renice this CPU-bound transcode ffmpeg down so
+    it can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
     playback."""
     try:
         os.nice(_SW_TRANSCODE_NICE)
@@ -536,13 +572,15 @@ def _ffmpeg_cmd(
     crf: int = 20,
     max_height: int = 0,
     loudnorm: bool = False,
+    cpu_decode: bool = False,
 ) -> list[str]:
     # For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
     # hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
     # identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
     # encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
-    venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
-    args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
+    # `cpu_decode` is the HW encode of CPU-decoded frames: a device for the filter graph, no -hwaccel.
+    venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height, cpu_decode)
+    args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap, cpu_decode)]
     # Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
     # keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
     # (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
@@ -641,29 +679,82 @@ def _probe_first_pts(seg: Path) -> float | None:
     return None
 
 
-def _kill(s: HlsSession) -> None:
+# Teardown is split in two, because every segment request and checkpoint takes `_lock`: waiting up to
+# 3 s for an ffmpeg to exit and deleting a whole segmented film under it stalled every other
+# viewer's playback for one viewer's seek or one reap pass. `_detach` (under the lock) signals the
+# process and renames its directory aside — so the deletion can never hit a NEW session that reuses
+# the path — and `_finish` (after the lock is released) waits and deletes.
+#
+# What the rename cannot cover is the dying ffmpeg itself: it writes by path, so until it has exited
+# it can still put a segment, or its final ENDLIST playlist, into a new session's directory on the
+# same path. `_dying` remembers it by that path, and `_spawn` waits it out — under the lock, on
+# purpose — before it creates the directory again.
+_dying: dict[Path, subprocess.Popen] = {}
+
+
+def _await_exit(key: str, proc: subprocess.Popen) -> None:
     try:
-        s.proc.terminate()
         try:
-            s.proc.wait(timeout=3)
+            proc.wait(timeout=3)
         # silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
         # seconds gets KILL, which is exactly what this branch does.
         except subprocess.TimeoutExpired:
-            s.proc.kill()
+            proc.kill()
     except Exception as exc:  # noqa: BLE001 — teardown finishes whatever the process does
         # Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
         # handle and a core, and the transcode cap that is supposed to bound them counts rows in
         # `_sessions` rather than processes. One line is the only way that ever surfaces.
+        log.info(
+            "could not stop the transcode process",
+            extra=logs.event("plex.stream.kill_failed", key=key, reason=type(exc).__name__),
+        )
+
+
+def _detach(s: HlsSession) -> tuple[HlsSession, Path | None]:
+    """Caller holds _lock and has taken `s` out of `_sessions`. Hand the result to `_finish` once the
+    lock is released."""
+    try:
+        s.proc.terminate()
+    except Exception as exc:  # noqa: BLE001 — teardown finishes whatever the process does
         log.info(
             "could not stop the transcode process",
             extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
         )
-    shutil.rmtree(s.dir, ignore_errors=True)
+    _dying[s.dir] = s.proc
+    tomb = s.dir.with_name(f"{s.dir.name}{_TOMB}{secrets.token_hex(4)}")
+    try:
+        s.dir.rename(tomb)
+    # silent-ok: nothing on disk (ffmpeg died before writing, or a sweep took it) — nothing to delete.
+    except FileNotFoundError:
+        return s, None
+    except OSError as exc:
+        # Cannot be moved aside, so it cannot be deleted outside the lock either: the path may be a
+        # new session's a moment from now. Deleted here instead, the old way.
+        log.info(
+            "a session directory could not be moved aside; deleted under the lock",
+            extra=logs.event("plex.stream.tomb_failed", key=s.key, reason=type(exc).__name__),
+        )
+        shutil.rmtree(s.dir, ignore_errors=True)
+        return s, None
+    return s, tomb
 
 
-def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
-    """Caller holds _lock. Three bounds, applied after a session is registered, each dropping the
-    least recently accessed sessions over it:
+def _finish(doomed: "list[tuple[HlsSession, Path | None]]") -> None:
+    """The slow half of a teardown, called WITHOUT `_lock`."""
+    for s, tomb in doomed:
+        _await_exit(s.key, s.proc)
+        with _lock:
+            if _dying.get(s.dir) is s.proc:
+                del _dying[s.dir]
+        if tomb is not None:
+            shutil.rmtree(tomb, ignore_errors=True)
+
+
+def _enforce_caps(
+    user_id: int, max_transcodes: int, max_sessions: int
+) -> "list[tuple[HlsSession, Path | None]]":
+    """Caller holds _lock, and `_finish`es what this returns once it has released it. Three bounds,
+    applied after a session is registered, each dropping the least recently accessed sessions over it:
 
     * this account's own sessions (`_MAX_SESSIONS_PER_USER`) — first, so a single account filling the
       box evicts itself before anyone else;
@@ -672,10 +763,12 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
       whole segmented copy of the film under `_HLS_ROOT`, written faster than realtime. Counted only
       by the transcode cap, N distinct items were N ffmpegs and N film-sized scratch directories.
 
-    Somebody's playback stops at each drop. The route turns the missing session into a retryable 404
+    Somebody's playback stops at each drop. The route turns the missing session into a 410
     and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
     rebuffering" has no other explanation anywhere, hence the line."""
 
+    doomed: list[tuple[HlsSession, Path | None]] = []
+
     def cull(pool: list[tuple[tuple[int, str], HlsSession]], cap: int, bound: str) -> None:
         if len(pool) <= cap:
             return
@@ -690,12 +783,13 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
                     sessions=len(pool),
                 ),
             )
-            _kill(sess)
             _sessions.pop(reg, None)
+            doomed.append(_detach(sess))
 
     cull([kv for kv in _sessions.items() if kv[0][0] == user_id], _MAX_SESSIONS_PER_USER, "user")
     cull([kv for kv in _sessions.items() if kv[1].transcode], max_transcodes, "transcode")
     cull(list(_sessions.items()), max_sessions, "total")
+    return doomed
 
 
 def start_session(
@@ -740,7 +834,11 @@ def start_session(
         return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
 
     def _spawn(
-        hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
+        hw: bool,
+        tonemap: bool = False,
+        cpu_decode: bool = False,
+        *,
+        replace_only: "HlsSession | None" = None,
     ) -> "HlsSession | None":
         # `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
         # check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
@@ -755,72 +853,88 @@ def start_session(
             preset = sysconfig.get_str(db, "plex_transcode_preset")
             crf = sysconfig.get_int(db, "plex_transcode_crf")
             max_height = sysconfig.get_int(db, "plex_transcode_max_height")
-        # Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
-        # can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
-        # platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
-        preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
-        with _lock:
-            if replace_only is not None and _sessions.get(reg) is not replace_only:
-                return None  # superseded/culled while we waited out of lock — don't clobber the newer one
-            old = _sessions.pop(reg, None)
-            if old is not None:
-                _kill(old)
-            shutil.rmtree(directory, ignore_errors=True)
-            directory.mkdir(parents=True, exist_ok=True)
-            proc = subprocess.Popen(
-                _ffmpeg_cmd(
-                    src,
-                    start_s,
+        # Nice the child that works the CPU — a SOFTWARE transcode, or a CPU decode feeding the GPU
+        # encoder (not the full-GPU path, and not remux) — so it can't starve the co-located
+        # API/playback. preexec_fn runs in the forked child (POSIX); on a platform without fork (the API
+        # always runs in a Linux container in prod) it's simply omitted.
+        cpu_bound = transcode and (not hw or cpu_decode)
+        preexec = _sw_child_preexec if (cpu_bound and os.name == "posix") else None
+        doomed: list[tuple[HlsSession, Path | None]] = []
+        # Finished even when the new session fails to start (no space, no ffmpeg, a DB error): what
+        # was moved aside under the lock is otherwise left for a restart's sweep.
+        try:
+            with _lock:
+                if replace_only is not None and _sessions.get(reg) is not replace_only:
+                    return None  # superseded/culled while we waited out of lock — don't clobber the newer one
+                old = _sessions.pop(reg, None)
+                if old is not None:
+                    doomed.append(_detach(old))
+                # An ffmpeg still exiting from THIS path (the old session, or one stopped a moment ago)
+                # would write into the directory about to be made; see `_dying`.
+                prev = _dying.pop(directory, None)
+                if prev is not None:
+                    _await_exit(key, prev)
+                shutil.rmtree(directory, ignore_errors=True)
+                directory.mkdir(parents=True, exist_ok=True)
+                proc = subprocess.Popen(
+                    _ffmpeg_cmd(
+                        src,
+                        start_s,
+                        directory,
+                        audio_ord,
+                        aoff,
+                        audio_langs if is_multi else None,
+                        transcode=transcode,
+                        hw=hw,
+                        tonemap=tonemap,
+                        preset=preset,
+                        crf=crf,
+                        max_height=max_height,
+                        loudnorm=loudnorm,
+                        cpu_decode=cpu_decode,
+                    ),
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                    preexec_fn=preexec,
+                )
+                s = HlsSession(
+                    key,
                     directory,
-                    audio_ord,
-                    aoff,
-                    audio_langs if is_multi else None,
+                    proc,
+                    start_s,
+                    variants,
                     transcode=transcode,
                     hw=hw,
                     tonemap=tonemap,
-                    preset=preset,
-                    crf=crf,
-                    max_height=max_height,
-                    loudnorm=loudnorm,
-                ),
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-                preexec_fn=preexec,
-            )
-            s = HlsSession(
-                key,
-                directory,
-                proc,
-                start_s,
-                variants,
-                transcode=transcode,
-                hw=hw,
-                tonemap=tonemap,
-                user_id=user_id,
-            )
-            _sessions[reg] = s
-            # The one line that says which pipeline a session actually got. Everything downstream
-            # — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
-            # now it was only inferable from the ffmpeg command nobody keeps.
-            log.info(
-                "an HLS session started",
-                extra=logs.event(
-                    "plex.stream.session_started",
-                    key=key,
-                    transcode=transcode,
-                    hw=hw,
-                    tonemap=tonemap,
-                    variants=len(variants or ()),
-                    start_s=round(start_s, 1),
-                ),
-            )
-            # Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
-            # capped to zero.
-            _enforce_caps(
-                user_id,
-                max(1, sysconfig.get_int(db, "plex_max_transcodes")),
-                max(1, sysconfig.get_int(db, "plex_max_sessions")),
-            )
+                    user_id=user_id,
+                    cpu_decode=cpu_decode,
+                )
+                _sessions[reg] = s
+                # The one line that says which pipeline a session actually got. Everything downstream
+                # — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
+                # now it was only inferable from the ffmpeg command nobody keeps.
+                log.info(
+                    "an HLS session started",
+                    extra=logs.event(
+                        "plex.stream.session_started",
+                        key=key,
+                        transcode=transcode,
+                        hw=hw,
+                        tonemap=tonemap,
+                        cpu_decode=cpu_decode,
+                        variants=len(variants or ()),
+                        start_s=round(start_s, 1),
+                    ),
+                )
+                # Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
+                # capped to zero.
+                doomed += _enforce_caps(
+                    user_id,
+                    max(1, sysconfig.get_int(db, "plex_max_transcodes")),
+                    max(1, sysconfig.get_int(db, "plex_max_sessions")),
+                )
+        finally:
+            _finish(doomed)
         return s
 
     s = _spawn(*_select_encode(db, item, transcode, src))
@@ -847,7 +961,7 @@ def start_session(
             # stale and stops the session it names (`stop_session`), which stopped the playback that won.
             # Never None either: the POST /session route turns a None return into a "Media file not found
             # on disk" 404 — dishonest for a file that exists. A dead session instead degrades to a
-            # retryable 404 ("No active playback session" / "Not available yet") the client re-requests.
+            # 410 ("session_gone") on its URLs, and the client asks again.
             return s
         s = respawned
         ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
@@ -866,7 +980,8 @@ def start_session(
             if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
                 break
     log.info(
-        "plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
+        "plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s"
+        " tonemap=%s cpu_decode=%s",
         key,
         start_s,
         s.media_start_s,
@@ -875,6 +990,7 @@ def start_session(
         transcode,
         s.hw,
         s.tonemap,
+        s.cpu_decode,
     )
     return s
 
@@ -974,6 +1090,11 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
     # session, absent for a single-audio one.
     if variant not in (s.variants or (None,)):
         return False
+    # And the name must be the one ffmpeg SPELLS for that sequence: `int()` also reads `seg_007.ts`
+    # and a non-ASCII digit run as a sequence the session may have reached, and neither file is
+    # ever written — a yes there parked the request for the whole wait.
+    if filename != segment_name(seq, variant):
+        return False
     if seq == 0:
         return True
     try:
@@ -1084,9 +1205,13 @@ def sweep_orphans(older_than: float | None = None) -> int:
     return dropped
 
 
-# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
-# it (see reap_idle).
-_FINISHED_IDLE_S = 30
+# The slowest heartbeat a live player sends: a browser cuts the timers of a hidden page that makes no
+# sound to one a minute (measured on Chromium: a paused and a muted player, about a minute after
+# hiding; a hidden player with sound keeps its 10 s).
+_THROTTLED_HEARTBEAT_S = 60
+# How long a session whose ffmpeg has FINISHED may go without a signal before `reap_idle` takes it:
+# the beat before the reaper looks, one lost on the way, and the time the next one takes to arrive.
+_FINISHED_IDLE_S = 3 * _THROTTLED_HEARTBEAT_S
 
 
 def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
@@ -1095,29 +1220,39 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
 
     A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
     ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
-    and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
-    killed a one-minute pause, and playback died ~30 s after resuming.
+    and with its forward buffer full a paused player fetches nothing at all.
 
-    What makes the short rule safe again is that fetching is no longer the only signal: a player
-    parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
-    viewer who is merely paused says so far more often than this rule asks. The heartbeat
+    What makes the short rule safe is that fetching is not the only signal: a player parked MID-ITEM
+    checkpoints its position and that route calls `touch_session`. That beat is every 10 s only while
+    the page is visible or making sound; a hidden, silent one sends it once a minute, which a 30 s
+    rule killed whenever the reaper happened to run more than 30 s after the beat (measured: a paused
+    session gone that way, and one that lived on the same cadence because its phase was luckier).
+    So the rule is sized to the throttled beat (`_THROTTLED_HEARTBEAT_S`). The heartbeat
     stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
-    states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
-    scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing":
+    states where reaping is correct, and where the player's 410 → reclaim recovers a viewer who
+    scrubs back in anyway. Do not restate this as "whether or not it is playing":
     a review found that claim in two comments and not in the code. An earlier blanket
     `reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
     one a whole segmented copy of the film on a volume shared with the download center."""
     now = time.time()
-    dropped = 0
+    doomed: list[tuple[HlsSession, Path | None]] = []
+    reaped: list[tuple[str, str, int]] = []
     with _lock:
         for reg, s in list(_sessions.items()):
             done = s.proc.poll() is not None
             idle = now - s.last_access
             if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
-                _kill(s)
                 _sessions.pop(reg, None)
-                dropped += 1
-    return dropped
+                doomed.append(_detach(s))
+                reaped.append((s.key, "finished_idle" if done else "idle", int(idle)))
+    # The viewer of a session wrongly taken sees only a 410 later, so this line is the one record of why.
+    for key, reason, idle_s in reaped:
+        log.info(
+            "an idle session was reaped",
+            extra=logs.event("plex.stream.session_reaped", key=key, reason=reason, idle_s=idle_s),
+        )
+    _finish(doomed)
+    return len(doomed)
 
 
 def stop_session(user_id: int, key: str, session_id: str) -> bool:
@@ -1132,8 +1267,9 @@ def stop_session(user_id: int, key: str, session_id: str) -> bool:
         s = _sessions.get((user_id, key))
         if s is None or s.id != session_id:
             return False
-        _kill(s)
         _sessions.pop((user_id, key), None)
+        doomed = [_detach(s)]
+    _finish(doomed)
     return True
 
 

backend/app/plex/sync.py

@@ -60,14 +60,20 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
 
 # Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
 # the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
-# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
-# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
-# own encoder).
-HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
+# the runtime encoder choice (`_select_encode`) and this catalog facet can never disagree by codec. VC-1
+# is NOT in it: measured on prod, the iHD driver answers "No support for codec vc1 profile 3" (VC-1
+# Advanced, what a Blu-ray remux carries), so every session that asked the GPU to decode one died and
+# fell back to libx264.
+HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "mpeg2video"})
+# Codecs the iGPU cannot decode but whose picture it can still ENCODE: the CPU decodes, the frames go up
+# to the GPU as 8-bit nv12, and h264_vaapi encodes (plex.stream's `cpu_decode` path). Measured on prod:
+# three segments in ~5 s for a 1080p VC-1 remux (libx264: ~17 s) and ~1 s for an SD XviD or DivX
+# episode. Only codecs that are always 8-bit SDR, and only ones measured; any other codec stays software.
+CPU_DECODE_CODECS = frozenset({"vc1", "mpeg4", "msmpeg4v3"})
 # Color-transfer values marking an HDR source (PQ / HLG). THE single source of this set: plex.stream
 # (the tonemap choice) and downloads.clip (a clip's tonemap) import it here, so a transfer added for
-# one is tonemapped by all. A 10-bit/HDR source breaks the VAAPI scale=nv12 path on Gen 9.5 (see
-# plex.stream) → software transcode, so it counts as NOT HW-capable.
+# one is tonemapped by all. An HDR source breaks the plain VAAPI scale=nv12 path on Gen 9.5, so it takes
+# the OpenCL tonemap graph (see plex.stream) — still the GPU encoder.
 HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
 
 
@@ -78,18 +84,24 @@ def transcode_hw_capable(
     hdr: bool = False,
 ) -> bool | None:
     """The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
-    iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
-    a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
-    breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
-    time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
-    bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
+    iGPU ENCODES it ("Transcoded on the hardware encoder"), else software libx264; None for a
+    non-transcode item. The same answer plex.stream's `_select_encode` gives on prod (render node and
+    OpenCL runtime present, policy "auto"):
+      * a HW-decodable codec: 8-bit SDR → the full-GPU path; HDR → the OpenCL tonemap, also on the GPU;
+        10-bit SDR → software (an exact downconvert, never the HDR tone curve);
+      * a CPU-decode codec (`CPU_DECODE_CODECS`) at 8 bits → CPU decode, GPU encode;
+      * any other codec → software.
+    `bit_depth`/`hdr` are unknown at sync time (the cheap listing carries no Stream), so they default to
+    8-bit SDR and the bit-depth backfill refines a 10-bit SDR HEVC to False later."""
     if playable != "transcode":
         return None
-    if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
-        return False
-    if bit_depth is not None and bit_depth > 8:
-        return False
-    return not hdr
+    codec = (vcodec or "").lower()
+    deep = bit_depth is not None and bit_depth > 8
+    if codec in HW_DECODABLE_CODECS:
+        return hdr or not deep
+    if codec in CPU_DECODE_CODECS:
+        return not deep and not hdr
+    return False
 
 
 def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
@@ -266,8 +278,9 @@ def sync(db: Session) -> dict:
                 # a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
                 log.exception("the Plex people-enrich batch failed")
             # Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
-            # playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
-            # successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
+            # playback-class filter's HW/SW split is precise (a 10-bit SDR HEVC → software) over
+            # successive runs. Only HW-decodable-codec transcodes are probed — the rest are classed
+            # by codec alone.
             try:
                 stats["media_probed"] = _probe_media_batch(db, plex)
             except Exception:  # best-effort; never fail the sync over a backfill
@@ -294,15 +307,15 @@ def sync(db: Session) -> dict:
 # whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
 _ENRICH_BATCH = 60
 # Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
-# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
+# probed (typically a small minority — the bulk of transcodes are old codecs classed by codec), so
 # a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
 _PROBE_BATCH = 60
 
 
 def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
     """Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
-    bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
-    False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
+    bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit SDR HEVC →
+    False/software; an HDR one stays on the GPU through the tonemap). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
     so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
     watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
     rows = (

backend/app/quota.py

@@ -44,6 +44,7 @@ class QuotaAction:
     VIDEOS_ENRICH = "videos_enrich"
     VIDEOS_LOOKUP = "videos_lookup"
     VIDEOS_SEARCH = "videos_search"
+    VIDEOS_RELATED = "videos_related"
     CHANNELS_DISCOVER = "channels_discover"
     CHANNELS_EXPLORE = "channels_explore"
     CHANNELS_SUBSCRIBE = "channels_subscribe"
@@ -142,7 +143,7 @@ def backfill_reserve_reached() -> bool:
     condition for every backfill/explore path (recent, deep, immediate full-history, explore, and
     rolling revalidation).
 
-    Reads in its OWN short-lived session (like record_usage / log_action), for two reasons: it
+    Reads in its OWN short-lived session (like record_usage / claim_action), for two reasons: it
     replaces the remaining/reserve comparison that was open-coded at every call site, and — critically
     for the per-page guards in paging loops — a guard must not hold the caller's transaction open
     across the network fetch that follows it. Quota is committed independently of any caller, so a
@@ -151,24 +152,40 @@ def backfill_reserve_reached() -> bool:
         return remaining_today(qs) <= sysconfig.get_int(qs, "backfill_quota_reserve")
 
 
-def log_action(user_id: int, action: str) -> None:
-    """Record a zero-cost action event so per-user daily caps still count it.
+def claim_action(user_id: int, action: str, limit: int) -> int | None:
+    """Take one of this user's `limit` daily `action` slots, or None when they are all taken — the
+    check and the take in one transaction, under a lock per user and action. Checked first and
+    logged after the work, N requests sent together at the last free slot all passed the check
+    (the work in between is seconds of scraping). Returns the event's id, for `refund_action` when
+    the work then produced nothing.
 
-    `record_usage` only logs when units are actually charged; a scrape-based search spends no
-    quota yet must still be rate-limited per user, so it logs its event here directly.
-
-    Writes in its OWN session (like record_usage) — quota accounting is durable independent of the
-    caller's transaction and must never commit the caller's session (that would flush a job's
-    partial state mid-run, defeating its own rollback)."""
+    The ONE way to count a capped action: a claim is a zero-unit event, and `actions_today` counts
+    those alone, so a call the action then pays for (search.list's own `record_usage` event, under
+    the same action) is accounting, never a second slot. Own session, committed at once — quota
+    accounting is durable whatever the caller's transaction does next, and must never commit it."""
     with SessionLocal() as qs:
-        qs.add(QuotaEvent(user_id=user_id, action=action, units=0))
+        qs.execute(select(func.pg_advisory_xact_lock(func.hashtext(action), user_id)))
+        if actions_today(qs, user_id, action) >= limit:
+            return None
+        event = QuotaEvent(user_id=user_id, action=action, units=0)
+        qs.add(event)
+        qs.commit()
+        return event.id
+
+
+def refund_action(event_id: int) -> None:
+    """Give a slot `claim_action` took back: the work it was taken for failed outright, and a
+    request that showed the user nothing must not count against them."""
+    with SessionLocal() as qs:
+        qs.execute(delete(QuotaEvent).where(QuotaEvent.id == event_id))
         qs.commit()
 
 
 def actions_today(db: Session, user_id: int, action: str) -> int:
-    """How many quota events of `action` this user has logged so far in the current Pacific
-    day — for per-user, per-action daily caps (e.g. the live-search limit). Counts events,
-    not units, so it only works for actions charged exactly once per user action."""
+    """How many slots of `action` this user has claimed so far in the current Pacific day — for
+    per-user, per-action daily caps (the live search, the related lists). Counts the zero-unit
+    claim events (`claim_action`) only: `record_usage`'s events for units the action spent are not
+    slots (it logs only when units were charged)."""
     return (
         db.scalar(
             select(func.count())
@@ -176,6 +193,7 @@ def actions_today(db: Session, user_id: int, action: str) -> int:
             .where(
                 owned_by(QuotaEvent.user_id, user_id),
                 QuotaEvent.action == action,
+                QuotaEvent.units == 0,
                 QuotaEvent.created_at >= pacific_day_start_utc(),
             )
         )

backend/app/ratelimit.py

@@ -42,6 +42,14 @@ class RateLimiter:
         with self._lock:
             return sum(c for t, c in self._hits.get(key, []) if t > cutoff) >= self.max_events
 
+    def room(self, key: str) -> float:
+        """How much cost ``key`` may still spend in the current window, WITHOUT counting anything —
+        for work counted only once it has succeeded: ask it first, ``allow`` what was done."""
+        cutoff = time.monotonic() - self.window
+        with self._lock:
+            used = sum(c for t, c in self._hits.get(key, []) if t > cutoff)
+        return max(0.0, self.max_events - used)
+
     def allow(self, key: str, cost: float = 1) -> bool:
         """Record an attempt costing ``cost`` for ``key`` and return whether it is within the limit.
 

backend/app/reader/models.py

@@ -4,6 +4,7 @@ and the comfort shelf on Home. Only this unit writes them; another unit reads th
 edge, or asks this unit's service."""
 
 from datetime import datetime
+from enum import StrEnum
 
 from sqlalchemy import (
     Boolean,
@@ -84,6 +85,21 @@ class VideoState(Base, UpdatedAtMixin):
     progress_updated_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
 
 
+class VideoUnmark(Base):
+    """When the user last took a video back to "new" here — un-marked it, or reset it. That leaves
+    no `VideoState` (its absence is "new"), so without this row a later import of what they watched
+    on YouTube could not tell a video they un-marked from one never touched, and marked it watched
+    again. Kept by user and video; the video's own deletion takes it."""
+
+    __tablename__ = "video_unmarks"
+    __table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_video_unmark"),)
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
+    video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
+    at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
+
+
 class Playlist(Base, TimestampMixin, UpdatedAtMixin):
     """A per-user named, ordered collection of videos from the shared catalog.
 
@@ -213,3 +229,70 @@ class ComfortPin(Base, TimestampMixin):
     # 0-based shelf order. The shelf is short and hand-curated, so a dense integer the reorder
     # endpoint rewrites wholesale beats a fractional index nobody would ever exercise here.
     position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
+
+
+class TakeoutKind(StrEnum):
+    """What a Google Takeout row says the user did with a video."""
+
+    WATCHED = "watched"  # history/watch-history.json
+    WATCH_LATER = "watch_later"  # playlists/Watch later-videos.csv
+
+
+class TakeoutOutcome(StrEnum):
+    """What the import did with a row; NULL = still waiting for its video to be known (and, for a
+    watched one, probed for Shorts)."""
+
+    WATCHED = "watched"  # the video is marked watched at the Takeout time
+    KEPT = "kept"  # the state the user set here (watched, hidden, or changed later) stays
+    SHORT = "short"  # a Short: swiping one past is not watching it
+    ADDED = "added"  # added to Watch later
+    PRESENT = "present"  # already in Watch later
+    MISSING = "missing"  # YouTube no longer has the video
+
+
+class TakeoutItem(Base, TimestampMixin):
+    """One video from a user's Google Takeout export: watched (with when) or in their YouTube Watch
+    later list (with when it was added). The rows are both the import job's input and its memory: a
+    video the catalog does not know yet waits here, with no outcome, until it enters the catalog.
+    The id is not a foreign key for that reason. A watched row's channel and length feed the
+    "channels you watch but do not follow" list: the length is asked of YouTube for that list alone
+    and kept here, so its videos need not enter the shared catalog to be counted."""
+
+    __tablename__ = "takeout_items"
+    __table_args__ = (
+        UniqueConstraint("user_id", "kind", "video_id", name="uq_takeout_item"),
+        CheckConstraint("kind IN ('watched', 'watch_later')", name="ck_takeout_item_kind"),
+        # The waiting rows, by video: what a newly known video looks up.
+        Index(
+            "ix_takeout_items_waiting",
+            "video_id",
+            postgresql_where=text("outcome IS NULL"),
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
+    kind: Mapped[str] = mapped_column(String(12))
+    video_id: Mapped[str] = mapped_column(String(16))
+    channel_id: Mapped[str | None] = mapped_column(String(32))
+    # Watched: the last time in the export; Watch later: when it was added.
+    at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
+    outcome: Mapped[str | None] = mapped_column(String(12))
+    # When YouTube was asked for the video's length (NULL = not yet), and the length it gave; a
+    # video it no longer has keeps a NULL length.
+    looked_up_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    duration_seconds: Mapped[int | None] = mapped_column(Integer)
+
+
+class TakeoutChannelDismissal(Base, TimestampMixin):
+    """A channel the user put out of their "watched but not followed" list. The id is not a foreign
+    key: the channel may never have entered the catalog, or may leave it, and the choice stays."""
+
+    __tablename__ = "takeout_channel_dismissals"
+    __table_args__ = (
+        UniqueConstraint("user_id", "channel_id", name="uq_takeout_channel_dismissal"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
+    channel_id: Mapped[str] = mapped_column(String(32))

backend/app/reader/queries.py

@@ -1,5 +1,5 @@
-"""Reader's reads: a user's labels, saved views, watch state and playlists, for the HTTP layer and
-for the other units that ask about them.
+"""Reader's reads: a user's labels, saved views, watch state, playlists and Takeout import, for the HTTP
+layer and for the other units that ask about them.
 
 Every read of a user's row goes through `userscope`, so "someone else's row" is the same answer as
 a missing one. Reader reads Catalog's videos along its declared edge (a playlist's cover and
@@ -7,14 +7,28 @@ length, whether a video exists); another unit asks the questions at the end of t
 than joining Reader's tables itself.
 """
 
+from collections.abc import Iterable
+from datetime import datetime
+
 from sqlalchemy import ColumnElement, and_, func, or_, select
 from sqlalchemy.orm import InstrumentedAttribute, Session, aliased
 
 from app.collation import collated
 from app.feedquery import feed_columns, serialize
 from app.models import User
-from app.catalog.models import Channel, Subscription, Video
-from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
+from app.catalog.models import BlockedChannel, Channel, Subscription, Video
+from app.reader.models import (
+    ChannelTag,
+    Playlist,
+    PlaylistItem,
+    SavedView,
+    Tag,
+    TakeoutChannelDismissal,
+    TakeoutItem,
+    TakeoutKind,
+    VideoState,
+    VideoUnmark,
+)
 from app.userscope import owned_by, owned_row
 
 # --- catalog, read along the edge ------------------------------------------------------------
@@ -27,11 +41,25 @@ def video(db: Session, video_id: str) -> Video | None:
 # --- tags ------------------------------------------------------------------------------------
 
 
-def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int]]:
-    """The tags this user sees (the shared system ones and their own), each with how many channel
-    links it has, by category then name."""
+def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int, int]]:
+    """The tags this user sees (the shared system ones and their own), by category then name, each
+    with how many channel links it has and how many of the user's own links sit on a channel they
+    subscribe to and have not hidden — the rows the Channels page's tag filter shows (a hidden
+    channel has its own tab). A link outlives an unsubscribe and a hide, so the two differ."""
+    subscribed = select(Subscription.channel_id).where(
+        owned_by(Subscription.user_id, user), Subscription.hidden.is_(False)
+    )
+    on_subscribed = dict(
+        db.execute(
+            select(ChannelTag.tag_id, func.count(ChannelTag.id))
+            .where(owned_by(ChannelTag.user_id, user), ChannelTag.channel_id.in_(subscribed))
+            .group_by(ChannelTag.tag_id)
+        )
+        .tuples()
+        .all()
+    )
     return [
-        (tag, count)
+        (tag, count, on_subscribed.get(tag.id, 0))
         for tag, count in db.execute(
             select(Tag, func.count(ChannelTag.id))
             .outerjoin(ChannelTag, ChannelTag.tag_id == Tag.id)
@@ -118,12 +146,17 @@ def item_counts(db: Session, ids: list[int]) -> dict[int, int]:
     )
 
 
-def durations(db: Session, ids: list[int]) -> dict[int, int]:
-    """Each playlist's running time: the sum of its videos' lengths."""
+def durations(db: Session, ids: list[int]) -> dict[int, tuple[int, int]]:
+    """Each playlist's running time — the sum of its videos' known lengths — and how many videos
+    it left out for having none (`Video.without_length`)."""
     return {
-        pid: int(total or 0)
-        for pid, total in db.execute(
-            select(PlaylistItem.playlist_id, func.coalesce(func.sum(Video.duration_seconds), 0))
+        pid: (int(total or 0), int(left_out or 0))
+        for pid, total, left_out in db.execute(
+            select(
+                PlaylistItem.playlist_id,
+                func.coalesce(func.sum(Video.length_seconds), 0),
+                func.count().filter(Video.without_length),
+            )
             .join(Video, Video.id == PlaylistItem.video_id)
             .where(PlaylistItem.playlist_id.in_(ids))
             .group_by(PlaylistItem.playlist_id)
@@ -163,11 +196,16 @@ def covers(db: Session, ids: list[int]) -> dict[int, str | None]:
     )
 
 
-def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None]:
-    """One playlist's item count, running time and cover. An outer join, so an empty list still
-    answers (0, 0, None)."""
-    count, total = db.execute(
-        select(func.count(PlaylistItem.id), func.coalesce(func.sum(Video.duration_seconds), 0))
+def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, int, str | None]:
+    """One playlist's item count, running time, the videos the running time left out for having no
+    length (`Video.without_length`) and cover. An outer join, so an empty list still answers
+    (0, 0, 0, None)."""
+    count, total, left_out = db.execute(
+        select(
+            func.count(PlaylistItem.id),
+            func.coalesce(func.sum(Video.length_seconds), 0),
+            func.count().filter(Video.without_length),
+        )
         .select_from(PlaylistItem)
         .outerjoin(Video, Video.id == PlaylistItem.video_id)
         .where(PlaylistItem.playlist_id == playlist_id)
@@ -179,7 +217,7 @@ def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None
         .order_by(PlaylistItem.position)
         .limit(1)
     )
-    return count or 0, total or 0, cover
+    return count or 0, total or 0, left_out or 0, cover
 
 
 def playlist_rows(db: Session, user: User, playlist_id: int) -> list:
@@ -230,6 +268,18 @@ def kept_by_a_reader(video_id: InstrumentedAttribute[str]) -> ColumnElement[bool
     )
 
 
+def in_a_watch_history(channel_id: InstrumentedAttribute[str]) -> ColumnElement[bool]:
+    """A channel someone's uploaded Takeout history shows watched. The "watched but not followed"
+    list names a channel from its catalog row, so the explore cleanup keeps the row of one (its
+    videos it may still reclaim: opening the list's videos brings them back). As a clause, so the
+    cleanup's DELETE stays one statement."""
+    return (
+        select(TakeoutItem.id)
+        .where(TakeoutItem.channel_id == channel_id, TakeoutItem.kind == TakeoutKind.WATCHED)
+        .exists()
+    )
+
+
 def playlisted_by(db: Session, video_ids: list[str]) -> list[tuple[int, str]]:
     """(user id, video id) for every user who has one of these videos in a playlist."""
     return list(
@@ -293,6 +343,36 @@ def held_by(db: Session, user: User, video_ids: list[str]) -> set[str]:
     )
 
 
+def seen_by(db: Session, user: User, video_ids: list[str]) -> set[str]:
+    """Which of these videos the user has already seen or set aside: watched or hidden here, or in
+    their uploaded Takeout history unless they took it back to "new" here since. The ids need not
+    be in the catalog — a Takeout row names a video the catalog may never have had — so a list
+    of what YouTube suggests can drop them before anything is ingested."""
+    if not video_ids:
+        return set()
+    marked = select(VideoState.video_id).where(
+        owned_by(VideoState.user_id, user),
+        VideoState.video_id.in_(video_ids),
+        VideoState.status.in_(("watched", "hidden")),
+    )
+    unmarked_since = (
+        select(VideoUnmark.id)
+        .where(
+            owned_by(VideoUnmark.user_id, user),
+            VideoUnmark.video_id == TakeoutItem.video_id,
+            VideoUnmark.at > TakeoutItem.at,
+        )
+        .exists()
+    )
+    in_history = select(TakeoutItem.video_id).where(
+        owned_by(TakeoutItem.user_id, user),
+        TakeoutItem.kind == TakeoutKind.WATCHED,
+        TakeoutItem.video_id.in_(video_ids),
+        ~unmarked_since,
+    )
+    return set(db.scalars(marked)) | set(db.scalars(in_history))
+
+
 def feed_rows_by_ids(db: Session, user: User, video_ids: list[str]) -> list[dict]:
     """These videos serialized with the feed's projection (their channel, this user's watch state,
     saved), in the order given."""
@@ -359,3 +439,355 @@ def user_tag_link(db: Session, user: User, channel_id: str, tag_id: int) -> Chan
             owned_by(ChannelTag.user_id, user),
         )
     ).scalar_one_or_none()
+
+
+# --- Google Takeout --------------------------------------------------------------------------
+
+
+def _settled_here() -> ColumnElement[bool]:
+    """A video a waiting "watched" row may be marked on: one the catalog keeps of its own, not a
+    discovery video only a search or an explore brought in. A mark is a `VideoState`, which the
+    cleanup jobs read as a reader keeping it (`kept_by_a_reader`): on one of those it would keep the
+    video, and its explored channel, for good. Such a row waits until the video is the catalog's own
+    (its channel subscribed to) — or is gone."""
+    subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
+    return and_(Video.via_explore.is_(False), or_(Video.via_search.is_(False), subscribed))
+
+
+def takeout_watched_ready(db: Session, user: User) -> list[tuple[int, str, datetime, bool]]:
+    """The user's waiting "watched" rows whose video the catalog knows and has probed for Shorts:
+    (row id, video id, Takeout time, is Short)."""
+    return list(
+        db.execute(
+            select(TakeoutItem.id, TakeoutItem.video_id, TakeoutItem.at, Video.is_short)
+            .join(Video, Video.id == TakeoutItem.video_id)
+            .where(
+                owned_by(TakeoutItem.user_id, user),
+                TakeoutItem.kind == TakeoutKind.WATCHED,
+                TakeoutItem.outcome.is_(None),
+                Video.shorts_probed,
+                _settled_here(),
+            )
+        ).tuples()
+    )
+
+
+def takeout_waiting(db: Session, user: User, kind: TakeoutKind) -> list[tuple[int, str]]:
+    """The user's rows of `kind` the import has not settled, oldest first: (row id, video id)."""
+    return list(
+        db.execute(
+            select(TakeoutItem.id, TakeoutItem.video_id)
+            .where(
+                owned_by(TakeoutItem.user_id, user),
+                TakeoutItem.kind == kind,
+                TakeoutItem.outcome.is_(None),
+            )
+            .order_by(TakeoutItem.at, TakeoutItem.id)
+        ).tuples()
+    )
+
+
+# Ids per IN list: a list is bound one parameter an id, and a statement takes at most 65,535.
+_IN_CHUNK = 5000
+
+
+def known_video_ids(db: Session, ids: list[str]) -> set[str]:
+    """Which of `ids` the catalog holds."""
+    known: set[str] = set()
+    for start in range(0, len(ids), _IN_CHUNK):
+        chunk = ids[start : start + _IN_CHUNK]
+        known.update(db.execute(select(Video.id).where(Video.id.in_(chunk))).scalars())
+    return known
+
+
+def unmarked_at(db: Session, user: User, ids: list[str]) -> dict[str, datetime]:
+    """When the user last took each of `ids` back to "new" here, for those they ever did."""
+    out: dict[str, datetime] = {}
+    for start in range(0, len(ids), _IN_CHUNK):
+        rows = db.execute(
+            select(VideoUnmark.video_id, VideoUnmark.at).where(
+                owned_by(VideoUnmark.user_id, user),
+                VideoUnmark.video_id.in_(ids[start : start + _IN_CHUNK]),
+            )
+        ).tuples()
+        out.update((vid, at) for vid, at in rows)
+    return out
+
+
+def in_playlist(db: Session, playlist_id: int, ids: list[str]) -> set[str]:
+    """Which of `ids` the playlist already holds."""
+    held: set[str] = set()
+    for start in range(0, len(ids), _IN_CHUNK):
+        chunk = ids[start : start + _IN_CHUNK]
+        held.update(
+            db.execute(
+                select(PlaylistItem.video_id).where(
+                    PlaylistItem.playlist_id == playlist_id, PlaylistItem.video_id.in_(chunk)
+                )
+            ).scalars()
+        )
+    return held
+
+
+def takeout_rows_besides(
+    db: Session, user: User, kind: TakeoutKind, video_ids: Iterable[str]
+) -> int:
+    """How many Takeout rows the user keeps other than `video_ids` of `kind`: what an upload of
+    those videos would be added to."""
+    total = (
+        db.scalar(select(func.count(TakeoutItem.id)).where(owned_by(TakeoutItem.user_id, user)))
+        or 0
+    )
+    ids = list(video_ids)
+    for start in range(0, len(ids), _IN_CHUNK):
+        total -= (
+            db.scalar(
+                select(func.count(TakeoutItem.id)).where(
+                    owned_by(TakeoutItem.user_id, user),
+                    TakeoutItem.kind == kind,
+                    TakeoutItem.video_id.in_(ids[start : start + _IN_CHUNK]),
+                )
+            )
+            or 0
+        )
+    return total
+
+
+def takeout_outcomes(db: Session, user: User) -> dict[tuple[str, str | None], int]:
+    """How many of the user's rows each kind and outcome holds (None = waiting)."""
+    rows = db.execute(
+        select(TakeoutItem.kind, TakeoutItem.outcome, func.count())
+        .where(owned_by(TakeoutItem.user_id, user))
+        .group_by(TakeoutItem.kind, TakeoutItem.outcome)
+    ).all()
+    return {(kind, outcome): n for kind, outcome, n in rows}
+
+
+def takeout_summary(db: Session, user: User) -> dict[str, dict[str, int]]:
+    """The user's kept Takeout videos by kind and outcome, for their settings page: `{"watched":
+    {"watched": n, "short": n, "waiting": n, …}, "watch_later": {…}}` — the import's outcome words,
+    "waiting" for a video not known yet."""
+    out: dict[str, dict[str, int]] = {kind.value: {} for kind in TakeoutKind}
+    for (kind, outcome), n in takeout_outcomes(db, user).items():
+        out[kind][outcome or "waiting"] = n
+    return out
+
+
+def takeout_users_ready(db: Session) -> list[int]:
+    """Every user with a waiting "watched" row whose video is now known and probed for Shorts: the
+    rows a later run of the Shorts probe made ready."""
+    return list(
+        db.execute(
+            select(TakeoutItem.user_id)
+            .join(Video, Video.id == TakeoutItem.video_id)
+            .where(
+                TakeoutItem.kind == TakeoutKind.WATCHED,
+                TakeoutItem.outcome.is_(None),
+                Video.shorts_probed,
+                _settled_here(),
+            )
+            .distinct()
+        ).scalars()
+    )
+
+
+def takeout_users_watch_later_waiting(db: Session) -> list[int]:
+    """Every user with a Watch later row a run could not put in yet."""
+    return list(
+        db.execute(
+            select(TakeoutItem.user_id)
+            .where(TakeoutItem.kind == TakeoutKind.WATCH_LATER, TakeoutItem.outcome.is_(None))
+            .distinct()
+        ).scalars()
+    )
+
+
+# --- channels watched but not followed ---------------------------------------------------------
+
+# A channel joins the list with this many watched videos of at least this length. A Short is three
+# minutes at most, so the length alone keeps them out, and a video under five minutes says too
+# little of what a channel is about to count towards it.
+WATCHED_CHANNEL_MIN_VIDEOS = 5
+WATCHED_CHANNEL_MIN_SECONDS = 300
+
+
+def _unfollowed(user: User) -> ColumnElement[bool]:
+    """A watched row whose channel the user neither follows, blocked, put out of the list, nor is."""
+    subscribed = select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
+    blocked = select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user))
+    dismissed = select(TakeoutChannelDismissal.channel_id).where(
+        owned_by(TakeoutChannelDismissal.user_id, user)
+    )
+    clauses = [
+        owned_by(TakeoutItem.user_id, user),
+        TakeoutItem.kind == TakeoutKind.WATCHED,
+        TakeoutItem.channel_id.is_not(None),
+        TakeoutItem.channel_id.not_in(subscribed),
+        TakeoutItem.channel_id.not_in(blocked),
+        TakeoutItem.channel_id.not_in(dismissed),
+    ]
+    if user.yt_channel_id:
+        clauses.append(TakeoutItem.channel_id != user.yt_channel_id)
+    return and_(*clauses)
+
+
+def _candidate_channels(user: User):
+    """The unfollowed channels with enough watched videos to make the list once their lengths are
+    known: (channel id, how many watched)."""
+    return (
+        select(TakeoutItem.channel_id, func.count().label("watched"))
+        .where(_unfollowed(user))
+        .group_by(TakeoutItem.channel_id)
+        .having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
+        .subquery()
+    )
+
+
+def takeout_lengths_to_ask(db: Session, user: User) -> list[str]:
+    """The watched videos whose length the list still needs: those of the candidate channels not
+    asked about yet, the most watched channels first, so a lookup cut short fills the top."""
+    cand = _candidate_channels(user)
+    return list(
+        db.execute(
+            select(TakeoutItem.video_id)
+            .join(cand, cand.c.channel_id == TakeoutItem.channel_id)
+            .where(_unfollowed(user), TakeoutItem.looked_up_at.is_(None))
+            .order_by(cand.c.watched.desc(), TakeoutItem.channel_id, TakeoutItem.at.desc())
+        ).scalars()
+    )
+
+
+def watched_channels_pending(db: Session, user: User) -> int:
+    """How many channels a later run may still add to the list: the candidates with a watched video
+    whose length is not known, and the ones that made it but have no catalog row to be listed from
+    yet. One the catalog holds without details is listed already (by the name it came in with), so
+    it is not one to come."""
+    cand = _candidate_channels(user)
+    unasked = db.execute(
+        select(TakeoutItem.channel_id)
+        .join(cand, cand.c.channel_id == TakeoutItem.channel_id)
+        .where(_unfollowed(user), TakeoutItem.looked_up_at.is_(None))
+        .distinct()
+    ).scalars()
+    counted = _counted_channels(user)
+    unlisted = db.execute(
+        select(counted.c.channel_id)
+        .outerjoin(Channel, Channel.id == counted.c.channel_id)
+        .where(Channel.id.is_(None))
+    ).scalars()
+    return len(set(unasked) | set(unlisted))
+
+
+def _counted_channels(user: User):
+    """The channels that make the list by their counted videos, whatever the catalog holds of them."""
+    return (
+        select(TakeoutItem.channel_id)
+        .where(_unfollowed(user), TakeoutItem.duration_seconds >= WATCHED_CHANNEL_MIN_SECONDS)
+        .group_by(TakeoutItem.channel_id)
+        .having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
+        .subquery()
+    )
+
+
+def watched_channels_to_describe(db: Session, user: User) -> list[str]:
+    """The channels on the list (by their counted videos) that the catalog lacks or has no details
+    of yet: what the list cannot name, or names with no more than a video brought."""
+    counted = _counted_channels(user)
+    return list(
+        db.execute(
+            select(counted.c.channel_id)
+            .outerjoin(Channel, Channel.id == counted.c.channel_id)
+            .where(Channel.details_synced_at.is_(None))
+        ).scalars()
+    )
+
+
+def watched_channels(db: Session, user: User) -> list[tuple[Channel, int, datetime]]:
+    """The channels the user watched often but does not follow: (channel, how many of its videos
+    over five minutes they watched, when last). Most watched first; on a tie the one watched more
+    lately, which is the likelier interest."""
+    n = func.count(TakeoutItem.id)
+    last = func.max(TakeoutItem.at)
+    return list(
+        db.execute(
+            select(Channel, n, last)
+            .join(Channel, Channel.id == TakeoutItem.channel_id)
+            .where(_unfollowed(user), TakeoutItem.duration_seconds >= WATCHED_CHANNEL_MIN_SECONDS)
+            .group_by(Channel.id)
+            .having(n >= WATCHED_CHANNEL_MIN_VIDEOS)
+            .order_by(n.desc(), last.desc(), Channel.id)
+        ).tuples()
+    )
+
+
+def has_watch_history(db: Session, user: User) -> bool:
+    """Whether the user has uploaded a Takeout watch history at all."""
+    return bool(
+        db.scalar(
+            select(
+                select(TakeoutItem.id)
+                .where(owned_by(TakeoutItem.user_id, user), TakeoutItem.kind == TakeoutKind.WATCHED)
+                .exists()
+            )
+        )
+    )
+
+
+def watched_in_history(db: Session, user: User, channel_id: str) -> bool:
+    """Whether the user's Takeout history shows them watching this channel at all."""
+    return bool(
+        db.scalar(
+            select(
+                select(TakeoutItem.id)
+                .where(
+                    owned_by(TakeoutItem.user_id, user),
+                    TakeoutItem.kind == TakeoutKind.WATCHED,
+                    TakeoutItem.channel_id == channel_id,
+                )
+                .exists()
+            )
+        )
+    )
+
+
+def watched_channel_videos(
+    db: Session, user: User, channel_id: str
+) -> tuple[list[tuple[str, datetime]], int]:
+    """The user's watched videos of one channel that the list counts, last watched first: (video
+    id, when). And how many YouTube no longer has."""
+    rows = db.execute(
+        select(TakeoutItem.video_id, TakeoutItem.at, TakeoutItem.duration_seconds)
+        .where(
+            owned_by(TakeoutItem.user_id, user),
+            TakeoutItem.kind == TakeoutKind.WATCHED,
+            TakeoutItem.channel_id == channel_id,
+            TakeoutItem.looked_up_at.is_not(None),
+        )
+        .order_by(TakeoutItem.at.desc(), TakeoutItem.video_id)
+    ).tuples()
+    counted: list[tuple[str, datetime]] = []
+    gone = 0
+    for vid, at, seconds in rows:
+        if seconds is None:
+            gone += 1
+        elif seconds >= WATCHED_CHANNEL_MIN_SECONDS:
+            counted.append((vid, at))
+    return counted, gone
+
+
+def takeout_users_with_list_work(db: Session) -> list[int]:
+    """Every user whose "watched but not followed" list a run can still add to: lengths to ask, or
+    channels to name. Asked of each user whose history holds a channel of enough watched videos,
+    since what is off a list (followed, blocked, put out, their own) is that user's own — a row of
+    one is never asked about, and would otherwise bring the user back on every run."""
+    per_channel = (
+        select(TakeoutItem.user_id)
+        .where(TakeoutItem.kind == TakeoutKind.WATCHED, TakeoutItem.channel_id.is_not(None))
+        .group_by(TakeoutItem.user_id, TakeoutItem.channel_id)
+        .having(func.count() >= WATCHED_CHANNEL_MIN_VIDEOS)
+        .subquery()
+    )
+    users = db.execute(
+        select(User).where(User.id.in_(select(per_channel.c.user_id))).order_by(User.id)
+    ).scalars()
+    return [user.id for user in users if watched_channels_pending(db, user)]

backend/app/reader/service.py

@@ -9,13 +9,27 @@ unique key, a row lock taken before a presence check. The comfort shelf's writes
 
 from datetime import datetime, timezone
 
-from sqlalchemy import exists, func, select, text, update
+from typing import Literal, cast
+
+from sqlalchemy import Table, bindparam, delete, exists, func, select, text, update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.orm import Session
 
 from app.models import User
 from app.reader import queries
-from app.reader.models import ChannelTag, Playlist, PlaylistItem, SavedView, Tag, VideoState
+from app.reader.models import (
+    ChannelTag,
+    Playlist,
+    PlaylistItem,
+    SavedView,
+    Tag,
+    TakeoutChannelDismissal,
+    TakeoutItem,
+    TakeoutKind,
+    TakeoutOutcome,
+    VideoState,
+    VideoUnmark,
+)
 from app.sync.playlists import lock_playlists, recompute_dirty
 from app.userscope import owned_by
 
@@ -196,6 +210,7 @@ def unmark(db: Session, user: User, video_id: str) -> None:
     the in-progress state, not where the user left off); otherwise the row goes. The player fires
     this beside a checkpoint that may delete the same row: the caller's commit then raises
     `StaleDataError`, which is that lost race and nothing worse."""
+    lock_takeout(db, user)  # an import settling meanwhile reads the un-mark below, or comes first
     row = queries.video_state(db, user, video_id)
     if row is None:
         return
@@ -204,9 +219,21 @@ def unmark(db: Session, user: User, video_id: str) -> None:
         row.watched_at = None
     else:
         db.delete(row)
+    _remember_unmark(db, user, video_id)
     db.flush()
 
 
+def _remember_unmark(db: Session, user: User, video_id: str) -> None:
+    """Note that the user took the video back to "new" now: the state they set leaves no row, and
+    an import of their YouTube history must not mark it watched again (`apply_takeout_watched`)."""
+    now = datetime.now(timezone.utc)
+    db.execute(
+        pg_insert(VideoUnmark)
+        .values(user_id=user.id, video_id=video_id, at=now)
+        .on_conflict_do_update(constraint="uq_video_unmark", set_={"at": now})
+    )
+
+
 def mark(db: Session, user: User, video_id: str, status: str) -> None:
     """ "watched" or "hidden", as one atomic upsert on (user, video). The player marks "watched" at
     the end of playback at the instant a checkpoint may delete the in-progress row; a SELECT then
@@ -228,15 +255,6 @@ def mark(db: Session, user: User, video_id: str, status: str) -> None:
     )
 
 
-def clear_state(db: Session, user: User, video_id: str) -> None:
-    """The video back to pristine for this user: status, resume point and `watched_at` all go.
-    Playlist membership lives elsewhere and stays."""
-    row = queries.video_state(db, user, video_id)
-    if row is not None:
-        db.delete(row)
-        db.flush()
-
-
 def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
     """The player's resume point. 0 clears it: a row holding nothing else goes, one with a status
     (watched, hidden) keeps it with the position zeroed. The status itself is never changed here.
@@ -272,12 +290,17 @@ def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
 # --- playlists -------------------------------------------------------------------------------
 
 
+# Every reason a move is refused; the UI names each one (`playlists.moveErrors.<reason>`, held by the
+# frontend's backendIds contract, which reads this Literal).
+MoveReason = Literal["same_playlist"]
+
+
 class MoveRefused(Exception):
     """A move between playlists that cannot happen: `reason` is the code the UI translates."""
 
-    def __init__(self, reason: str):
+    def __init__(self, reason: MoveReason):
         super().__init__(reason)
-        self.reason = reason
+        self.reason: MoveReason = reason
 
 
 def touch(pl: Playlist) -> None:
@@ -383,30 +406,6 @@ def remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -
     return True
 
 
-def place_item(db: Session, pl: Playlist, video_id: str, position: int) -> None:
-    """Put a video already in the list at `position` (clamped to the end), renumbering the whole
-    list: positions can have gaps after removals, so "index N" is only meaningful over the
-    ordered rows."""
-    rows = queries.items(db, pl.id)
-    new = next(r for r in rows if r.video_id == video_id)
-    rows.remove(new)
-    rows.insert(min(position, len(rows)), new)
-    for pos, row in enumerate(rows):
-        row.position = pos
-    db.flush()
-
-
-def add_to_playlist(db: Session, pl: Playlist, video_id: str, position: int | None) -> None:
-    """Add a video to one of the user's lists, at `position` when given (the list page's "Undo" of
-    a removal puts the row back where it was in ONE request), else at the end. A video already in
-    the list stays where it is."""
-    added = add_item(db, pl, video_id, mark_dirty=False)
-    if added and position is not None:
-        place_item(db, pl, video_id, position)
-    recompute_dirty(db, pl)
-    db.flush()
-
-
 def rename_playlist(db: Session, pl: Playlist, name: str) -> None:
     """A new name moves `dirty`; the caller holds the playlist's lock (the scheduled mirror refresh
     reads `dirty` under it)."""
@@ -424,39 +423,163 @@ def delete_playlist(db: Session, pl: Playlist) -> None:
     db.flush()
 
 
-def move_item(db: Session, src: Playlist, dst: Playlist, video_id: str) -> None:
-    """Move one video from `src` into `dst`, in the caller's one transaction.
+def remove_items(db: Session, pl: Playlist, video_ids: list[str]) -> list[str]:
+    """Remove these videos from a playlist, in one transaction; a video not in it any more is simply
+    not among them. Says which went: an Undo must not bring back one this removal never took."""
+    lock_playlists(db, pl.id)
+    wanted = set(video_ids)
+    rows = [it for it in queries.items(db, pl.id) if it.video_id in wanted]
+    for row in rows:
+        db.delete(row)
+    if rows:
+        recompute_dirty(db, pl)
+        touch(pl)
+    db.flush()
+    return [row.video_id for row in rows]
 
-    The client-side shape — add to the target, then remove from the source — leaves the video in
-    BOTH lists whenever the second call fails. Idempotent: a video already in the target is simply
-    dropped from the source (moving into a list that has it is a removal, not a duplicate). Raises
-    `MoveRefused` for the same list twice, or a video a concurrent edit already took out."""
+
+def _insert_at(
+    db: Session,
+    pl: Playlist,
+    placed: list[tuple[int, str]],
+    leaving: dict[str, PlaylistItem] | None = None,
+    staying_out: frozenset[str] = frozenset(),
+) -> int:
+    """Add videos at their indices (ascending, so each index is where the row stood when the rows
+    before it were back) and renumber the list 0..n-1, as positions can have gaps. A video already in
+    the list stays where it is, and one the catalog has lost or in `staying_out` is left out, the
+    ones after it closing up. A video with a row in `leaving` brings that row along, for the reason
+    `move_items` re-parents. Says how many were added; the caller holds the playlist's lock."""
+    leaving = leaving or {}
+    rows = queries.items(db, pl.id)
+    present = {r.video_id for r in rows}
+    known = queries.known_video_ids(db, [vid for _, vid in placed])
+    added = 0
+    left_out = 0
+    for at, vid in sorted(placed, key=lambda p: p[0]):
+        if vid in present or vid not in known or vid in staying_out:
+            left_out += 1
+            continue
+        row = leaving.pop(vid, None)
+        if row is None:
+            row = PlaylistItem(playlist_id=pl.id, video_id=vid, position=0)
+            db.add(row)
+        else:
+            row.playlist_id = pl.id
+        rows.insert(min(at - left_out, len(rows)), row)
+        present.add(vid)
+        added += 1
+    for pos, row in enumerate(rows):
+        row.position = pos
+    return added
+
+
+def restore_items(db: Session, pl: Playlist, video_ids: list[str], positions: list[int]) -> int:
+    """Put removed videos back at their places, in ONE request: an append plus a whole-list reorder
+    leaves them at the end whenever the second call fails. Says how many came back."""
+    lock_playlists(db, pl.id)
+    added = _insert_at(db, pl, list(zip(positions, video_ids, strict=True)))
+    if added:
+        recompute_dirty(db, pl)
+        touch(pl)
+    db.flush()
+    return added
+
+
+class ListGone(Exception):
+    """A list of a two-list write was deleted between the route's read and the lock."""
+
+    def __init__(self, *, source: bool):
+        super().__init__("source" if source else "target")
+        self.source = source
+
+
+def _lock_both(db: Session, src: Playlist, dst: Playlist) -> None:
+    """Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise. A list
+    already gone is said here: a write into it would answer with its foreign key's 500."""
+    found = lock_playlists(db, src.id, dst.id)
+    if src.id not in found:
+        raise ListGone(source=True)
+    if dst.id not in found:
+        raise ListGone(source=False)
+
+
+def move_items(
+    db: Session, src: Playlist, dst: Playlist, video_ids: list[str]
+) -> tuple[list[str], list[str]]:
+    """Move these videos from `src` into `dst`, in the caller's one transaction, in the source's
+    order whatever order they were picked in. Returns the ones that landed in the target and the ones
+    it already held, which only left the source: the Undo needs to tell them apart. A video a
+    concurrent edit already took out of the source is in neither. Raises `MoveRefused` for the same
+    list twice."""
     if src.id == dst.id:
         raise MoveRefused("same_playlist")
-    # Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
-    lock_playlists(db, src.id, dst.id)
-    row = queries.item(db, src.id, video_id)
-    if row is None:
-        raise MoveRefused("not_in_playlist")
-    dst_changed = False
-    if queries.item(db, dst.id, video_id) is None:
-        # Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
-        # survives the move, and there is no window where the video belongs to neither list. The
-        # check above holds: both lists are locked, and an insert into the target (another tab's
-        # add, a YouTube merge) waits for the lock — its foreign key needs the target row.
-        row.playlist_id = dst.id
-        row.position = next_item_position(db, dst.id)
-        dst_changed = True
-    if not dst_changed:
-        db.delete(row)
-    # The source always changed; the target only when the row landed in it. Touching a target
-    # that already had the video promoted an untouched list in the rail's "recently changed" sort.
-    recompute_dirty(db, src)
-    touch(src)
-    if dst_changed:
+    _lock_both(db, src, dst)
+    wanted = set(video_ids)
+    rows = [it for it in queries.items(db, src.id) if it.video_id in wanted]
+    held = queries.in_playlist(db, dst.id, [r.video_id for r in rows])
+    position = next_item_position(db, dst.id)
+    moved: list[str] = []
+    already_there: list[str] = []
+    for row in rows:
+        if row.video_id in held:
+            already_there.append(row.video_id)
+            db.delete(row)
+        else:
+            moved.append(row.video_id)
+            # Re-parented rather than deleted and inserted: `added_at` ("when I saved this") survives
+            # the move, and there is no window where the video belongs to neither list. `held` stays
+            # true: both lists are locked, and an insert into the target (another tab's add, a
+            # YouTube merge) waits for the lock, as its foreign key needs the target row.
+            row.playlist_id = dst.id
+            row.position = position
+            position += 1
+    if rows:
+        recompute_dirty(db, src)
+        touch(src)
+    # Only when a row landed in it: touching a target that already had the video promoted an untouched
+    # list in the rail's "recently changed" sort.
+    if moved:
         recompute_dirty(db, dst)
         touch(dst)
     db.flush()
+    return moved, already_there
+
+
+def undo_move(
+    db: Session,
+    src: Playlist,
+    dst: Playlist,
+    moved: list[str],
+    already_there: list[str],
+    positions: list[int],
+) -> int:
+    """Take a move back in one transaction: the `moved` videos leave `dst` and return to their places
+    in `src`, keeping their rows, as the move did. The `already_there` ones return too and stay in
+    `dst`, which held them before. A moved video `dst` no longer holds has gone on from there, and
+    bringing it back would leave it in lists the reader never put it in, so it stays out, as a
+    move skips a video a concurrent edit took out. `positions` pairs with `moved` + `already_there`.
+    Says how many came back. Raises `MoveRefused` for the same list twice, as a move does."""
+    if src.id == dst.id:
+        raise MoveRefused("same_playlist")
+    _lock_both(db, src, dst)
+    wanted = set(moved)
+    rows = {it.video_id: it for it in queries.items(db, dst.id) if it.video_id in wanted}
+    changed = bool(rows)
+    placed = list(zip(positions, [*moved, *already_there], strict=True))
+    back = _insert_at(db, src, placed, leaving=rows, staying_out=frozenset(wanted - rows.keys()))
+    # What `src` holds again by now only leaves `dst`.
+    for row in rows.values():
+        db.delete(row)
+    db.flush()
+    if changed:
+        recompute_dirty(db, dst)
+        touch(dst)
+    if back:
+        recompute_dirty(db, src)
+        touch(src)
+    db.flush()
+    return back
 
 
 def reorder_items(db: Session, pl: Playlist, order: list[str]) -> int:
@@ -495,3 +618,192 @@ def detach_user_tag(db: Session, user: User, channel_id: str, tag_id: int) -> No
     if row is not None:
         db.delete(row)
         db.flush()
+
+
+# --- Google Takeout --------------------------------------------------------------------------
+
+_TAKEOUT_BATCH = 1000
+# The advisory-lock namespace of one user's Takeout rows ("TO"), so a key cannot meet another lock's.
+_TAKEOUT_LOCK_NS = 0x544F
+
+
+def lock_takeout(db: Session, user: User) -> None:
+    """Hold the user's Takeout rows for the rest of the transaction. An upload's staging and an
+    import's settling each lock many of the same rows, one statement after another and in no common
+    order, so without one lock over both the two can deadlock. An un-mark takes it too: the import
+    reads the un-marks before it marks, and one landing between the two would be overwritten."""
+    db.execute(select(func.pg_advisory_xact_lock(_TAKEOUT_LOCK_NS, user.id)))
+
+
+def stage_takeout(
+    db: Session, user: User, kind: TakeoutKind, rows: dict[str, tuple[datetime, str | None]]
+) -> None:
+    """Keep the videos of one Takeout file, `{video id: (time, channel id)}`, for the import job.
+    A second upload of the same export, or a newer one, adds what is new: a video already kept keeps
+    the later time, and a row the import has settled keeps its outcome."""
+    lock_takeout(db, user)
+    values = [
+        {"user_id": user.id, "kind": kind, "video_id": vid, "at": at, "channel_id": channel}
+        for vid, (at, channel) in rows.items()
+    ]
+    for start in range(0, len(values), _TAKEOUT_BATCH):
+        stmt = pg_insert(TakeoutItem).values(values[start : start + _TAKEOUT_BATCH])
+        db.execute(
+            stmt.on_conflict_do_update(
+                constraint="uq_takeout_item",
+                set_={
+                    "at": func.greatest(TakeoutItem.at, stmt.excluded.at),
+                    "channel_id": func.coalesce(stmt.excluded.channel_id, TakeoutItem.channel_id),
+                },
+            )
+        )
+    db.flush()
+
+
+def _settle(db: Session, ids: list[int], outcome: TakeoutOutcome) -> None:
+    for start in range(0, len(ids), _TAKEOUT_BATCH):
+        db.execute(
+            update(TakeoutItem)
+            .where(TakeoutItem.id.in_(ids[start : start + _TAKEOUT_BATCH]))
+            .values(outcome=outcome)
+        )
+
+
+def apply_takeout_watched(db: Session, user: User) -> dict[str, int]:
+    """Mark watched what the user watched on YouTube, for the videos the catalog knows and has probed
+    for Shorts; the rest wait. `watched_at` is the Takeout time. A Short is not marked (swiping one
+    past is not watching it), and a state the user set here wins: watched or hidden, anything
+    changed after the Takeout time, or an un-mark since then (`VideoUnmark`). The upsert's own
+    condition is that rule, so a state set while the import runs wins too."""
+    lock_takeout(db, user)
+    short: list[int] = []
+    todo: dict[str, tuple[int, datetime]] = {}
+    ready = queries.takeout_watched_ready(db, user)
+    unmarked = queries.unmarked_at(db, user, [video_id for _, video_id, _, _ in ready])
+    kept: list[int] = []
+    for row_id, video_id, at, is_short in ready:
+        if is_short:
+            short.append(row_id)
+        elif video_id in unmarked and unmarked[video_id] >= at:
+            # Un-marked here since they watched it there: theirs, as a state they set is.
+            kept.append(row_id)
+        else:
+            todo[video_id] = (row_id, at)
+    marked: set[str] = set()
+    items = list(todo.items())
+    now = datetime.now(timezone.utc)
+    for start in range(0, len(items), _TAKEOUT_BATCH):
+        stmt = pg_insert(VideoState).values(
+            [
+                {"user_id": user.id, "video_id": vid, "status": "watched", "watched_at": at}
+                for vid, (_, at) in items[start : start + _TAKEOUT_BATCH]
+            ]
+        )
+        marked.update(
+            db.execute(
+                stmt.on_conflict_do_update(
+                    constraint="uq_user_video",
+                    set_={
+                        "status": "watched",
+                        "watched_at": stmt.excluded.watched_at,
+                        "updated_at": now,
+                    },
+                    where=(VideoState.status == "new")
+                    & (VideoState.updated_at < stmt.excluded.watched_at),
+                ).returning(VideoState.video_id)
+            ).scalars()
+        )
+    kept += [row_id for vid, (row_id, _) in items if vid not in marked]
+    _settle(db, [todo[vid][0] for vid in marked], TakeoutOutcome.WATCHED)
+    _settle(db, kept, TakeoutOutcome.KEPT)
+    _settle(db, short, TakeoutOutcome.SHORT)
+    db.flush()
+    return {"watched": len(marked), "kept": len(kept), "shorts": len(short)}
+
+
+def apply_takeout_watch_later(
+    db: Session, user: User, rows: list[tuple[int, str]], asked: set[str]
+) -> dict[str, int]:
+    """Put the videos of the user's waiting Watch later `rows`, (row id, video id) oldest first, into
+    their list here, after what it already holds. `asked` is the videos YouTube was just asked for: one
+    of those the catalog still lacks is gone from YouTube; any other unknown one keeps waiting."""
+    if not rows:
+        return {}
+    lock_takeout(db, user)
+    ids = [vid for _, vid in rows]
+    known = queries.known_video_ids(db, ids)
+    pl = watch_later(db, user)
+    # What add_item does a video at a time, for the whole list at once: one presence read and one
+    # insert, not four statements a video (a full list is thousands). Locked before the read, as there.
+    next_position = next_item_position(db, pl.id)
+    held = queries.in_playlist(db, pl.id, [vid for vid in ids if vid in known])
+    done: dict[TakeoutOutcome, list[int]] = {o: [] for o in TakeoutOutcome}
+    for row_id, vid in rows:
+        if vid not in known:
+            if vid in asked:
+                done[TakeoutOutcome.MISSING].append(row_id)
+        elif vid in held:
+            done[TakeoutOutcome.PRESENT].append(row_id)
+        else:
+            db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=next_position))
+            next_position += 1
+            held.add(vid)
+            done[TakeoutOutcome.ADDED].append(row_id)
+    if done[TakeoutOutcome.ADDED]:
+        touch(pl)
+    for outcome, ids in done.items():
+        _settle(db, ids, outcome)
+    db.flush()
+    return {
+        "watch_later_added": len(done[TakeoutOutcome.ADDED]),
+        "watch_later_present": len(done[TakeoutOutcome.PRESENT]),
+        "watch_later_missing": len(done[TakeoutOutcome.MISSING]),
+    }
+
+
+# --- channels watched but not followed ---------------------------------------------------------
+
+
+def record_takeout_lengths(
+    db: Session, user: User, asked: list[str], lengths: dict[str, int | None]
+) -> None:
+    """Keep what YouTube said of the user's watched videos `asked` about: each one's length, or none
+    for a video it no longer has (`lengths` leaves it out). Every asked row is marked asked, so a
+    video YouTube lost is not asked about again."""
+    if not asked:
+        return
+    lock_takeout(db, user)  # these rows are an upload's to update too
+    now = datetime.now(timezone.utc)
+    table = cast(Table, TakeoutItem.__table__)
+    db.execute(
+        table.update()
+        .where(
+            owned_by(TakeoutItem.user_id, user),
+            table.c.kind == TakeoutKind.WATCHED,
+            table.c.video_id == bindparam("vid"),
+        )
+        .values(looked_up_at=now, duration_seconds=bindparam("seconds")),
+        [{"vid": vid, "seconds": lengths.get(vid)} for vid in asked],
+    )
+    db.flush()
+
+
+def dismiss_watched_channel(db: Session, user: User, channel_id: str) -> None:
+    """Put a channel out of the user's "watched but not followed" list (again is a no-op)."""
+    db.execute(
+        pg_insert(TakeoutChannelDismissal)
+        .values(user_id=user.id, channel_id=channel_id)
+        .on_conflict_do_nothing(constraint="uq_takeout_channel_dismissal")
+    )
+    db.flush()
+
+
+def restore_watched_channel(db: Session, user: User, channel_id: str) -> None:
+    """Let a channel the user put out of the list back on it."""
+    db.execute(
+        delete(TakeoutChannelDismissal).where(
+            owned_by(TakeoutChannelDismissal.user_id, user),
+            TakeoutChannelDismissal.channel_id == channel_id,
+        )
+    )
+    db.flush()

backend/app/reader/takeout.py

@@ -0,0 +1,497 @@
+"""A Google Takeout export, imported: what the user watched on YouTube becomes watched here, and their
+YouTube Watch later list joins theirs. The YouTube API reads neither of them, so the export is the
+only way in.
+
+Two files of the export are read, each uploaded on its own: `history/watch-history.json` and
+`playlists/Watch later-videos.csv`. Their videos are kept (`TakeoutItem`) and the import job applies
+what it can; a video the catalog does not know yet waits for it. The watched videos' channels also
+make the list of channels the user watches but does not follow (`queries.watched_channels`): the
+job asks YouTube how long those videos are, and what the channels are, for it.
+
+The format, measured on a real export (2026-10): the history is a JSON list, one entry per view,
+newest first. A view of a video has a `titleUrl` with `watch?v=<id>` (on www.youtube.com or
+music.youtube.com) and a `time`, with or without milliseconds. The channel is the first `subtitles`
+entry's url, absent for a video since removed. An ad is an entry whose `details` say "From Google
+Ads". The title is in the account's language, so nothing here reads it. A Short is a plain
+`watch?v=` view, told apart only once the catalog has probed the video. The Watch later csv is a
+header row (in the account's language) and `<id>,<ISO time added>` rows.
+"""
+
+import codecs
+import csv
+import io
+import json
+import re
+import threading
+from collections.abc import Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from datetime import datetime
+from itertools import chain
+
+from sqlalchemy.orm import Session
+
+from app import logs, quota
+from app.bodylimit import TAKEOUT_MAX_BYTES
+from app.db import SessionLocal
+from app.models import User
+from app.ratelimit import RateLimiter
+from app.reader import queries, service
+from app.reader.models import TakeoutKind
+from app.sync.runner import get_service_user
+from app.sync.subscriptions import describe_channels
+from app.sync.videos import ensure_videos, parse_iso8601_duration
+from app.youtube.client import YouTubeClient, YouTubeError
+
+log = logs.get_logger(__name__)
+
+# The largest file the upload takes (its edge cap and the route's own are one number).
+MAX_BYTES = TAKEOUT_MAX_BYTES
+# The most videos a Watch later file may hold: YouTube's own cap on one playlist. Each unknown one is
+# looked up on the shared API budget, so a longer list is not one YouTube made.
+WATCH_LATER_MAX = 5000
+# The most Takeout videos one user keeps: twenty times the real export measured here (~9,000 videos
+# across both files), so it only stops uploads of made-up ones piling up for good.
+ROWS_MAX = 200_000
+
+_VIDEO_ID = re.compile(r"^[A-Za-z0-9_-]{11}$")
+_WATCH_URL = re.compile(
+    r"^https?://(?:www\.|music\.|m\.)?youtube\.com/watch\?v=([A-Za-z0-9_-]{11})"
+)
+_CHANNEL_URL = re.compile(r"/channel/(UC[A-Za-z0-9_-]{22})")
+
+
+class TakeoutError(ValueError):
+    """The file is not one the import reads; `reason` is the client's word for why."""
+
+    def __init__(self, reason: str):
+        super().__init__(reason)
+        self.reason = reason
+
+
+@dataclass
+class Parsed:
+    """One file's videos, `{video id: (time, channel id)}`, and what it left out, counted."""
+
+    kind: TakeoutKind
+    videos: dict[str, tuple[datetime, str | None]] = field(default_factory=dict)
+    entries: int = 0
+    ads: int = 0
+    other: int = 0  # not a view of a video (a post, a story) or a video with no link left
+    malformed: int = 0
+
+
+def _time(raw: object) -> datetime | None:
+    if not isinstance(raw, str):
+        return None
+    try:
+        at = datetime.fromisoformat(raw.replace("Z", "+00:00"))
+    # silent-ok: the row is counted as malformed, and the count is in the upload's answer.
+    except ValueError:
+        return None
+    return at if at.tzinfo is not None else None
+
+
+def _keep(parsed: Parsed, vid: str, at: datetime, channel: str | None) -> None:
+    old = parsed.videos.get(vid)
+    if old is None or at > old[0]:
+        parsed.videos[vid] = (at, channel or (old[1] if old else None))
+
+
+def _history(data: Iterator[object]) -> Parsed:
+    parsed = Parsed(TakeoutKind.WATCHED)
+    for entry in data:
+        parsed.entries += 1
+        if not isinstance(entry, dict):
+            parsed.malformed += 1
+            continue
+        details = entry.get("details")
+        if isinstance(details, list) and any(
+            isinstance(d, dict) and d.get("name") == "From Google Ads" for d in details
+        ):
+            parsed.ads += 1
+            continue
+        url = entry.get("titleUrl")
+        match = _WATCH_URL.match(url) if isinstance(url, str) else None
+        if match is None:
+            parsed.other += 1
+            continue
+        at = _time(entry.get("time"))
+        if at is None:
+            parsed.malformed += 1
+            continue
+        channel = None
+        subtitles = entry.get("subtitles")
+        if isinstance(subtitles, list) and subtitles and isinstance(subtitles[0], dict):
+            found = _CHANNEL_URL.search(str(subtitles[0].get("url", "")))
+            channel = found.group(1) if found else None
+        _keep(parsed, match.group(1), at, channel)
+    return parsed
+
+
+def _watch_later(text: str) -> Parsed:
+    parsed = Parsed(TakeoutKind.WATCH_LATER)
+    rows = (row for row in csv.reader(io.StringIO(text)) if any(cell.strip() for cell in row))
+    first = next(rows, None)
+    if first is None or len(first) < 2:
+        raise TakeoutError("not_takeout")
+    # The first row is the header, in the account's language, unless it is already a video.
+    for row in chain([first], rows) if _VIDEO_ID.match(first[0].strip()) else rows:
+        parsed.entries += 1
+        vid = row[0].strip()
+        at = _time(row[1].strip()) if len(row) > 1 else None
+        if not _VIDEO_ID.match(vid) or at is None:
+            parsed.malformed += 1
+            continue
+        _keep(parsed, vid, at, None)
+        if len(parsed.videos) > WATCH_LATER_MAX:
+            raise TakeoutError("too_many")
+    return parsed
+
+
+def _skip(text: str, at: int) -> int:
+    """The first index at or after `at` that is not JSON whitespace."""
+    while at < len(text) and text[at] in " \t\n\r":
+        at += 1
+    return at
+
+
+# How much of the file is decoded at a time. Decoded whole, the text costs up to four bytes a
+# character (one emoji in a title is enough), several times the file on top of its bytes.
+_WINDOW = 1024 * 1024
+
+
+class _Window:
+    """The file's text, decoded a window at a time: `text` holds what is not read yet from `at` on.
+    The bytes are the caller's, so only the window is this reader's memory."""
+
+    def __init__(self, body: bytes, start: int):
+        self._body, self._next = body, start
+        self._decoder = codecs.getincrementaldecoder("utf-8")()  # strict: bad UTF-8 raises
+        self.text, self.at = "", 0
+
+    @property
+    def done(self) -> bool:
+        return self._next >= len(self._body)
+
+    def more(self, at_least: int = 0) -> None:
+        """Decode the next window (at least `at_least` bytes), dropping what is read already."""
+        self.text, self.at = self.text[self.at :], 0
+        size = max(_WINDOW, at_least)
+        chunk = self._body[self._next : self._next + size]
+        self._next += len(chunk)
+        self.text += self._decoder.decode(chunk, final=self.done)
+
+    def skip(self) -> None:
+        """Past the whitespace, reading on as far as it goes."""
+        while True:
+            self.at = _skip(self.text, self.at)
+            if self.at < len(self.text) or self.done:
+                return
+            self.more()
+
+    def startswith(self, char: str) -> bool:
+        self.skip()
+        return self.text.startswith(char, self.at)
+
+
+def _decode(decoder: json.JSONDecoder, text: str, at: int) -> tuple[object, int] | None:
+    """The JSON value at `at` and where it ends, or None where none reads (yet)."""
+    try:
+        return decoder.raw_decode(text, at)
+    # silent-ok: not a failure here — the caller reads on, or refuses the file once it has all.
+    except json.JSONDecodeError:
+        return None
+
+
+def _json_list(body: bytes, start: int) -> Iterator[object]:
+    """The entries of the JSON list at byte `start` (its "["), one at a time: the whole tree is never
+    built, nor the whole text, so a file of a million tiny entries costs a window and one entry, not
+    a million dicts at once."""
+    decoder = json.JSONDecoder()
+    window = _Window(body, start + 1)  # past the "[" the caller saw
+    if window.startswith("]"):
+        window.at += 1
+    else:
+        while True:
+            window.skip()
+            read = _decode(decoder, window.text, window.at)
+            # An entry cut by the window's edge fails, or (a number) reads short: read on and try
+            # again, with a window twice what is left, so one entry larger than any window is read
+            # in a few tries, not one a window.
+            if read is None or (read[1] == len(window.text) and not window.done):
+                if window.done:
+                    raise ValueError("not a JSON list")
+                window.more(2 * (len(window.text) - window.at))
+                continue
+            entry, window.at = read
+            yield entry
+            if window.startswith(","):
+                window.at += 1
+            elif window.startswith("]"):
+                window.at += 1
+                break
+            else:
+                raise ValueError("not a JSON list")
+    window.skip()
+    if window.at != len(window.text):
+        raise ValueError("text after the JSON list")
+
+
+# The largest Watch later file read: YouTube's longest list is some 200 KB of rows, so a file ten
+# times that is not one — and, unlike the history, it is decoded whole.
+CSV_MAX_BYTES = 2 * 1024 * 1024
+_BOM = codecs.BOM_UTF8
+
+
+def parse(body: bytes) -> Parsed:
+    """Read an uploaded Takeout file: the watch history (JSON) or the Watch later list (csv), told
+    apart by their content, not their name. Raises `TakeoutError` for anything else, and for a file
+    with nothing to import in it."""
+    start = len(_BOM) if body.startswith(_BOM) else 0
+    while start < len(body) and body[start] in b" \t\n\r\f\v":
+        start += 1
+    first = body[start : start + 1]
+    if first in (b"{", b"<"):
+        # Another JSON file of the export, or the history's HTML variant.
+        raise TakeoutError("not_takeout")
+    if first != b"[" and len(body) - start > CSV_MAX_BYTES:
+        raise TakeoutError("too_many")
+    try:
+        # A bad UTF-8 sequence is a ValueError too (UnicodeDecodeError), read below as not a file.
+        parsed = (
+            _history(_json_list(body, start))
+            if first == b"["
+            else _watch_later(body[start:].decode("utf-8"))
+        )
+    # A file the import cannot read, however it fails: bad JSON (a ValueError, as is a number too
+    # long to convert), nesting deeper than the decoder recurses, a csv field past csv's limit.
+    except (ValueError, RecursionError, csv.Error) as exc:
+        if isinstance(exc, TakeoutError):
+            raise
+        raise TakeoutError("not_takeout") from None
+    if not parsed.videos:
+        raise TakeoutError("no_videos")
+    return parsed
+
+
+# One import per user at a time: a second file uploaded while the first one's run is under way runs
+# after it (the scheduler's ad-hoc queue), but the Shorts job settles the same users' rows on a
+# thread of its own, and the two would settle the same rows. One API process runs the scheduler, so a
+# lock in this module holds it (as the channel backfill's does).
+_user_locks: dict[int, threading.Lock] = {}
+_user_locks_guard = threading.Lock()
+
+
+@contextmanager
+def _one_import(user_id: int) -> Iterator[None]:
+    with _user_locks_guard:
+        lock = _user_locks.setdefault(user_id, threading.Lock())
+    with lock:
+        yield
+
+
+def run_target(user_id: int) -> str:
+    """The import run's target, `job_runs.target`: one user's import."""
+    return f"user:{user_id}"
+
+
+def target_title(db: Session, target: str) -> str | None:
+    """Who an import run's target is, for the admin's run lists: the user's name, else their email."""
+    _, _, raw = target.partition("user:")
+    user = db.get(User, int(raw)) if raw.isdigit() else None
+    return (user.display_name or user.email) if user is not None else None
+
+
+class TooMany(Exception):
+    """The upload would take the user past `ROWS_MAX` kept videos."""
+
+
+def stage(db: Session, user: User, parsed: Parsed) -> None:
+    """Keep an uploaded file's videos for the import, unless that would take the user past
+    `ROWS_MAX` (`TooMany`). The caller commits."""
+    service.lock_takeout(db, user)  # the count and the insert are one user's, not two uploads' each
+    if (
+        queries.takeout_rows_besides(db, user, parsed.kind, parsed.videos) + len(parsed.videos)
+        > ROWS_MAX
+    ):
+        raise TooMany
+    service.stage_takeout(db, user, parsed.kind, parsed.videos)
+
+
+# How many unknown Watch later videos one user may have looked up on the shared API budget in a day:
+# two full Watch later lists (two units a hundred ids). The rest wait for the next day's runs. Both
+# shares pace the work, they do not bound its cost: they are held in memory, so a restart starts
+# them afresh, which only brings the rest forward — every video is asked about once (a looked-up
+# row is marked), and `quota.backfill_reserve_reached` still guards the shared budget.
+_lookups = RateLimiter(max_events=2 * WATCH_LATER_MAX, window_seconds=24 * 3600)
+# How many watched videos' lengths one user may ask about in a day for the "watched but not followed"
+# list (a unit a fifty). A share of its own: a large history asks for thousands, and on the Watch
+# later share it would leave the export's Watch later file waiting a day behind it.
+_lengths = RateLimiter(max_events=2 * WATCH_LATER_MAX, window_seconds=24 * 3600)
+
+
+def _put_in_watch_later(db: Session, user: User) -> dict:
+    """Put the user's waiting Watch later videos into theirs here: the ones the catalog knows at once,
+    the others once YouTube was asked for them — when a YouTube-connected account can ask, today's
+    budget is above the backfill reserve, and the user's own daily share of lookups allows it. A video
+    not asked about yet waits for a later run. The lookup commits what it fetched; the caller commits
+    the rest."""
+    waiting = queries.takeout_waiting(db, user, TakeoutKind.WATCH_LATER)
+    if not waiting:
+        return {}
+    ids = [vid for _, vid in waiting]
+    known = queries.known_video_ids(db, ids)
+    ask = [vid for vid in ids if vid not in known][:WATCH_LATER_MAX]
+    asked: set[str] = set()
+    fetcher = get_service_user(db) if ask else None
+    if (
+        fetcher is not None
+        and not quota.backfill_reserve_reached()
+        and _lookups.room(str(user.id)) >= len(ask)
+    ):
+        with quota.attribute(user.id, quota.QuotaAction.VIDEOS_LOOKUP):
+            with YouTubeClient(db, fetcher) as yt:
+                ensure_videos(db, yt, ask)
+        # Counted once done: a lookup that failed spent none of the share (one run per user at a
+        # time, so the room asked above is still there).
+        _lookups.allow(str(user.id), cost=len(ask))
+        asked = set(ask)
+    # Only the rows read above: one a second upload staged meanwhile was never asked about.
+    return service.apply_takeout_watch_later(db, user, waiting, asked)
+
+
+def _ask_lengths(db: Session, user: User) -> dict:
+    """Ask YouTube how long the user's watched videos are that the "watched but not followed" list
+    still needs, and keep the answer — on the same terms as the Watch later lookups, on a daily
+    share of its own. What is not asked yet (no YouTube-connected account, the reserve reached, the
+    share spent) waits for a later run, and the list says it is not complete. A YouTube failure is
+    logged and left for a later run too: it is no reason to fail the import it follows. The caller
+    commits."""
+    ask = queries.takeout_lengths_to_ask(db, user)
+    fetcher = get_service_user(db) if ask else None
+    if fetcher is None or quota.backfill_reserve_reached():
+        return {}
+    ask = ask[: int(_lengths.room(str(user.id)))]
+    if not ask:
+        return {}
+    try:
+        with quota.attribute(user.id, quota.QuotaAction.VIDEOS_LOOKUP):
+            with YouTubeClient(db, fetcher) as yt:
+                items = yt.get_videos(ask)
+    except YouTubeError as exc:
+        log.warning(
+            "Takeout: asking the watched videos' lengths failed: %s",
+            exc,
+            extra=logs.event("takeout.lengths.failed", user_id=user.id, reason=exc.reason),
+        )
+        return {}
+    _lengths.allow(str(user.id), cost=len(ask))
+    lengths = {
+        it["id"]: parse_iso8601_duration(it.get("contentDetails", {}).get("duration"))
+        for it in items
+        if it.get("id")
+    }
+    service.record_takeout_lengths(db, user, ask, lengths)
+    return {"lengths_asked": len(ask)}
+
+
+def _describe_channels(db: Session, user: User) -> dict:
+    """Bring the channels the user's list holds into the catalog with their details (name, picture,
+    description), the ones it lacks: without them the list cannot show a channel. A YouTube failure
+    is logged and left for a later run. The caller commits."""
+    need = queries.watched_channels_to_describe(db, user)
+    fetcher = get_service_user(db) if need else None
+    if fetcher is None or quota.backfill_reserve_reached():
+        return {}
+    try:
+        with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
+            with YouTubeClient(db, fetcher) as yt:
+                describe_channels(db, yt, need)
+    except YouTubeError as exc:
+        db.rollback()
+        log.warning(
+            "Takeout: describing the watched channels failed: %s",
+            exc,
+            extra=logs.event("takeout.channels.failed", user_id=user.id, reason=exc.reason),
+        )
+        return {}
+    return {"channels_described": len(need)}
+
+
+def _channel_list(db: Session, user: User) -> dict:
+    """The "watched but not followed" list's part of a run: the lengths, landed, then the channels.
+    Commits each."""
+    result = _ask_lengths(db, user)
+    db.commit()
+    result |= _describe_channels(db, user)
+    db.commit()
+    return result
+
+
+def run_import(user_id: int) -> dict:
+    """The import job: apply what the user's kept Takeout rows can say now, as one recorded run.
+    The watched part commits before the Watch later part asks YouTube for the videos the catalog
+    lacks, so a YouTube failure there loses nothing already marked; the "watched but not followed"
+    list's lookups come last. Returns the counts."""
+    with _one_import(user_id), SessionLocal() as db:
+        user = db.get(User, user_id)
+        if user is None:
+            return {"skipped": "no user"}
+        result = service.apply_takeout_watched(db, user)
+        db.commit()
+        result |= _put_in_watch_later(db, user)
+        db.commit()
+        result |= _channel_list(db, user)
+        outcomes = queries.takeout_outcomes(db, user)
+        result["waiting"] = outcomes.get((TakeoutKind.WATCHED, None), 0)
+        if outcomes.get((TakeoutKind.WATCH_LATER, None)):
+            result["watch_later_waiting"] = outcomes[(TakeoutKind.WATCH_LATER, None)]
+        return result
+
+
+def apply_waiting(db: Session) -> dict:
+    """Settle what every user's import left waiting and can be settled now, one commit each: a
+    watched video the catalog has come to know and the Shorts probe has cleared (by its job or a
+    search's), the Watch later videos a run could not put in yet, and what the "watched but not
+    followed" list still needs asked of YouTube. Run after each run of the
+    `shorts` job, which is what clears the most of them. Returns its counts, or an empty dict when
+    nothing was waiting that could move."""
+    marked = added = asked = failed = 0
+    ready = set(queries.takeout_users_ready(db))
+    later = set(queries.takeout_users_watch_later_waiting(db))
+    lengths = set(queries.takeout_users_with_list_work(db))
+    for user_id in sorted(ready | later | lengths):
+        user = db.get(User, user_id)
+        if user is None:
+            continue
+        # One user's failure (a lookup YouTube refused) is theirs: it neither fails the job that
+        # runs this nor keeps the users after them waiting. What it did not settle waits.
+        try:
+            with _one_import(user_id):
+                if user_id in ready:
+                    marked += service.apply_takeout_watched(db, user)["watched"]
+                    db.commit()
+                if user_id in later:
+                    added += _put_in_watch_later(db, user).get("watch_later_added", 0)
+                    db.commit()
+                if user_id in lengths:
+                    asked += _channel_list(db, user).get("lengths_asked", 0)
+        except Exception:
+            db.rollback()
+            failed += 1
+            log.exception(
+                "Takeout settling failed for user %s",
+                user_id,
+                extra=logs.event("takeout.settle.failed", user_id=user_id),
+            )
+    out = {}
+    if ready:
+        out["takeout_watched"] = marked
+    if added:
+        out["takeout_watch_later_added"] = added
+    if asked:
+        out["takeout_lengths_asked"] = asked
+    if failed:
+        out["takeout_failed"] = failed
+    return out

backend/app/refusal.py

@@ -1,29 +1,39 @@
 """A request the app refuses, said in a way the UI can put in the reader's language.
 
 Every refusal the API answers carries a structured `detail` — `{code, reason, …params}` — and the client
-translates it (`lib/api/core.ts` `localizeDetail`). A bare English string reached the Hungarian UI
+translates it (`lib/api/refusalText.ts` `localizeDetail`). A bare English string reached the Hungarian UI
 verbatim: the blocking error dialog, and every inline surface that printed `HttpError.detail`, showed
 whatever sentence the route happened to hand-write. `refused` is the generic family
 (`errors.refused.<reason>`, the params interpolated into the sentence); the older specific codes
 (`not_found`, `youtube`, `playlist`, the admin refusals, …) stay, since each is already translated.
 
-A test (`test_http_details.py`) reads every `HTTPException` in the app and fails on a string detail,
-and the frontend's contract test requires every `refused(…, "<reason>")` to be named in both
-languages — so the rule is enforced, not described.
+A test (`test_http_details.py`) reads every place an `HTTPException` or a `JSONResponse` sends a detail
+and fails on one the client cannot say: a sentence, a dict without a `code`, a variable, none at all.
+`tests/refusal_sites.py` reads every `{"code": …}` detail from the syntax tree, written where it is sent
+(or returned by a factory that only ever is), with any `**` spread before its `code` and `reason`, and
+follows each reason along a closed list of shapes: a literal, a `Literal` type on the parameter or
+attribute it travels in, every call of a helper like `refused` below, a map's values; anything else is
+reported, never guessed. The frontend's contract test
+(`i18n/backendIds.contract.test.ts`) runs it and requires each code in one record of
+`lib/api/refusalCodes.ts`, and each reason's sentence in both languages; a code or reason it cannot
+follow fails it, unless no screen shows that code. So the rule is enforced, not described.
 """
 
 from fastapi import HTTPException
 
+from app.sysconfig import InvalidSetting, SettingReason
+
 
 def refused(status: int, reason: str, **params: object) -> HTTPException:
     """The refusal to raise: `raise refused(409, "view_name_taken", name=name)`. `params` travel in
-    the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see."""
-    return HTTPException(status_code=status, detail={"code": "refused", "reason": reason, **params})
+    the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see.
+    They are spread first, so a `code` or `reason` among them could never replace these two."""
+    return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
 
 
-# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the frontend's
-# contract test can read every reason this can produce.
-_SETTING_REASONS = {
+# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the refusal
+# scan reads every reason this can produce, and holds its keys to `SettingReason`.
+_SETTING_REASONS: dict[SettingReason, str] = {
     "not_a_number": "setting_not_a_number",
     "too_low": "setting_too_low",
     "too_high": "setting_too_high",
@@ -32,6 +42,6 @@ _SETTING_REASONS = {
 }
 
 
-def setting_refused(exc) -> HTTPException:
+def setting_refused(exc: InvalidSetting) -> HTTPException:
     """A value the admin entered that `sysconfig.set_value` refused (`exc`: its `InvalidSetting`)."""
     return refused(400, _SETTING_REASONS[exc.reason], key=exc.key, **exc.params)

backend/app/routes/_common.py

@@ -6,9 +6,9 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
 one existence-leak policy.
 """
 
-from typing import Any, Protocol, TypeVar
+from typing import Annotated, Any, Protocol, TypeVar
 
-from fastapi import HTTPException, Request
+from fastapi import HTTPException, Query, Request
 from pydantic import BaseModel, ConfigDict, Field
 from sqlalchemy.orm import Session
 
@@ -21,6 +21,10 @@ from app.youtube.client import YouTubeError, refusal_reason
 
 M = TypeVar("M", bound=Base)
 
+# A shuffle / surprise seed off the query string. The client draws one under a billion; bounded so a
+# hand-typed 400-digit one is a 422, not an int no float or bigint can hold further down.
+Seed = Annotated[int, Query(ge=-(2**53), le=2**53)]
+
 # Resume-position thresholds of a player that only keeps a resume point (the YouTube player, the
 # library's): positions below this are "didn't really start" and within this of the end are
 # "basically finished" — neither is worth storing, so the position is cleared instead (no progress
@@ -124,6 +128,14 @@ async def capped_body(request: Request, limit: int) -> bytes:
     return bytes(body)
 
 
+async def drain_body(request: Request) -> None:
+    """Read the rest of the body to nowhere, before a refusal is raised: answered mid-upload, the
+    client reports a network failure, not the refusal (`capped_stream_to` says why). `bodylimit`
+    bounds how much there can be."""
+    async for _ in request.stream():
+        pass
+
+
 class ByteSink(Protocol):
     """Where `capped_stream_to` writes: a file, or anything that takes the bytes on their way to one."""
 

backend/app/routes/auth.py

@@ -5,9 +5,10 @@ helpers) stays in app/auth.py, imported below."""
 
 import secrets
 from datetime import datetime, timedelta, timezone
-from typing import NamedTuple
+from typing import Literal, NamedTuple
 from urllib.parse import urlencode
 
+from authlib.common.security import generate_token
 from authlib.integrations.starlette_client import OAuth, OAuthError
 from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
 from fastapi.responses import JSONResponse, RedirectResponse
@@ -25,10 +26,11 @@ from app import sysconfig
 from app.auth import (
     _YOUTUBE_SCOPES,
     BASE_SCOPES,
-    MAX_SESSION_ACCOUNTS,
+    EXPECTED_ACCOUNT_HEADER,
     PASSWORD_MIN_LEN,
     READ_SCOPES,
     WRITE_SCOPES,
+    built_for_another_account,
     current_user,
     env_admin_canonicals,
     env_allowed_canonicals,
@@ -37,7 +39,9 @@ from app.auth import (
     is_allowed,
     log,
     operator_contact,
-    resolved_user_id,
+    optional_current_user,
+    session_user_id,
+    stamp_session,
 )
 from app.config import settings
 from app.db import SessionLocal, get_db
@@ -320,6 +324,25 @@ class SetPasswordIn(BaseModel):
     current_password: str | None = None
 
 
+# A connect in flight, in the session: `{state, uid, explicit}`. It names the OAuth state of the flow
+# that set it, because the session is the browser's, not the flow's: a callback coming back with
+# another state (an older flow, which authlib then refuses) must not spend the marker of the connect
+# still open, or that connect would return as a plain sign-in — signing someone else in, or storing
+# a YouTube grant on whichever account the Google identity already belongs to.
+_LINK_MARKER = "oauth_link"
+
+# Where a navigation from a tab showing another account than the one signed in lands: the app, which
+# boots as the account signed in now and says why (the frontend's `ACCOUNT_CHANGED_FLAG`).
+ACCOUNT_CHANGED_LANDING = "/?account=changed"
+
+
+def _take_link_marker(request: Request) -> dict | None:
+    marker = request.session.get(_LINK_MARKER)
+    if not isinstance(marker, dict) or marker.get("state") != request.query_params.get("state"):
+        return None
+    return request.session.pop(_LINK_MARKER)
+
+
 @router.get("/login")
 async def login(request: Request, db: Session = Depends(get_db)):
     # access_type=offline ensures a refresh_token on first authorization. We avoid
@@ -330,6 +353,10 @@ async def login(request: Request, db: Session = Depends(get_db)):
         # Google sign-in isn't configured on this instance — land back on the login page (the UI
         # hides the button, so this is just a safety net for a stale/direct hit).
         return RedirectResponse(url="/")
+    # A sign-in, never a link. Starting it retires every earlier flow's state (authlib keeps only the
+    # newest), so a connect's marker left from before — abandoned at Google, or still open in
+    # another tab — names a flow that can no longer finish.
+    request.session.pop(_LINK_MARKER, None)
     return await client.authorize_redirect(
         request,
         _oauth_redirect_uri(request),
@@ -350,24 +377,23 @@ async def callback(
     if client is None:
         return RedirectResponse(url="/")
     # Linking mode: an already-authenticated account is attaching this Google identity (a
-    # password account enabling YouTube or connecting SSO). The markers are set by /auth/link and
-    # /auth/upgrade; when present we attach to the current user instead of identifying by sub.
-    # Pop them BEFORE the token exchange: if the user aborts Google's consent screen the exchange
-    # throws below, and a marker left behind would silently turn the NEXT clean sign-in into a link
-    # attempt. Popping up front means an aborted link flow leaves no residue in the session.
-    link_uid = request.session.pop("oauth_link_uid", None)
-    link_explicit = request.session.pop("oauth_link_explicit", False)
+    # password account enabling YouTube or connecting SSO). The marker is set by
+    # /auth/google/connect; when it is this flow's we attach to the current user instead of
+    # identifying by sub. Taken BEFORE the token exchange: if the user aborts Google's consent screen
+    # the exchange throws below, and a marker left behind would silently turn the NEXT clean sign-in
+    # into a link attempt. A connect that never comes back here at all is cleared by /auth/login.
+    link = _take_link_marker(request)
     try:
         token = await client.authorize_access_token(request)
     except OAuthError as exc:
         # The user cancelled or denied consent at Google — land back with a friendly notice instead
-        # of a raw 400 JSON error page. An already-signed-in user (adding another account via
-        # /auth/login, or linking/upgrading via /auth/link|upgrade) never sees the pre-auth login
-        # banner, so surface a neutral toast via ?oauth=cancelled; a fresh sign-in cancel lands on
-        # the login page with the oauth_cancelled banner instead. Keyed on the live session, not on
-        # link_uid, so the add-account path (no marker) is covered too.
+        # of a raw 400 JSON error page. An already-signed-in user (signing in again via /auth/login,
+        # or connecting via /auth/google/connect) never sees the pre-auth login banner, so
+        # surface a neutral toast via ?oauth=cancelled; a fresh sign-in cancel lands on the login page
+        # with the oauth_cancelled banner instead. Keyed on the live session, not on link_uid, so a
+        # sign-in started while signed in (no marker) is covered too.
         log.info("Google OAuth cancelled/failed: %s", exc.error)
-        if request.session.get("user_id") is not None:
+        if session_user_id(request) is not None:
             return RedirectResponse(url="/?oauth=cancelled")
         return RedirectResponse(url="/?login=oauth_cancelled")
 
@@ -378,8 +404,8 @@ async def callback(
     sub = userinfo["sub"]
     email = (userinfo.get("email") or "").lower()
 
-    if link_uid is not None:
-        return _complete_link(request, db, link_uid, link_explicit, sub, userinfo, token)
+    if link is not None:
+        return _complete_link(request, db, link["uid"], link["explicit"], sub, userinfo, token)
 
     if not is_allowed(db, email):
         # No uid to name — having no account here is the branch. And the address is left out on
@@ -580,19 +606,16 @@ def _complete_link(
     token: dict,
 ):
     """Attach the just-authorized Google identity to the already-signed-in account that started
-    the link/upgrade. No is_allowed gate — they're already an active user. Refuses to hijack a
+    the connect. No is_allowed gate — they're already an active user. Refuses to hijack a
     Google identity owned by another account, or to silently swap a different linked identity.
-    `explicit` (the popped oauth_link_explicit marker) distinguishes a "Connect Google" click from
-    a YouTube upgrade — the caller pops it up front so an aborted flow leaves nothing behind."""
-    # SEC-11: the marker must name an account signed into THIS browser — the session default OR any
-    # account in the wallet — not only the default. A tab acting as a secondary account links Google to
-    # ITSELF (the marker was set from current_user, which resolves the ?account= it navigated with), so
-    # accepting wallet membership stops the identity/token landing on the default account A instead of B.
-    accounts = request.session.get("account_ids") or []
-    if link_uid != request.session.get("user_id") and link_uid not in accounts:
-        return RedirectResponse(url="/?link=error")
-    user = db.get(User, link_uid)
-    if user is None:
+    `explicit` (from the link marker) distinguishes a "Connect Google" click from a YouTube grant —
+    the caller takes the marker up front so an aborted flow leaves nothing behind."""
+    # The marker must name the account still signed into THIS browser, and that session must still
+    # hold: a sign-in as someone else between the start and the callback replaced it (the identity
+    # must not land on the newcomer), and a session revoked meanwhile ("log out everywhere", a
+    # password change, a suspension) must not finish what it started.
+    user = optional_current_user(request, db)
+    if user is None or user.id != link_uid:
         return RedirectResponse(url="/?link=error")
     other = db.execute(
         select(User).where(User.google_sub == sub, User.id != user.id)
@@ -612,45 +635,31 @@ def _complete_link(
     _store_token(db, user, token)
     db.commit()
     log.info("Google linked: uid=%s scopes=%s", user.id, token.get("scope"))
-    # An explicit "Connect Google" lands on Settings with a confirmation; a YouTube upgrade
+    # An explicit "Connect Google" lands on Settings with a confirmation; a YouTube grant
     # (wizard / Settings access rows) returns to the app so the onboarding flow resumes as before.
     return RedirectResponse(url="/?link=ok" if explicit else "/")
 
 
 @router.post("/logout")
 async def logout(request: Request):
-    """Sign the requesting tab's active account out of this browser (removing it from the wallet).
-    Per-tab aware: the account is taken from the X-Siftlode-Account header when present, so logging
-    out in one tab doesn't disturb another tab running a different account. If the account being
-    removed was the session default, promote the most recent remaining account (or clear the
-    session when none remain)."""
-    # A tab still pinned to an account already signed out in another tab resolves to nobody, so it
-    # signs nobody out — it used to fall back to the default and sign THAT account out in its place.
-    active, is_default = resolved_user_id(request)
-    remaining = [a for a in (request.session.get("account_ids") or []) if a != active]
-    request.session["account_ids"] = remaining
-    if is_default:
-        if remaining:
-            request.session["user_id"] = remaining[-1]
-            return JSONResponse({"ok": True, "switched": True})
-        request.session.clear()
-        return JSONResponse({"ok": True, "switched": False})
-    # A non-default (per-tab) account signed out: it's gone from the wallet and the session default
-    # is untouched. The tab drops its own override and falls back to the default on reload.
-    return JSONResponse({"ok": True, "switched": False})
+    """Sign this browser out: the session holds one account, and it ends — unless the tab asking was
+    built for another account than the one signed in now (a sign-in as someone else in another tab):
+    its Sign out would end THAT person's session (see `EXPECTED_ACCOUNT_HEADER`)."""
+    signed_in = session_user_id(request)
+    if signed_in is not None and built_for_another_account(
+        signed_in, request.headers.get(EXPECTED_ACCOUNT_HEADER)
+    ):
+        raise refused(401, "account_changed")
+    request.session.clear()
+    return JSONResponse({"ok": True})
 
 
 def _establish_session(request: Request, user: User) -> None:
     """Sign `user` in for this browser session — the ONE sign-in path, shared by the Google callback
-    and password login, so both join the same account switcher and both stamp `last_login_at` (the
-    caller commits). Multi-session: every account that authenticated in this browser is remembered so
-    the user can switch between them; only accounts that actually proved themselves HERE land in that
-    wallet (it is the proof they may be switched to)."""
-    accounts = [a for a in (request.session.get("account_ids") or []) if a != user.id]
-    accounts.append(user.id)
-    request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
+    and password login, so both stamp `last_login_at` (the caller commits). One account per session:
+    the sign-in replaces whoever was signed in here before."""
     request.session["user_id"] = user.id
-    _remember_epoch(request, user)
+    stamp_session(request, user)
     user.last_login_at = datetime.now(timezone.utc)
 
 
@@ -967,7 +976,16 @@ def _resend_verify_if_eligible(email: str) -> None:
             send_verification_link(db, user)
 
 
-def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
+# Every refusal a password sign-in answers with; the UI names each one (`welcome.auth.blocked.<reason>`,
+# held by the frontend's backendIds contract, which reads this Literal).
+LoginRefusal = Literal[
+    "rate_limited", "invalid", "suspended", "email_unverified", "pending_approval"
+]
+
+
+def _login_refused(
+    status: int, reason: LoginRefusal, user: User | None = None, **extra
+) -> HTTPException:
     """Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
     shape so the SPA can say it in the user's own language (the old bare English strings were
     unreadable under a bilingual UI, and unusable as anything but a one-line error).
@@ -983,15 +1001,15 @@ def _login_refused(status: int, reason: str, user: User | None = None, **extra)
     `email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
     rather than assuming a sequence.
     """
-    detail: dict = {"code": "login", "reason": reason}
-    if user is not None:
-        detail |= {
-            "email": user.email,
-            "email_verified": user.email_verified,
-            "approved": user.is_active,
-        }
-    detail |= {k: v for k, v in extra.items() if v is not None}
-    return HTTPException(status_code=status, detail=detail)
+    account = (
+        {"email": user.email, "email_verified": user.email_verified, "approved": user.is_active}
+        if user is not None
+        else {}
+    )
+    given = {k: v for k, v in extra.items() if v is not None}
+    return HTTPException(
+        status_code=status, detail={**account, **given, "code": "login", "reason": reason}
+    )
 
 
 @router.post("/password-login")
@@ -1135,15 +1153,6 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
     return {"ok": True}
 
 
-def _remember_epoch(request: Request, user: User) -> None:
-    """Record this account's current `session_epoch` in the signed cookie (SA4), so `current_user`
-    can later reject the cookie if the account's epoch was bumped (password reset/change, or
-    "log out everywhere"). Per-account so one account's revocation doesn't evict the others."""
-    epochs = dict(request.session.get("epochs") or {})
-    epochs[str(user.id)] = user.session_epoch
-    request.session["epochs"] = epochs
-
-
 def bump_session_epoch(db: Session, user: User) -> None:
     """Invalidate every EXISTING signed cookie for this account by advancing its epoch (SA4). Cookies
     minted before the bump carry a now-stale epoch and are rejected by `current_user`. The account's
@@ -1220,35 +1229,11 @@ def logout_others(
     # session at a newer epoch than the DB (see set_password).
     bump_session_epoch(db, user)
     db.commit()
-    _remember_epoch(request, user)
+    stamp_session(request, user)
     log.info("Logout-others: uid=%s (epoch=%s)", user.id, user.session_epoch)
     return {"ok": True}
 
 
-@router.get("/link")
-async def link_google(
-    request: Request,
-    user: User = Depends(current_user),
-    db: Session = Depends(get_db),
-):
-    """Start linking a Google account to the signed-in (e.g. password) account. Requests only
-    the identity scopes — YouTube access is granted separately via /auth/upgrade. The callback
-    sees `oauth_link_uid` and attaches the identity instead of creating a new account."""
-    client = google_oauth(db)
-    if client is None:
-        return RedirectResponse(url="/?link=error")
-    request.session["oauth_link_uid"] = user.id
-    request.session["oauth_link_explicit"] = True
-    return await client.authorize_redirect(
-        request,
-        _oauth_redirect_uri(request),
-        access_type="offline",
-        prompt="select_account",
-        include_granted_scopes="true",
-        scope=BASE_SCOPES,
-    )
-
-
 @router.post("/set-password")
 def set_password(
     body: SetPasswordIn,
@@ -1272,35 +1257,57 @@ def set_password(
     # can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
     bump_session_epoch(db, user)
     db.commit()
-    _remember_epoch(request, user)
+    stamp_session(request, user)
     log.info("Password set/changed: uid=%s", user.id)
     return {"ok": True}
 
 
-@router.get("/upgrade")
-async def upgrade(
+# What each Google connect asks Google for. `identity` attaches a Google sign-in to the account
+# (Settings → Connect Google) and lets the person pick which Google account; `read` and `write` grant
+# YouTube access (the onboarding wizard, the Settings access rows) and always show the consent screen,
+# so a scope granted earlier and since revoked at Google is asked for again.
+_CONNECT: dict[str, tuple[str, str]] = {
+    "identity": (BASE_SCOPES, "select_account"),
+    "read": (READ_SCOPES, "consent"),
+    "write": (WRITE_SCOPES, "consent"),
+}
+
+
+@router.get("/google/connect")
+async def google_connect(
     request: Request,
-    access: str = "read",
+    access: Literal["identity", "read", "write"],
+    expect: str | None = None,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ):
-    """Incremental consent for the onboarding wizard. `access=read` grants YouTube
-    read-only (enough to build the feed); `access=write` additionally grants management
-    (unsubscribe). The shared callback stores whatever Google returns, so `can_read` /
-    `can_write` flip on once granted. Anything other than an explicit `write` defaults to
-    the least-privileged read scope."""
+    """Attach Google to the signed-in account: its identity, or YouTube read/write access on top.
+    The shared callback sees the link marker and attaches what Google returns to THIS account —
+    without it a password account (no google_sub) would get a brand-new, separate Google user
+    created instead. It stores whatever scopes Google grants, so `can_read` / `can_write` flip on
+    once granted.
+
+    `expect` is the account the tab was built for. A navigation carries no headers, so the check
+    every request gets (`EXPECTED_ACCOUNT_HEADER`) rides the query here: a tab still showing the
+    account signed in before would otherwise attach a Google identity to the one signed in now."""
+    if built_for_another_account(user.id, expect):
+        return RedirectResponse(url=ACCOUNT_CHANGED_LANDING)
     client = google_oauth(db)
     if client is None:
-        return RedirectResponse(url="/")
-    # Attach the grant to THIS account in the callback. Without it a password account (no
-    # google_sub) would get a brand-new, separate Google user created instead of being linked.
-    request.session["oauth_link_uid"] = user.id
-    scope = WRITE_SCOPES if access == "write" else READ_SCOPES
+        return RedirectResponse(url="/?link=error")
+    scope, prompt = _CONNECT[access]
+    state = generate_token(48)
+    request.session[_LINK_MARKER] = {
+        "state": state,
+        "uid": user.id,
+        "explicit": access == "identity",
+    }
     return await client.authorize_redirect(
         request,
         _oauth_redirect_uri(request),
+        state=state,
         access_type="offline",
-        prompt="consent",
+        prompt=prompt,
         include_granted_scopes="true",
         scope=scope,
     )

backend/app/routes/channels.py

@@ -76,6 +76,10 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
             "details_synced": ch.details_synced_at is not None,
             "recent_synced": ch.recent_synced_at is not None,
             "backfill_done": ch.backfill_done,
+            # When the follow reached this list: written by the subscription import the run that
+            # first saw it, or by a subscribe made here — not the channel row's own date, which an
+            # explored or discovered channel had before it was followed.
+            "added_at": iso(sub.created_at),
         }
         for sub, ch in rows
     ]
@@ -307,9 +311,9 @@ def explore_channel(
     """Open an un-subscribed channel for browsing. Records the exploration (a per-user,
     grace-clocked row that gates this user's access to the channel's ephemeral videos and
     drives the later cleanup), flags the channel ephemeral unless someone organically follows
-    it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
-    Returns the next page token so the page can "load more". Guarded against the shared
-    backfill quota reserve."""
+    it, then ingests one page of its uploads (newest-first), enriched — marked via_explore while
+    nobody follows it. Returns the next page token so the page can "load more". Guarded against
+    the shared backfill quota reserve."""
     channel = _channel_or_404(db, channel_id)
     if queries.is_blocked(db, user, channel_id):
         raise refused(403, "channel_blocked")

backend/app/routes/discover.py

@@ -23,7 +23,7 @@ from app.discover import pins, rails
 from app.discover.budget import Budget, DEFAULT_BUDGET
 from app.models import User
 from app.plex.cards import norm_lang
-from app.routes._common import not_found
+from app.routes._common import Seed, not_found
 
 router = APIRouter(prefix="/api/discover", tags=["discover"])
 
@@ -44,7 +44,7 @@ class ReorderIn(BaseModel):
 @router.get("")
 def get_discover(
     budget: Budget = DEFAULT_BUDGET,
-    seed: int = 0,
+    seed: Seed = 0,
     lang: str = "en",
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
@@ -79,7 +79,7 @@ MAX_EXCLUDE = 32
 @router.get("/pick")
 def get_pick(
     budget: Budget = DEFAULT_BUDGET,
-    seed: int = 0,
+    seed: Seed = 0,
     exclude: str = "",
     start: int = 0,
     lang: str = "en",

backend/app/routes/downloads.py

@@ -208,7 +208,7 @@ def set_credential(
     try:
         cleaned = dlcreds.validate_cookies(body.cookies)
     except dlcreds.InvalidCookies as e:
-        raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
+        raise HTTPException(status_code=400, detail={"code": "cookies", "reason": e.reason}) from e
     dlcreds.upsert_credential(db, user.id, host_key, cleaned)
     audit.record(
         db,
@@ -359,8 +359,9 @@ def enqueue_clip(
         job = service.enqueue_clip(db, user.id, spec, display_name=display_name)
     except clipmod.ClipError as e:
         status = 404 if e.reason == "not_found" else 422
-        detail = {"code": "clip", "reason": e.reason, **e.params}
-        raise HTTPException(status_code=status, detail=detail) from e
+        raise HTTPException(
+            status_code=status, detail={**e.params, "code": "clip", "reason": e.reason}
+        ) from e
     except quota.QuotaExceeded as e:
         raise HTTPException(status_code=422, detail=quota_detail(e)) from e
     db.commit()
@@ -610,7 +611,7 @@ def list_links(
     job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
 ) -> list[dict]:
     own_job(db, user, job_id)
-    return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
+    return [linksmod.owner_view(db, link) for link in queries.links_to(db, job_id=job_id)]
 
 
 class CreateLinkIn(BaseModel):
@@ -652,7 +653,7 @@ def create_link(
         raise refused(409, "share_unfinished")
     link, token = _new_link(db, user, body, job_id=job_id)
     db.commit()
-    return linksmod.owner_view(link, token=token)
+    return linksmod.owner_view(db, link, token=token)
 
 
 def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
@@ -683,7 +684,7 @@ def list_folder_links(
     folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
 ) -> list[dict]:
     own_folder(db, user, folder_id)
-    return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
+    return [linksmod.owner_view(db, link) for link in queries.links_to(db, folder_id=folder_id)]
 
 
 @router.post("/folders/{folder_id}/links")
@@ -696,7 +697,7 @@ def create_folder_link(
     _shareable_folder(db, user, folder_id)
     link, token = _new_link(db, user, body, folder_id=folder_id)
     db.commit()
-    return linksmod.owner_view(link, token=token)
+    return linksmod.owner_view(db, link, token=token)
 
 
 class UpdateLinkIn(BaseModel):
@@ -723,7 +724,7 @@ def update_link(
         sent=frozenset(body.model_fields_set),
     )
     db.commit()
-    return linksmod.owner_view(link)
+    return linksmod.owner_view(db, link)
 
 
 @router.delete("/links/{link_id}")

backend/app/routes/feed.py

@@ -1,19 +1,33 @@
+import threading
+from collections.abc import AsyncIterator
 from datetime import date
+from functools import partial
 
-from fastapi import APIRouter, Depends, Query
+from fastapi import APIRouter, Depends, Path, Query, Request
 from pydantic import BaseModel
 from sqlalchemy.orm import Session
 from sqlalchemy.orm.exc import StaleDataError
 
-from app import quota
+from app import jobruns, quota, scheduler
+from app.catalog import queries as catalog_queries, service as catalog_service
+from app.catalog.models import known_length
 from app.auth import current_user
-from app.db import get_db
+from app.db import get_db, release_read_connection
 from app.mailtext import lang_for_user
+from app.ratelimit import RateLimiter
 from app.feedquery import BadCursor, facet_counts, feed_count, feed_page, feed_summary, feed_years
 from app.models import User
-from app.reader import queries, service
-from app.routes._common import ProgressIn, kept_position, not_found, youtube_refusal
-from app.sync.videos import parse_iso8601_duration
+from app.reader import queries, service, takeout
+from app.routes._common import (
+    ProgressIn,
+    Seed,
+    capped_body,
+    drain_body,
+    kept_position,
+    not_found,
+    youtube_refusal,
+)
+from app.sync.videos import ensure_videos, parse_iso8601_duration
 from app.utils import iso
 from app.youtube.client import YouTubeClient, YouTubeError
 from app.refusal import refused
@@ -76,7 +90,7 @@ def _feed_params(
 def get_feed(
     params: dict = Depends(_feed_params),
     sort: str = "newest",
-    seed: int = 0,
+    seed: Seed = 0,
     limit: int = Query(default=60, le=200),
     cursor: str | None = None,
     user: User = Depends(current_user),
@@ -192,22 +206,21 @@ def set_video_state(
     return {"ok": True, "video_id": video_id, "status": status}
 
 
-@router.delete("/videos/{video_id}/state")
-def clear_video_state(
+@router.delete("/videos/{video_id}/progress")
+def clear_video_progress(
     video_id: str,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
-    """Reset a video to pristine for this user — drop the whole VideoState row (status,
-    watch position and watched_at), as if it had never been opened. Saved/playlist membership
-    lives elsewhere and is untouched."""
+    """Forget where the user left a video: its resume point goes, its status (watched, hidden) stays.
+    Saved/playlist membership lives elsewhere and is untouched."""
     try:
-        service.clear_state(db, user, video_id)
+        service.set_progress(db, user, video_id, 0)
         db.commit()
-    # silent-ok: the row went under the delete, which is the outcome the delete wanted.
+    # silent-ok: the row went under the write, which is the outcome the write wanted.
     except StaleDataError:
         db.rollback()
-    return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
+    return {"ok": True, "video_id": video_id, "position_seconds": 0}
 
 
 @router.post("/videos/{video_id}/progress")
@@ -237,6 +250,21 @@ def set_video_progress(
     return {"ok": True, "video_id": video_id, "position_seconds": position}
 
 
+@router.get("/videos/{video_id}/row")
+def get_video_row(
+    video_id: str = Path(max_length=32),
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """One catalog video as the feed lists it (its channel, this user's watch state and resume point,
+    saved): what a player opened from an address starts from. The catalog only, so it costs nothing;
+    every video a player can open (a list's row, a related one) is in it."""
+    rows = queries.feed_rows_by_ids(db, user, [video_id])
+    if not rows:
+        raise not_found("video")
+    return rows[0]
+
+
 @router.get("/videos/{video_id}")
 def get_video_detail(
     video_id: str,
@@ -261,7 +289,7 @@ def get_video_detail(
             "channel_title": v.channel.title if v.channel else None,
             "published_at": iso(v.published_at),
             "view_count": v.view_count,
-            "duration_seconds": v.duration_seconds,
+            "duration_seconds": v.length_seconds,
         }
 
     # The DB path above is free; the off-catalog YouTube lookup below spends a quota unit.
@@ -289,7 +317,218 @@ def get_video_detail(
         "channel_title": snippet.get("channelTitle"),
         "published_at": snippet.get("publishedAt"),
         "view_count": int(views) if views is not None else None,
-        "duration_seconds": parse_iso8601_duration(
-            items[0].get("contentDetails", {}).get("duration")
+        "duration_seconds": known_length(
+            parse_iso8601_duration(items[0].get("contentDetails", {}).get("duration"))
         ),
     }
+
+
+# --- Google Takeout ------------------------------------------------------------------------------
+
+
+# One upload at a time per user, ten an hour, and two at once across every user: a file's bytes are
+# held whole while it is read (its text a window at a time), so these bound what the accounts can
+# make the process hold. The section uploads its files one after another.
+_takeout_limiter = RateLimiter(max_events=10, window_seconds=3600)
+_takeout_uploading: set[int] = set()
+_takeout_uploading_guard = threading.Lock()
+_TAKEOUT_AT_ONCE = 2
+
+
+async def _takeout_body(
+    request: Request, user: User = Depends(current_user), db: Session = Depends(get_db)
+) -> AsyncIterator[bytes]:
+    # The sign-in check has run, and it was a SELECT: hand the connection back before a file of
+    # megabytes arrives, so a slow upload does not hold one `idle in transaction`. The user's id is
+    # taken first: the release expires the user, and reading it after would reload it, taking a
+    # connection straight back out.
+    uid = user.id
+    release_read_connection(db)
+    with _takeout_uploading_guard:
+        busy = uid in _takeout_uploading
+        full = not busy and len(_takeout_uploading) >= _TAKEOUT_AT_ONCE
+        taken = not busy and not full
+        if taken:
+            _takeout_uploading.add(uid)
+    if not taken or not _takeout_limiter.allow(str(uid)):
+        if taken:
+            with _takeout_uploading_guard:
+                _takeout_uploading.discard(uid)
+        # Read to the end before answering: refused mid-upload, the browser reports a network
+        # failure instead of the refusal (capped_stream_to says why).
+        await drain_body(request)
+        if busy:
+            raise refused(429, "takeout_busy")
+        if full:
+            raise refused(429, "takeout_server_busy")
+        raise refused(429, "takeout_cooldown")
+    try:
+        yield await capped_body(request, takeout.MAX_BYTES)
+    finally:
+        with _takeout_uploading_guard:
+            _takeout_uploading.discard(uid)
+
+
+@router.post("/takeout")
+def upload_takeout(
+    # The sign-in check FIRST: the body's own dependency needs the signed-in user, so a stranger's
+    # file is refused before a byte of it is read.
+    user: User = Depends(current_user),
+    body: bytes = Depends(_takeout_body),
+    db: Session = Depends(get_db),
+):
+    """One file of the user's Google Takeout export — the watch history or the Watch later list.
+    Its videos are kept and the import runs in the background; its outcome reaches the user's inbox."""
+    try:
+        parsed = takeout.parse(body)
+    except takeout.TakeoutError as exc:
+        if exc.reason == "no_videos":
+            raise refused(422, "takeout_no_videos") from None
+        if exc.reason == "too_many":
+            raise refused(422, "takeout_list_too_long", limit=takeout.WATCH_LATER_MAX) from None
+        raise refused(422, "takeout_unreadable") from None
+    try:
+        takeout.stage(db, user, parsed)
+    except takeout.TooMany:
+        db.rollback()
+        raise refused(422, "takeout_too_many", limit=takeout.ROWS_MAX) from None
+    db.commit()
+    scheduler.run_ad_hoc(
+        scheduler.TAKEOUT_IMPORT,
+        partial(takeout.run_import, user.id),
+        actor_id=user.id,
+        target=takeout.run_target(user.id),
+        notify=True,
+    )
+    return {
+        "ok": True,
+        "kind": parsed.kind,
+        "videos": len(parsed.videos),
+        "entries": parsed.entries,
+        "ads": parsed.ads,
+        "other": parsed.other,
+        "malformed": parsed.malformed,
+    }
+
+
+@router.get("/takeout")
+def takeout_status(user: User = Depends(current_user), db: Session = Depends(get_db)):
+    """What the user's Takeout import has done: its last run (none before the first upload), and
+    every video kept from the export, by kind and outcome — a video the catalog did not know yet
+    counts as "waiting" until it does, and is marked then."""
+    target = takeout.run_target(user.id)
+    run = jobruns.last_run(db, scheduler.TAKEOUT_IMPORT, target)
+    return {
+        # Queued and not started: its run row is not written yet, and the last run is still the
+        # previous one, so the page waits for it rather than taking that one for this upload's.
+        "queued": scheduler.ad_hoc_queued(scheduler.TAKEOUT_IMPORT, target),
+        "max_bytes": takeout.MAX_BYTES,
+        "run": None
+        if run is None
+        else {
+            "status": run.status,
+            "started_at": iso(run.started_at),
+            "finished_at": iso(run.finished_at),
+            "result": run.result,
+            "failure": run.failure,
+        },
+        **queries.takeout_summary(db, user),
+    }
+
+
+# --- channels watched but not followed -----------------------------------------------------------
+
+
+@router.get("/takeout/channels")
+def watched_channels(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
+    """The channels the user's Takeout history shows them watching often but not following: each
+    with how many of its videos over five minutes they watched, and when last. `pending` is how many
+    more channels may join once the import has asked YouTube for their videos' lengths or their
+    names, so a list still being made does not read as the whole of it; `imported` tells "no history
+    uploaded" from "nothing to suggest"."""
+    return {
+        "imported": queries.has_watch_history(db, user),
+        "pending": queries.watched_channels_pending(db, user),
+        "channels": [
+            {
+                "id": ch.id,
+                "title": ch.title,
+                "handle": ch.handle,
+                "thumbnail_url": ch.thumbnail_url,
+                "subscriber_count": ch.subscriber_count,
+                "video_count": ch.video_count,
+                "description": ch.description,
+                "watched": int(n),
+                "last_watched_at": iso(last),
+            }
+            for ch, n, last in queries.watched_channels(db, user)
+        ],
+    }
+
+
+@router.post("/takeout/channels/{channel_id}/videos")
+def watched_channel_videos(
+    channel_id: str, user: User = Depends(current_user), db: Session = Depends(get_db)
+) -> dict:
+    """The videos of a channel that the user's list counts, ready to play, last watched first. The
+    channel is explored for the user (as its page does), and the videos the catalog lacks come in as
+    the exploration's: the player needs them in the catalog, the shared Library does not get them,
+    and the cleanup reclaims them once nobody keeps them. Of a channel someone follows they come in
+    as the catalog's own, as its uploads do. `gone`: how many YouTube no longer has."""
+    channel = catalog_queries.channel(db, channel_id)
+    if channel is None:
+        raise not_found("channel")
+    if catalog_queries.is_blocked(db, user, channel_id):
+        raise refused(403, "channel_blocked")
+    counted, gone = queries.watched_channel_videos(db, user, channel_id)
+    ids = [vid for vid, _ in counted]
+    known = queries.known_video_ids(db, ids)
+    missing = [vid for vid in ids if vid not in known]
+    if missing and quota.backfill_reserve_reached():
+        raise refused(429, "quota_reserve")
+    catalog_service.explore(db, user, channel)
+    db.commit()
+    if missing:
+        try:
+            with (
+                quota.attribute(user.id, quota.QuotaAction.CHANNELS_EXPLORE),
+                YouTubeClient(db, user) as yt,
+            ):
+                ensure_videos(db, yt, missing, explore=channel.from_explore)
+        except YouTubeError as exc:
+            raise youtube_refusal(exc) from exc
+        catalog_service.settle_explored(db, channel)
+        db.commit()
+    items = queries.feed_rows_by_ids(db, user, ids)
+    watched_at = dict(counted)
+    for item in items:
+        item["takeout_watched_at"] = iso(watched_at[item["id"]])
+    # A video YouTube still had when its length was asked may have gone since.
+    return {"ok": True, "items": items, "gone": gone + len(ids) - len(items)}
+
+
+@router.put("/takeout/channels/{channel_id}/dismissal")
+def dismiss_watched_channel(
+    channel_id: str = Path(max_length=32),
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """Put a channel out of the user's list (it stays out until restored). Only one their own history
+    shows them watching: a made-up id would only pile up rows every list read then has to skip."""
+    if not queries.watched_in_history(db, user, channel_id):
+        raise not_found("channel")
+    service.dismiss_watched_channel(db, user, channel_id)
+    db.commit()
+    return {"ok": True, "channel_id": channel_id}
+
+
+@router.delete("/takeout/channels/{channel_id}/dismissal")
+def restore_watched_channel(
+    channel_id: str = Path(max_length=32),
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """Let a channel the user put out of the list back on it."""
+    service.restore_watched_channel(db, user, channel_id)
+    db.commit()
+    return {"ok": True, "channel_id": channel_id}

backend/app/routes/files.py

@@ -139,7 +139,7 @@ def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
 
 def quota_detail(e: quota.QuotaExceeded) -> dict:
     """Structured error the trilingual client localizes (reason key + numbers), instead of a
-    pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
+    pre-baked English string with a dot-decimal GB value. See lib/api/refusalText.ts localizeDetail()."""
     return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
 
 

backend/app/routes/me.py

@@ -9,11 +9,9 @@ from app.auth import (
     count_admins,
     count_pending_activations,
     current_user,
-    drop_from_wallet,
     google_enabled,
     has_read_scope,
     has_write_scope,
-    is_allowed,
     optional_current_user,
     purge_user,
 )
@@ -26,79 +24,10 @@ from app.db import get_db
 from app.models import Invite, User
 from app.utils import iso
 from app.refusal import refused
-from app.routes._common import not_found
 
 router = APIRouter(prefix="/api/me", tags=["me"])
 
 
-@router.get("/accounts")
-def my_accounts(
-    request: Request,
-    user: User = Depends(current_user),
-    db: Session = Depends(get_db),
-) -> list[dict]:
-    """The accounts that have authenticated in this browser session (switchable without a
-    new Google sign-in). The active one is flagged."""
-    ids = request.session.get("account_ids") or [user.id]
-    rows = {u.id: u for u in db.execute(select(User).where(User.id.in_(ids))).scalars()}
-    out = []
-    for uid in ids:  # preserve recency order from the session
-        u = rows.get(uid)
-        if u is not None:
-            out.append(
-                {
-                    "id": u.id,
-                    "email": u.email,
-                    "display_name": u.display_name,
-                    "avatar_url": u.avatar_url,
-                    "active": u.id == user.id,
-                }
-            )
-    return out
-
-
-class SwitchAccountIn(BaseModel):
-    user_id: int
-
-
-@router.post("/switch")
-def switch_account(
-    body: SwitchAccountIn,
-    request: Request,
-    user: User = Depends(current_user),
-    db: Session = Depends(get_db),
-) -> dict:
-    """Switch the active account to another one already authenticated in this browser. No
-    Google round-trip — but only accounts present in the session list (proof they signed in
-    here) are allowed, and access is re-checked in case the invite was revoked since."""
-    target = body.user_id
-    if target not in (request.session.get("account_ids") or []):
-        raise refused(403, "not_your_account")
-    # An account `_session_user` would refuse — gone, deactivated, suspended, or its session revoked — is
-    # refused here too, and dropped from the switch list the way `_session_user` drops it. Made the session
-    # default instead, the next request's current_user would refuse it as the DEFAULT and clear the whole
-    # wallet session, logging out every account here. (B-core-9)
-    u = db.get(User, target)
-    if u is None:
-        drop_from_wallet(request.session, target)
-        raise not_found("account")
-    if not is_allowed(db, u.email):
-        raise refused(403, "account_no_access")
-    if not u.is_active:
-        drop_from_wallet(request.session, target)
-        raise refused(403, "account_no_access")
-    if u.is_suspended:
-        drop_from_wallet(request.session, target)
-        raise refused(403, "account_suspended")
-    # Revoked: a password reset/change or "log out everywhere" bumped the account's epoch.
-    epochs = request.session.get("epochs") or {}
-    if int(epochs.get(str(target), 0)) != (u.session_epoch or 0):
-        drop_from_wallet(request.session, target)
-        raise refused(403, "account_signed_out")
-    request.session["user_id"] = target
-    return {"ok": True, "user_id": target}
-
-
 @router.get("")
 def get_me(
     user: User | None = Depends(optional_current_user), db: Session = Depends(get_db)
@@ -150,6 +79,10 @@ def get_me(
         # Which live-search source runs, so the search button can say what a search costs: "scrape"
         # spends no API quota, "api" spends 100 units a page.
         "search_source": sysconfig.get_str(db, "search_source"),
+        # Whether live search and related videos are on: an admin turns either off with a daily
+        # limit of 0, and the UI then leaves out what would only be refused.
+        "search_enabled": sysconfig.get_int(db, "search_daily_limit_per_user") > 0,
+        "related_enabled": sysconfig.get_int(db, "related_daily_limit_per_user") > 0,
         "can_read": has_read_scope(user),
         "can_write": has_write_scope(user),
         "pending_invites": pending_invites,
@@ -187,17 +120,11 @@ def delete_my_account(
     surfaces)."""
     if user.role == "admin" and count_admins(db) <= 1:
         raise refused(400, "only_admin_delete")
-    user_id = user.id
     # Full erasure (cascades) + access-request cleanup + Google-grant revocation; shared with the
-    # admin delete path. Capture the id first — `user` is detached once purge_user deletes the row.
+    # admin delete path.
     purge_user(db, user, background)
-    # Drop this account from the browser session; switch to another signed-in account if any.
-    remaining = [a for a in (request.session.get("account_ids") or []) if a != user_id]
-    if remaining:
-        request.session["account_ids"] = remaining
-        request.session["user_id"] = remaining[-1]
-    else:
-        request.session.clear()
+    # The session's one account is gone: so is the session.
+    request.session.clear()
     return {"deleted": True}
 
 
@@ -213,7 +140,13 @@ def update_preferences(
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
-    # Merge so partial updates don't wipe other keys.
+    # Merge so partial updates don't wipe other keys — over the stored row read again under a row
+    # lock: a client saves keys one at a time (several at once on a boot that moves this browser's
+    # copies to the account), and two merges over the same earlier read would keep only the last.
+    # FOR NO KEY UPDATE, as bump_session_epoch: it still orders two merges, and a transaction that
+    # only added a row referring to the account (a playlist pull, then minutes of YouTube) does not
+    # hold the save up.
+    db.refresh(user, with_for_update={"key_share": True})
     merged = dict(user.preferences or {})
     merged.update(preferences.root)
     user.preferences = merged

backend/app/routes/playlists.py

@@ -3,10 +3,10 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
 mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
 out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Literal
 
 from fastapi import APIRouter, Depends, HTTPException
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, model_validator
 from sqlalchemy.orm import Session
 
 from app import quota
@@ -41,11 +41,16 @@ router = APIRouter(prefix="/api/playlists", tags=["playlists"])
 # It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
 # scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
 # two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
-# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
-# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
-# legitimate work. Their exposure is tracked as SEC-18 instead.
 _playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
 
+# SEC-18, the push side: `push-plan` reads the live list (1 unit a page) and `push` reads and writes
+# it, both against the same instance-wide budget. They get a budget of their own, counted in calls
+# and far wider than the pull's: a press in the UI is one plan and, unless nothing changed, one push,
+# so no person reaches it, while a scripted loop stops at it. A big push is not this limit's job —
+# `push` already refuses one the day's remaining budget cannot cover.
+PUSH_ALLOWANCE = 20
+_playlist_push_limiter = RateLimiter(max_events=PUSH_ALLOWANCE, window_seconds=600)
+
 
 def _check_pull_cooldown(user: User) -> None:
     """Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
@@ -53,7 +58,23 @@ def _check_pull_cooldown(user: User) -> None:
         raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
 
 
-def _refused(status: int, reason: str) -> HTTPException:
+# Every playlist action the server refuses; the UI names each one (`playlists.errors.<reason>`, held by
+# the frontend's backendIds contract, which reads this Literal).
+PlaylistRefusal = Literal[
+    "name_required",
+    "not_on_youtube",
+    "push_cooldown",
+    "quota_exceeded",
+    "sync_all_busy",
+    "sync_busy",
+    "watch_later_not_pushable",
+    "watch_later_undeletable",
+    "youtube_editing_off",
+    "youtube_read_scope",
+]
+
+
+def _refused(status: int, reason: PlaylistRefusal) -> HTTPException:
     """A playlist action the server refuses, as a reason the UI translates
     (`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
     return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
@@ -74,10 +95,13 @@ def _video_or_404(db: Session, video_id: str) -> None:
 def _summary(
     pl: "Playlist",
     count: int,
-    total_duration: int = 0,
+    duration: tuple[int, int] = (0, 0),
     has_video: bool | None = None,
     cover: str | None = None,
 ) -> dict:
+    """`duration` is `queries.durations`'s pair: the known running time, and the videos it left out
+    for having no length — the "+N without length" beside it."""
+    total_duration, left_out = duration
     out = {
         "id": pl.id,
         "name": pl.name,
@@ -87,6 +111,7 @@ def _summary(
         "dirty": pl.dirty,
         "item_count": count,
         "total_duration_seconds": total_duration,
+        "without_length": left_out,
         "cover_thumbnail": cover,
         # When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
         # "recently changed first" sort reads it. See `service.touch` for why it isn't automatic.
@@ -117,7 +142,7 @@ def list_playlists(
         _summary(
             p,
             counts.get(p.id, 0),
-            durations.get(p.id, 0),
+            durations.get(p.id, (0, 0)),
             (p.id in member) if contains else None,
             covers.get(p.id),
         )
@@ -229,9 +254,16 @@ def revert_youtube(
             raise youtube_refusal(exc, playlist=True) from exc
 
 
+def _check_push_cooldown(user: User) -> None:
+    """Shared gate of the two push routes, so they spend one allowance and cannot drift apart."""
+    if not _playlist_push_limiter.allow(str(user.id)):
+        raise _refused(429, "push_cooldown")
+
+
 def _pushable(db: Session, user: User, playlist_id: int) -> "Playlist":
-    """A local playlist the user may push to YouTube (create or update). The built-in
-    Watch later and read-only YouTube mirrors are never pushable."""
+    """A playlist the user may push to YouTube: a local one (created there on its first push) or a
+    mirror of their own YouTube list (its local edits merged back). Only the built-in Watch later is
+    never pushable."""
     pl = _own_playlist(db, user, playlist_id)
     if pl.kind == "watch_later":
         raise _refused(400, "watch_later_not_pushable")
@@ -248,6 +280,7 @@ def push_plan(
     the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
     YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
     pl = _pushable(db, user, playlist_id)
+    _check_push_cooldown(user)  # after the refusal: a list that cannot be pushed spends nothing
     try:
         with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
             plan = plan_merge(db, user, pl)
@@ -273,6 +306,8 @@ def push(
     with sync_claim(db, pl.id) as free:
         if not free:
             raise _refused(409, "sync_busy")
+        # Spent only by a push that runs, as Revert's pull: a busy list or Watch later costs nothing.
+        _check_push_cooldown(user)
         try:
             with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
                 # Read the live playlist ONCE (linked playlists only) and share it with both the
@@ -334,8 +369,8 @@ def rename_playlist(
             raise _refused(400, "name_required")
         service.rename_playlist(db, pl, name)
     db.commit()
-    count, total, cover = queries.playlist_totals(db, pl.id)
-    return _summary(pl, count, total, cover=cover)
+    count, total, left_out, cover = queries.playlist_totals(db, pl.id)
+    return _summary(pl, count, (total, left_out), cover=cover)
 
 
 @router.delete("/{playlist_id}")
@@ -371,24 +406,17 @@ def delete_playlist(
     return {"deleted": playlist_id}
 
 
-class AddItemIn(VideoRefIn):
-    # Where the video lands among the list's items, in their stored order (0 = first). Omitted, it
-    # is appended. The list page's "Undo" of a removal puts the row back where it was with this —
-    # ONE request, because append-then-reorder leaves it at the end whenever the second call fails.
-    position: int | None = Field(default=None, ge=0)
-
-
 @router.post("/{playlist_id}/items")
 def add_item(
     playlist_id: int,
-    body: AddItemIn,
+    body: VideoRefIn,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
-    pl = _own_playlist(db, user, playlist_id)
+    pl = _own_playlist(db, user, playlist_id, lock=True)
     video_id = body.video_id
     _video_or_404(db, video_id)
-    service.add_to_playlist(db, pl, video_id, body.position)
+    service.add_item(db, pl, video_id, mark_dirty=True)
     db.commit()
     return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
 
@@ -400,58 +428,158 @@ def remove_item(
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
-    pl = _own_playlist(db, user, playlist_id)
+    pl = _own_playlist(db, user, playlist_id, lock=True)
     service.remove_item(db, pl, video_id, mark_dirty=True)
     db.commit()
     return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
 
 
-class MoveItemIn(BaseModel):
+class VideoIdsIn(BaseModel):
+    # Required: with a default, a misnamed key answered 200 having acted on no video.
+    video_ids: list[str]
+
+
+def _paired(video_count: int, positions: list[int]) -> None:
+    """One place per video, none before the start and none twice: both undo bodies come back from
+    the client, and the insert's arithmetic counts on places that only rise."""
+    if len(positions) != video_count:
+        raise ValueError("positions must pair one-to-one with the videos")
+    if any(at < 0 for at in positions):
+        raise ValueError("positions cannot be negative")
+    if len(set(positions)) != len(positions):
+        raise ValueError("positions cannot repeat")
+
+
+class RestoreItemsIn(VideoIdsIn):
+    # Where each video goes back: an index over the ordered rows, one per video.
+    positions: list[int] = []
+
+    @model_validator(mode="after")
+    def _one_position_per_video(self):
+        _paired(len(self.video_ids), self.positions)
+        return self
+
+
+class MoveItemsIn(VideoIdsIn):
     target_playlist_id: int
 
 
-@router.post("/{playlist_id}/items/{video_id}/move")
-def move_item(
+class UndoMoveIn(BaseModel):
+    from_playlist_id: int
+    # Required, as `video_ids` is. An empty one is still a real body: the target held them all.
+    moved: list[str]
+    already_there: list[str] = []
+    # One per video of `moved` + `already_there`, in that order.
+    positions: list[int]
+
+    @model_validator(mode="after")
+    def _one_position_per_video(self):
+        _paired(len(self.moved) + len(self.already_there), self.positions)
+        return self
+
+
+def _list_gone(e: service.ListGone) -> HTTPException:
+    """Each list of a two-list write under the reason the route's own read gives it."""
+    return not_found("playlist") if e.source else not_found("move_target")
+
+
+def _move_refused(e: service.MoveRefused) -> HTTPException:
+    return HTTPException(status_code=400, detail={"code": "playlist_move", "reason": e.reason})
+
+
+def _other_list(db: Session, user: User, playlist_id: int) -> "Playlist":
+    """The other list of a move. Its own reason: "this playlist doesn't exist any more" about a list
+    that is gone was read as the one on screen, which is still there."""
+    other = queries.own_playlist(db, user, playlist_id)
+    if other is None:
+        raise not_found("move_target")
+    return other
+
+
+@router.post("/{playlist_id}/items/remove-bulk")
+def remove_items(
     playlist_id: int,
-    video_id: str,
-    body: MoveItemIn,
+    body: VideoIdsIn,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:
-    """Move one video from a playlist into another of the caller's playlists, atomically.
+    """Remove many videos from a playlist at once. Returns the ones that went."""
+    pl = _own_playlist(db, user, playlist_id, lock=True)
+    removed = service.remove_items(db, pl, body.video_ids)
+    db.commit()
+    return {"ok": True, "playlist_id": pl.id, "removed": removed}
+
+
+@router.post("/{playlist_id}/items/restore")
+def restore_items(
+    playlist_id: int,
+    body: RestoreItemsIn,
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """The "Undo" of a removal: the videos back at their places. Returns how many came back."""
+    pl = _own_playlist(db, user, playlist_id, lock=True)
+    added = service.restore_items(db, pl, body.video_ids, body.positions)
+    db.commit()
+    return {"ok": True, "playlist_id": pl.id, "added": added}
+
+
+@router.post("/{playlist_id}/items/move-bulk")
+def move_items(
+    playlist_id: int,
+    body: MoveItemsIn,
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """Move videos into another of the caller's playlists, atomically.
 
     ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
-    from the source — leaves the video in BOTH lists whenever the second call fails, which is
-    exactly the kind of half-applied state this codebase treats as a bug rather than an edge
-    case. Idempotent: if the video is already in the target, the source row is simply dropped
-    (moving into a list that already has it is a removal, not a duplicate)."""
+    from the source — leaves a video in BOTH lists whenever the second call fails. A video the
+    target already has only leaves the source, and one a concurrent edit (another tab, a YouTube
+    re-pull) already took out is skipped. Says which landed there and which it already held, for
+    the "Undo"."""
     src = _own_playlist(db, user, playlist_id)
-    # Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
-    # the list on screen — which is still there.
-    dst = queries.own_playlist(db, user, body.target_playlist_id)
-    if dst is None:
-        raise not_found("move_target")
+    dst = _other_list(db, user, body.target_playlist_id)
     try:
-        service.move_item(db, src, dst, video_id)
+        moved, already_there = service.move_items(db, src, dst, body.video_ids)
     except service.MoveRefused as e:
-        # A same-list move is a bad request; a video a concurrent edit (another tab, a YouTube
-        # re-pull) already took out is gone.
-        status = 400 if e.reason == "same_playlist" else 404
-        raise HTTPException(
-            status_code=status, detail={"code": "playlist_move", "reason": e.reason}
-        ) from None
+        raise _move_refused(e) from None
+    except service.ListGone as e:
+        raise _list_gone(e) from None
     db.commit()
-    return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
+    return {
+        "ok": True,
+        "from_playlist_id": src.id,
+        "to_playlist_id": dst.id,
+        "moved": moved,
+        "already_there": already_there,
+    }
 
 
-class ReorderItemsIn(BaseModel):
-    video_ids: list[str] = []
+@router.post("/{playlist_id}/items/undo-move")
+def undo_move(
+    playlist_id: int,
+    body: UndoMoveIn,
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """The "Undo" of a move, back into this playlist in one transaction. Returns how many came back."""
+    src = _own_playlist(db, user, playlist_id)
+    dst = _other_list(db, user, body.from_playlist_id)
+    try:
+        added = service.undo_move(db, src, dst, body.moved, body.already_there, body.positions)
+    except service.MoveRefused as e:
+        raise _move_refused(e) from None
+    except service.ListGone as e:
+        raise _list_gone(e) from None
+    db.commit()
+    return {"ok": True, "playlist_id": src.id, "added": added}
 
 
 @router.put("/{playlist_id}/order")
 def reorder_items(
     playlist_id: int,
-    body: ReorderItemsIn,
+    body: VideoIdsIn,
     user: User = Depends(current_user),
     db: Session = Depends(get_db),
 ) -> dict:

backend/app/routes/plex/detail.py

@@ -432,6 +432,7 @@ def item_detail(
         "summary": (it.summary_hu or it.summary) if hu else it.summary,
         "year": it.year,
         "duration_seconds": it.duration_s,
+        "size_bytes": it.size_bytes,
         "playable": it.playable,
         # Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
         "transcode_hw": it.transcode_hw,

backend/app/routes/plex/playback.py

@@ -19,7 +19,7 @@ from fastapi.responses import FileResponse
 from pydantic import BaseModel
 from sqlalchemy.orm import Session
 
-from app.auth import current_user, resolved_user_id
+from app.auth import current_user, session_user_id
 from app.db import SessionLocal, get_db
 from app.models import User
 from app.plex import imgcache, queries, service, subtitles
@@ -42,7 +42,7 @@ def _require_session(request: Request) -> int:
     """Lightweight auth for the high-fan-out image proxies: validate the signed session cookie WITHOUT
     a DB user-load, so these endpoints never hold a pool connection for the auth. Poster images are
     low-sensitivity; a valid session is enough."""
-    uid, _ = resolved_user_id(request)
+    uid = session_user_id(request)
     if not uid:
         raise refused(401, "not_authenticated")
     return uid
@@ -234,6 +234,15 @@ def person_image(u: str, request: Request) -> Response:
     return imgcache.response(data, ct)
 
 
+def _supersede_resume_pushes(user_id: int, item_id: int, now: datetime) -> None:
+    """A watched mark supersedes every resume push still in flight for the item: stamping it as the
+    latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep the map
+    small) removed the very guard, and an older resume push could then land at Plex AFTER the
+    scrobble and revive a resume point on a watched item. Eviction bounds the map."""
+    with _push_throttle_lock:
+        _last_push_at[(user_id, item_id)] = now
+
+
 def _push_resume_guarded(
     user_id: int,
     item_id: int,
@@ -314,13 +323,14 @@ def item_progress(
     `_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
     resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
     it = item_or_404(db, rating_key)
-    # A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
+    # A player parked mid-item — playing or PAUSED — keeps checkpointing (every 10 s, once a minute
+    # in a hidden silent page), so this is the signal
     # that such a viewer still wants their HLS session: the one thing `last_access` could not see,
     # since it moved only on a file fetch and a paused player fetches nothing (see
     # stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
     # finish latch is up, or before the first position is known. Those are the states in which
     # reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
-    # the same 404 → reclaim the player already handles.
+    # the same 410 → reclaim the player already handles.
     plex_stream.touch_session(user.id, str(rating_key))
     position = max(0, int(body.position_seconds))
     duration = int(body.duration_seconds)
@@ -329,12 +339,7 @@ def item_progress(
     near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
     if near_end:
         flipped = service.mark_finished(db, user.id, it.id, now)
-        # The finish supersedes every resume push still in flight for this item: stamping it as the
-        # latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep
-        # the map small) removed the very guard, and an older resume push could then land at Plex
-        # AFTER the scrobble and revive a resume point on a watched item. Eviction bounds the map.
-        with _push_throttle_lock:
-            _last_push_at[(user.id, it.id)] = now
+        _supersede_resume_pushes(user.id, it.id, now)
         db.commit()
         # Only the checkpoint that flipped the row scrobbles: a later checkpoint of the same finished
         # item — or a second one racing this one — would otherwise bump Plex's viewCount again.
@@ -352,7 +357,11 @@ def item_progress(
     push_now = _should_push_resume(user.id, it.id, now, final)
     # Marked dirty before the (background) push so the reconcile knows to re-push if it fails, and so
     # `push_state_to_plex` can settle synced_to_plex once Plex confirms.
-    service.checkpoint(db, user, it.id, st, position, now, dirty=push_now)
+    if not service.checkpoint(db, user, it.id, st, position, now, dirty=push_now):
+        # Marked watched a moment ago: this is the finishing player's late tick (see
+        # `service.checkpoint`), and the answer is the same one a near-end checkpoint gets.
+        db.commit()
+        return {"ok": True, "status": "watched", "position_seconds": 0}
     db.commit()
     if push_now:
         _push_watch(
@@ -408,6 +417,8 @@ def item_state(
             _push_watch(background, db, user.id, it.id, rating_key, "unwatched")
         return {"ok": True, "status": "new"}
     service.set_status(db, user, it.id, st, status)
+    if status == "watched":
+        _supersede_resume_pushes(user.id, it.id, datetime.now(timezone.utc))
     db.commit()
     if status == "watched" and prev_status != "watched":
         _push_watch(background, db, user.id, it.id, rating_key, "watched")
@@ -427,6 +438,10 @@ def _bulk_state(
     always reads the committed marks."""
     pushable = plex_watch.link_for_push(db, user_id) is not None
     to_push = service.bulk_state(db, user_id, eps, watched)
+    if watched:
+        now = datetime.now(timezone.utc)
+        for item_id, _rating_key, _action in to_push:
+            _supersede_resume_pushes(user_id, item_id, now)
     if pushable and to_push:
         # One coalesced background task for the whole batch (reuses a single session + Plex client),
         # not one task per episode — a 200-ep show mark is 1 task, not 200.
@@ -595,7 +610,9 @@ def stream_hls_file(
     is per (key, viewer, offset, tag) — so resolving by the key alone handed the old player the new
     session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
     bytes with a 200, which the player has no way to notice. A request for a
-    session that is no longer the registered one is a 404, and the player reclaims at the playhead.
+    session that is no longer the registered one is a 410, and the player reclaims at the playhead.
+    410, not 404: a 404 here means "that segment is not there yet", worth hls.js's retries, and the
+    player can only read the status (a gone session retried for half a minute behind that 404).
 
     That reclaim is also why a request with no live session is no longer auto-started here: the
     auto-start could only ever begin at 0, silently throwing away the viewer's position."""
@@ -611,7 +628,7 @@ def stream_hls_file(
         raise not_found("file")
     s = plex_stream.current_session(user.id, str(rating_key))
     if s is None or s.id != session_id:
-        raise refused(404, "session_gone")
+        raise refused(410, "session_gone")
     path = s.dir / filename
     # Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
     # segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
@@ -632,14 +649,15 @@ def stream_hls_file(
     try:
         content = path.read_bytes()
     except OSError:
-        # The session was torn down between the wait and the read — `_kill` rmtrees the whole
-        # directory, and nothing holds `_lock` across those two steps. It has to present as the same
-        # 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
+        # The session was torn down between the wait and the read — teardown moves the whole
+        # directory aside and deletes it, and nothing holds `_lock` across those two steps. It has
+        # to present as the same 410 a gone session does: the player's recovery is keyed on the
+        # STATUS, so
         # a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
         # restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
         # kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
         # instance's in-flight segment request races the rmtree on every seek-restart.
-        raise refused(404, "session_gone") from None
+        raise refused(410, "session_gone") from None
     return Response(
         content=content,
         media_type=_HLS_CT[ext],

backend/app/routes/plex/playlists.py

@@ -183,7 +183,9 @@ def playlist_remove_item(
 
 
 class PlexItemKeysIn(BaseModel):
-    item_rating_keys: list[str] = []
+    # Required: with a default, a misnamed key answered 200 having acted on no item, and a reorder
+    # took it for an empty order.
+    item_rating_keys: list[str]
 
 
 class PlexItemAddBulkIn(PlexItemKeysIn):

backend/app/routes/public.py

@@ -176,8 +176,15 @@ def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
     returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
     link = _link_or_404(db, token)
     if link.password_hash:
-        # Which kind it opens, for the password prompt's title — never what is in it.
-        return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
+        # Which kind it opens, for the password prompt's words — never what is in it. `video` tells
+        # a video (watched) from any other file (opened); the address is only a hint, every
+        # prefix opens every link.
+        opens = linksmod.opens(db, link)
+        return {
+            "needs_password": True,
+            "kind": "folder" if opens == "folder" else "file",
+            "video": opens == "video",
+        }
     meta = _opened(db, link, token)
     db.commit()
     return meta

backend/app/routes/runs.py

@@ -3,7 +3,7 @@
 Admin-only and read-only — rows leave the table through the retention job
 (`job_run_retention_days`). Filtered on the server and paged by `_keyset`, like the server log and
 the audit log beside it on the activity page. A row names what a reader needs without a second
-request: the channel a user's ad-hoc run worked on, and how many log lines the run left (only
+request: what a user's ad-hoc run worked on (the channel, the importing user), and how many log lines the run left (only
 WARNING and up are stored, so most runs have none — the count says so before the click).
 """
 
@@ -116,8 +116,12 @@ def list_runs(
         limit,
         lambda r: r.started_at,
     )
-    channels = {r.target for r in runs if r.job in scheduler.AD_HOC_JOBS and r.target}
+    channels = {r.target for r in runs if r.job == scheduler.CHANNEL_BACKFILL and r.target}
     titles = catalog_queries.channel_titles(db, channels)
+    for target in {r.target for r in runs if r.job == scheduler.TAKEOUT_IMPORT and r.target}:
+        title = scheduler.ad_hoc_target_title(db, scheduler.TAKEOUT_IMPORT, target)
+        if title is not None:
+            titles[target] = title
     lines: dict[int, int] = {}
     if runs:
         counted = db.execute(

backend/app/routes/saved_views.py

@@ -128,7 +128,8 @@ def update_view(
         db.commit()
     except IntegrityError:
         db.rollback()
-        raise refused(409, "view_name_taken", name=view.name) from None
+        # The name asked for: the rollback has put the view's old name back on `view`.
+        raise refused(409, "view_name_taken", name=name) from None
     return _serialize(view)
 
 

backend/app/routes/search.py

@@ -13,10 +13,18 @@ this search-discovered "noise" by default. Shorts and currently-live/upcoming vi
 ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
 dropped from the search response up front).
 
+A video's related videos (`/related/{video_id}`, the list YouTube shows beside it) come in the same
+way: read from the website (`related_scrape`), ingested as search finds are, under their own daily
+cap — with what this user has already seen left out.
+
 This is the HTTP side — the paging, the caps and the refusals; what a page of results leaves in the
 catalog is `catalog/search`."""
 
-from fastapi import APIRouter, Depends
+import threading
+from collections.abc import Callable
+from typing import Literal
+
+from fastapi import APIRouter, Depends, Path
 from pydantic import BaseModel
 from sqlalchemy.orm import Session
 
@@ -26,8 +34,10 @@ from app.catalog import queries
 from app.catalog import search as catalog_search
 from app.db import get_db
 from app.models import User
+from app.ratelimit import RateLimiter
 from app.reader import queries as reader_queries
-from app.youtube import search_scrape
+from app.youtube import related_scrape, search_scrape
+from app.youtube.innertube import ScrapeError
 from app.youtube.client import YouTubeClient, YouTubeError
 from app.refusal import refused
 from app.routes._common import youtube_refusal
@@ -37,6 +47,115 @@ router = APIRouter(prefix="/api/search", tags=["search"])
 
 # search.list costs this many quota units per page.
 SEARCH_COST = 100
+# The scrape source reads at most this many pages for one search.
+SEARCH_MAX_PAGES = 12
+
+# A related list is filled to this many unseen videos, reading at most this many pages (about 20
+# videos each, measured) — the ones the user has seen are left out, so a page alone can fall short.
+RELATED_TARGET = 20
+RELATED_MAX_PAGES = 4
+# The locale YouTube fills a related list in: on the video's topic, or with what trends in Hungary.
+RELATED_REGIONS = {"topical": ("en", "US"), "hu": ("hu", "HU")}
+
+# A request that fails outright gives its slot back, but its shape is the user's (a cursor is a
+# free string): one built to fail would read YouTube for nothing, without end. So only this many
+# refunds an hour per user and list; past them a failure keeps its slot, as a success does.
+REFUNDS_PER_HOUR = 5
+_refunds = RateLimiter(max_events=REFUNDS_PER_HOUR, window_seconds=3600)
+# The API source's budget check and the search.list call it pays for run one at a time: checked
+# apart, searches sent together at the last 100 units all passed, and all spent. One process
+# serves the API (a single uvicorn worker, as `ratelimit` says), so a process lock holds them all.
+# A search waits this long for it, then is refused as busy: each waiter holds a request worker, and
+# a slow YouTube (30 s a call at worst) would otherwise park them all behind one another.
+_api_search = threading.Lock()
+API_SEARCH_WAIT_SECONDS = 10.0
+
+
+def _gather(
+    db: Session,
+    yt: YouTubeClient,
+    user: User,
+    *,
+    cursor: str | None,
+    max_pages: int,
+    fetch: Callable[[str | None], dict],
+    candidates_of: Callable[[list[dict], list[str]], list[dict]],
+    enough: Callable[[list[str]], bool],
+    scrape_failed: Callable[[Exception], None],
+    ingest_failed: Callable[[int, int], None],
+) -> tuple[list[str], str | None]:
+    """Page through a YouTube list into the catalog: fetch a page, keep its `candidates_of`, ingest
+    them, and go on from its continuation until `enough` or `max_pages`. Returns the kept ids and
+    the cursor to read further from. A page that fails after earlier ones gathered something ends
+    the list with what it has; a first page that fails raises the 422 refusal.
+
+    422, not 502: this is a JSON endpoint, and the FE treats 502/503/504 as a gateway dropout
+    (auto-retry + "connection lost" banner — lib/api/core RETRIABLE_GATEWAY), which hides the real
+    message and repeats the work. A Data API failure is logged where it happened; the scraper's is
+    not, and its text does not ride the answer — so `scrape_failed` says it. An ingest (DB writes +
+    enrich) fails the same way, guarded for any exception so a write or shape failure is a 422 with
+    a message too, not a bare 500; `ingest_failed` logs it inside the `except`, with the trace."""
+    collected: list[str] = []
+    next_cursor = cursor
+    for _ in range(max_pages):
+        try:
+            page = fetch(next_cursor)
+        except (YouTubeError, ScrapeError) as exc:
+            if collected:
+                break  # keep what earlier pages already gathered
+            if isinstance(exc, ScrapeError):
+                scrape_failed(exc)
+            raise youtube_refusal(exc) from exc
+        candidates = candidates_of(page["items"], collected)
+        try:
+            collected.extend(catalog_search.ingest_candidates(db, yt, user, candidates))
+        except Exception as exc:  # any ingest failure, not just YouTubeError
+            db.rollback()
+            ingest_failed(len(candidates), len(collected))
+            if collected:
+                break
+            raise youtube_refusal(exc) from exc
+        next_cursor = page["next_page_token"]
+        if enough(collected) or not next_cursor:
+            break
+    return collected, next_cursor
+
+
+def _give_back(claimed: int, feature: str, user_id: int) -> None:
+    """Refund the slot of a request that failed outright, while this user has refunds left."""
+    if _refunds.allow(f"{feature}:{user_id}"):
+        quota.refund_action(claimed)
+        return
+    log.info(
+        "a failed request kept its slot: this hour's refunds are spent",
+        extra=logs.event(f"{feature}.refund_refused", per_hour=REFUNDS_PER_HOUR),
+    )
+
+
+def _budget_spent() -> None:
+    """Say that a live search was refused because the shared YouTube budget cannot pay for it."""
+    log.info(
+        "a live search was refused: the shared YouTube budget is spent",
+        extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
+    )
+
+
+def _busy() -> None:
+    """Say that a live search was refused because other API searches held the line too long."""
+    log.info(
+        "a live search was refused: other searches held the line",
+        extra=logs.event("search.refused", reason="busy", waited=API_SEARCH_WAIT_SECONDS),
+    )
+
+
+def _cap_spent(feature: str, limit: int) -> None:
+    """Say that a request was refused for the per-user daily cap. The 429 is in the request line
+    either way; which of the caps produced it is not, and they are answered by different things —
+    this one by tomorrow, the shared budget by the admin."""
+    log.info(
+        "a request was refused: the per-user daily cap is spent",
+        extra=logs.event(f"{feature}.refused", reason="daily_cap", limit=limit),
+    )
 
 
 @router.get("/youtube")
@@ -59,109 +178,122 @@ def search_youtube(
     # Source: "scrape" (InnerTube, no API quota) or "api" (search.list, 100 units/page).
     source = sysconfig.get_str(db, "search_source")
 
-    # Per-user daily cap applies to both sources (anti-abuse); with scrape it's the only guard
-    # since the search itself spends no quota.
+    # Every request takes a slot of the per-user daily cap, a cursor's too (anti-abuse); with
+    # scrape it's the only guard since the search itself spends no quota. A limit of 0 turns live
+    # search off on this instance — a refusal tomorrow will not change.
     daily_limit = sysconfig.get_int(db, "search_daily_limit_per_user")
-    used = quota.actions_today(db, user.id, quota.QuotaAction.VIDEOS_SEARCH)
-    if used >= daily_limit:
-        # The 429 is in the request line either way; which of the two caps produced it is not, and
-        # they are answered by different things — this one by tomorrow, the other by the admin.
-        log.info(
-            "a live search was refused: the per-user daily cap is spent",
-            extra=logs.event("search.refused", reason="daily_cap", limit=daily_limit),
-        )
-        raise refused(429, "search_daily_limit", limit=daily_limit)
-    # The 100-unit budget pre-check only applies to the API source.
+    if daily_limit <= 0:
+        raise refused(409, "search_disabled")
+    # The 100-unit budget pre-check only applies to the API source. It is asked again under the
+    # lock, right before the call (`fetch`); this one refuses before a slot is taken.
     if source == "api" and not quota.can_spend(db, SEARCH_COST):
-        log.info(
-            "a live search was refused: the shared YouTube budget is spent",
-            extra=logs.event("search.refused", reason="shared_quota", cost=SEARCH_COST),
-        )
+        _budget_spent()
         raise refused(429, "quota_spent")
+    # The check and the take are one step (`quota.claim_action`), whichever the source: checked
+    # first and logged after, requests sent together at the last free slot all passed — and with
+    # the API source each of them spent 100 units. A search that fails outright gives it back below.
+    claimed = quota.claim_action(user.id, quota.QuotaAction.VIDEOS_SEARCH, daily_limit)
+    if claimed is None:
+        _cap_spent("search", daily_limit)
+        raise refused(429, "search_daily_limit", limit=daily_limit)
 
     # Channels this user has blocked are dropped before ingestion — never shown, never stored.
     blocked = queries.blocked_ids(db, user)
 
-    collected: list[str] = []
-    next_cursor = cursor
-    MAX_PAGES = 12  # scrape safety bound
-    with YouTubeClient(db, user) as yt:
-        for _ in range(MAX_PAGES):
-            try:
-                if source == "api":
-                    with quota.attribute(user.id, quota.QuotaAction.VIDEOS_SEARCH):
-                        page = yt.search_videos(term, page_token=next_cursor)
-                else:
-                    page = search_scrape.search(term, continuation=next_cursor)
-            except (YouTubeError, search_scrape.ScrapeError) as exc:
-                if collected:
-                    break  # keep what we already gathered
-                # 422, not 502: this is a JSON endpoint, and the FE treats 502/503/504 as a gateway
-                # dropout (auto-retry + "connection lost" banner — lib/api/core RETRIABLE_GATEWAY),
-                # which hides the real message and burns a second quota-costed search. Upstream failure
-                # on a JSON endpoint = 422 + message; reserve 5xx for the raw byte-proxies. (B-core-1)
-                # A Data API failure is logged where it happened; the scraper's is not, and its text
-                # no longer rides the answer — so it is said here.
-                if isinstance(exc, search_scrape.ScrapeError):
-                    log.warning(
-                        "the YouTube search scrape failed",
-                        extra=logs.event("search.scrape_failed", error=str(exc)),
-                    )
-                raise youtube_refusal(exc) from exc
+    # Whether search.list was paid for: YouTube charges a call it refuses too, so once the client
+    # recorded its units — or the request went out and no answer came back — the search has spent
+    # the shared budget. One that never reached YouTube (no connection, no token) has not.
+    charged = False
 
-            # Drop currently-live/upcoming, channel-less, and blocked-channel results.
-            candidates = [
-                it
-                for it in page["items"]
-                if it["live_broadcast"] == "none"
-                and it["channel_id"]
-                and it["channel_id"] not in blocked
-            ]
-            # Ingest (DB writes + enrich) runs outside the page-fetch try above; guard it too so a
-            # write/shape failure degrades to a 422 with a message on this JSON endpoint instead of a
-            # bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
-            try:
-                collected.extend(catalog_search.ingest_candidates(db, yt, user, candidates))
-            except Exception as exc:  # any ingest failure, not just YouTubeError
-                db.rollback()
-                # The term is NOT in this line. A search string is exactly what the privacy
-                # policy keeps out of the log unless the user turned Teljes/TRACE on for
-                # themselves, and `redact_text` does not cover it — it scrubs addresses and
-                # credential params, not what someone typed into the search box. What the
-                # ERROR line carries instead is the shape of the failure, which is what tells
-                # an ingest bug from an upstream one; the term follows at TRACE, and only
-                # under this user's own consent — not whenever the process runs at TRACE.
-                log.exception(
-                    "live search ingest failed",
-                    extra=logs.event(
-                        "search.live.ingest_failed",
-                        source=source,
-                        candidates=len(candidates),
-                        collected=len(collected),
-                    ),
-                )
-                if logs.consented_trace():
-                    log.log(
-                        logs.TRACE,
-                        "the term the ingest failed on",
-                        # log-pii-ok: only under this user's own Teljes consent
-                        extra=logs.event("search.live.term", term=term),
-                    )
-                if collected:
-                    break  # keep what earlier pages already gathered
-                raise youtube_refusal(exc) from exc
-            next_cursor = page["next_page_token"]
-            # The API source pages cost 100 units each — never auto-page; one page per request.
-            if source == "api" or len(collected) >= limit or not next_cursor:
-                break
+    def fetch(token: str | None) -> dict:
+        if source != "api":
+            return search_scrape.search(term, continuation=token)
+        if not _api_search.acquire(timeout=API_SEARCH_WAIT_SECONDS):
+            _busy()
+            raise refused(429, "search_busy")
+        try:
+            return paid_search(token)
+        finally:
+            _api_search.release()
 
-    # Count this as ONE per-user search against the daily cap. The scrape source spends no
-    # quota, so it never logs an event on its own; the API source already logged exactly one
-    # (via record_usage — it never auto-pages). Logging once here, after a search that produced
-    # at least one page (a total failure raises 502 above and never reaches this), keeps the cap
-    # counting user searches instead of internal continuation pages.
-    if source != "api":
-        quota.log_action(user.id, quota.QuotaAction.VIDEOS_SEARCH)
+    def paid_search(token: str | None) -> dict:
+        nonlocal charged
+        with quota.attribute(user.id, quota.QuotaAction.VIDEOS_SEARCH):
+            if not quota.can_spend(db, SEARCH_COST):
+                _budget_spent()
+                raise refused(429, "quota_spent")
+            spent = yt.spent
+            try:
+                page = yt.search_videos(term, page_token=token)
+            except YouTubeError as exc:
+                charged = yt.spent > spent or exc.sent
+                raise
+            charged = True
+            return page
+
+    def candidates_of(items: list[dict], _collected: list[str]) -> list[dict]:
+        # Drop currently-live/upcoming, channel-less, and blocked-channel results.
+        return [
+            it
+            for it in items
+            if it["live_broadcast"] == "none"
+            and it["channel_id"]
+            and it["channel_id"] not in blocked
+        ]
+
+    def scrape_failed(exc: Exception) -> None:
+        log.warning(
+            "the YouTube search scrape failed",
+            extra=logs.event("search.scrape_failed", error=str(exc)),
+        )
+
+    def ingest_failed(candidates: int, collected: int) -> None:
+        # The term is NOT in this line. A search string is exactly what the privacy policy keeps
+        # out of the log unless the user turned Teljes/TRACE on for themselves, and `redact_text`
+        # does not cover it — it scrubs addresses and credential params, not what someone typed
+        # into the search box. What the ERROR line carries instead is the shape of the failure,
+        # which is what tells an ingest bug from an upstream one; the term follows at TRACE, and
+        # only under this user's own consent — not whenever the process runs at TRACE.
+        log.exception(
+            "live search ingest failed",
+            extra=logs.event(
+                "search.live.ingest_failed",
+                source=source,
+                candidates=candidates,
+                collected=collected,
+            ),
+        )
+        if logs.consented_trace():
+            log.log(
+                logs.TRACE,
+                "the term the ingest failed on",
+                # log-pii-ok: only under this user's own Teljes consent
+                extra=logs.event("search.live.term", term=term),
+            )
+
+    try:
+        with YouTubeClient(db, user) as yt:
+            collected, next_cursor = _gather(
+                db,
+                yt,
+                user,
+                cursor=cursor,
+                # The API source pages cost 100 units each — never auto-page; one page per request.
+                max_pages=1 if source == "api" else SEARCH_MAX_PAGES,
+                fetch=fetch,
+                candidates_of=candidates_of,
+                enough=lambda got: len(got) >= limit,
+                scrape_failed=scrape_failed,
+                ingest_failed=ingest_failed,
+            )
+    except Exception:
+        # A search that showed the user nothing does not count against them, unless search.list
+        # was already paid for: then the slot is what stops a failing search (a bogus cursor gets
+        # a 400 every time) from spending the shared budget without end. A busy refusal paid
+        # nothing and gives it back.
+        if not charged:
+            _give_back(claimed, "search", user.id)
+        raise
 
     ordered = collected[:limit]
 
@@ -182,7 +314,8 @@ def search_youtube(
 
 
 class ClearSearchIn(BaseModel):
-    video_ids: list[str] = []
+    # Required: a default read a misnamed key as an empty list, a 200 that cleared nothing.
+    video_ids: list[str]
 
 
 @router.post("/clear")
@@ -201,3 +334,78 @@ def clear_search(
     removed = catalog_search.clear(db, user, ids)
     db.commit()
     return {"ok": True, "removed": removed}
+
+
+@router.get("/related/{video_id}")
+def related_videos(
+    video_id: str = Path(max_length=16),
+    cursor: str | None = None,
+    region: Literal["topical", "hu"] = "topical",
+    user: User = Depends(current_user),
+    db: Session = Depends(get_db),
+) -> dict:
+    """The videos YouTube shows beside this one, materialised into the catalog like search results,
+    without the ones this user has watched or hidden here or in their Takeout history, and without
+    blocked channels. Pages on (`cursor`) until `RELATED_TARGET` are gathered; `next_cursor` reads
+    further. `region` picks the locale YouTube fills the list in (`RELATED_REGIONS`)."""
+    # Every request takes a slot, a cursor's too: each one scrapes pages and enriches what they
+    # hold, and a cursor is a free string — counting only first pages let a replayed cursor read
+    # without end. The check and the take are one step (`quota.claim_action`); a request that fails
+    # outright gives its slot back, a few an hour (`_give_back`). A limit of 0 turns related videos
+    # off on this instance.
+    daily_limit = sysconfig.get_int(db, "related_daily_limit_per_user")
+    if daily_limit <= 0:
+        raise refused(409, "related_disabled")
+    claimed = quota.claim_action(user.id, quota.QuotaAction.VIDEOS_RELATED, daily_limit)
+    if claimed is None:
+        _cap_spent("related", daily_limit)
+        raise refused(429, "related_daily_limit", limit=daily_limit)
+
+    hl, gl = RELATED_REGIONS[region]
+    blocked = queries.blocked_ids(db, user)
+
+    def candidates_of(items: list[dict], collected: list[str]) -> list[dict]:
+        ids = [it["id"] for it in items]
+        drop = reader_queries.seen_by(db, user, ids) | set(collected) | {video_id}
+        return [it for it in items if it["id"] not in drop and it["channel_id"] not in blocked]
+
+    def scrape_failed(exc: Exception) -> None:
+        log.warning(
+            "the related-videos scrape failed",
+            extra=logs.event("related.scrape_failed", error=str(exc)),
+        )
+
+    def ingest_failed(candidates: int, collected: int) -> None:
+        log.exception(
+            "a related list's ingest failed",
+            extra=logs.event("related.ingest_failed", candidates=candidates, collected=collected),
+        )
+
+    try:
+        with YouTubeClient(db, user) as yt:
+            collected, next_cursor = _gather(
+                db,
+                yt,
+                user,
+                cursor=cursor,
+                max_pages=RELATED_MAX_PAGES,
+                fetch=lambda token: related_scrape.related(
+                    video_id, continuation=token, hl=hl, gl=gl
+                ),
+                candidates_of=candidates_of,
+                enough=lambda got: len(got) >= RELATED_TARGET,
+                scrape_failed=scrape_failed,
+                ingest_failed=ingest_failed,
+            )
+    except Exception:
+        _give_back(claimed, "related", user.id)
+        raise
+
+    rows = reader_queries.feed_rows_by_ids(db, user, collected)
+    # A related list mixes channels the user follows with ones they never met: each row says which.
+    followed = queries.followed_among(db, user, {r["channel_id"] for r in rows})
+    return {
+        "items": [{**r, "followed": r["channel_id"] in followed} for r in rows],
+        "next_cursor": next_cursor,
+        "has_more": bool(next_cursor),
+    }

backend/app/routes/tags.py

@@ -48,8 +48,9 @@ def list_tags(user: User = Depends(current_user), db: Session = Depends(get_db))
             "category": tag.category,
             "system": tag.user_id is None,
             "channel_count": count,
+            "subscribed_count": subscribed,
         }
-        for tag, count in queries.tags_with_counts(db, user)
+        for tag, count, subscribed in queries.tags_with_counts(db, user)
     ]
 
 
@@ -107,7 +108,8 @@ def update_tag(
         db.commit()
     except IntegrityError:
         db.rollback()
-        raise refused(409, "tag_name_taken", name=tag.name) from None
+        # The name asked for: the rollback has put the tag's old name back on `tag`.
+        raise refused(409, "tag_name_taken", name=name) from None
     return {
         "ok": True,
         "id": tag.id,

backend/app/scheduler.py

@@ -22,6 +22,7 @@ from app.plex.tmdb import run_people_enrich
 from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
 from app.notifications import create_notification
 from app.runstatus import RunStatus, RunTrigger
+from app.reader import takeout
 from app.state import is_sync_paused
 from app.sync.autotag import run_autotag_all
 from app.sync.explore import purge_ephemeral
@@ -118,8 +119,9 @@ def _notify_done(
     summary: str | None,
     result: dict | None = None,
 ) -> None:
-    """Post a completion notice to the triggering admin's inbox (manual runs only). Best
-    effort — a notification failure must never mask the job's own outcome."""
+    """Post a completion notice to the inbox of who started the run: an admin's manual run, or a
+    user's ad-hoc run that asks for one. Best effort — a notification failure must never mask the
+    job's own outcome."""
     try:
         create_notification(
             db,
@@ -272,7 +274,6 @@ def scheduler_snapshot(db) -> dict:
     open_runs = _open_run_ids()
     recent = jobruns.recent_runs(db, [*JOB_INTERVALS, *AD_HOC_JOBS], RECENT_RUNS)
     open_runs |= _open_run_ids()
-    going_id = max((r for r, j in open_runs.items() if j == CHANNEL_BACKFILL), default=None)
     jobs = []
     for job_id, default_interval in JOB_INTERVALS.items():
         a = acts.get(job_id, {})
@@ -288,30 +289,11 @@ def scheduler_snapshot(db) -> dict:
         )
     # Work a user starts on one target: no interval, no "run now" (it needs its target), but its
     # runs are shown beside the jobs' — it spends the same quota. Its one in-memory guard is the
-    # target's own (sync/runner), so "running" is the runs this process has open: two channels can
-    # backfill at once, so a run still going counts even when a newer one already ended. That run then
-    # leads the row, so its channel, its "no outcome yet" and its dot are one run's, never the
-    # newer run's outcome under the older one's channel.
-    backfills = recent.get(CHANNEL_BACKFILL, [])
-    going = db.get(JobRun, going_id) if going_id is not None else None
-    if going is not None and going.status != RunStatus.RUNNING:
-        going = None  # its outcome is written; only the bookkeeping has not caught up
-    if going is not None:
-        backfills = [going, *(r for r in backfills if r.id != going.id)][:RECENT_RUNS]
-    shown = backfills[0] if backfills else None
-    ad_hoc = [
-        {
-            "id": CHANNEL_BACKFILL,
-            "running": going is not None,
-            "target": shown.target if shown else None,
-            "target_title": (
-                catalog_queries.channel_title(db, shown.target)
-                if shown is not None and shown.target
-                else None
-            ),
-            **_last_runs(backfills, open_runs),
-        }
-    ]
+    # target's own (sync/runner, the import's per-user lock), so "running" is the runs this process
+    # has open: two targets can run at once, so a run still going counts even when a newer one already
+    # ended. That run then leads the row, so its target, its "no outcome yet" and its dot are one
+    # run's, never the newer run's outcome under the older one's target.
+    ad_hoc = [_ad_hoc_row(db, job, recent.get(job, []), open_runs) for job in AD_HOC_JOBS]
     return {
         "running_here": running_here,
         "enabled": settings.scheduler_enabled,
@@ -333,6 +315,27 @@ def _seen(run, open_runs: dict[int, str]) -> RunStatus:
     return RunStatus(run.status)
 
 
+def _ad_hoc_row(db, job: str, runs: list[JobRun], open_runs: dict[int, str]) -> dict:
+    going_id = max((r for r, j in open_runs.items() if j == job), default=None)
+    going = db.get(JobRun, going_id) if going_id is not None else None
+    if going is not None and going.status != RunStatus.RUNNING:
+        going = None  # its outcome is written; only the bookkeeping has not caught up
+    if going is not None:
+        runs = [going, *(r for r in runs if r.id != going.id)][:RECENT_RUNS]
+    shown = runs[0] if runs else None
+    return {
+        "id": job,
+        "running": going is not None,
+        "target": shown.target if shown else None,
+        "target_title": (
+            ad_hoc_target_title(db, job, shown.target)
+            if shown is not None and shown.target
+            else None
+        ),
+        **_last_runs(runs, open_runs),
+    }
+
+
 def _last_runs(runs: list, open_runs: dict[int, str]) -> dict:
     """A row's run fields from its newest runs (newest first): how the last one went, and the few
     before it for the history dots."""
@@ -402,7 +405,12 @@ def _autotag_job() -> None:
 
 
 def _shorts_job() -> None:
-    _job("shorts", run_shorts)
+    def work(db):
+        # A video waiting in someone's Takeout import becomes markable the moment the probe has
+        # said it is not a Short, so the import's waiting rows are settled right after it.
+        return run_shorts(db) | takeout.apply_waiting(db)
+
+    _job("shorts", work)
 
 
 def _subscriptions_job() -> None:
@@ -608,48 +616,140 @@ def trigger_all(actor_id: int | None = None) -> list[str]:
 # one channel. It runs on the same scheduler as a recorded run (trigger `ad_hoc`), so what it did and
 # spent is in `job_runs` like any job's, not a background task only the log ever heard of.
 CHANNEL_BACKFILL = "channel_backfill"
+# A user's Google Takeout import (`reader.takeout`): its target is the user.
+TAKEOUT_IMPORT = "takeout_import"
 # Every ad-hoc job id: the page names each one (a frontend contract test reads this tuple).
-AD_HOC_JOBS = (CHANNEL_BACKFILL,)
+AD_HOC_JOBS = (CHANNEL_BACKFILL, TAKEOUT_IMPORT)
+# What each one's target is called on the admin's pages: a channel's title, an importing user's name.
+_TARGET_TITLES: dict[str, Callable[..., str | None]] = {
+    CHANNEL_BACKFILL: catalog_queries.channel_title,
+    TAKEOUT_IMPORT: takeout.target_title,
+}
 
 
-def run_ad_hoc(job: str, fn: Callable[[], dict], *, actor_id: int, target: str) -> None:
+def ad_hoc_target_title(db, job: str, target: str) -> str | None:
+    """The name of an ad-hoc run's target (None for a job with none, or a target since gone)."""
+    title = _TARGET_TITLES.get(job)
+    return title(db, target) if title is not None else None
+
+
+# The ad-hoc runs added and not yet recorded, `job@target`: between the two the target's newest
+# run row is still the previous run's, so a page watching it asks `ad_hoc_queued` too.
+_queued: set[str] = set()
+_queued_guard = threading.Lock()
+# What the scheduler is told to run is only the key: the request itself waits here, the newest one
+# per key, until a fire of that key takes it. A request made while its key's run is under way is so
+# kept for the run after it, which that run asks for as it ends (`_running`): the scheduler would
+# drop a second fire of an id whose run has not returned (one instance per job id).
+_pending: dict[str, tuple[Callable[[], dict], int, bool]] = {}
+_running: set[str] = set()
+# Two at once, for the moment between a run letting go of its key and its thread returning: a fire
+# then is what takes the next request, and is not dropped.
+_AD_HOC_INSTANCES = 2
+
+
+def ad_hoc_queued(job: str, target: str) -> bool:
+    """Whether a run of `job` on `target` is queued and its run row not written yet."""
+    with _queued_guard:
+        return f"{job}@{target}" in _queued
+
+
+def run_ad_hoc(
+    job: str, fn: Callable[[], dict], *, actor_id: int, target: str, notify: bool = False
+) -> None:
     """Run `fn` now as `actor_id`'s run of `job` on `target`, a one-shot job on the scheduler. Its
     own exclusion (if any) lives in `fn`, which says so in its result; one id per job and target, so
-    a second request before the first has even started is that same run."""
+    a second request before the first has even started is that same run, and one made while it runs
+    is the run after it. `notify` posts its outcome to the user's inbox, for work whose page does not
+    watch it run."""
     if job not in AD_HOC_JOBS:
         raise KeyError(job)  # an unnamed job would show on the page as its bare id
-    sched = _scheduler
-    if sched is None:
+    if _scheduler is None:
         # Only between shutdown and exit: start_scheduler runs at every API startup.
         raise RuntimeError("the scheduler is not running")
+    key = f"{job}@{target}"
+    with _queued_guard:
+        _pending[key] = (fn, actor_id, notify)
+        _queued.add(key)
+        running = key in _running
+    if running:
+        log.info("ad-hoc %s for %s runs again once the run under way ends", job, target)
+        return
+    _fire(job, target)
+
+
+def _fire(job: str, target: str) -> None:
+    """Add the one-shot that takes `job@target`'s pending request. Raises (the request dropped) only
+    when the scheduler cannot take it."""
+    key = f"{job}@{target}"
+    sched = _scheduler
     try:
+        if sched is None:
+            raise RuntimeError("the scheduler is not running")
         sched.add_job(
-            _ad_hoc,
+            _ad_hoc_fired,
             "date",
-            args=(job, fn, actor_id, target),
-            id=f"{job}@{target}",
+            args=(job, target),
+            id=key,
             misfire_grace_time=None,
+            max_instances=_AD_HOC_INSTANCES,
         )
-    # silent-ok: not a failure — the run for this target is already queued, and it does this work.
+    # silent-ok: not a failure — a fire of this key is already queued, and it takes this request.
     except ConflictingIdError:
         log.info("ad-hoc %s for %s already queued", job, target)
+    except Exception:
+        with _queued_guard:
+            _pending.pop(key, None)
+            _queued.discard(key)
+        raise
 
 
-def _ad_hoc(job: str, fn: Callable[[], dict], actor_id: int, target: str) -> None:
+def _ad_hoc_fired(job: str, target: str) -> None:
+    key = f"{job}@{target}"
+    with _queued_guard:
+        request = _pending.pop(key, None)
+        if request is None:
+            return  # an earlier fire took it
+        _running.add(key)
+    fn, actor_id, notify = request
+    try:
+        _ad_hoc(job, fn, actor_id, target, notify)
+    finally:
+        with _queued_guard:
+            _running.discard(key)
+            again = key in _pending
+        if again:
+            try:
+                _fire(job, target)
+            except Exception:
+                # The request made meanwhile is lost with the scheduler: shut down, nothing runs.
+                log.exception("ad-hoc %s for %s not run again", job, target)
+
+
+def _ad_hoc(
+    job: str, fn: Callable[[], dict], actor_id: int, target: str, notify: bool = False
+) -> None:
+    key = f"{job}@{target}"
     try:
         run_id = jobruns.start(job, RunTrigger.AD_HOC, actor_id, target)
     except Exception:
         # No row to tell it in: the database is what failed. The log is the record.
         log.exception("ad-hoc %s for %s not started: its run could not be recorded", job, target)
         return
+    finally:
+        # Recorded (or never will be): the run row now speaks for it — unless another request came
+        # meanwhile, which is still queued.
+        with _queued_guard:
+            if key not in _pending:
+                _queued.discard(key)
     _opened(run_id, job)
     try:
-        _ad_hoc_work(job, fn, run_id)
+        _ad_hoc_work(job, fn, run_id, actor_id if notify else None)
     finally:
         _closed(run_id)
 
 
-def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
+def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int, notify_id: int | None) -> None:
     with logs.bind(run_id=run_id):
         # As in _run: only the work is inside the try.
         try:
@@ -658,8 +758,23 @@ def _ad_hoc_work(job: str, fn: Callable[[], dict], run_id: int) -> None:
             log.exception("job %s failed", job, extra=logs.event("scheduler.job.failed", job=job))
             err, failure = _failure(exc)
             _close(run_id, RunStatus.ERROR, error=err, failure=failure)
+            # The user's notice names the failure, never the upstream's own text: that is an English
+            # diagnostic, for the admin's run row and the log, not a user's inbox.
+            _notify_ad_hoc(
+                notify_id, job, RunStatus.ERROR, None, failure or {"failure": "unexpected"}
+            )
             return
-        _finish_ok(job, run_id, result)
+        summary, data = _finish_ok(job, run_id, result)
+        _notify_ad_hoc(notify_id, job, RunStatus.OK, summary, data)
+
+
+def _notify_ad_hoc(
+    user_id: int | None, job: str, status: RunStatus, summary: str | None, result: dict | None
+) -> None:
+    if user_id is None:
+        return
+    with SessionLocal() as db:
+        _notify_done(db, user_id, job, status, summary, result)
 
 
 # Room for every job at once (a "run all") beside a few ad-hoc runs: APScheduler's default pool of
@@ -700,3 +815,7 @@ def shutdown_scheduler() -> None:
     if _scheduler is not None:
         _scheduler.shutdown(wait=False)
         _scheduler = None
+    with _queued_guard:
+        # What it held will not run.
+        _queued.clear()
+        _pending.clear()

backend/app/sync/explore.py

@@ -14,6 +14,7 @@ from sqlalchemy import delete, or_, select
 from sqlalchemy.orm import Session
 
 from app import logs, sysconfig
+from app.catalog import service as catalog_service
 from app.catalog.models import Channel, ExploredChannel, SearchFind, Subscription, Video
 from app.reader import queries as reader_queries
 from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_details
@@ -66,11 +67,13 @@ def _enrich_new(db: Session, yt: YouTubeClient, video_ids: list[str]) -> int:
 def explore_ingest_page(
     db: Session, yt: YouTubeClient, channel: Channel, page_token: str | None = None
 ) -> dict:
-    """Ingest one page (up to 50, newest-first) of the channel's uploads, marking the new
-    videos via_explore, then enrich them. Returns the count ingested and the next page token
-    (None when the back-catalogue is exhausted) so the caller can offer "load more". Stateless
-    pagination: the caller round-trips the token, so the channel's own backfill cursor (used by
-    the organic deep backfill) is never touched."""
+    """Ingest one page (up to 50, newest-first) of the channel's uploads, marking the new videos
+    via_explore while the channel is an exploration's (`from_explore`): of a channel someone follows
+    they are its uploads like any other, and a flag there would never be cleared, since subscribing
+    clears only an explored channel's. Then enrich them. Returns the count ingested and the next
+    page token (None when the back-catalogue is exhausted) so the caller can offer "load more".
+    Stateless pagination: the caller round-trips the token, so the channel's own backfill cursor
+    (used by the organic deep backfill) is never touched."""
     if not channel.uploads_playlist_id:
         return {"ingested": 0, "next_page_token": None}
     try:
@@ -86,9 +89,10 @@ def explore_ingest_page(
     for item in data.get("items", []):
         stub = stub_from_playlist_item(item, channel.id)
         if stub is not None:
-            stub["via_explore"] = True
+            stub["via_explore"] = channel.from_explore
             stubs.append(stub)
     inserted = insert_stubs(db, stubs)
+    catalog_service.settle_explored(db, channel)
     self_enriched = _enrich_new(db, yt, [s["id"] for s in stubs])
     log.info(
         "explore ingest channel=%s ingested=%s enriched=%s", channel.id, inserted, self_enriched
@@ -105,7 +109,9 @@ def purge_unkept_explored(db: Session) -> dict:
       - a live ExploredChannel row (someone explored it within the grace window — channel-level), or
       - a video has per-user state: watched/in-progress (VideoState), in a playlist (PlaylistItem),
         or surfaced by the user's own search (SearchFind) — these survive even if the channel goes.
-    A channel row is dropped only once it holds no videos at all. 0 quota."""
+    A channel row is dropped only once it holds no videos at all, and not while someone's Takeout
+    history shows it watched (their "watched but not followed" list names it from that row).
+    0 quota."""
     grace_days = sysconfig.get_int(db, "explore_grace_days")
     cutoff = datetime.now(timezone.utc) - timedelta(days=grace_days)
 
@@ -135,7 +141,9 @@ def purge_unkept_explored(db: Session) -> dict:
 
     has_videos = select(Video.id).where(Video.channel_id == Channel.id).exists()
     channels_deleted = db.execute(
-        delete(Channel).where(Channel.id.in_(orphan_ids), ~has_videos)
+        delete(Channel).where(
+            Channel.id.in_(orphan_ids), ~has_videos, ~reader_queries.in_a_watch_history(Channel.id)
+        )
     ).rowcount
     db.commit()
     return {

backend/app/sync/subscriptions.py

@@ -8,6 +8,7 @@ from sqlalchemy.orm import Session
 
 from app import logs
 from app.models import User
+from app.catalog import service as catalog_service
 from app.catalog.models import Channel, Subscription
 from app.utils import normalize_description
 from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail
@@ -109,6 +110,21 @@ def apply_channel_details(db: Session, items: list[dict], asked: Iterable[str])
         channel.details_synced_at = now
 
 
+def describe_channels(db: Session, yt: YouTubeClient, channel_ids: list[str]) -> None:
+    """Make sure each given channel is in the catalog with its details: a channel it lacks is added
+    as a stub, then all of them are asked for (`channels.list`, 1 unit a 50). Flushes; the caller
+    commits."""
+    if not channel_ids:
+        return
+    have = set(db.execute(select(Channel.id).where(Channel.id.in_(channel_ids))).scalars())
+    for cid in dict.fromkeys(channel_ids):
+        if cid not in have:
+            db.add(Channel(id=cid))
+    db.flush()
+    apply_channel_details(db, yt.get_channels(channel_ids), channel_ids)
+    db.flush()
+
+
 def should_prune(fetched_count: int, existing_count: int) -> bool:
     """Whether an import may delete the local subscriptions missing from this fetch.
 
@@ -182,6 +198,10 @@ def import_subscriptions(db: Session, user: User) -> dict:
                 db.add(sub_row)
             sub_row.yt_subscription_id = sub.get("yt_subscription_id")
         db.flush()
+        # A follow made on youtube.com keeps an explored channel as a follow made here does.
+        explored = select(Channel).where(Channel.id.in_(fetched), Channel.from_explore.is_(True))
+        for channel in db.execute(explored).scalars():
+            catalog_service.settle_explored(db, channel)
 
         # Prune subscriptions removed on YouTube — but only when this fetch is trustworthy
         # (see `should_prune`).

backend/app/sync/videos.py

@@ -420,11 +420,15 @@ def run_shorts_classification(db: Session, limit: int | None = None) -> dict:
     return {"probed": probed, "shorts": shorts}
 
 
-def ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None:
+def ensure_videos(
+    db: Session, yt: YouTubeClient, video_ids: list[str], *, explore: bool = False
+) -> None:
     """Make sure every given video id exists in the catalog, fetching + ingesting the
     missing ones (and a stub channel for any unknown channel). Deleted/private videos that
     the API won't return are simply left out. The catalog's write for a playlist sync that names
-    videos it has never seen; commits."""
+    videos it has never seen; commits. `explore` adds the new ones as an exploration's
+    (`via_explore`): visible to who explores or follows their channel, and reclaimed by the cleanup
+    once nobody keeps them."""
     if not video_ids:
         return
     have = set(db.execute(select(Video.id).where(Video.id.in_(video_ids))).scalars().all())
@@ -458,7 +462,12 @@ def ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None:
         seen.add(vid)
         # published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
         # fallback here too so the constructor never carries an explicit NULL.
-        v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
+        v = Video(
+            id=vid,
+            channel_id=cid,
+            published_at=parse_dt(sn.get("publishedAt")) or now,
+            via_explore=explore,
+        )
         apply_video_details(v, it)
         v.enriched_at = now
         db.add(v)

backend/app/sysconfig.py

@@ -140,6 +140,15 @@ SPECS: tuple[ConfigSpec, ...] = (
         max=100_000,
         applies=LIVE,
     ),
+    ConfigSpec(
+        "related_daily_limit_per_user",
+        "int",
+        "quota",
+        "related_daily_limit_per_user",
+        min=0,
+        max=100_000,
+        applies=LIVE,
+    ),
     # Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
     ConfigSpec("search_source", "choice", "quota", "search_source", applies=LIVE),
     ConfigSpec(
@@ -434,15 +443,20 @@ def _default(s: ConfigSpec):
     return getattr(settings, s.default_attr)
 
 
+# Every reason `set_value` refuses with. `refusal.setting_refused` looks each one up in its map, which
+# the refusal scan holds to exactly these keys: a reason the map lacks would answer a 500.
+SettingReason = Literal["not_a_number", "too_low", "too_high", "not_a_choice", "no_encryption_key"]
+
+
 class InvalidSetting(ValueError):
     """A value `set_value` refuses, as a reason the UI translates (`errors.refused.setting_<reason>`)
     plus what the sentence needs — the English message is for logs only. A ValueError still, so a
     caller that only needs "refused or not" keeps working."""
 
-    def __init__(self, key: str, reason: str, **params: object):
+    def __init__(self, key: str, reason: SettingReason, **params: object):
         super().__init__(f"{key}: {reason} {params}")
         self.key = key
-        self.reason = reason
+        self.reason: SettingReason = reason
         self.params = params
 
 

backend/app/youtube/client.py

@@ -30,12 +30,26 @@ class YouTubeError(Exception):
     message is prose, it changes under us, and it is deliberately the poorer of the two here.
 
     `path` is the API endpoint that ANSWERED with the failure (`playlistItems`); empty when no
-    endpoint did — the request never got through (`transport`) or the token step refused first."""
+    endpoint did — the request never got through (`transport`) or the token step refused first.
 
-    def __init__(self, message: str, *, reason: str = "", path: str = ""):
+    `sent` is a `transport` failure that came after the request may have gone out (a read timeout,
+    a dropped answer): nothing answered, but YouTube may have counted the call."""
+
+    def __init__(self, message: str, *, reason: str = "", path: str = "", sent: bool = False):
         super().__init__(message)
         self.reason = reason
         self.path = path
+        self.sent = sent
+
+
+# The transport failures that stop a request before it reaches YouTube: nothing went out.
+_UNSENT = (
+    httpx.ConnectError,
+    httpx.ConnectTimeout,
+    httpx.PoolTimeout,
+    httpx.ProxyError,
+    httpx.UnsupportedProtocol,
+)
 
 
 def _error_reason(resp: httpx.Response) -> str:
@@ -162,6 +176,9 @@ def best_thumbnail(thumbnails: dict | None) -> str | None:
 
 
 class YouTubeClient:
+    # The quota units this client has recorded: what its calls have spent of the shared budget.
+    spent: int = 0
+
     def __init__(self, db, user: User):
         self.db = db
         self.user = user
@@ -189,9 +206,16 @@ class YouTubeClient:
             return self._http.request(method, url, **kwargs)
         except httpx.HTTPError as exc:
             raise YouTubeError(
-                f"YouTube API unreachable ({exc.__class__.__name__}: {exc})", reason="transport"
+                f"YouTube API unreachable ({exc.__class__.__name__}: {exc})",
+                reason="transport",
+                sent=not isinstance(exc, _UNSENT),
             ) from exc
 
+    def _spend(self, units: int) -> None:
+        """Record what a call YouTube answered cost — a refused one too: it is charged all the same."""
+        quota.record_usage(units)
+        self.spent += units
+
     # --- auth ---
     def _access_token(self) -> str:
         tok = self.user.token
@@ -269,7 +293,7 @@ class YouTubeClient:
         else:
             headers["Authorization"] = f"Bearer {self._access_token()}"
         resp = self._send("GET", f"{API_BASE}/{path}", params=p, headers=headers)
-        quota.record_usage(cost)
+        self._spend(cost)
         if resp.status_code != 200:
             raise _upstream_failure("GET", path, resp, expected)
         return resp.json()
@@ -401,7 +425,7 @@ class YouTubeClient:
             params={"id": subscription_id},
             headers={"Authorization": f"Bearer {self._access_token()}"},
         )
-        quota.record_usage(50)
+        self._spend(50)
         if resp.status_code not in (200, 204):
             raise _upstream_failure("DELETE", "subscriptions", resp)
 
@@ -431,7 +455,7 @@ class YouTubeClient:
     def _write(self, method: str, path: str, *, params: dict, json: dict | None = None) -> dict:
         headers = {"Authorization": f"Bearer {self._access_token()}"}
         resp = self._send(method, f"{API_BASE}/{path}", params=params, json=json, headers=headers)
-        quota.record_usage(50)
+        self._spend(50)
         if resp.status_code not in (200, 204):
             raise _upstream_failure(method, path, resp)
         return resp.json() if resp.content else {}

backend/app/youtube/innertube.py

@@ -0,0 +1,160 @@
+"""The website's own InnerTube API (`youtubei/v1/*`), read the way the website reads it — no quota.
+
+The zero-quota search (`search_scrape`) and the related videos (`related_scrape`) both POST here with
+the public WEB key and a client version, read out of a live results page. This deliberately calls an
+undocumented endpoint: if YouTube changes the JSON shape or blocks the host, each caller says so as a
+`ScrapeError`, never as an empty answer.
+
+The SOCS=CAI consent-skip cookie is sent like the Shorts probe's, so a server-side request isn't
+bounced to consent.youtube.com."""
+
+import re
+
+import httpx
+
+from app import logs
+
+log = logs.get_logger(__name__)
+
+_API_URL = "https://www.youtube.com/youtubei/v1/"
+_RESULTS_URL = "https://www.youtube.com/results"
+# Public WEB InnerTube key + a recent client version. These are also embedded in every results
+# page; we fetch the live pair once and cache it, falling back to these constants.
+_FALLBACK_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"  # gitleaks:allow -- the public WEB key, in every results page
+_FALLBACK_CLIENT_VERSION = "2.20240101.00.00"
+
+_HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
+    # SOCS skips YouTube's cookie-consent interstitial (otherwise every server-side request is
+    # redirected to consent.youtube.com).
+    "Cookie": "SOCS=CAI",
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
+_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
+
+# Cached (api_key, client_version) extracted from a results page; refreshed on demand.
+_config: tuple[str, str] | None = None
+
+
+class ScrapeError(Exception):
+    """An InnerTube read could not be completed (network, blocked, or unexpected shape)."""
+
+
+def runs_text(node: dict | None) -> str | None:
+    """The text of an InnerTube text node: a `simpleText`, or its `runs` joined."""
+    if not node:
+        return None
+    if node.get("simpleText"):
+        return node["simpleText"]
+    runs = node.get("runs") or []
+    if runs:
+        return "".join(r.get("text", "") for r in runs)
+    return None
+
+
+def scraped_video(
+    vid: str, channel_id: object, channel_title: object, title: object, thumbnails: object
+) -> dict:
+    """One scraped video in the shape `YouTubeClient.search_videos` answers, so a page of either
+    source is ingested the same way. `published_at` is None (the website shows only a relative
+    "3 years ago"; the enrich fills the real date) and `live_broadcast` "none" (live/upcoming are
+    judged after the enrich, by `live_status`). The largest thumbnail is the last."""
+    thumbs = thumbnails if isinstance(thumbnails, list) else []
+    last = thumbs[-1] if thumbs else None
+    return {
+        "id": vid,
+        "channel_id": channel_id if isinstance(channel_id, str) and channel_id else None,
+        "channel_title": channel_title if isinstance(channel_title, str) else None,
+        "title": title if isinstance(title, str) else None,
+        "published_at": None,
+        "thumbnail_url": last.get("url") if isinstance(last, dict) else None,
+        "live_broadcast": "none",
+    }
+
+
+def video_renderer(node: dict) -> dict | None:
+    """A `videoRenderer` (search) or `compactVideoRenderer` (the older related column) flattened —
+    the two draw a video the same way. None without a video id; the channel is None when the byline
+    names none."""
+    vid = node.get("videoId")
+    if not isinstance(vid, str) or not vid:
+        return None
+    owner = node.get("ownerText") or node.get("longBylineText") or {}
+    runs = owner.get("runs") if isinstance(owner, dict) else None
+    first = runs[0] if isinstance(runs, list) and runs and isinstance(runs[0], dict) else {}
+    endpoint = (first.get("navigationEndpoint") or {}).get("browseEndpoint") or {}
+    return scraped_video(
+        vid,
+        endpoint.get("browseId"),
+        first.get("text"),
+        runs_text(node.get("title")),
+        (node.get("thumbnail") or {}).get("thumbnails"),
+    )
+
+
+def continuation_token(node: object) -> str | None:
+    """The "load more" cursor of a `continuationItemRenderer`, or None when `node` is not one."""
+    if not isinstance(node, dict):
+        return None
+    command = (node.get("continuationEndpoint") or {}).get("continuationCommand") or {}
+    token = command.get("token")
+    return token if isinstance(token, str) and token else None
+
+
+def _get_config(client: httpx.Client) -> tuple[str, str]:
+    """The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
+
+    The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
+    keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
+    the life of the process — and when those went stale, search returned nothing, with no error
+    anywhere. Now the next call simply tries the live pair again."""
+    global _config
+    if _config is not None:
+        return _config
+    try:
+        html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
+    except httpx.HTTPError as exc:
+        log.warning(
+            "the InnerTube config could not be fetched; falling back to the pinned key",
+            extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
+        )
+        return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
+    key = _KEY_RE.search(html)
+    ver = _VER_RE.search(html)
+    if key and ver:
+        _config = (key.group(1), ver.group(1))
+        return _config
+    log.warning(
+        "the results page carried no InnerTube config; falling back to the pinned key",
+        extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
+    )
+    return (
+        key.group(1) if key else _FALLBACK_KEY,
+        ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
+    )
+
+
+def post(endpoint: str, body: dict, *, hl: str = "en", gl: str = "US") -> dict:
+    """POST `body` to `youtubei/v1/<endpoint>` as the WEB client in the `hl`/`gl` locale and return
+    the decoded JSON object. A network failure, an HTTP error or a body that is not a JSON object (a
+    block, consent or captcha page) raises `ScrapeError`; the caller judges the shape."""
+    with httpx.Client(timeout=20.0, headers=_HEADERS) as client:
+        api_key, client_version = _get_config(client)
+        context = {
+            "client": {"clientName": "WEB", "clientVersion": client_version, "hl": hl, "gl": gl}
+        }
+        try:
+            resp = client.post(
+                _API_URL + endpoint, params={"key": api_key}, json={"context": context, **body}
+            )
+            resp.raise_for_status()
+            data = resp.json()
+        except httpx.HTTPError as exc:
+            raise ScrapeError(f"InnerTube request failed: {exc}") from exc
+        except ValueError as exc:  # non-JSON body (blocked / consent / captcha)
+            raise ScrapeError("InnerTube returned a non-JSON response") from exc
+    if not isinstance(data, dict):
+        raise ScrapeError("InnerTube returned an unexpected shape")
+    return data

backend/app/youtube/related_scrape.py

@@ -0,0 +1,141 @@
+"""A video's related videos, read from the watch page's InnerTube `next` endpoint — no quota.
+
+The Data API has no related-videos call any more, so this is the only source: the same list the
+website shows beside a playing video. A page holds about 20 videos and a continuation token for the
+next 20. The returned page has the search scrape's shape (``{"items": [...], "next_page_token"}``)
+so the catalog ingests it the same way (`catalog.search.ingest_candidates`).
+
+YouTube draws each item as a `lockupViewModel` now; the older `compactVideoRenderer` is read too, so
+a rollback on its side does not empty the list. An item that names no single channel — a video by
+several channels draws an avatar stack, not one avatar — is left out: the catalog row needs its
+channel up front, and the stack does not say which one uploaded it.
+
+`hl`/`gl` pick the locale YouTube fills the list in: ``en``/``US`` keeps it on the video's topic,
+``hu``/``HU`` mixes in what is trending in Hungary (measured)."""
+
+from app.youtube.innertube import (
+    ScrapeError,
+    continuation_token,
+    post,
+    scraped_video,
+    video_renderer,
+)
+
+__all__ = ["ScrapeError", "related"]
+
+# Where a first page and a continuation page put their item list.
+_FIRST = (
+    "contents",
+    "twoColumnWatchNextResults",
+    "secondaryResults",
+    "secondaryResults",
+    "results",
+)
+_APPENDS = ("appendContinuationItemsAction", "reloadContinuationItemsCommand")
+
+
+def _at(node: object, *path: str | int) -> object:
+    """The value under `path` in nested dicts/lists, or None where the path stops."""
+    for key in path:
+        if isinstance(node, dict) and isinstance(key, str):
+            node = node.get(key)
+        elif isinstance(node, list) and isinstance(key, int) and -len(node) <= key < len(node):
+            node = node[key]
+        else:
+            return None
+    return node
+
+
+def _lockup(node: dict) -> dict | None:
+    if node.get("contentType") != "LOCKUP_CONTENT_TYPE_VIDEO" or not node.get("contentId"):
+        return None
+    meta = _at(node, "metadata", "lockupMetadataViewModel")
+    channel_id = _at(
+        meta,
+        "image",
+        "decoratedAvatarViewModel",
+        "rendererContext",
+        "commandContext",
+        "onTap",
+        "innertubeCommand",
+        "browseEndpoint",
+        "browseId",
+    )
+    channel_title = _at(
+        meta, "metadata", "contentMetadataViewModel", "metadataRows", 0, "metadataParts", 0
+    )
+    return scraped_video(
+        node["contentId"],
+        channel_id,
+        _at(channel_title, "text", "content"),
+        _at(meta, "title", "content"),
+        _at(node, "contentImage", "thumbnailViewModel", "image", "sources"),
+    )
+
+
+def _unavailable(data: dict) -> bool:
+    """Whether the watch page says the video is not available: its main column draws a
+    `backgroundPromoRenderer` ("This video isn't available anymore") and there is no related column.
+    YouTube's own word for "nothing related" — the one empty page that is not a shape we lost."""
+    sections = _at(data, "contents", "twoColumnWatchNextResults", "results", "results", "contents")
+    for section in sections if isinstance(sections, list) else []:
+        inner = _at(section, "itemSectionRenderer", "contents")
+        for node in inner if isinstance(inner, list) else []:
+            if isinstance(node, dict) and "backgroundPromoRenderer" in node:
+                return True
+    return False
+
+
+def _items(data: dict) -> list | None:
+    """The page's item list (first page or continuation), or None when it has neither frame."""
+    first = _at(data, *_FIRST)
+    if isinstance(first, list):
+        return first
+    if _unavailable(data):
+        return []
+    for endpoint in data.get("onResponseReceivedEndpoints") or []:
+        for key in _APPENDS:
+            items = _at(endpoint, key, "continuationItems")
+            if isinstance(items, list):
+                return items
+    return None
+
+
+def related(
+    video_id: str, continuation: str | None = None, *, hl: str = "en", gl: str = "US"
+) -> dict:
+    """One page of `video_id`'s related videos (`continuation` = the token a previous page gave).
+
+    A video YouTube says it no longer has is an empty page. An answer of a shape this does not know
+    — no related column, or one whose entries are all renderers it cannot read — is a `ScrapeError`:
+    not an empty page, which would read as "nothing related"."""
+    body = {"continuation": continuation} if continuation else {"videoId": video_id}
+    data = post("next", body, hl=hl, gl=gl)
+    items = _items(data)
+    if items is None:
+        raise ScrapeError("InnerTube returned no related list")
+    out: list[dict] = []
+    seen: set[str] = set()
+    token = None
+    known = 0  # entries drawn by a renderer read here (a video's, or the cursor)
+    for entry in items:
+        if not isinstance(entry, dict):
+            continue
+        if "lockupViewModel" in entry:
+            item = _lockup(entry["lockupViewModel"])
+        elif "compactVideoRenderer" in entry:
+            item = video_renderer(entry["compactVideoRenderer"])
+        else:
+            found = continuation_token(entry.get("continuationItemRenderer"))
+            known += found is not None
+            token = token or found
+            continue
+        known += 1
+        # A video by several channels draws an avatar stack, not one: no channel, left out.
+        if item and item["channel_id"] and item["id"] not in seen:
+            seen.add(item["id"])
+            out.append(item)
+    # A list with entries but none of them one we read: YouTube draws its items some new way.
+    if items and not known:
+        raise ScrapeError("InnerTube's related list holds no renderer this reads")
+    return {"items": out, "next_page_token": token}

backend/app/youtube/search_scrape.py

@@ -6,9 +6,9 @@ calls — and read the result list straight out of the JSON. The returned video
 through the normal enrich path (videos.list, 1 unit / 50), exactly as the official search does,
 so the search itself is effectively free and only the (cheap, shared) enrichment is charged.
 
-The SOCS=CAI consent-skip cookie is used like the Shorts probe so a server-side request isn't
-bounced to consent.youtube.com. This deliberately scrapes an undocumented endpoint: if YouTube
-changes the JSON shape or blocks the host, an admin can flip ``search_source`` back to ``api``.
+The request itself (the live key, the consent cookie) is `innertube`'s. This deliberately scrapes an
+undocumented endpoint: if YouTube changes the JSON shape or blocks the host, an admin can flip
+``search_source`` back to ``api``.
 
 The returned page mirrors :meth:`YouTubeClient.search_videos` exactly
 (``{"items": [...], "next_page_token": str | None}``) so the route is source-agnostic — the
@@ -17,110 +17,16 @@ only difference is the cursor is an InnerTube *continuation token*, not a Data A
 the real date is filled by the videos.list enrich. ``live_broadcast`` is reported as ``"none"``
 and live/upcoming filtering happens post-enrich via ``live_status`` (same as the API path)."""
 
-import re
+from app.youtube.innertube import ScrapeError, continuation_token, post, video_renderer
 
-import httpx
+__all__ = ["ScrapeError", "search"]
 
-from app import logs
-
-log = logs.get_logger(__name__)
-
-_SEARCH_URL = "https://www.youtube.com/youtubei/v1/search"
-_RESULTS_URL = "https://www.youtube.com/results"
-# Public WEB InnerTube key + a recent client version. These are also embedded in every results
-# page; we fetch the live pair once and cache it, falling back to these constants.
-_FALLBACK_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
-_FALLBACK_CLIENT_VERSION = "2.20240101.00.00"
 # sp / params value selecting the "type: video" search filter (base64 protobuf EgIQAQ==).
 _TYPE_VIDEO = "EgIQAQ=="
 
-_HEADERS = {
-    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
-    # SOCS skips YouTube's cookie-consent interstitial (otherwise every server-side request is
-    # redirected to consent.youtube.com).
-    "Cookie": "SOCS=CAI",
-    "Accept-Language": "en-US,en;q=0.9",
-}
-
 # The top-level keys a first page (contents) and a continuation page (onResponseReceivedCommands)
 # put their result list under.
 _RESULT_FRAMES = {"contents", "onResponseReceivedCommands"}
-_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
-_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
-
-# Cached (api_key, client_version) extracted from a results page; refreshed on demand.
-_config: tuple[str, str] | None = None
-
-
-class ScrapeError(Exception):
-    """A scrape-based search could not be completed (network, blocked, or unexpected shape)."""
-
-
-def _runs_text(node: dict | None) -> str | None:
-    if not node:
-        return None
-    if node.get("simpleText"):
-        return node["simpleText"]
-    runs = node.get("runs") or []
-    if runs:
-        return "".join(r.get("text", "") for r in runs)
-    return None
-
-
-def _get_config(client: httpx.Client) -> tuple[str, str]:
-    """The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
-
-    The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
-    keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
-    the life of the process — and when those went stale, search returned nothing, with no error
-    anywhere. Now the next search simply tries the live pair again."""
-    global _config
-    if _config is not None:
-        return _config
-    try:
-        html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
-    except httpx.HTTPError as exc:
-        log.warning(
-            "the InnerTube config could not be fetched; falling back to the pinned key",
-            extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
-        )
-        return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
-    key = _KEY_RE.search(html)
-    ver = _VER_RE.search(html)
-    if key and ver:
-        _config = (key.group(1), ver.group(1))
-        return _config
-    log.warning(
-        "the results page carried no InnerTube config; falling back to the pinned key",
-        extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
-    )
-    return (
-        key.group(1) if key else _FALLBACK_KEY,
-        ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
-    )
-
-
-def _video_item(vr: dict) -> dict | None:
-    """Flatten a YouTube `videoRenderer` into the same shape as YouTubeClient.search_videos."""
-    vid = vr.get("videoId")
-    if not vid:
-        return None
-    owner = vr.get("ownerText") or vr.get("longBylineText") or {}
-    runs = owner.get("runs") or []
-    channel_id = channel_title = None
-    if runs:
-        channel_title = runs[0].get("text")
-        channel_id = runs[0].get("navigationEndpoint", {}).get("browseEndpoint", {}).get("browseId")
-    thumbs = (vr.get("thumbnail") or {}).get("thumbnails") or []
-    return {
-        "id": vid,
-        "channel_id": channel_id,
-        "channel_title": channel_title,
-        "title": _runs_text(vr.get("title")),
-        "published_at": None,  # scrape exposes only a relative time; enrich fills the real date
-        "thumbnail_url": thumbs[-1]["url"] if thumbs else None,
-        "live_broadcast": "none",  # live/upcoming filtered post-enrich via live_status
-    }
 
 
 def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
@@ -130,7 +36,7 @@ def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
     if isinstance(obj, dict):
         vr = obj.get("videoRenderer")
         if isinstance(vr, dict):
-            item = _video_item(vr)
+            item = video_renderer(vr)
             if item and item["id"] not in seen:
                 seen.add(item["id"])
                 out.append(item)
@@ -144,11 +50,9 @@ def _collect_videos(obj, out: list[dict], seen: set[str]) -> None:
 def _find_continuation(obj) -> str | None:
     """First continuationItemRenderer token (the 'load more' cursor), or None at the end."""
     if isinstance(obj, dict):
-        cir = obj.get("continuationItemRenderer")
-        if isinstance(cir, dict):
-            tok = cir.get("continuationEndpoint", {}).get("continuationCommand", {}).get("token")
-            if tok:
-                return tok
+        tok = continuation_token(obj.get("continuationItemRenderer"))
+        if tok:
+            return tok
         for v in obj.values():
             tok = _find_continuation(v)
             if tok:
@@ -167,34 +71,14 @@ def search(q: str, continuation: str | None = None) -> dict:
     `continuation` is the InnerTube continuation token from a previous page (None for the first
     page). Returns ``{"items": [...], "next_page_token": str | None}`` — the same shape as
     YouTubeClient.search_videos so the route handles both sources identically."""
-    with httpx.Client(timeout=20.0, headers=_HEADERS) as client:
-        api_key, client_version = _get_config(client)
-        context = {
-            "client": {
-                "clientName": "WEB",
-                "clientVersion": client_version,
-                "hl": "en",
-                "gl": "US",
-            }
-        }
-        body: dict = {"context": context}
-        if continuation:
-            body["continuation"] = continuation
-        else:
-            body["query"] = q
-            body["params"] = _TYPE_VIDEO
-        try:
-            resp = client.post(_SEARCH_URL, params={"key": api_key}, json=body)
-            resp.raise_for_status()
-            data = resp.json()
-        except httpx.HTTPError as exc:
-            raise ScrapeError(f"InnerTube request failed: {exc}") from exc
-        except ValueError as exc:  # non-JSON body (blocked / consent / captcha)
-            raise ScrapeError("InnerTube returned a non-JSON response") from exc
+    body: dict = (
+        {"continuation": continuation} if continuation else {"query": q, "params": _TYPE_VIDEO}
+    )
+    data = post("search", body)
 
     # Even a search with no results carries its result list's frame. Without one this is not "no
     # results" but a shape we do not know (YouTube moved, or a block page in JSON) — say so.
-    if not isinstance(data, dict) or not _RESULT_FRAMES & data.keys():
+    if not _RESULT_FRAMES & data.keys():
         raise ScrapeError("InnerTube returned an unexpected shape")
     items: list[dict] = []
     _collect_videos(data, items, set())

backend/requirements.lock

@@ -5,9 +5,11 @@
 # scans it, blocking. A security fix, or a yt-dlp release YouTube's changes force, may be ONE pin
 # bumped in place instead of a full regeneration, which would move every other package too — then
 # `pip install -r requirements.lock && pip check` must pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177); yt-dlp 2026.7.4 -> 2026.8.19
-# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403).
+# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403);
+# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2); brotli 1.2.0 -> 1.2.1 (PYSEC-2026-4322);
+# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616); joserfc 1.7.4 -> 1.7.6 (CVE-2026-119283, CVE-2026-119284).
 APScheduler==3.11.3
-Authlib==1.7.2
+Authlib==1.9.1
 Mako==1.4.1
 MarkupSafe==3.0.3
 PyYAML==6.0.3
@@ -20,8 +22,8 @@ anyio==4.14.2
 argon2-cffi-bindings==25.1.0
 argon2-cffi==25.1.0
 bgutil-ytdlp-pot-provider==1.3.1
-brotli==1.2.0
-certifi==2026.7.22
+brotli==1.2.1
+certifi==2026.10.19
 cffi==2.1.1
 charset-normalizer==3.5.1
 click==8.4.2
@@ -37,7 +39,7 @@ httptools==0.8.0
 httpx==0.28.1
 idna==3.18
 itsdangerous==2.2.0
-joserfc==1.7.4
+joserfc==1.7.6
 markdown-it-py==4.2.0
 mdurl==0.1.2
 mutagen==1.48.1

backend/tests/conftest.py

@@ -231,7 +231,7 @@ def _configured(monkeypatch):
 @pytest.fixture
 def client(db, override_dep, _configured):
     """A TestClient wired to the test DB that goes through the REAL auth/session stack — no
-    `current_user` override — so login, the wallet gate and epoch revoke are exercised end to end.
+    `current_user` override — so login, the one-account session and epoch revoke are exercised end to end.
     Starlette's SessionMiddleware round-trips the signed cookie, so a login in one call authenticates
     the next on this same instance. No lifespan (the scheduler / HLS sweep stay off)."""
     override_dep(get_db, db.info["request"])
@@ -327,3 +327,15 @@ def _reset_log_erasures():
     from app import logs
 
     logs._erased.clear()
+
+
+@pytest.fixture(autouse=True)
+def _reset_ad_hoc_queue():
+    """The scheduler's ad-hoc queue is module-level too: a request a test left pending would read as
+    a later test's user's import still queued (ids restart from 1)."""
+    from app import scheduler
+
+    with scheduler._queued_guard:
+        scheduler._queued.clear()
+        scheduler._pending.clear()
+        scheduler._running.clear()

backend/tests/fixtures/innertube_next_continuation.json

@@ -0,0 +1,527 @@
+{
+ "onResponseReceivedEndpoints": [
+  {
+   "appendContinuationItemsAction": {
+    "continuationItems": [
+     {
+      "lockupViewModel": {
+       "contentImage": {
+        "thumbnailViewModel": {
+         "image": {
+          "sources": [
+           {
+            "url": "https://i.ytimg.com/vi/_HfdncCbMOE/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLARvB39pkdsiuUUQTsZjkhvbKrfcQ",
+            "width": 168,
+            "height": 94
+           },
+           {
+            "url": "https://i.ytimg.com/vi/_HfdncCbMOE/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLB15aZU5-tzwYCEbNTnZK3UX6ZCzQ",
+            "width": 336,
+            "height": 188
+           }
+          ]
+         },
+         "overlays": [
+          {
+           "thumbnailBottomOverlayViewModel": {
+            "badges": [
+             {
+              "thumbnailBadgeViewModel": {
+               "text": "1:08:18",
+               "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
+               "animationActivationTargetId": "_HfdncCbMOE",
+               "animatedText": "Now playing",
+               "animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
+              }
+             }
+            ]
+           }
+          }
+         ]
+        }
+       },
+       "metadata": {
+        "lockupMetadataViewModel": {
+         "title": {
+          "content": "MIT 6.S191 (2025): Large Language Models (Liquid AI)"
+         },
+         "image": {
+          "decoratedAvatarViewModel": {
+           "avatar": {
+            "avatarViewModel": {
+             "image": {
+              "sources": [
+               {
+                "url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
+                "width": 68,
+                "height": 68
+               }
+              ]
+             },
+             "avatarImageSize": "AVATAR_SIZE_M"
+            }
+           },
+           "a11yLabel": "Go to channel Alexander Amini",
+           "rendererContext": {
+            "commandContext": {
+             "onTap": {
+              "innertubeCommand": {
+               "browseEndpoint": {
+                "browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
+               }
+              }
+             }
+            }
+           }
+          }
+         },
+         "metadata": {
+          "contentMetadataViewModel": {
+           "metadataRows": [
+            {
+             "metadataParts": [
+              {
+               "text": {
+                "content": "Alexander Amini"
+               }
+              }
+             ]
+            },
+            {
+             "metadataParts": [
+              {
+               "text": {
+                "content": "62K"
+               },
+               "accessibilityLabel": "62 thousand views",
+               "leadingIcon": {
+                "name": "PLAY_ARROW_OUTLINED",
+                "height": 12,
+                "width": 12,
+                "marginEndToken": {
+                 "spacing": 2
+                }
+               }
+              },
+              {
+               "text": {
+                "content": "1y ago"
+               },
+               "accessibilityLabel": "1 year ago"
+              }
+             ]
+            }
+           ],
+           "delimiter": " "
+          }
+         }
+        }
+       },
+       "contentId": "_HfdncCbMOE",
+       "contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
+       "themedPalette": {
+        "lightThemePalette": {
+         "baseBackground": 4294964720,
+         "raisedBackground": 4294964720,
+         "additiveBackground": 566850118,
+         "textPrimary": 4281800969,
+         "textSecondary": 4285876787,
+         "outline": 860759827,
+         "keyLight": 4292983720,
+         "collectionStack": 4291337092,
+         "invertedBackground": 4287578649,
+         "textPrimaryInverse": 4294964720,
+         "textDisabled": 2135828243,
+         "dropShadow": 432632390,
+         "cardOutline": 180974150,
+         "overlayBackground": 3426162953,
+         "overlayTextPrimary": 4294964720,
+         "solidWash": 4283311891,
+         "solidRim": 4281800969,
+         "overlaySolidWash": 4294699488,
+         "overlaySolidRim": 4294964720,
+         "tonalWash": 2583688688,
+         "tonalRim": 202310912,
+         "overlayTonalWash": 436205040,
+         "overlayTonalRim": 436205040,
+         "outlineRim": 1280190227,
+         "overlayOutlineRim": 871618504,
+         "solidWashInverse": 4294170568,
+         "solidRimInverse": 2147215840,
+         "tonalWashInverse": 654308848,
+         "tonalRimInverse": 654043616,
+         "outlineRimInverse": 654308848,
+         "solidBackground": 4283311891,
+         "solidBackgroundInverse": 4291337092,
+         "tonalBackground": 868785028,
+         "tonalBackgroundInverse": 435410888,
+         "outlineInverse": 872147424,
+         "directionalLightStopPrimary": 4294699488,
+         "directionalLightStopSecondary": 4294964720,
+         "primary": 4279174400,
+         "filledStateLayer": 4281800969,
+         "tonalTouchLight": 4294964720,
+         "tonalHoverLight": 3439061472,
+         "tonalStateLayer": 420414720,
+         "overlayDisabledContainer": 871618504,
+         "overlayTonalTouchLight": 1291577824,
+         "overlayTonalHoverLight": 1291577824,
+         "overlayTonalStateLayer": 436205040,
+         "disabledContainer": 420414720,
+         "filledTouchLight": 4285876787,
+         "filledHoverLight": 4284627235,
+         "onOverlayDisabledContainer": 2583688688,
+         "overlayTonalContainer": 2583423456,
+         "overlayOutline": 872412656,
+         "onFilledContainer": 4294964720,
+         "filledContainer": 4280027906,
+         "tonalContainer": 641145097,
+         "overlayPrimary": 4294964720,
+         "onOverlayFilledContainer": 4279174400,
+         "overlayFilledTouchLight": 4294964720,
+         "overlayFilledHoverLight": 4294699488,
+         "overlayFilledStateLayer": 4292983720,
+         "overlayFilledContainer": 4294170568,
+         "onDisabledContainer": 1712260352,
+         "temporaryOutline": 860759827
+        },
+        "darkThemePalette": {
+         "baseBackground": 4280027906,
+         "raisedBackground": 4281800969,
+         "additiveBackground": 734622278,
+         "textPrimary": 4294699488,
+         "textSecondary": 4292983720,
+         "outline": 1072945096,
+         "keyLight": 4284627235,
+         "collectionStack": 4285876787,
+         "invertedBackground": 4293107837,
+         "textPrimaryInverse": 4280027906,
+         "textDisabled": 2146686920,
+         "dropShadow": 432632390,
+         "cardOutline": 180974150,
+         "overlayBackground": 3426162953,
+         "overlayTextPrimary": 4294964720,
+         "solidWash": 4294699488,
+         "solidRim": 4294964720,
+         "overlaySolidWash": 4294699488,
+         "overlaySolidRim": 4294964720,
+         "tonalWash": 217307080,
+         "tonalRim": 218101232,
+         "overlayTonalWash": 436205040,
+         "overlayTonalRim": 436205040,
+         "outlineRim": 871618504,
+         "overlayOutlineRim": 871618504,
+         "solidWashInverse": 4285876787,
+         "solidRimInverse": 871618504,
+         "tonalWashInverse": 872412656,
+         "tonalRimInverse": 870431656,
+         "outlineRimInverse": 1291843056,
+         "solidBackground": 4291337092,
+         "solidBackgroundInverse": 4283311891,
+         "tonalBackground": 435410888,
+         "tonalBackgroundInverse": 868785028,
+         "outlineInverse": 860759827,
+         "directionalLightStopPrimary": 4281800969,
+         "directionalLightStopSecondary": 4280027906,
+         "primary": 4294964720,
+         "filledStateLayer": 4292983720,
+         "tonalTouchLight": 1073474016,
+         "tonalHoverLight": 654043616,
+         "tonalStateLayer": 436205040,
+         "overlayDisabledContainer": 871618504,
+         "overlayTonalTouchLight": 1291577824,
+         "overlayTonalHoverLight": 1291577824,
+         "overlayTonalStateLayer": 436205040,
+         "disabledContainer": 436205040,
+         "filledTouchLight": 4294964720,
+         "filledHoverLight": 4294699488,
+         "onOverlayDisabledContainer": 2583688688,
+         "overlayTonalContainer": 2583423456,
+         "overlayOutline": 872412656,
+         "onFilledContainer": 4279174400,
+         "filledContainer": 4294170568,
+         "tonalContainer": 435410888,
+         "overlayPrimary": 4294964720,
+         "onOverlayFilledContainer": 4279174400,
+         "overlayFilledTouchLight": 4294964720,
+         "overlayFilledHoverLight": 4294699488,
+         "overlayFilledStateLayer": 4292983720,
+         "overlayFilledContainer": 4294170568,
+         "onDisabledContainer": 1728050672,
+         "temporaryOutline": 871618504
+        }
+       }
+      }
+     },
+     {
+      "lockupViewModel": {
+       "contentImage": {
+        "thumbnailViewModel": {
+         "image": {
+          "sources": [
+           {
+            "url": "https://i.ytimg.com/vi/P7Hkh2zOGQ0/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLAHInNtMierSAgtrGl5rLSKc-vdzQ",
+            "width": 168,
+            "height": 94
+           },
+           {
+            "url": "https://i.ytimg.com/vi/P7Hkh2zOGQ0/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLDXrm5g-rdjRIqqK309ImTdqpw4CA",
+            "width": 336,
+            "height": 188
+           }
+          ]
+         },
+         "overlays": [
+          {
+           "thumbnailBottomOverlayViewModel": {
+            "badges": [
+             {
+              "thumbnailBadgeViewModel": {
+               "text": "1:08:19",
+               "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
+               "animationActivationTargetId": "P7Hkh2zOGQ0",
+               "animatedText": "Now playing",
+               "animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
+              }
+             }
+            ]
+           }
+          }
+         ]
+        }
+       },
+       "metadata": {
+        "lockupMetadataViewModel": {
+         "title": {
+          "content": "MIT 6.S191 (2024): Google - Generative AI for Media"
+         },
+         "image": {
+          "decoratedAvatarViewModel": {
+           "avatar": {
+            "avatarViewModel": {
+             "image": {
+              "sources": [
+               {
+                "url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
+                "width": 68,
+                "height": 68
+               }
+              ]
+             },
+             "avatarImageSize": "AVATAR_SIZE_M"
+            }
+           },
+           "a11yLabel": "Go to channel Alexander Amini",
+           "rendererContext": {
+            "commandContext": {
+             "onTap": {
+              "innertubeCommand": {
+               "browseEndpoint": {
+                "browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
+               }
+              }
+             }
+            }
+           }
+          }
+         },
+         "metadata": {
+          "contentMetadataViewModel": {
+           "metadataRows": [
+            {
+             "metadataParts": [
+              {
+               "text": {
+                "content": "Alexander Amini"
+               }
+              }
+             ]
+            },
+            {
+             "metadataParts": [
+              {
+               "text": {
+                "content": "49K"
+               },
+               "accessibilityLabel": "49 thousand views",
+               "leadingIcon": {
+                "name": "PLAY_ARROW_OUTLINED",
+                "height": 12,
+                "width": 12,
+                "marginEndToken": {
+                 "spacing": 2
+                }
+               }
+              },
+              {
+               "text": {
+                "content": "2y ago"
+               },
+               "accessibilityLabel": "2 years ago"
+              }
+             ]
+            }
+           ],
+           "delimiter": " "
+          }
+         }
+        }
+       },
+       "contentId": "P7Hkh2zOGQ0",
+       "contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
+       "themedPalette": {
+        "lightThemePalette": {
+         "baseBackground": 4293981951,
+         "raisedBackground": 4293981951,
+         "additiveBackground": 558201801,
+         "textPrimary": 4278655801,
+         "textSecondary": 4280831104,
+         "outline": 856631890,
+         "keyLight": 4288855783,
+         "collectionStack": 4286094803,
+         "invertedBackground": 4279847311,
+         "textPrimaryInverse": 4293981951,
+         "textDisabled": 2131700306,
+         "dropShadow": 423984073,
+         "cardOutline": 172325833,
+         "overlayBackground": 3423017785,
+         "overlayTextPrimary": 4293981951,
+         "solidWash": 4279183954,
+         "solidRim": 4278655801,
+         "overlaySolidWash": 4292864764,
+         "overlaySolidRim": 4293981951,
+         "tonalWash": 2582705919,
+         "tonalRim": 201328143,
+         "overlayTonalWash": 435222271,
+         "overlayTonalRim": 435222271,
+         "outlineRim": 1276062290,
+         "overlayOutlineRim": 868669942,
+         "solidWashInverse": 4291222006,
+         "solidRimInverse": 2145381116,
+         "tonalWashInverse": 653326079,
+         "tonalRimInverse": 652208892,
+         "outlineRimInverse": 653326079,
+         "solidBackground": 4279183954,
+         "solidBackgroundInverse": 4286094803,
+         "tonalBackground": 863542739,
+         "tonalBackgroundInverse": 432462326,
+         "outlineInverse": 870312700,
+         "directionalLightStopPrimary": 4292864764,
+         "directionalLightStopSecondary": 4293981951,
+         "primary": 4278191631,
+         "filledStateLayer": 4278655801,
+         "tonalTouchLight": 4293981951,
+         "tonalHoverLight": 3437226748,
+         "tonalStateLayer": 419431951,
+         "overlayDisabledContainer": 868669942,
+         "overlayTonalTouchLight": 1289743100,
+         "overlayTonalHoverLight": 1289743100,
+         "overlayTonalStateLayer": 435222271,
+         "disabledContainer": 419431951,
+         "filledTouchLight": 4280831104,
+         "filledHoverLight": 4280040297,
+         "onOverlayDisabledContainer": 2582705919,
+         "overlayTonalContainer": 2581588732,
+         "overlayOutline": 871429887,
+         "onFilledContainer": 4293981951,
+         "filledContainer": 4278324509,
+         "tonalContainer": 637999929,
+         "overlayPrimary": 4293981951,
+         "onOverlayFilledContainer": 4278191631,
+         "overlayFilledTouchLight": 4293981951,
+         "overlayFilledHoverLight": 4292864764,
+         "overlayFilledStateLayer": 4288855783,
+         "overlayFilledContainer": 4291222006,
+         "onDisabledContainer": 1711277583,
+         "temporaryOutline": 856631890
+        },
+        "darkThemePalette": {
+         "baseBackground": 4278324509,
+         "raisedBackground": 4278655801,
+         "additiveBackground": 725973961,
+         "textPrimary": 4292864764,
+         "textSecondary": 4288855783,
+         "outline": 1069996534,
+         "keyLight": 4280040297,
+         "collectionStack": 4280831104,
+         "invertedBackground": 4286359267,
+         "textPrimaryInverse": 4278324509,
+         "textDisabled": 2143738358,
+         "dropShadow": 423984073,
+         "cardOutline": 172325833,
+         "overlayBackground": 3423017785,
+         "overlayTextPrimary": 4293981951,
+         "solidWash": 4292864764,
+         "solidRim": 4293981951,
+         "overlaySolidWash": 4292864764,
+         "overlaySolidRim": 4293981951,
+         "tonalWash": 214358518,
+         "tonalRim": 217118463,
+         "overlayTonalWash": 435222271,
+         "overlayTonalRim": 435222271,
+         "outlineRim": 868669942,
+         "overlayOutlineRim": 868669942,
+         "solidWashInverse": 4280831104,
+         "solidRimInverse": 868669942,
+         "tonalWashInverse": 871429887,
+         "tonalRimInverse": 866303719,
+         "outlineRimInverse": 1290860287,
+         "solidBackground": 4286094803,
+         "solidBackgroundInverse": 4279183954,
+         "tonalBackground": 432462326,
+         "tonalBackgroundInverse": 863542739,
+         "outlineInverse": 856631890,
+         "directionalLightStopPrimary": 4278655801,
+         "directionalLightStopSecondary": 4278324509,
+         "primary": 4293981951,
+         "filledStateLayer": 4288855783,
+         "tonalTouchLight": 1071639292,
+         "tonalHoverLight": 652208892,
+         "tonalStateLayer": 435222271,
+         "overlayDisabledContainer": 868669942,
+         "overlayTonalTouchLight": 1289743100,
+         "overlayTonalHoverLight": 1289743100,
+         "overlayTonalStateLayer": 435222271,
+         "disabledContainer": 435222271,
+         "filledTouchLight": 4293981951,
+         "filledHoverLight": 4292864764,
+         "onOverlayDisabledContainer": 2582705919,
+         "overlayTonalContainer": 2581588732,
+         "overlayOutline": 871429887,
+         "onFilledContainer": 4278191631,
+         "filledContainer": 4291222006,
+         "tonalContainer": 432462326,
+         "overlayPrimary": 4293981951,
+         "onOverlayFilledContainer": 4278191631,
+         "overlayFilledTouchLight": 4293981951,
+         "overlayFilledHoverLight": 4292864764,
+         "overlayFilledStateLayer": 4288855783,
+         "overlayFilledContainer": 4291222006,
+         "onDisabledContainer": 1727067903,
+         "temporaryOutline": 868669942
+        }
+       }
+      }
+     },
+     {
+      "continuationItemRenderer": {
+       "trigger": "CONTINUATION_TRIGGER_ON_ITEM_SHOWN",
+       "continuationEndpoint": {
+        "commandMetadata": {
+         "webCommandMetadata": {
+          "sendPost": true,
+          "apiUrl": "/youtubei/v1/next"
+         }
+        },
+        "continuationCommand": {
+         "token": "CCgSDRILU2RUWkFNREtyTlkYACqPDE1nb0lodVQ4eG9EdXVLSUYyczZMNnd6NENBcjFDQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhQ2dQeVBnQUtEc0ktQ3dqaDRleUUzTFAzdV93QkNnUHlQZ0FLRGNJLUNnaU5zcmptOXBENTJEOEtBX0ktQUFvTndqNEtDTExudjR5ZXg3U1dTUW9EOGo0QUNnM0NQZ29JNnZHODhaR1JpTWNnQ2dQeVBnQUtEc0ktQ3dpRF85S3Z5WS1hbThJQkNnUHlQZ0FLRHNJLUN3alppNmJYbWR6YWxZb0JDZ1B5UGdBS0RzSS1Dd2lkemJfS2dPdWN0YUFCQ2dQeVBnQUtEY0ktQ2dqMXktMjYyb2JQeFdNS0FfSS1BQW9Pd2o0TENPLUE4SWVaMUlESXJRRUtBX0ktQUFvTndqNEtDUHlZNDVhUzhkeWpJZ29EOGo0QUNnM0NQZ29Jb191aWtfSE05ZHdTQ2dQeVBnQUtEc0ktQ3dqaTBiemlrdnpBbkxvQkNnUHlQZ0FLRHNJLUN3aW05cXI1eXNtTWlzOEJDZ1B5UGdBS0RjSS1DZ2k3M2V5NXlzQ1pzMG9LQV9JLUFBb093ajRMQ0xmT281WFU1X2JPbFFFS0FfSS1BQW9Od2o0S0NLTEVwYXlXOTliWkdnb0Q4ajRBQ2c3Q1Bnc0k2dWZSMnRPM3hLem9BUW9EOGo0QUNnN0NQZ3NJN2ZlZXZQRHV2WlRXQVFvRDhqNEFDZzNDUGdvSXFibS0zYXVBdDVWcUNnUHlQZ0FLRGNJLUNnaUc1UHpHZ082NG9nVVNLQUFDQkFZSUNnd09FQklVRmhnYUhCNGdJaVFtS0Nvc0xqQXlORFk0T2p3LVFFSkVSa2hLVEU0YUJBZ0FFQUVhQkFnQ0VBTWFCQWdFRUFVYUJBZ0dFQWNhQkFnSUVBa2FCQWdLRUFzYUJBZ01FQTBhQkFnT0VBOGFCQWdRRUJFYUJBZ1NFQk1hQkFnVUVCVWFCQWdXRUJjYUJBZ1lFQmthQkFnYUVCc2FCQWdjRUIwYUJBZ2VFQjhhQkFnZ0VDRWFCQWdpRUNNYUJBZ2tFQ1VhQkFnbUVDY2FCQWdvRUNrYUJBZ3FFQ3NhQkFnc0VDMGFCQWd1RUM4YUJBZ3dFREVhQkFneUVETWFCQWcwRURVYUJBZzJFRGNhQkFnNEVEa2FCQWc2RURzYUJBZzhFRDBhQkFnLUVEOGFCQWhBRUVFYUJBaENFRU1hQkFoRUVFVWFCQWhHRUVjYUJBaElFRWthQkFoS0VFc2FCQWhNRUUwYUJBaE9FRThxS0FBQ0JBWUlDZ3dPRUJJVUZoZ2FIQjRnSWlRbUtDb3NMakF5TkRZNE9qdy1RRUpFUmtoS1RFNGoPd2F0Y2gtbmV4dC1mZWVk",
+         "request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
+        }
+       }
+      }
+     }
+    ]
+   }
+  }
+ ]
+}

backend/tests/fixtures/innertube_next_first.json

@@ -0,0 +1,805 @@
+{
+ "contents": {
+  "twoColumnWatchNextResults": {
+   "secondaryResults": {
+    "secondaryResults": {
+     "results": [
+      {
+       "lockupViewModel": {
+        "contentImage": {
+         "thumbnailViewModel": {
+          "image": {
+           "sources": [
+            {
+             "url": "https://i.ytimg.com/vi/to-lHJfK4pw/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLBKP_oFChvoqvwVrKAy_PG9bn2_AA",
+             "width": 168,
+             "height": 94
+            },
+            {
+             "url": "https://i.ytimg.com/vi/to-lHJfK4pw/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLDyH3Z8DJhXMmFJ5eKEWMufk7ql9g",
+             "width": 336,
+             "height": 188
+            }
+           ]
+          },
+          "overlays": [
+           {
+            "thumbnailBottomOverlayViewModel": {
+             "badges": [
+              {
+               "thumbnailBadgeViewModel": {
+                "text": "1:02:00",
+                "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
+                "animationActivationTargetId": "to-lHJfK4pw",
+                "animatedText": "Now playing",
+                "animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
+               }
+              }
+             ]
+            }
+           }
+          ]
+         }
+        },
+        "metadata": {
+         "lockupMetadataViewModel": {
+          "title": {
+           "content": "MIT 6.S191 (2025): Reinforcement Learning"
+          },
+          "image": {
+           "decoratedAvatarViewModel": {
+            "avatar": {
+             "avatarViewModel": {
+              "image": {
+               "sources": [
+                {
+                 "url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
+                 "width": 68,
+                 "height": 68
+                }
+               ]
+              },
+              "avatarImageSize": "AVATAR_SIZE_M"
+             }
+            },
+            "a11yLabel": "Go to channel Alexander Amini",
+            "rendererContext": {
+             "commandContext": {
+              "onTap": {
+               "innertubeCommand": {
+                "browseEndpoint": {
+                 "browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
+                }
+               }
+              }
+             }
+            }
+           }
+          },
+          "metadata": {
+           "contentMetadataViewModel": {
+            "metadataRows": [
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "Alexander Amini"
+                }
+               }
+              ]
+             },
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "128K"
+                },
+                "accessibilityLabel": "128 thousand views",
+                "leadingIcon": {
+                 "name": "PLAY_ARROW_OUTLINED",
+                 "height": 12,
+                 "width": 12,
+                 "marginEndToken": {
+                  "spacing": 2
+                 }
+                }
+               },
+               {
+                "text": {
+                 "content": "1y ago"
+                },
+                "accessibilityLabel": "1 year ago"
+               }
+              ]
+             }
+            ],
+            "delimiter": " "
+           }
+          }
+         }
+        },
+        "contentId": "to-lHJfK4pw",
+        "contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
+        "themedPalette": {
+         "lightThemePalette": {
+          "baseBackground": 4293981439,
+          "raisedBackground": 4293981439,
+          "additiveBackground": 558196425,
+          "textPrimary": 4278850870,
+          "textSecondary": 4282009454,
+          "outline": 857088843,
+          "keyLight": 4289509597,
+          "collectionStack": 4287338944,
+          "invertedBackground": 4279842703,
+          "textPrimaryInverse": 4293981439,
+          "textDisabled": 2132157259,
+          "dropShadow": 423978697,
+          "cardOutline": 172320457,
+          "overlayBackground": 3423212854,
+          "overlayTextPrimary": 4293981439,
+          "solidWash": 4279640907,
+          "solidRim": 4278850870,
+          "overlaySolidWash": 4292929530,
+          "overlaySolidRim": 4293981439,
+          "tonalWash": 2582705407,
+          "tonalRim": 201327631,
+          "overlayTonalWash": 435221759,
+          "overlayTonalRim": 435221759,
+          "outlineRim": 1276519243,
+          "overlayOutlineRim": 868930802,
+          "solidWashInverse": 4291482866,
+          "solidRimInverse": 2145445882,
+          "tonalWashInverse": 653325567,
+          "tonalRimInverse": 652273658,
+          "outlineRimInverse": 653325567,
+          "solidBackground": 4279640907,
+          "solidBackgroundInverse": 4287338944,
+          "tonalBackground": 864786880,
+          "tonalBackgroundInverse": 432723186,
+          "outlineInverse": 870377466,
+          "directionalLightStopPrimary": 4292929530,
+          "directionalLightStopSecondary": 4293981439,
+          "primary": 4278191119,
+          "filledStateLayer": 4278850870,
+          "tonalTouchLight": 4293981439,
+          "tonalHoverLight": 3437291514,
+          "tonalStateLayer": 419431439,
+          "overlayDisabledContainer": 868930802,
+          "overlayTonalTouchLight": 1289807866,
+          "overlayTonalHoverLight": 1289807866,
+          "overlayTonalStateLayer": 435221759,
+          "disabledContainer": 419431439,
+          "filledTouchLight": 4282009454,
+          "filledHoverLight": 4280825181,
+          "onOverlayDisabledContainer": 2582705407,
+          "overlayTonalContainer": 2581653498,
+          "overlayOutline": 871429375,
+          "onFilledContainer": 4293981439,
+          "filledContainer": 4278323484,
+          "tonalContainer": 638194998,
+          "overlayPrimary": 4293981439,
+          "onOverlayFilledContainer": 4278191119,
+          "overlayFilledTouchLight": 4293981439,
+          "overlayFilledHoverLight": 4292929530,
+          "overlayFilledStateLayer": 4289509597,
+          "overlayFilledContainer": 4291482866,
+          "onDisabledContainer": 1711277071,
+          "temporaryOutline": 857088843
+         },
+         "darkThemePalette": {
+          "baseBackground": 4278323484,
+          "raisedBackground": 4278850870,
+          "additiveBackground": 725968585,
+          "textPrimary": 4292929530,
+          "textSecondary": 4289509597,
+          "outline": 1070257394,
+          "keyLight": 4280825181,
+          "collectionStack": 4282009454,
+          "invertedBackground": 4286355427,
+          "textPrimaryInverse": 4278323484,
+          "textDisabled": 2143999218,
+          "dropShadow": 423978697,
+          "cardOutline": 172320457,
+          "overlayBackground": 3423212854,
+          "overlayTextPrimary": 4293981439,
+          "solidWash": 4292929530,
+          "solidRim": 4293981439,
+          "overlaySolidWash": 4292929530,
+          "overlaySolidRim": 4293981439,
+          "tonalWash": 214619378,
+          "tonalRim": 217117951,
+          "overlayTonalWash": 435221759,
+          "overlayTonalRim": 435221759,
+          "outlineRim": 868930802,
+          "overlayOutlineRim": 868930802,
+          "solidWashInverse": 4282009454,
+          "solidRimInverse": 868930802,
+          "tonalWashInverse": 871429375,
+          "tonalRimInverse": 866957533,
+          "outlineRimInverse": 1290859775,
+          "solidBackground": 4287338944,
+          "solidBackgroundInverse": 4279640907,
+          "tonalBackground": 432723186,
+          "tonalBackgroundInverse": 864786880,
+          "outlineInverse": 857088843,
+          "directionalLightStopPrimary": 4278850870,
+          "directionalLightStopSecondary": 4278323484,
+          "primary": 4293981439,
+          "filledStateLayer": 4289509597,
+          "tonalTouchLight": 1071704058,
+          "tonalHoverLight": 652273658,
+          "tonalStateLayer": 435221759,
+          "overlayDisabledContainer": 868930802,
+          "overlayTonalTouchLight": 1289807866,
+          "overlayTonalHoverLight": 1289807866,
+          "overlayTonalStateLayer": 435221759,
+          "disabledContainer": 435221759,
+          "filledTouchLight": 4293981439,
+          "filledHoverLight": 4292929530,
+          "onOverlayDisabledContainer": 2582705407,
+          "overlayTonalContainer": 2581653498,
+          "overlayOutline": 871429375,
+          "onFilledContainer": 4278191119,
+          "filledContainer": 4291482866,
+          "tonalContainer": 432723186,
+          "overlayPrimary": 4293981439,
+          "onOverlayFilledContainer": 4278191119,
+          "overlayFilledTouchLight": 4293981439,
+          "overlayFilledHoverLight": 4292929530,
+          "overlayFilledStateLayer": 4289509597,
+          "overlayFilledContainer": 4291482866,
+          "onDisabledContainer": 1727067391,
+          "temporaryOutline": 868930802
+         }
+        }
+       }
+      },
+      {
+       "lockupViewModel": {
+        "contentImage": {
+         "thumbnailViewModel": {
+          "image": {
+           "sources": [
+            {
+             "url": "https://i.ytimg.com/vi/R8V8CbuxryI/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLDqXMjrpgDyQKHL2lWrxczwxKoefw",
+             "width": 168,
+             "height": 94
+            },
+            {
+             "url": "https://i.ytimg.com/vi/R8V8CbuxryI/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLB7JX9Z238s6DGqIS5-fhcEIkUxKw",
+             "width": 336,
+             "height": 188
+            }
+           ]
+          },
+          "overlays": [
+           {
+            "thumbnailBottomOverlayViewModel": {
+             "badges": [
+              {
+               "thumbnailBadgeViewModel": {
+                "text": "49:36",
+                "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
+                "animationActivationTargetId": "R8V8CbuxryI",
+                "animatedText": "Now playing",
+                "animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
+               }
+              }
+             ]
+            }
+           }
+          ]
+         }
+        },
+        "metadata": {
+         "lockupMetadataViewModel": {
+          "title": {
+           "content": "MIT 6.S191: Deep Generative Modeling"
+          },
+          "image": {
+           "decoratedAvatarViewModel": {
+            "avatar": {
+             "avatarViewModel": {
+              "image": {
+               "sources": [
+                {
+                 "url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
+                 "width": 68,
+                 "height": 68
+                }
+               ]
+              },
+              "avatarImageSize": "AVATAR_SIZE_M"
+             }
+            },
+            "a11yLabel": "Go to channel Alexander Amini",
+            "rendererContext": {
+             "commandContext": {
+              "onTap": {
+               "innertubeCommand": {
+                "browseEndpoint": {
+                 "browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
+                }
+               }
+              }
+             }
+            }
+           }
+          },
+          "metadata": {
+           "contentMetadataViewModel": {
+            "metadataRows": [
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "Alexander Amini"
+                }
+               }
+              ]
+             },
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "38K"
+                },
+                "accessibilityLabel": "38 thousand views",
+                "leadingIcon": {
+                 "name": "PLAY_ARROW_OUTLINED",
+                 "height": 12,
+                 "width": 12,
+                 "marginEndToken": {
+                  "spacing": 2
+                 }
+                }
+               },
+               {
+                "text": {
+                 "content": "6mo ago"
+                },
+                "accessibilityLabel": "6 months ago"
+               }
+              ]
+             }
+            ],
+            "delimiter": " "
+           }
+          }
+         }
+        },
+        "contentId": "R8V8CbuxryI",
+        "contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
+        "themedPalette": {
+         "lightThemePalette": {
+          "baseBackground": 4294965488,
+          "raisedBackground": 4294965488,
+          "additiveBackground": 566856261,
+          "textPrimary": 4281737226,
+          "textSecondary": 4285420858,
+          "outline": 860565782,
+          "keyLight": 4292724140,
+          "collectionStack": 4290815883,
+          "invertedBackground": 4287583769,
+          "textPrimaryInverse": 4294965488,
+          "textDisabled": 2135634198,
+          "dropShadow": 432638533,
+          "cardOutline": 180980293,
+          "overlayBackground": 3426099210,
+          "overlayTextPrimary": 4294965488,
+          "solidWash": 4283117846,
+          "solidRim": 4281737226,
+          "overlaySolidWash": 4294635232,
+          "overlaySolidRim": 4294965488,
+          "tonalWash": 2583689456,
+          "tonalRim": 202311680,
+          "overlayTonalWash": 436205808,
+          "overlayTonalRim": 436205808,
+          "outlineRim": 1279996182,
+          "overlayOutlineRim": 871554762,
+          "solidWashInverse": 4294106826,
+          "solidRimInverse": 2147151584,
+          "tonalWashInverse": 654309616,
+          "tonalRimInverse": 653979360,
+          "outlineRimInverse": 654309616,
+          "solidBackground": 4283117846,
+          "solidBackgroundInverse": 4290815883,
+          "tonalBackground": 868263819,
+          "tonalBackgroundInverse": 435347146,
+          "outlineInverse": 872083168,
+          "directionalLightStopPrimary": 4294635232,
+          "directionalLightStopSecondary": 4294965488,
+          "primary": 4279175168,
+          "filledStateLayer": 4281737226,
+          "tonalTouchLight": 4294965488,
+          "tonalHoverLight": 3438997216,
+          "tonalStateLayer": 420415488,
+          "overlayDisabledContainer": 871554762,
+          "overlayTonalTouchLight": 1291513568,
+          "overlayTonalHoverLight": 1291513568,
+          "overlayTonalStateLayer": 436205808,
+          "disabledContainer": 420415488,
+          "filledTouchLight": 4285420858,
+          "filledHoverLight": 4284302120,
+          "onOverlayDisabledContainer": 2583689456,
+          "overlayTonalContainer": 2583359200,
+          "overlayOutline": 872413424,
+          "onFilledContainer": 4294965488,
+          "filledContainer": 4280029186,
+          "tonalContainer": 641081354,
+          "overlayPrimary": 4294965488,
+          "onOverlayFilledContainer": 4279175168,
+          "overlayFilledTouchLight": 4294965488,
+          "overlayFilledHoverLight": 4294635232,
+          "overlayFilledStateLayer": 4292724140,
+          "overlayFilledContainer": 4294106826,
+          "onDisabledContainer": 1712261120,
+          "temporaryOutline": 860565782
+         },
+         "darkThemePalette": {
+          "baseBackground": 4280029186,
+          "raisedBackground": 4281737226,
+          "additiveBackground": 734628421,
+          "textPrimary": 4294635232,
+          "textSecondary": 4292724140,
+          "outline": 1072881354,
+          "keyLight": 4284302120,
+          "collectionStack": 4285420858,
+          "invertedBackground": 4293112700,
+          "textPrimaryInverse": 4280029186,
+          "textDisabled": 2146623178,
+          "dropShadow": 432638533,
+          "cardOutline": 180980293,
+          "overlayBackground": 3426099210,
+          "overlayTextPrimary": 4294965488,
+          "solidWash": 4294635232,
+          "solidRim": 4294965488,
+          "overlaySolidWash": 4294635232,
+          "overlaySolidRim": 4294965488,
+          "tonalWash": 217243338,
+          "tonalRim": 218102000,
+          "overlayTonalWash": 436205808,
+          "overlayTonalRim": 436205808,
+          "outlineRim": 871554762,
+          "overlayOutlineRim": 871554762,
+          "solidWashInverse": 4285420858,
+          "solidRimInverse": 871554762,
+          "tonalWashInverse": 872413424,
+          "tonalRimInverse": 870172076,
+          "outlineRimInverse": 1291843824,
+          "solidBackground": 4290815883,
+          "solidBackgroundInverse": 4283117846,
+          "tonalBackground": 435347146,
+          "tonalBackgroundInverse": 868263819,
+          "outlineInverse": 860565782,
+          "directionalLightStopPrimary": 4281737226,
+          "directionalLightStopSecondary": 4280029186,
+          "primary": 4294965488,
+          "filledStateLayer": 4292724140,
+          "tonalTouchLight": 1073409760,
+          "tonalHoverLight": 653979360,
+          "tonalStateLayer": 436205808,
+          "overlayDisabledContainer": 871554762,
+          "overlayTonalTouchLight": 1291513568,
+          "overlayTonalHoverLight": 1291513568,
+          "overlayTonalStateLayer": 436205808,
+          "disabledContainer": 436205808,
+          "filledTouchLight": 4294965488,
+          "filledHoverLight": 4294635232,
+          "onOverlayDisabledContainer": 2583689456,
+          "overlayTonalContainer": 2583359200,
+          "overlayOutline": 872413424,
+          "onFilledContainer": 4279175168,
+          "filledContainer": 4294106826,
+          "tonalContainer": 435347146,
+          "overlayPrimary": 4294965488,
+          "onOverlayFilledContainer": 4279175168,
+          "overlayFilledTouchLight": 4294965488,
+          "overlayFilledHoverLight": 4294635232,
+          "overlayFilledStateLayer": 4292724140,
+          "overlayFilledContainer": 4294106826,
+          "onDisabledContainer": 1728051440,
+          "temporaryOutline": 871554762
+         }
+        }
+       }
+      },
+      {
+       "lockupViewModel": {
+        "contentImage": {
+         "thumbnailViewModel": {
+          "image": {
+           "sources": [
+            {
+             "url": "https://i.ytimg.com/vi/HLKo4fJx_7k/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLAQcvMQEX1ny9Z8qu2mIMogpEMaVA",
+             "width": 168,
+             "height": 94
+            },
+            {
+             "url": "https://i.ytimg.com/vi/HLKo4fJx_7k/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLCkoPSkHifs5ifpWXXcb96XG_Ly1w",
+             "width": 336,
+             "height": 188
+            }
+           ]
+          },
+          "overlays": [
+           {
+            "thumbnailBottomOverlayViewModel": {
+             "badges": [
+              {
+               "thumbnailBadgeViewModel": {
+                "text": "47:36",
+                "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT",
+                "animationActivationTargetId": "HLKo4fJx_7k",
+                "animatedText": "Now playing",
+                "animationActivationEntitySelectorType": "THUMBNAIL_BADGE_ANIMATION_ENTITY_SELECTOR_TYPE_PLAYER_STATE"
+               }
+              }
+             ]
+            }
+           }
+          ]
+         }
+        },
+        "metadata": {
+         "lockupMetadataViewModel": {
+          "title": {
+           "content": "MIT 6.S191 (2025): Language Models and New Frontiers"
+          },
+          "image": {
+           "decoratedAvatarViewModel": {
+            "avatar": {
+             "avatarViewModel": {
+              "image": {
+               "sources": [
+                {
+                 "url": "https://yt3.ggpht.com/ytc/AIdro_k517G0Dt8QgsF86GKzYkG3Rs8jSGCiYGJw04lx4_1KXQ=s68-c-k-c0x00ffffff-no-rj",
+                 "width": 68,
+                 "height": 68
+                }
+               ]
+              },
+              "avatarImageSize": "AVATAR_SIZE_M"
+             }
+            },
+            "a11yLabel": "Go to channel Alexander Amini",
+            "rendererContext": {
+             "commandContext": {
+              "onTap": {
+               "innertubeCommand": {
+                "browseEndpoint": {
+                 "browseId": "UCtslD4DGH6PKyG_1gFAX7sg"
+                }
+               }
+              }
+             }
+            }
+           }
+          },
+          "metadata": {
+           "contentMetadataViewModel": {
+            "metadataRows": [
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "Alexander Amini"
+                }
+               }
+              ]
+             },
+             {
+              "metadataParts": [
+               {
+                "text": {
+                 "content": "63K"
+                },
+                "accessibilityLabel": "63 thousand views",
+                "leadingIcon": {
+                 "name": "PLAY_ARROW_OUTLINED",
+                 "height": 12,
+                 "width": 12,
+                 "marginEndToken": {
+                  "spacing": 2
+                 }
+                }
+               },
+               {
+                "text": {
+                 "content": "1y ago"
+                },
+                "accessibilityLabel": "1 year ago"
+               }
+              ]
+             }
+            ],
+            "delimiter": " "
+           }
+          }
+         }
+        },
+        "contentId": "HLKo4fJx_7k",
+        "contentType": "LOCKUP_CONTENT_TYPE_VIDEO",
+        "themedPalette": {
+         "lightThemePalette": {
+          "baseBackground": 4294964208,
+          "raisedBackground": 4294964208,
+          "additiveBackground": 566847045,
+          "textPrimary": 4281734410,
+          "textSecondary": 4285417018,
+          "outline": 860561942,
+          "keyLight": 4292720812,
+          "collectionStack": 4290812043,
+          "invertedBackground": 4287575065,
+          "textPrimaryInverse": 4294964208,
+          "textDisabled": 2135630358,
+          "dropShadow": 432629317,
+          "cardOutline": 180971077,
+          "overlayBackground": 3426096394,
+          "overlayTextPrimary": 4294964208,
+          "solidWash": 4283114006,
+          "solidRim": 4281734410,
+          "overlaySolidWash": 4294633184,
+          "overlaySolidRim": 4294964208,
+          "tonalWash": 2583688176,
+          "tonalRim": 202310656,
+          "overlayTonalWash": 436204528,
+          "overlayTonalRim": 436204528,
+          "outlineRim": 1279992342,
+          "overlayOutlineRim": 871551946,
+          "solidWashInverse": 4294104010,
+          "solidRimInverse": 2147149536,
+          "tonalWashInverse": 654308336,
+          "tonalRimInverse": 653977312,
+          "outlineRimInverse": 654308336,
+          "solidBackground": 4283114006,
+          "solidBackgroundInverse": 4290812043,
+          "tonalBackground": 868259979,
+          "tonalBackgroundInverse": 435344330,
+          "outlineInverse": 872081120,
+          "directionalLightStopPrimary": 4294633184,
+          "directionalLightStopSecondary": 4294964208,
+          "primary": 4279174144,
+          "filledStateLayer": 4281734410,
+          "tonalTouchLight": 4294964208,
+          "tonalHoverLight": 3438995168,
+          "tonalStateLayer": 420414464,
+          "overlayDisabledContainer": 871551946,
+          "overlayTonalTouchLight": 1291511520,
+          "overlayTonalHoverLight": 1291511520,
+          "overlayTonalStateLayer": 436204528,
+          "disabledContainer": 420414464,
+          "filledTouchLight": 4285417018,
+          "filledHoverLight": 4284298280,
+          "onOverlayDisabledContainer": 2583688176,
+          "overlayTonalContainer": 2583357152,
+          "overlayOutline": 872412144,
+          "onFilledContainer": 4294964208,
+          "filledContainer": 4280027394,
+          "tonalContainer": 641078538,
+          "overlayPrimary": 4294964208,
+          "onOverlayFilledContainer": 4279174144,
+          "overlayFilledTouchLight": 4294964208,
+          "overlayFilledHoverLight": 4294633184,
+          "overlayFilledStateLayer": 4292720812,
+          "overlayFilledContainer": 4294104010,
+          "onDisabledContainer": 1712260096,
+          "temporaryOutline": 860561942
+         },
+         "darkThemePalette": {
+          "baseBackground": 4280027394,
+          "raisedBackground": 4281734410,
+          "additiveBackground": 734619205,
+          "textPrimary": 4294633184,
+          "textSecondary": 4292720812,
+          "outline": 1072878538,
+          "keyLight": 4284298280,
+          "collectionStack": 4285417018,
+          "invertedBackground": 4293105020,
+          "textPrimaryInverse": 4280027394,
+          "textDisabled": 2146620362,
+          "dropShadow": 432629317,
+          "cardOutline": 180971077,
+          "overlayBackground": 3426096394,
+          "overlayTextPrimary": 4294964208,
+          "solidWash": 4294633184,
+          "solidRim": 4294964208,
+          "overlaySolidWash": 4294633184,
+          "overlaySolidRim": 4294964208,
+          "tonalWash": 217240522,
+          "tonalRim": 218100720,
+          "overlayTonalWash": 436204528,
+          "overlayTonalRim": 436204528,
+          "outlineRim": 871551946,
+          "overlayOutlineRim": 871551946,
+          "solidWashInverse": 4285417018,
+          "solidRimInverse": 871551946,
+          "tonalWashInverse": 872412144,
+          "tonalRimInverse": 870168748,
+          "outlineRimInverse": 1291842544,
+          "solidBackground": 4290812043,
+          "solidBackgroundInverse": 4283114006,
+          "tonalBackground": 435344330,
+          "tonalBackgroundInverse": 868259979,
+          "outlineInverse": 860561942,
+          "directionalLightStopPrimary": 4281734410,
+          "directionalLightStopSecondary": 4280027394,
+          "primary": 4294964208,
+          "filledStateLayer": 4292720812,
+          "tonalTouchLight": 1073407712,
+          "tonalHoverLight": 653977312,
+          "tonalStateLayer": 436204528,
+          "overlayDisabledContainer": 871551946,
+          "overlayTonalTouchLight": 1291511520,
+          "overlayTonalHoverLight": 1291511520,
+          "overlayTonalStateLayer": 436204528,
+          "disabledContainer": 436204528,
+          "filledTouchLight": 4294964208,
+          "filledHoverLight": 4294633184,
+          "onOverlayDisabledContainer": 2583688176,
+          "overlayTonalContainer": 2583357152,
+          "overlayOutline": 872412144,
+          "onFilledContainer": 4279174144,
+          "filledContainer": 4294104010,
+          "tonalContainer": 435344330,
+          "overlayPrimary": 4294964208,
+          "onOverlayFilledContainer": 4279174144,
+          "overlayFilledTouchLight": 4294964208,
+          "overlayFilledHoverLight": 4294633184,
+          "overlayFilledStateLayer": 4292720812,
+          "overlayFilledContainer": 4294104010,
+          "onDisabledContainer": 1728050160,
+          "temporaryOutline": 871551946
+         }
+        }
+       }
+      },
+      {
+       "continuationItemRenderer": {
+        "trigger": "CONTINUATION_TRIGGER_ON_ITEM_SHOWN",
+        "continuationEndpoint": {
+         "commandMetadata": {
+          "webCommandMetadata": {
+           "sendPost": true,
+           "apiUrl": "/youtubei/v1/next"
+          }
+         },
+         "continuationCommand": {
+          "token": "CBQSDRILU2RUWkFNREtyTlkYACqXBk1nb0l5ZGl1dXRUNDdQc2EyczZMNnd5LUJBcTdCQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhRWhRQUFnUUdDQW9NRGhBU0ZCWVlHaHdlSUNJa0pob0VDQUFRQVJvRUNBSVFBeG9FQ0FRUUJSb0VDQVlRQnhvRUNBZ1FDUm9FQ0FvUUN4b0VDQXdRRFJvRUNBNFFEeG9FQ0JBUUVSb0VDQklRRXhvRUNCUVFGUm9FQ0JZUUZ4b0VDQmdRR1JvRUNCb1FHeG9FQ0J3UUhSb0VDQjRRSHhvRUNDQVFJUm9FQ0NJUUl4b0VDQ1FRSlJvRUNDWVFKeW9VQUFJRUJnZ0tEQTRRRWhRV0dCb2NIaUFpSkNZag93YXRjaC1uZXh0LWZlZWQ%3D",
+          "request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
+         }
+        },
+        "button": {
+         "buttonRenderer": {
+          "style": "STYLE_SUGGESTIVE",
+          "size": "SIZE_DEFAULT",
+          "isDisabled": false,
+          "text": {
+           "runs": [
+            {
+             "text": "Show more"
+            }
+           ]
+          },
+          "command": {
+           "commandMetadata": {
+            "webCommandMetadata": {
+             "sendPost": true,
+             "apiUrl": "/youtubei/v1/next"
+            }
+           },
+           "continuationCommand": {
+            "token": "CBQSDRILU2RUWkFNREtyTlkYACqXBk1nb0l5ZGl1dXRUNDdQc2EyczZMNnd5LUJBcTdCQW9EOGo0QUNnN0NQZ3NJbk1XcnZzbWo2Y2UyQVFvRDhqNEFDZzNDUGdvSW90N0czWnVCMy1KSENnUHlQZ0FLRGNJLUNnaTVfOGVUbjV5cTJSd0tBX0ktQUFvT3dqNExDSkQydWJfNXp2ZXZfUUVLQV9JLUFBb053ajRLQ1BXWjVQakt2cjZWU2dvRDhqNEFDZzNDUGdvSW9aTDd1OUtGdF85NkNnUHlQZ0FLRGNJLUNnaUh3N2FLMzd6YzVIZ0tBX0ktQUFvT3dqNExDUE94aHRQTWtfVzB1UUVLQV9JLUFBb093ajRMQ092eTBzaXNsSWZScGdFS0FfSS1BQW9Pd2o0TENKS1FxdHpBeE5lZHJBRUtBX0ktQUFvTndqNEtDTFdteWE2cHNPMkJPZ29EOGo0QUNnM0NQZ29JanYtdy1wQ3k1YVozQ2dQeVBnQUtEc0ktQ3dqMmtmVF9tYlB1XzRvQkNnUHlQZ0FLRHNJLUN3aVpuYWZjc0tQVmpZUUJDZ1B5UGdBS0RjSS1DZ2lxM3BxM3FyX2s0U0VLQV9JLUFBb053ajRLQ0phNXU5eVJvTmJOS0FvRDhqNEFDZzNDUGdvSXFMeWR3T1NhNGJkV0NnUHlQZ0FLRGNJLUNnakh3X2ZucktlSDdXUUtBX0ktQUFvTndqNEtDUG4xak1mcXg4QzJNQW9EOGo0QUNnM0NQZ29JeWRpdXV0VDQ3UHNhRWhRQUFnUUdDQW9NRGhBU0ZCWVlHaHdlSUNJa0pob0VDQUFRQVJvRUNBSVFBeG9FQ0FRUUJSb0VDQVlRQnhvRUNBZ1FDUm9FQ0FvUUN4b0VDQXdRRFJvRUNBNFFEeG9FQ0JBUUVSb0VDQklRRXhvRUNCUVFGUm9FQ0JZUUZ4b0VDQmdRR1JvRUNCb1FHeG9FQ0J3UUhSb0VDQjRRSHhvRUNDQVFJUm9FQ0NJUUl4b0VDQ1FRSlJvRUNDWVFKeW9VQUFJRUJnZ0tEQTRRRWhRV0dCb2NIaUFpSkNZag93YXRjaC1uZXh0LWZlZWQ%3D",
+            "request": "CONTINUATION_REQUEST_TYPE_WATCH_NEXT"
+           }
+          }
+         }
+        }
+       }
+      }
+     ]
+    }
+   }
+  }
+ }
+}

backend/tests/refusal_sites.py

@@ -0,0 +1,738 @@
+"""Every refusal the backend can answer, read from its syntax tree: each `{"code": …, "reason": …}` detail,
+with every reason its code can be sent with.
+
+The frontend's contract (`i18n/backendIds.contract.test.ts`) runs this and requires a sentence for each
+code and reason in both languages. A scan that tries to follow every shape opens a silent gap with each one
+it learns, and following how a value flows through Python (variables, scopes, aliases, a dict changed in
+place) never closes. So the CODE keeps to a narrow form, and this reads a CLOSED list of it, reporting
+everything else, never guessing:
+
+- where a detail stands: where it is sent — `HTTPException`'s second argument or `detail=`, the `"detail"`
+  value of a `JSONResponse`'s content dict — through the arms of `a if c else b`, or as what a module-level
+  factory returns when every return of it is a detail and every reference to it is a call sent as one. A
+  detail held in a variable is not followed;
+- the code: one literal. A `**` spread, or a key that is not a literal, stands BEFORE a literal `code` and
+  `reason`, both of which the detail then carries: the later key wins, so no spread can change them;
+- a reason: a string literal (`None` sends none: the site counts as reasonless); both arms of
+  `a if c else b`; a function's local bound only by plain `x = …`; a module constant reached as `mod.NAME`;
+  a call's `Literal` return annotation;
+- a parameter typed with a `Literal` (basedpyright holds every value passed in), or a `str` one of a
+  module-level helper (`refused`, `not_found`), followed to every call, each of which must pass it by
+  position or by name; a parameter its body binds again is unread;
+- an attribute: `e.x` of the exception an `except C as e` catches, or of a parameter typed `C`, when a
+  method of `C` declares `self.x: <Literal>`;
+- a map lookup (`_SETTING_REASONS[exc.reason]`): the map's values, and its keys must be exactly what the
+  index can be.
+
+A name bound twice at module level, inside a module-level `if`/`try`/loop, named by a `global`, or imported
+from two places is not looked up; nor is a local a lambda or a comprehension binds, or one a `nonlocal` or
+`global` names. A module constant or map that anything stores into, deletes from or (a map) uses but by
+`map[k]` or a read method is unread. A reference named like a followed helper that the scan cannot resolve
+(`import app.x`, a relative import, an app module that does not bind it) is unread. Anything else is UNREAD,
+or an unread code, and the contract fails on it unless no screen shows that code. Stdlib only: it runs on
+the host, beside the frontend's tests (`python -I backend/tests/refusal_sites.py` → JSON).
+"""
+
+import ast
+import json
+import pathlib
+import sys
+from collections.abc import Iterator
+from dataclasses import dataclass, field
+
+APP = pathlib.Path(__file__).resolve().parents[1] / "app"
+
+# Files whose `"code"` dicts are data, not refusals: a subtitle track's status payload, TMDB country codes.
+NOT_REFUSALS = {
+    "plex/subtitles.py": "a subtitle track's status payload (its own `code`)",
+    "plex/tmdb.py": "a production country's ISO code",
+}
+
+# What a map may be used by besides `map[k]`: a method that only reads it.
+READS = {"get", "items", "keys", "values"}
+
+Node = ast.AST
+FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
+SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
+COMPREHENSIONS = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
+COMPOUND = (
+    ast.If,
+    ast.Try,
+    ast.TryStar,
+    ast.For,
+    ast.AsyncFor,
+    ast.While,
+    ast.With,
+    ast.AsyncWith,
+    ast.Match,
+)
+
+
+@dataclass
+class Res:
+    values: set[str] = field(default_factory=set)
+    unread: list[str] = field(default_factory=list)
+    none: bool = False  # an arm sends no reason
+
+    def __or__(self, other: "Res") -> "Res":
+        return Res(self.values | other.values, self.unread + other.unread, self.none or other.none)
+
+
+@dataclass
+class Module:
+    path: str  # relative to app/, posix
+    name: str  # dotted, `app.…`
+    tree: ast.Module
+    parents: dict[Node, Node]
+    # local name → (module, symbol or None for the module)
+    imports: dict[str, tuple[str, str | None]]
+    top: dict[str, Node]  # module-level bindings: def, class, or the assignment
+    ambiguous: set[
+        str
+    ]  # bound more than once, conditionally, by a `global`, or imported from two places
+    slots: set[int]  # the ids of the nodes sent as a detail
+
+
+def _own_scope(node: Node) -> Iterator[Node]:
+    """`node` and what it holds in its own scope: a nested def, class or lambda is itself, not its body."""
+    stack = [node]
+    while stack:
+        n = stack.pop()
+        yield n
+        if n is not node and isinstance(n, SCOPES):
+            continue
+        stack.extend(ast.iter_child_nodes(n))
+
+
+def _stored(target: Node) -> set[str]:
+    """The names an assignment target binds, not the ones it reads (`LABELS[k] = v` binds none)."""
+    return {
+        n.id for n in ast.walk(target) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)
+    }
+
+
+def _binds(node: Node) -> set[str]:
+    """The names one node binds in the scope it sits in."""
+    if isinstance(node, ast.Assign):
+        return set().union(*(_stored(t) for t in node.targets))
+    if isinstance(node, (ast.AnnAssign, ast.AugAssign, ast.For, ast.AsyncFor, ast.NamedExpr)):
+        return _stored(node.target)
+    if isinstance(node, (ast.With, ast.AsyncWith)):
+        return set().union(*(_stored(i.optional_vars) for i in node.items if i.optional_vars))
+    if isinstance(node, (ast.Import, ast.ImportFrom)):
+        return {(a.asname or a.name).split(".")[0] for a in node.names}
+    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
+        return {node.name}
+    if isinstance(node, (ast.MatchAs, ast.MatchStar, ast.ExceptHandler)) and node.name:
+        return {node.name}
+    return set()
+
+
+def _sender(call: ast.Call) -> str | None:
+    return getattr(call.func, "id", None) or getattr(call.func, "attr", None)
+
+
+def detail_slots(tree: Node) -> Iterator[tuple[ast.Call, Node | None]]:
+    """Every place a detail is sent from, with what it holds: `HTTPException`'s second argument or
+    `detail=` (None: sent without one), and the `"detail"` value of a `JSONResponse`'s content dict (a
+    content without one is an answer, not a refusal)."""
+    for node in ast.walk(tree):
+        if not isinstance(node, ast.Call):
+            continue
+        kw = {k.arg: k.value for k in node.keywords}
+        if _sender(node) == "HTTPException":
+            yield node, kw.get("detail", node.args[1] if len(node.args) > 1 else None)
+        elif _sender(node) == "JSONResponse":
+            content = kw.get("content", node.args[0] if node.args else None)
+            if isinstance(content, ast.Dict):
+                for k, v in zip(content.keys, content.values, strict=True):
+                    if isinstance(k, ast.Constant) and k.value == "detail":
+                        yield node, v
+
+
+def _code_dict(node: Node) -> bool:
+    """A `{"code": …}` dict, or a `dict(code=…)` call."""
+    if isinstance(node, ast.Dict):
+        return any(isinstance(k, ast.Constant) and k.value == "code" for k in node.keys)
+    return (
+        isinstance(node, ast.Call)
+        and getattr(node.func, "id", None) == "dict"
+        and any(k.arg == "code" for k in node.keywords)
+    )
+
+
+def _module_name(path: str) -> str:
+    parts = path.removesuffix(".py").split("/")
+    if parts[-1] == "__init__":
+        parts = parts[:-1]
+    return ".".join(["app", *parts])
+
+
+def _load(sources: dict[str, str]) -> dict[str, Module]:
+    mods: dict[str, Module] = {}
+    for path, src in sources.items():
+        tree = ast.parse(src)
+        parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
+        origins: dict[str, set[tuple[str, str | None]]] = {}
+        ambiguous: set[str] = set()
+        for node in ast.walk(tree):
+            if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
+                for a in node.names:
+                    origins.setdefault(a.asname or a.name, set()).add((node.module, a.name))
+            elif isinstance(node, ast.Import):
+                for a in node.names:
+                    if a.asname:
+                        origins.setdefault(a.asname, set()).add((a.name, None))
+            elif isinstance(node, ast.Global):
+                ambiguous |= set(node.names)
+        top: dict[str, Node] = {}
+        count: dict[str, int] = {}
+        ambiguous |= {name for name, o in origins.items() if len(o) > 1}
+        for stmt in tree.body:
+            if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
+                top[stmt.name] = stmt
+            elif isinstance(stmt, ast.Assign):
+                for t in stmt.targets:
+                    if isinstance(t, ast.Name):
+                        top[t.id] = stmt
+            elif isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name):
+                top[stmt.target.id] = stmt
+            for n in [stmt] if isinstance(stmt, SCOPES) else _own_scope(stmt):
+                for name in _binds(n):
+                    count[name] = count.get(name, 0) + 1
+                    if isinstance(stmt, COMPOUND):
+                        ambiguous.add(name)
+        ambiguous |= {name for name, c in count.items() if c > 1}
+        imports = {name: next(iter(o)) for name, o in origins.items() if name not in ambiguous}
+        slots = {id(d) for _, d in detail_slots(tree) if d is not None}
+        name = _module_name(path)
+        mods[name] = Module(path, name, tree, parents, imports, top, ambiguous, slots)
+    return mods
+
+
+class Scan:
+    def __init__(self, sources: dict[str, str]):
+        self.mods = _load(sources)
+        self.errors: list[str] = []
+        self._following: set[tuple[str, str, str]] = set()
+        self._followed: dict[tuple[str, str, str], Res] = {}
+        self._resolving: set[int] = set()
+        self._factory: dict[int, bool] = {}
+        self._changed: dict[int, bool] = {}
+
+    # --- names ---------------------------------------------------------------------------------------
+
+    def lookup(
+        self, m: Module, name: str, seen: frozenset[tuple[str, str]] = frozenset()
+    ) -> tuple[Module, Node] | tuple[str, None] | None:
+        """What a module-level name is: (its module, its binding), or (a module's name, None)."""
+        if name in m.ambiguous:
+            return None
+        if name in m.top:
+            return m, m.top[name]
+        if name in m.imports:
+            mod, sym = m.imports[name]
+            if sym is None:
+                return (mod, None) if mod in self.mods else None
+            if f"{mod}.{sym}" in self.mods:
+                return f"{mod}.{sym}", None
+            if mod in self.mods and (mod, sym) not in seen:
+                return self.lookup(self.mods[mod], sym, seen | {(mod, sym)})
+        return None
+
+    def lookup_expr(self, m: Module, e: Node) -> tuple[Module, Node] | tuple[str, None] | None:
+        if isinstance(e, ast.Name):
+            return None if self.binder(m, e) is not None else self.lookup(m, e.id)
+        if isinstance(e, ast.Attribute):
+            base = self.lookup_expr(m, e.value)
+            if base and base[1] is None and isinstance(base[0], str):
+                return self.lookup(self.mods[base[0]], e.attr)
+        return None
+
+    def binder(self, m: Module, e: ast.Name) -> Node | None:
+        """The innermost scope `e` is bound in: a function (a parameter, or a binding in its own body), a
+        lambda (a parameter), a comprehension (its target); None: a module-level name."""
+        node = m.parents.get(e)
+        while node is not None:
+            if isinstance(node, (*FUNCS, ast.Lambda)):
+                a = node.args
+                params = [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg]
+                if e.id in {p.arg for p in params if p}:
+                    return node
+                if isinstance(node, FUNCS) and self.assignments(node, e.id):
+                    return node
+            elif isinstance(node, COMPREHENSIONS) and any(
+                e.id in _stored(g.target) for g in node.generators
+            ):
+                return node
+            node = m.parents.get(node)
+        return None
+
+    def at(self, m: Module, e: Node) -> str:
+        return f"{m.path}:{getattr(e, 'lineno', '?')}: {ast.unparse(e)}"
+
+    def enclosing(self, m: Module, e: Node, kind: type | tuple[type, ...]) -> Node | None:
+        node = m.parents.get(e)
+        while node is not None and not isinstance(node, kind):
+            node = m.parents.get(node)
+        return node
+
+    def assignments(self, fn: Node, name: str) -> list[Node]:
+        """What binds `name` in this function's own body (not in a nested one)."""
+        return [n for n in _own_scope(fn) if n is not fn and name in _binds(n)]
+
+    def caught_as(self, m: Module, e: Node, name: str) -> ast.ExceptHandler | None:
+        node = m.parents.get(e)
+        while node is not None and not isinstance(node, FUNCS):
+            if isinstance(node, ast.ExceptHandler) and node.name == name:
+                return node
+            node = m.parents.get(node)
+        return None
+
+    def references(
+        self, m: Module, target: Node
+    ) -> Iterator[tuple[Module, Node, ast.Call | None, bool]]:
+        """Every reference to a module-level def or class: the call it makes (None: not a call), and
+        whether it resolved. A reference named like it that the scan cannot resolve is yielded too."""
+        name = getattr(target, "name", None)
+        for cm in self.mods.values():
+            for node in ast.walk(cm.tree):
+                if not isinstance(node, (ast.Name, ast.Attribute)) or not isinstance(
+                    node.ctx, ast.Load
+                ):
+                    continue
+                if (node.id if isinstance(node, ast.Name) else node.attr) != name:
+                    continue
+                parent = cm.parents.get(node)
+                if isinstance(parent, ast.Attribute) and parent.value is node:
+                    continue
+                found = self.lookup_expr(cm, node)
+                if found:
+                    if not (found[0] is m and found[1] is target):
+                        continue
+                elif not self.unresolved(cm, node):
+                    continue
+                call = parent if isinstance(parent, ast.Call) and parent.func is node else None
+                yield cm, node, call, found is not None
+
+    def unresolved(self, m: Module, node: Node) -> bool:
+        """A name, or a dotted path, the app could mean that the scan cannot resolve: not a local, nor an
+        outside module's (an attribute of an object the function holds is not one)."""
+        root = node
+        while isinstance(root, ast.Attribute):
+            root = root.value
+        if not isinstance(root, ast.Name) or self.binder(m, root) is not None:
+            return False
+        if root.id in m.ambiguous or (root.id not in m.top and root.id not in m.imports):
+            return True
+        if root.id in m.top:
+            return False
+        if root is node:  # imported from an app module that does not bind it where the scan can see
+            mod = m.imports[root.id][0]
+            return (mod == "app" or mod.startswith("app.")) and self.lookup(m, root.id) is None
+        found = self.lookup(m, root.id)
+        return (
+            found is not None and found[1] is None
+        )  # an app module whose dotted path went nowhere
+
+    def changed(self, m: Module, stmt: Node, name: str, *, only_read: bool) -> bool:
+        """Something in the app stores into, deletes or rebinds the module-level `name` that `stmt` binds,
+        or (`only_read`, a map) uses it but by `map[k]` or a method that only reads it."""
+        key = id(stmt) * 2 + only_read
+        if key in self._changed:
+            return self._changed[key]
+        own = stmt.targets if isinstance(stmt, ast.Assign) else [getattr(stmt, "target", None)]
+        out = False
+        for cm in self.mods.values():
+            for node in ast.walk(cm.tree):
+                if out:
+                    break
+                if not (
+                    isinstance(node, ast.Name)
+                    and node.id == name
+                    or isinstance(node, ast.Attribute)
+                    and node.attr == name
+                ) or (cm is m and node in own):
+                    continue
+                found = self.lookup_expr(cm, node)
+                if not (found and found[0] is m and found[1] is stmt):
+                    continue
+                if not isinstance(node.ctx, ast.Load):
+                    out = True
+                elif only_read:
+                    p = cm.parents.get(node)
+                    out = not (
+                        isinstance(p, ast.Subscript)
+                        and p.value is node
+                        and isinstance(p.ctx, ast.Load)
+                        or isinstance(p, ast.Attribute)
+                        and p.attr in READS
+                    )
+        self._changed[key] = out
+        return out
+
+    # --- types ---------------------------------------------------------------------------------------
+
+    def ann_values(self, m: Module, ann: Node | None, depth: int = 0) -> set[str] | None:
+        """The values a `Literal[…]` annotation, or an alias of one, allows; None when it is not one."""
+        if isinstance(ann, ast.Subscript) and getattr(ann.value, "id", None) == "Literal":
+            items = ann.slice.elts if isinstance(ann.slice, ast.Tuple) else [ann.slice]
+            if all(isinstance(i, ast.Constant) and isinstance(i.value, str) for i in items):
+                return {i.value for i in items}
+            return None
+        if isinstance(ann, (ast.Name, ast.Attribute)) and depth < 8:
+            found = self.lookup_expr(m, ann)
+            if found and isinstance(found[0], Module) and isinstance(found[1], ast.Assign):
+                return self.ann_values(found[0], found[1].value, depth + 1)
+        return None
+
+    def classes(self, m: Module, e: Node | None) -> list[tuple[Module, ast.ClassDef]] | None:
+        found = self.lookup_expr(m, e) if e is not None else None
+        if found and isinstance(found[0], Module) and isinstance(found[1], ast.ClassDef):
+            return [(found[0], found[1])]
+        return None
+
+    def typed(self, m: Module, at: Node, name: str) -> list[tuple[Module, ast.ClassDef]] | None:
+        """The class `name` holds where `at` sits: the one an `except C as name` catches, or a parameter's
+        annotation (a parameter its body binds again holds no known class)."""
+        handler = self.caught_as(m, at, name)
+        if handler is not None:
+            return self.classes(m, handler.type)
+        fn = self.enclosing(m, at, FUNCS)
+        if fn is None:
+            return None
+        params = {a.arg: a for a in [*fn.args.posonlyargs, *fn.args.args, *fn.args.kwonlyargs]}
+        if name in params and not self.assignments(fn, name):
+            return self.classes(m, params[name].annotation)
+        return None
+
+    def attr_values(self, cm: Module, cls: ast.ClassDef, attr: str) -> set[str] | None:
+        """An attribute's `Literal` values, from `self.attr: T` in a method of the class."""
+        for node in ast.walk(cls):
+            if (
+                isinstance(node, ast.AnnAssign)
+                and isinstance(node.target, ast.Attribute)
+                and node.target.attr == attr
+                and getattr(node.target.value, "id", None) == "self"
+            ):
+                return self.ann_values(cm, node.annotation)
+        return None
+
+    # --- reasons -------------------------------------------------------------------------------------
+
+    def resolve(self, m: Module, e: Node) -> Res:
+        if isinstance(e, ast.Constant):
+            if e.value is None:
+                return Res(none=True)
+            if isinstance(e.value, str):
+                return Res({e.value})
+        if isinstance(e, ast.IfExp):
+            return self.resolve(m, e.body) | self.resolve(m, e.orelse)
+        if isinstance(e, ast.Name):
+            return self.resolve_name(m, e)
+        if isinstance(e, ast.Attribute):
+            return self.resolve_attr(m, e)
+        if isinstance(e, ast.Call):
+            found = self.lookup_expr(m, e.func)
+            if found and isinstance(found[0], Module) and isinstance(found[1], FUNCS):
+                v = self.ann_values(found[0], found[1].returns)
+                if v is not None:
+                    return Res(v)
+        if isinstance(e, ast.Subscript):
+            return self.resolve_lookup(m, e)
+        return Res(unread=[self.at(m, e)])
+
+    def resolve_name(self, m: Module, e: ast.Name) -> Res:
+        if self.caught_as(m, e, e.id) is not None:
+            return Res(unread=[self.at(m, e)])  # the exception itself, not a reason
+        fn = self.binder(m, e)
+        if fn is None:
+            return Res(unread=[self.at(m, e)])
+        if not isinstance(fn, FUNCS):
+            return Res(unread=[f"{self.at(m, e)} (bound by a lambda or a comprehension)"])
+        if any(isinstance(n, (ast.Nonlocal, ast.Global)) and e.id in n.names for n in ast.walk(fn)):
+            return Res(unread=[f"{self.at(m, e)} (named by a nonlocal or a global)"])
+        a = fn.args
+        params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
+        assigned = self.assignments(fn, e.id)
+        if e.id in params:
+            if assigned:
+                return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
+            return self.resolve_param(m, fn, params[e.id], e)
+        out = Res(unread=[] if assigned else [self.at(m, e)])  # `*args`, `**kwargs`
+        for s in assigned:
+            if not (isinstance(s, ast.Assign) and all(isinstance(t, ast.Name) for t in s.targets)):
+                out = out | Res(unread=[self.at(m, e)])
+            elif id(s) in self._resolving:
+                out = out | Res(unread=[f"{self.at(m, e)} (reached again while it is resolved)"])
+            else:
+                self._resolving.add(id(s))
+                out = out | self.resolve(m, s.value)
+                self._resolving.discard(id(s))
+        return out
+
+    def resolve_param(self, m: Module, fn: Node, param: ast.arg, e: Node) -> Res:
+        v = self.ann_values(m, param.annotation)
+        if v is not None:
+            return Res(v)
+        if param.annotation is not None and getattr(param.annotation, "id", None) != "str":
+            return Res(unread=[self.at(m, e)])
+        return self.follow(m, fn, param)
+
+    def follow(self, m: Module, fn: Node, param: ast.arg) -> Res:
+        """A `str` parameter of a helper: what every call of the helper passes it."""
+        assert isinstance(fn, FUNCS)
+        key = (m.name, fn.name, param.arg)
+        if key in self._followed:
+            return self._followed[key]
+        where = f"{m.path}:{fn.lineno}: {fn.name}({param.arg})"
+        if key in self._following:
+            return Res(unread=[f"{where} reached again while it is followed"])
+        if m.top.get(fn.name) is not fn or fn.name in m.ambiguous:
+            return Res(unread=[f"{where} is not a module-level helper"])
+        self._following.add(key)
+        positional = [a.arg for a in [*fn.args.posonlyargs, *fn.args.args]]
+        by_name = param not in fn.args.posonlyargs
+        out = Res()
+        for cm, node, call, resolved in self.references(m, fn):
+            if not resolved:
+                out = out | Res(
+                    unread=[f"{self.at(cm, node)} (a reference the scan cannot resolve)"]
+                )
+                continue
+            if call is None:
+                out = out | Res(unread=[f"{self.at(cm, node)} (the helper used as a value)"])
+                continue
+            kw = {k.arg: k.value for k in call.keywords}
+            given = param.arg in positional and positional.index(param.arg) < len(call.args)
+            if by_name and param.arg in kw:
+                out = out | self.resolve(cm, kw[param.arg])
+            elif any(isinstance(a, ast.Starred) for a in call.args):
+                out = out | Res(unread=[self.at(cm, call)])
+            elif given:
+                out = out | self.resolve(cm, call.args[positional.index(param.arg)])
+            else:
+                out = out | Res(unread=[f"{self.at(cm, call)} (does not pass {param.arg})"])
+        self._followed[key] = out  # once followed, its key answers from here: never reached again
+        return out
+
+    def resolve_attr(self, m: Module, e: ast.Attribute) -> Res:
+        base = e.value
+        if isinstance(base, ast.Name):
+            typed = self.typed(m, e, base.id)
+            if typed is not None:
+                out = Res()
+                for cm, cls in typed:
+                    v = self.attr_values(cm, cls, e.attr)
+                    out = out | (
+                        Res(v)
+                        if v is not None
+                        else Res(
+                            unread=[f"{self.at(m, e)} ({cls.name}.{e.attr} has no Literal type)"]
+                        )
+                    )
+                return out
+        found = self.lookup_expr(m, e)
+        if (
+            found
+            and isinstance(found[0], Module)
+            and isinstance(found[1], (ast.Assign, ast.AnnAssign))
+            and isinstance(found[1].value, ast.Constant)
+        ):
+            if self.changed(found[0], found[1], e.attr, only_read=False):
+                return Res(unread=[f"{self.at(m, e)} (bound again or deleted elsewhere)"])
+            return self.resolve(found[0], found[1].value)
+        return Res(unread=[self.at(m, e)])
+
+    def resolve_lookup(self, m: Module, e: ast.Subscript) -> Res:
+        found = self.lookup_expr(m, e.value)
+        if not (
+            found
+            and isinstance(found[0], Module)
+            and isinstance(found[1], (ast.Assign, ast.AnnAssign))
+        ):
+            return Res(unread=[self.at(m, e)])
+        table = found[1].value
+        if not isinstance(table, ast.Dict) or not all(
+            isinstance(k, ast.Constant) and isinstance(k.value, str) for k in table.keys
+        ):
+            return Res(unread=[self.at(m, e)])
+        name = e.value.attr if isinstance(e.value, ast.Attribute) else getattr(e.value, "id", "")
+        if self.changed(found[0], found[1], name, only_read=True):
+            return Res(unread=[f"{self.at(m, e)} (the map is changed after it is built)"])
+        out = Res()
+        for v in table.values:
+            out = out | self.resolve(found[0], v)
+        index = self.resolve(m, e.slice)
+        keys = {k.value for k in table.keys if isinstance(k, ast.Constant)}
+        if index.unread or index.none or index.values != keys:
+            self.errors.append(
+                f"{self.at(m, e)}: the map's keys {sorted(keys)} are not what its index can be "
+                f"{sorted(index.values)}{' + unread ' + str(index.unread) if index.unread else ''}"
+            )
+        return out
+
+    # --- sites ---------------------------------------------------------------------------------------
+
+    def is_detail(self, node: Node | None) -> bool:
+        if isinstance(node, ast.IfExp):
+            return self.is_detail(node.body) and self.is_detail(node.orelse)
+        return _code_dict(node)
+
+    def factory(self, m: Module, fn: Node) -> bool:
+        """A module-level function each return of which is a detail, and each reference to which is a
+        call sent as one (`quota_detail`)."""
+        assert isinstance(fn, FUNCS)
+        if id(fn) in self._factory:
+            return self._factory[id(fn)]
+        returns = [n for n in _own_scope(fn) if isinstance(n, ast.Return)]
+        ok = (
+            m.top.get(fn.name) is fn  # a name bound twice: its calls are unresolved below
+            and bool(returns)
+            and all(self.is_detail(r.value) for r in returns)
+        )
+        for cm, _, call, resolved in self.references(m, fn) if ok else []:
+            ok = resolved and id(call) in cm.slots  # a bare reference is no call: never sent
+            if not ok:
+                break
+        self._factory[id(fn)] = ok
+        return ok
+
+    def placed(self, m: Module, d: Node) -> bool:
+        """`d` is sent as a detail: from a sender's slot, through the arms of `a if c else b`, or as what a
+        factory returns."""
+        holder, p = d, m.parents.get(d)
+        while isinstance(p, ast.IfExp) and holder is not p.test:
+            holder, p = p, m.parents.get(p)
+        if id(holder) in m.slots:
+            return True
+        fn = self.enclosing(m, p, FUNCS) if isinstance(p, ast.Return) else None
+        return fn is not None and self.factory(m, fn)
+
+    def sayable(self, m: Module, d: Node | None) -> bool:
+        """What a slot holds is a detail the client can say: a code dict, or a factory's call."""
+        if isinstance(d, ast.IfExp):
+            return self.sayable(m, d.body) and self.sayable(m, d.orelse)
+        if _code_dict(d):
+            return True
+        found = self.lookup_expr(m, d.func) if isinstance(d, ast.Call) else None
+        return bool(
+            found
+            and isinstance(found[0], Module)
+            and isinstance(found[1], FUNCS)
+            and self.factory(found[0], found[1])
+        )
+
+    def entries(self, d: Node) -> list[tuple[str | None, Node]]:
+        """A dict's (key, value) pairs: None for a `**` spread, "?" for a key that is not a literal."""
+        if isinstance(d, ast.Dict):
+            return [
+                (
+                    k.value
+                    if isinstance(k, ast.Constant) and isinstance(k.value, str)
+                    else None
+                    if k is None
+                    else "?",
+                    v,
+                )
+                for k, v in zip(d.keys, d.values, strict=True)
+            ]
+        assert isinstance(d, ast.Call)
+        return [(None, a) for a in d.args] + [(k.arg, k.value) for k in d.keywords]
+
+    def site(self, m: Module, d: Node, out: dict) -> None:
+        items = self.entries(d)
+        code = Res()
+        for v in (v for k, v in items if k == "code"):
+            code = code | self.resolve(m, v)
+        problems = [
+            f"{self.at(m, v)} (a spread in dict())"
+            for k, v in items
+            if k is None and isinstance(d, ast.Call)
+        ] + [
+            f"{self.at(m, v)} (a spread or a computed key the code and the reason do not follow)"
+            for i, (k, v) in enumerate(items)
+            if k in (None, "?") and not {"code", "reason"} <= {k2 for k2, _ in items[i + 1 :]}
+        ]
+        if not self.placed(m, d):
+            problems.append(f"{self.at(m, d)} (a detail put where the scan cannot follow it)")
+        if code.unread or code.none or len(code.values) != 1 or problems:
+            out["unreadCodes"] += problems or [self.at(m, d)]
+            return
+        entry = out["codes"].setdefault(
+            next(iter(code.values)), {"reasons": set(), "unread": [], "sites": 0, "reasonless": []}
+        )
+        entry["sites"] += 1
+        res, has_reason = Res(), False
+        for k, v in items:
+            if k == "reason":
+                res, has_reason = res | self.resolve(m, v), True
+        if not has_reason or res.none:
+            entry["reasonless"].append(f"{m.path}:{d.lineno}")
+        entry["reasons"] |= res.values
+        entry["unread"] += res.unread
+
+    def other_details(self) -> list[tuple[str, int, str | None]]:
+        """Each slot that sends what the client cannot say — nothing, or anything but a code dict or a
+        factory's call: (its file, its line, the function it is sent from)."""
+        out = []
+        for m in self.mods.values():
+            for call, d in detail_slots(m.tree):
+                if self.sayable(m, d):
+                    continue
+                fn = self.enclosing(m, call, FUNCS)
+                out.append((m.path, call.lineno, getattr(fn, "name", None)))
+        return sorted(out)
+
+    def run(self) -> dict:
+        out: dict = {
+            "codes": {},
+            "unreadCodes": [],
+            "errors": self.errors,
+            "excluded": {},
+            "aliases": {},
+            "notRefusals": sorted(NOT_REFUSALS),
+        }
+        for m in self.mods.values():
+            for name, stmt in m.top.items():
+                if (
+                    isinstance(stmt, ast.Assign)
+                    and isinstance(stmt.value, ast.Subscript)
+                    and (v := self.ann_values(m, stmt.value))
+                ):
+                    out["aliases"].setdefault(m.path, {})[name] = sorted(v)
+            for node in ast.walk(m.tree):
+                if not _code_dict(node):
+                    continue
+                if m.path in NOT_REFUSALS:
+                    out["excluded"][m.path] = out["excluded"].get(m.path, 0) + 1
+                    continue
+                self.site(m, node, out)
+        for entry in out["codes"].values():
+            entry["reasons"] = sorted(entry["reasons"])
+        return out
+
+
+def scan(sources: dict[str, str]) -> dict:
+    return Scan(sources).run()
+
+
+def app_sources(app: pathlib.Path = APP) -> dict[str, str]:
+    return {
+        p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
+        for p in sorted(app.rglob("*.py"))
+        if "__pycache__" not in p.parts
+    }
+
+
+def scan_app(app: pathlib.Path = APP) -> dict:
+    return scan(app_sources(app))
+
+
+if __name__ == "__main__":
+    json.dump(
+        scan_app(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else APP),
+        sys.stdout,
+        indent=1,
+        sort_keys=True,
+    )

backend/tests/test_abuse_limits.py

@@ -125,3 +125,109 @@ def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_u
     r = c.post("/api/playlists/sync-youtube")
     assert r.status_code == 429
     assert r.json()["detail"] == {"code": "sync_cooldown"}
+
+
+def _pushable_list(db, make_user, monkeypatch, plroutes):
+    """A local (never exported) list and a user with the write scope; the merge itself is stubbed —
+    the limiter is under test, not the YouTube round trip."""
+    from app import auth
+    from app.reader.models import Playlist
+
+    user = make_user()
+    pl = Playlist(user_id=user.id, name="Local", source="local")
+    db.add(pl)
+    db.commit()
+    monkeypatch.setattr(auth, "has_write_scope", lambda u: True)
+    monkeypatch.setattr(plroutes, "plan_merge", lambda d, u, p, live=None: {"units_estimate": 0})
+    monkeypatch.setattr(plroutes, "merge_playlist", lambda d, u, p, live=None: {"added": 0})
+    return user, pl
+
+
+def test_the_push_side_goes_on_cooldown_after_its_allowance(db, as_user, make_user, monkeypatch):
+    # SEC-18: push-plan reads the live list and push writes it, both against the instance-wide
+    # YouTube budget. A press in the UI is one plan + one push, so the allowance is generous — but a
+    # scripted loop stops at it, with a reason the UI can say.
+    from app.routes import playlists as plroutes
+
+    user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
+    c = as_user(user)
+    allowance = plroutes.PUSH_ALLOWANCE
+    codes = [c.get(f"/api/playlists/{pl.id}/push-plan").status_code for _ in range(allowance)]
+    assert codes == [200] * allowance
+    r = c.get(f"/api/playlists/{pl.id}/push-plan")
+    assert r.status_code == 429
+    assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
+
+
+def test_plan_and_push_draw_on_one_allowance(db, as_user, make_user, monkeypatch):
+    # The asymmetric fix again, avoided up front: a limit on one of the two routes leaves the same
+    # drain open next door.
+    from app.routes import playlists as plroutes
+
+    user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
+    c = as_user(user)
+    for i in range(plroutes.PUSH_ALLOWANCE):
+        if i % 2:
+            r = c.get(f"/api/playlists/{pl.id}/push-plan")
+        else:
+            r = c.post(f"/api/playlists/{pl.id}/push")
+        assert r.status_code == 200, (i, r.json())
+    r = c.post(f"/api/playlists/{pl.id}/push")
+    assert r.status_code == 429
+    assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
+
+
+def test_a_refused_push_spends_no_allowance(db, as_user, make_user, monkeypatch):
+    # The allowance counts pushes that run: Watch later (never pushable) and a list another sync
+    # holds are refused before any YouTube call, so they must not leave the next real press at 429.
+    from contextlib import contextmanager
+
+    from app.reader.models import Playlist
+    from app.routes import playlists as plroutes
+
+    user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
+    wl = Playlist(user_id=user.id, name="Watch later", kind="watch_later")
+    db.add(wl)
+    db.commit()
+    c = as_user(user)
+    over = plroutes.PUSH_ALLOWANCE + 1
+    assert {c.get(f"/api/playlists/{wl.id}/push-plan").status_code for _ in range(over)} == {400}
+    assert {c.post(f"/api/playlists/{wl.id}/push").status_code for _ in range(over)} == {400}
+
+    @contextmanager
+    def busy(_db, _id):
+        yield False
+
+    with monkeypatch.context() as m:
+        m.setattr(plroutes, "sync_claim", busy)
+        assert {c.post(f"/api/playlists/{pl.id}/push").status_code for _ in range(over)} == {409}
+    assert c.post(f"/api/playlists/{pl.id}/push").status_code == 200
+
+
+def test_a_mirrored_youtube_list_is_pushable(db, as_user, make_user, monkeypatch):
+    # A mirror of the user's own YouTube list takes local edits (`dirty`) and syncs them back; only
+    # Watch later is refused.
+    from app.reader.models import Playlist
+    from app.routes import playlists as plroutes
+
+    user, _ = _pushable_list(db, make_user, monkeypatch, plroutes)
+    mirror = Playlist(
+        user_id=user.id, name="Mine", source="youtube", yt_playlist_id="PLx", dirty=True
+    )
+    db.add(mirror)
+    db.commit()
+    assert as_user(user).get(f"/api/playlists/{mirror.id}/push-plan").status_code == 200
+
+
+def test_the_push_cooldown_is_per_user(db, as_user, make_user, monkeypatch):
+    from app.routes import playlists as plroutes
+
+    a, pa = _pushable_list(db, make_user, monkeypatch, plroutes)
+    b, pb = _pushable_list(db, make_user, monkeypatch, plroutes)
+    ca = as_user(a)
+    codes = [
+        ca.get(f"/api/playlists/{pa.id}/push-plan").status_code
+        for _ in range(plroutes.PUSH_ALLOWANCE + 1)
+    ]
+    assert codes == [200] * plroutes.PUSH_ALLOWANCE + [429]
+    assert as_user(b).get(f"/api/playlists/{pb.id}/push-plan").status_code == 200

backend/tests/test_activity_reads.py

@@ -159,6 +159,15 @@ def test_the_run_list_filters_pages_and_names_the_channel_and_the_log_lines(db,
     assert facets["total"] == 4
 
 
+def test_the_run_list_names_the_user_a_takeout_import_ran_for(db, make_user, as_user):
+    admin = make_user(role="admin")
+    importer = make_user("importer@example.com")
+    imported = _run(db, "takeout_import", "ok", importer.id, T0, f"user:{importer.id}", "ad_hoc")
+    db.commit()
+    (row,) = as_user(admin).get("/api/admin/runs").json()["items"]
+    assert (row["id"], row["target_title"]) == (imported, "importer@example.com")
+
+
 def test_the_quota_breakdown_covers_its_range_and_one_account(db, make_user, as_user):
     admin = make_user(role="admin")
     now = datetime.now(timezone.utc)

backend/tests/test_auth_session.py

@@ -1,21 +1,25 @@
 """Auth / session security core (TEST-1), DB-lane.
 
-The wallet impersonation gate, epoch-based session revocation, single-use tokens and the
-enumeration-safe uniform 401 are the security mag, and had zero tests (only `_oauth_redirect_uri` was
-covered). Starlette's SessionMiddleware round-trips the signed cookie under TestClient, so the whole
-browser-less flow is exercisable. Uses the real auth stack via the `client` fixture (NO current_user
-override) so login, the wallet gate and epoch revoke are the code under test.
+One account per browser session, epoch-based session revocation, single-use tokens and the
+enumeration-safe uniform 401 are the security mag. Starlette's SessionMiddleware round-trips the
+signed cookie under TestClient, so the whole browser-less flow is exercisable. Uses the real auth
+stack via the `client` fixture (NO current_user override) so login, the session and epoch revoke are
+the code under test.
 
 Note: `/api/me` is the bootstrap probe — it returns 200 `{authenticated: False}` when logged out, not a
 401. A revoked session is therefore checked on a PROTECTED endpoint (`/api/feed`), which does 401.
 """
 
+import base64
+import json
 from datetime import timedelta
+from pathlib import Path
 
 import pytest
 from fastapi.testclient import TestClient
+from itsdangerous import TimestampSigner
 
-from app.auth import ACTIVE_ACCOUNT_HEADER
+from app.config import settings
 from app.main import app
 
 _PW = "correct horse battery"
@@ -46,74 +50,48 @@ def test_wrong_password_and_unknown_email_give_a_byte_identical_401(client, make
     assert wrong_pw.json() == unknown.json()
 
 
-def test_a_non_wallet_account_header_falls_back_to_the_default(client, make_user):
-    # The X-Siftlode-Account header may only re-target an account already signed into THIS browser
-    # (in the session wallet). A header naming someone else must be ignored, not honored — otherwise
-    # it's an impersonation gate hole.
-    make_user(email="alice2@example.com", password=_PW)
-    mallory = make_user(email="mallory@example.com", password=_PW)
-    assert _login(client, "alice2@example.com", _PW).status_code == 200
-    me = client.get("/api/me", headers={ACTIVE_ACCOUNT_HEADER: str(mallory.id)})
-    assert me.status_code == 200
-    assert me.json()["email"] == "alice2@example.com"  # still Alice, not Mallory
+def _cookie(payload: dict) -> str:
+    """A session cookie signed the way SessionMiddleware signs one — for a cookie an older build wrote."""
+    data = base64.b64encode(json.dumps(payload).encode())
+    return TimestampSigner(str(settings.secret_key)).sign(data).decode()
 
 
-def test_a_tab_pinned_to_an_account_signed_out_elsewhere_acts_as_nobody(client, make_user):
-    # Two accounts in one browser: a tab pinned to Alice, Bob the default. Alice signs out in another
-    # tab. The pinned tab used to go on acting as the default, Bob — its player checkpoints and pref
-    # saves landed in his account, and its own Sign out signed HIM out.
-    alice = make_user(email="alice3@example.com", password=_PW)
+def test_signing_in_as_another_account_replaces_the_first(client, make_user):
+    # One account per browser session: a second sign-in is a fresh session, not a second account
+    # beside the first — so signing out leaves nobody signed in here.
+    make_user(email="alice3@example.com", password=_PW)
     make_user(email="bob3@example.com", password=_PW)
     assert _login(client, "alice3@example.com", _PW).status_code == 200
-    assert _login(client, "bob3@example.com", _PW).status_code == 200  # Bob is now the default
-    tab = {ACTIVE_ACCOUNT_HEADER: str(alice.id)}
-    assert client.get("/api/feed", headers=tab).status_code == 200
-    assert client.post("/auth/logout", headers=tab).status_code == 200  # Alice signs out
+    assert _login(client, "bob3@example.com", _PW).status_code == 200
+    assert client.get("/api/me").json()["email"] == "bob3@example.com"
 
-    assert client.get("/api/feed", headers=tab).status_code == 401
-    assert client.post("/auth/logout", headers=tab).status_code == 200
-    assert client.get("/api/feed").status_code == 200  # Bob is still signed in, the session intact
-    # The bootstrap probe still answers as the default: how the stale tab learns who it is now.
-    assert client.get("/api/me", headers=tab).json()["email"] == "bob3@example.com"
+    assert client.post("/auth/logout").status_code == 200
+    assert client.get("/api/feed").status_code == 401
+    assert client.get("/api/me").json() == {"authenticated": False}
 
 
-@pytest.mark.parametrize("ended", ["revoked", "suspended"])
-def test_a_tab_pinned_to_an_account_whose_session_ended_comes_back_as_the_default(
-    client, make_user, db, ended
-):
-    # A tab pinned to Alice, Bob the default. Alice's session then ends while she is still in this
-    # browser's wallet ("log out everywhere" on another device, or an admin suspends her). The tab's
-    # bootstrap probe used to answer "signed out": the landing page with Bob still signed in, and no way
-    # off it but closing the tab (signing in as anyone else there kept the tab pinned to Alice).
+def test_no_header_or_parameter_picks_another_account(client, make_user):
+    # Alice signed in here before Bob: nothing a request carries makes it act as her any more.
     alice = make_user(email="alice4@example.com", password=_PW)
     make_user(email="bob4@example.com", password=_PW)
     assert _login(client, "alice4@example.com", _PW).status_code == 200
     assert _login(client, "bob4@example.com", _PW).status_code == 200
-    tab = {ACTIVE_ACCOUNT_HEADER: str(alice.id)}
-    assert client.get("/api/feed", headers=tab).status_code == 200
-    if ended == "revoked":
-        alice.session_epoch = (alice.session_epoch or 0) + 1
-    else:
-        alice.is_suspended = True
-    db.commit()
+    me = client.get(f"/api/me?account={alice.id}", headers={"X-Siftlode-Account": str(alice.id)})
+    assert me.json()["email"] == "bob4@example.com"
 
-    assert client.get("/api/me", headers=tab).json()["email"] == "bob4@example.com"
-    assert client.get("/api/feed", headers=tab).status_code == 401  # still nobody, never Bob
-    assert client.get("/api/feed").status_code == 200  # Bob's session is intact
+
+@pytest.mark.parametrize("path", ["/api/me/accounts", "/api/me/switch"])
+def test_there_is_no_account_list_or_switch(client, make_user, path):
+    make_user(email="alice5@example.com", password=_PW)
+    assert _login(client, "alice5@example.com", _PW).status_code == 200
+    r = client.get(path) if path.endswith("accounts") else client.post(path, json={"user_id": 1})
+    assert r.status_code in (404, 405)
 
 
 @pytest.mark.parametrize("ended", ["revoked", "suspended", "deactivated"])
-def test_switching_to_an_account_whose_session_ended_refuses_it_and_drops_it_from_the_list(
-    client, make_user, db, monkeypatch, ended
-):
-    # The account switcher lists this browser's wallet. A revoked account was dropped from it on a refused
-    # switch, but a suspended one stayed, refused on every click; a deactivated one was not refused at all —
-    # made the default, the next request found it inactive and signed every account here out.
-    monkeypatch.setattr("app.routes.me.is_allowed", lambda _db, _email: True)  # both hold an invite
-    alice = make_user(email="alice5@example.com", password=_PW)
-    make_user(email="bob5@example.com", password=_PW)
-    assert _login(client, "alice5@example.com", _PW).status_code == 200
-    assert _login(client, "bob5@example.com", _PW).status_code == 200
+def test_a_session_whose_account_ended_signs_the_browser_out(client, make_user, db, ended):
+    alice = make_user(email="alice6@example.com", password=_PW)
+    assert _login(client, "alice6@example.com", _PW).status_code == 200
     if ended == "revoked":
         alice.session_epoch = (alice.session_epoch or 0) + 1
     elif ended == "suspended":
@@ -121,10 +99,111 @@ def test_switching_to_an_account_whose_session_ended_refuses_it_and_drops_it_fro
     else:
         alice.is_active = False
     db.commit()
+    assert client.get("/api/feed").status_code == 401
+    assert "user_id" not in _session(client)  # the refusal ended the session, not just this request
+    assert client.get("/api/me").json() == {"authenticated": False}
 
-    assert client.post("/api/me/switch", json={"user_id": alice.id}).status_code == 403
-    assert [a["email"] for a in client.get("/api/me/accounts").json()] == ["bob5@example.com"]
-    assert client.get("/api/feed").status_code == 200  # Bob is still signed in
+
+def test_a_cookie_from_the_multi_account_build_signs_in_again(client, make_user):
+    # The old session held a wallet and one epoch per account. It is not read: whoever holds one
+    # signs in once more.
+    alice = make_user(email="alice7@example.com", password=_PW)
+    old = {"user_id": alice.id, "account_ids": [alice.id], "epochs": {str(alice.id): 0}}
+    client.cookies.set("session", _cookie(old))
+    assert client.get("/api/feed").status_code == 401
+    assert client.get("/api/me").json() == {"authenticated": False}
+
+
+def test_deleting_the_account_signs_the_browser_out(client, make_user):
+    make_user(email="alice8@example.com", password=_PW)
+    assert _login(client, "alice8@example.com", _PW).status_code == 200
+    assert client.delete("/api/me/account").status_code == 200
+    # The cookie names nobody now: the probe answers signed out without looking the account up.
+    assert "user_id" not in _session(client)
+
+
+def _session(client) -> dict:
+    """The decoded payload of the client's session cookie."""
+    raw = client.cookies.get("session")
+    if not raw:
+        return {}
+    data = TimestampSigner(str(settings.secret_key)).unsign(raw.encode())
+    return json.loads(base64.b64decode(data))
+
+
+@pytest.mark.parametrize("signed_in_as", ["starter", "someone_else"])
+def test_a_google_link_lands_only_on_the_account_still_signed_in(db, make_user, signed_in_as):
+    # The link marker names who started it; a sign-in as someone else before the callback must not
+    # get that Google identity attached to the account that started it from the other person's session.
+    from starlette.requests import Request
+
+    from app.routes.auth import _complete_link
+
+    starter, other = make_user(), make_user()
+    session = {"user_id": starter.id if signed_in_as == "starter" else other.id, "epoch": 0}
+    request = Request({"type": "http", "session": session, "headers": []})
+    resp = _complete_link(request, db, starter.id, True, "sub-123", {"email": ""}, {"scope": ""})
+    db.refresh(starter)
+    if signed_in_as == "starter":
+        assert resp.headers["location"] == "/?link=ok"
+        assert starter.google_sub == "sub-123"
+    else:
+        assert resp.headers["location"] == "/?link=error"
+        assert starter.google_sub is None
+
+
+def test_no_app_code_reads_a_second_account():
+    # The multi-account machinery is gone; a module reading its session keys or its header again
+    # would bring back a session that names more than one account.
+    app_dir = Path(__file__).parent.parent / "app"
+    hits = [
+        f"{path.relative_to(app_dir)}: {word}"
+        for path in app_dir.rglob("*.py")
+        for word in ("account_ids", '"epochs"', "x-siftlode-account", 'query_params.get("account")')
+        if word in path.read_text(encoding="utf-8").lower()
+    ]
+    assert hits == []
+
+
+def test_a_tab_built_for_the_account_signed_in_before_is_refused(client, make_user):
+    # Two tabs of one browser: one signed in as someone else, so the cookie is Bob's under both. The
+    # tab still showing Alice says so with every request; acting as Bob there would put her player
+    # positions, pref saves and Sign out on his account. Refused, and the session stays Bob's.
+    alice = make_user(email="alice5@example.com", password=_PW)
+    bob = make_user(email="bob5@example.com", password=_PW)
+    assert _login(client, "alice5@example.com", _PW).status_code == 200
+    assert _login(client, "bob5@example.com", _PW).status_code == 200
+
+    stale = client.get("/api/feed", headers={"X-Siftlode-Expect-Account": str(alice.id)})
+    assert stale.status_code == 401
+    assert stale.json()["detail"]["reason"] == "account_changed"
+    assert (
+        client.get("/api/feed", headers={"X-Siftlode-Expect-Account": str(bob.id)}).status_code
+        == 200
+    )
+    assert client.get("/api/feed").status_code == 200
+    # Nor may its Sign out end Bob's session.
+    assert (
+        client.post(
+            "/auth/logout", headers={"X-Siftlode-Expect-Account": str(alice.id)}
+        ).status_code
+        == 401
+    )
+    # The probe answers who is signed in now, so the stale tab can reload as Bob.
+    probe = client.get("/api/me", headers={"X-Siftlode-Expect-Account": str(alice.id)})
+    assert probe.json()["email"] == "bob5@example.com"
+
+
+def test_setting_a_password_keeps_the_caller_signed_in(client, make_user):
+    # The change ends every OTHER session (the epoch moves); the caller is stamped with the new one.
+    make_user(email="dave@example.com", password=_PW)
+    assert _login(client, "dave@example.com", _PW).status_code == 200
+    r = client.post(
+        "/auth/set-password",
+        json={"password": "a brand new passphrase", "current_password": _PW},
+    )
+    assert r.status_code == 200
+    assert client.get("/api/feed").status_code == 200
 
 
 def test_logout_others_revokes_the_other_session_but_keeps_the_caller(client, make_user):

backend/tests/test_body_limit.py

@@ -23,6 +23,7 @@ from app import bodylimit, debugsession, sysconfig
 from app.config import settings
 from app.models import SystemConfig
 from app.bodylimit import DEFAULT_MAX_BODY_BYTES, BodyLimitMiddleware
+from app.reader import takeout
 from app.routes import diagnostics as diagnostics_routes
 
 SMALL = b"x" * 16
@@ -221,6 +222,7 @@ def test_the_default_ceiling_clears_the_largest_body_a_default_route_takes():
 CAPPED_BODY_SITES = {
     ("diagnostics.py", "_parsed_events", "capped_body", "BODY_MAX_BYTES"),
     ("debug_sessions.py", "_image_body", "capped_body", "debugsession.SCREENSHOT_MAX_BYTES"),
+    ("feed.py", "_takeout_body", "capped_body", "takeout.MAX_BYTES"),
     ("files.py", "spool_upload", "capped_stream_to", "cap"),
 }
 
@@ -368,6 +370,9 @@ def test_every_body_reading_site_is_known_and_fits_under_its_ceiling():
     # The upload site's limit is the sysconfig knob, which is bounded BY the edge ceiling rather
     # than being a constant to compare against — that pairing is its own test below.
     assert ("POST", "/api/files/upload") in bodylimit.ROUTE_MAX_BODY_BYTES
+    # A Takeout file is larger than the default: without its ceiling every real history is a 413.
+    assert takeout.MAX_BYTES > DEFAULT_MAX_BODY_BYTES
+    assert bodylimit.ROUTE_MAX_BODY_BYTES[("POST", "/api/takeout")] == takeout.MAX_BYTES
 
 
 def test_release_read_connection_ends_the_transaction_it_was_called_for(db):

backend/tests/test_catalog_routes.py

@@ -72,3 +72,23 @@ def test_the_sync_status_counts_the_callers_subscriptions(db, make_user, as_user
     assert (body["subscriptions"], body["channels_total"]) == (1, 1)
     mine = as_user(user).get("/api/sync/my-status").json()
     assert (mine["channels_total"], mine["channels_recent_pending"]) == (1, 1)
+
+
+def test_a_channel_lists_when_it_entered_this_users_list_not_when_the_channel_did(
+    db, make_user, as_user
+):
+    from datetime import datetime, timezone
+
+    user = make_user()
+    db.add(Channel(id="UCa", title="A", created_at=datetime(2026, 6, 10, tzinfo=timezone.utc)))
+    db.flush()
+    db.add(
+        Subscription(
+            user_id=user.id,
+            channel_id="UCa",
+            created_at=datetime(2026, 10, 12, 8, 30, tzinfo=timezone.utc),
+        )
+    )
+    db.commit()
+    listed = as_user(user).get("/api/channels").json()
+    assert [(c["id"], c["added_at"]) for c in listed] == [("UCa", "2026-10-12T08:30:00+00:00")]

backend/tests/test_catalog_service.py

@@ -12,6 +12,7 @@ from datetime import datetime, timedelta, timezone
 from sqlalchemy import select
 
 from app.catalog import queries, search, service
+from app.db import SessionLocal
 from app.catalog.models import (
     BlockedChannel,
     Channel,
@@ -22,6 +23,8 @@ from app.catalog.models import (
 )
 from app.reader import queries as reader_queries
 from app.reader import service as reader_service
+from app.sync import explore as sync_explore
+from app.sync import subscriptions as sync_subscriptions
 
 
 def _channel(db, cid="UCc", **kw):
@@ -141,6 +144,89 @@ def test_subscribing_keeps_an_explored_channel_and_its_videos(db, make_user):
     assert queries.subscription(db, user, "UCc") is None
 
 
+def test_exploring_a_channel_someone_follows_brings_its_uploads_in_as_the_catalogs_own(
+    db, make_user, monkeypatch
+):
+    follower, explorer = make_user(), make_user()
+    ch = _channel(db, uploads_playlist_id="UUc")
+    db.add(Subscription(user_id=follower.id, channel_id="UCc"))
+    db.commit()
+    service.explore(db, explorer, ch)
+    db.commit()
+
+    class _Yt:
+        def get_playlist_items(self, playlist_id, page_token):
+            item = {"contentDetails": {"videoId": "v1", "videoPublishedAt": "2026-05-01T00:00:00Z"}}
+            return {"items": [item]}
+
+    monkeypatch.setattr(sync_explore, "_enrich_new", lambda session, yt, ids: 0)
+    sync_explore.explore_ingest_page(db, _Yt(), ch)
+    db.commit()
+    db.expire_all()
+    # The channel is followed, so not the exploration's: a video flagged as one would stay flagged
+    # (subscribing clears only an explored channel's) and its watched marks would wait for good.
+    assert (ch.from_explore, db.get(Video, "v1").via_explore) == (False, False)
+
+
+def test_a_follow_pulled_in_from_youtube_makes_an_explored_channel_the_catalogs_own(
+    db, make_user, monkeypatch
+):
+    explorer, follower = make_user(), make_user()
+    ch = _channel(db, details_synced_at=datetime.now(timezone.utc))
+    service.explore(db, explorer, ch)
+    db.commit()
+    _video(db, "v1", via_explore=True)
+
+    class _Yt:
+        def __init__(self, session, user):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def iter_subscriptions(self):
+            yield {"channel_id": "UCc", "title": "UCc", "yt_subscription_id": "sub-UCc"}
+
+        def get_channels(self, ids):
+            return []
+
+    monkeypatch.setattr(sync_subscriptions, "YouTubeClient", _Yt)
+    sync_subscriptions.import_subscriptions(db, follower)
+    db.commit()
+    db.expire_all()
+    # Followed on youtube.com is followed all the same: left flagged, the channel's later videos
+    # came in as an exploration's, hidden from the Library and their watched marks waiting for good.
+    assert (db.get(Channel, "UCc").from_explore, db.get(Video, "v1").via_explore) == (False, False)
+
+
+def test_a_follow_landing_while_an_explored_page_comes_in_leaves_none_of_it_flagged(
+    db, make_user, monkeypatch
+):
+    explorer, follower = make_user(), make_user()
+    ch = _channel(db, uploads_playlist_id="UUc")
+    service.explore(db, explorer, ch)
+    db.commit()
+
+    class _Yt:
+        def get_playlist_items(self, playlist_id, page_token):
+            # Someone subscribes, in a request of their own, while YouTube answers this one.
+            with SessionLocal() as other:
+                service.record_subscription(other, follower, other.get(Channel, "UCc"), "sub-UCc")
+                other.commit()
+            item = {"contentDetails": {"videoId": "v1", "videoPublishedAt": "2026-05-01T00:00:00Z"}}
+            return {"items": [item]}
+
+    monkeypatch.setattr(sync_explore, "_enrich_new", lambda session, yt, ids: 0)
+    sync_explore.explore_ingest_page(db, _Yt(), ch)
+    db.commit()
+    db.expire_all()
+    # The subscribe cleared what was flagged before it; a page that lands after it must not be.
+    assert (db.get(Channel, "UCc").from_explore, db.get(Video, "v1").via_explore) == (False, False)
+
+
 def test_a_block_is_once_and_ends_the_exploration(db, make_user):
     user = make_user()
     ch = _channel(db)

backend/tests/test_clip.py

@@ -531,6 +531,9 @@ def test_a_youtube_clip_hands_the_range_to_yt_dlp(monkeypatch, tmp_path):
         def __enter__(self):
             return self
 
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
         def __exit__(self, *_a):
             return False
 

backend/tests/test_discover.py

@@ -1105,10 +1105,16 @@ def test_two_overlapping_reorders_never_share_a_slot(db, as_user):
 
         worker = threading.Thread(target=second_tab)
         worker.start()
-        # Commit A only once B is blocked on a row lock, so B's read is the stale snapshot.
+        # Commit A only once B is blocked on a row lock, so B's read is the stale snapshot. pg_locks is
+        # the whole server's: only a wait in this database is B's (a parallel worker's is not).
         waiting = 0
         for _ in range(200):
-            waiting = probe.scalar(text("SELECT count(*) FROM pg_locks WHERE NOT granted"))
+            waiting = probe.scalar(
+                text(
+                    "SELECT count(*) FROM pg_locks WHERE NOT granted AND pid IN"
+                    " (SELECT pid FROM pg_stat_activity WHERE datname = current_database())"
+                )
+            )
             probe.rollback()
             if waiting:
                 break

backend/tests/test_edit_crop_frame.py

@@ -0,0 +1,209 @@
+"""The editor's crop lands on the part of the picture the frame was drawn on, also when a video's
+pixels are not square. The browser reports such a video by its shown size (320 stored, 640 shown),
+and ffmpeg's `crop` counts stored pixels: measured on a real clip, a frame over the middle half kept
+half the frame's red surroundings. So a crop carries the size it was measured on.
+
+Real ffmpeg: the pixels of the output are the oracle.
+"""
+
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+from app.config import settings
+from app.downloads import edit, pipeline, service, storage
+from app.downloads import upload as uploadmod
+from app.downloads.models import DownloadJob, MediaAsset
+
+pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
+
+
+def _source(path: Path, size: str, box: str, sar: int, rotate: int = 0) -> None:
+    """Red, with a green box: a crop of the box holds no red. `rotate` tags the file to be shown
+    turned by that many degrees counter-clockwise, as a phone records a portrait video."""
+    tag = ["-display_rotation:v:0", str(rotate)] if rotate else []
+    subprocess.run(
+        [
+            "ffmpeg", "-y", "-loglevel", "error",
+            "-f", "lavfi", "-i", f"color=c=red:s={size}:d=2:r=10",
+            "-f", "lavfi", "-i", "sine=duration=2",
+            "-vf", f"drawbox={box}:c=lime:t=fill,setsar={sar}",
+            "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path),
+        ],
+        check=True,
+    )  # fmt: skip
+    if tag:
+        tagged = path.with_name("tagged-" + path.name)
+        subprocess.run(
+            ["ffmpeg", "-y", "-loglevel", "error", *tag, "-i", str(path), "-c", "copy", str(tagged)],
+            check=True,
+        )  # fmt: skip
+        tagged.replace(path)
+
+
+def _red_and_green(path: Path) -> tuple[int, int]:
+    raw = subprocess.run(
+        ["ffmpeg", "-loglevel", "error", "-i", str(path), "-frames:v", "1",
+         "-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
+        check=True,
+        capture_output=True,
+    ).stdout  # fmt: skip
+    red = green = 0
+    for i in range(0, len(raw) - 2, 3):
+        if raw[i] > 180 and raw[i + 1] < 80:
+            red += 1
+        elif raw[i + 1] > 180 and raw[i] < 80:
+            green += 1
+    return red, green
+
+
+def _cut_asset(db, make_user, src: Path, spec: dict, then=lambda: None) -> MediaAsset:
+    user = make_user()
+    source = uploadmod.ingest_upload(
+        db, user.id, filename="clip.mp4", src_path=src, size=src.stat().st_size
+    )
+    assert source.status == "done"
+    then()
+    job = service.enqueue_edit(db, user.id, source, spec)
+    asset = db.get(MediaAsset, job.asset_id)
+    job.status, asset.status = "running", "downloading"  # the worker's claim
+    db.commit()
+
+    pipeline._process_edit(job.id, asset.id)
+
+    db.expire_all()
+    job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
+    assert (job.status, job.error, asset.status) == ("done", None, "ready")
+    return asset
+
+
+def _cut(db, make_user, tmp_path, src: Path, crop: dict, **spec) -> Path:
+    asset = _cut_asset(db, make_user, src, {"crop": crop, **spec})
+    return storage.abs_path(str(tmp_path), asset.rel_path)
+
+
+def test_a_crop_of_a_video_with_wide_pixels_keeps_what_the_frame_showed(
+    db, make_user, tmp_path, monkeypatch
+):
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    _source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
+
+    # What the editor sends for the middle half of this video: the browser shows it at 640 x 240.
+    final = _cut(
+        db, make_user, tmp_path, src,
+        {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
+    )  # fmt: skip
+
+    red, green = _red_and_green(final)
+    assert (red, green) == (0, 160 * 120)
+
+
+def test_a_crop_of_square_pixels_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    _source(src, "480x640", "x=120:y=160:w=240:h=320", sar=1)
+
+    final = _cut(
+        db, make_user, tmp_path, src,
+        {"x": 120, "y": 160, "w": 240, "h": 320, "frame_w": 480, "frame_h": 640},
+    )  # fmt: skip
+
+    assert _red_and_green(final) == (0, 240 * 320)
+
+
+def test_a_crop_stored_before_it_carried_its_frame_is_cut_as_it_always_was():
+    spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50}})
+    assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
+    cmd = edit.build_edit_cmd(Path("in.mp4"), Path("out.mp4"), spec, "mp4")
+    assert cmd[cmd.index("-vf") + 1].startswith("crop=100:50:4:6,")
+
+
+@pytest.mark.parametrize("frame", [{"frame_w": 0, "frame_h": 240}, {"frame_w": "x", "frame_h": 2}])
+def test_a_frame_that_cannot_be_read_is_not_divided_by(frame):
+    spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50, **frame}})
+    assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
+
+
+def test_a_crop_of_a_join_keeps_what_the_frame_showed(db, make_user, tmp_path, monkeypatch):
+    """A cut-list joins through `filter_complex`, a crop path of its own."""
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    _source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
+
+    final = _cut(
+        db, make_user, tmp_path, src,
+        {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
+        segments=[{"start_s": 0, "end_s": 0.5}, {"start_s": 1, "end_s": 1.5}], accurate=True,
+    )  # fmt: skip
+
+    assert _red_and_green(final) == (0, 160 * 120)
+
+
+def test_a_crop_of_a_rotated_video_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
+    """The browser shows a rotated file upright, so the frame was drawn on the upright picture."""
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    # Stored 320 x 240, green on its left half; shown turned a quarter, 240 x 320.
+    _source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
+
+    # Turned counter-clockwise, the stored left half is the shown bottom half.
+    final = _cut(
+        db, make_user, tmp_path, src,
+        {"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320},
+    )  # fmt: skip
+
+    assert _red_and_green(final) == (0, 240 * 160)
+
+
+def test_a_clip_ffprobe_cannot_size_gets_the_size_its_crop_cut(
+    db, make_user, tmp_path, monkeypatch
+):
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    _source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
+
+    def probe_fails():
+        monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
+
+    asset = _cut_asset(
+        db, make_user, src,
+        {"crop": {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240}},
+        then=probe_fails,
+    )  # fmt: skip
+
+    # Half the shown width of a 320-wide file: 160 stored, not the 320 the frame counted.
+    assert (asset.width, asset.height) == (160, 120)
+
+
+def test_a_rotated_clip_ffprobe_cannot_size_gets_the_upright_size_its_crop_cut(
+    db, make_user, tmp_path, monkeypatch
+):
+    """The frame was drawn on the upright picture, which is the stored one turned: so is its cut."""
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    src = tmp_path / "in.mp4"
+    _source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
+
+    def probe_fails():
+        monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
+
+    asset = _cut_asset(
+        db, make_user, src,
+        {"crop": {"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320}},
+        then=probe_fails,
+    )  # fmt: skip
+
+    # What ffmpeg cut (the rotated test above): 240 x 160, not the stored 320 x 120.
+    assert (asset.width, asset.height) == (240, 160)
+
+
+@pytest.mark.parametrize("key", ["frame_w", "x"])
+def test_an_infinite_number_in_a_crop_is_dropped_not_a_crash(key):
+    crop = {"x": 4, "y": 6, "w": 100, "h": 50, "frame_w": 640, "frame_h": 240}
+    spec = edit.normalize_edit_spec({"crop": {**crop, key: float("inf")}})
+    if key == "frame_w":
+        assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
+    else:
+        assert "crop" not in spec

backend/tests/test_egress.py

@@ -0,0 +1,412 @@
+"""SEC-17: the egress gate judges the address a guarded download actually dials.
+
+Two loopback addresses stand in for the two sides: 127.0.0.2 plays a public host, 127.0.0.1 our own
+network. The gate under test refuses exactly 127.0.0.1, so every assertion is about the judgement and
+the wiring, with real sockets, a real yt-dlp and a real ffmpeg — no mocked transport.
+"""
+
+import contextlib
+import http.server
+import ipaddress
+import socket
+import socketserver
+import struct
+import threading
+import time
+from pathlib import Path
+
+import pytest
+import yt_dlp
+from yt_dlp.utils import download_range_func
+
+from app.downloads import adapters, egress, errors, formats
+
+PUBLIC = "127.0.0.2"
+INTERNAL = "127.0.0.1"
+
+
+def _forbid_internal(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
+    return str(ip) == INTERNAL
+
+
+@pytest.fixture
+def gate():
+    g = egress.start_gate(forbidden=_forbid_internal)
+    yield g
+    g.shutdown()
+    g.server_close()
+
+
+class _Listener:
+    """A TCP listener that only counts who connected — the internal service under attack."""
+
+    def __init__(self, host: str = INTERNAL, port: int = 0):
+        self.hits = 0
+        self.sock = socket.socket()
+        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        self.sock.bind((host, port))
+        self.sock.listen()
+        self.port = self.sock.getsockname()[1]
+        threading.Thread(target=self._serve, daemon=True).start()
+
+    def _serve(self):
+        while True:
+            try:
+                conn, _ = self.sock.accept()
+            except OSError:
+                return
+            self.hits += 1
+            with contextlib.suppress(OSError):
+                conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
+            conn.close()
+
+    def close(self):
+        self.sock.close()
+
+
+def _public_server(routes: dict, port: int = 0, directory: Path | None = None):
+    """An HTTP server on the 'public' address: `routes` maps a path to (status, headers); anything
+    else is served from `directory`."""
+
+    class H(http.server.SimpleHTTPRequestHandler):
+        def __init__(self, *a, **k):
+            super().__init__(*a, directory=str(directory or "."), **k)
+
+        def log_message(self, *a):
+            pass
+
+        def do_GET(self):
+            if self.path in routes:
+                status, headers, body = routes[self.path]
+                self.send_response(status)
+                for k, v in headers.items():
+                    self.send_header(k, v)
+                self.send_header("Content-Length", str(len(body)))
+                self.end_headers()
+                self.wfile.write(body)
+                return
+            super().do_GET()
+
+    srv = socketserver.ThreadingTCPServer((PUBLIC, port), H)
+    srv.daemon_threads = True
+    threading.Thread(target=srv.serve_forever, daemon=True).start()
+    return srv
+
+
+def _guarded_opts(tmp_path: Path, gate_url: str, **extra) -> dict:
+    opts = formats.build_ydl_opts({}, str(tmp_path / "%(id)s.%(ext)s"), lambda d: None, egress=True)
+    opts.update(retries=0, fragment_retries=0, extractor_retries=0, socket_timeout=5)
+    opts["proxy"] = gate_url  # the test's gate (it lets 127.0.0.2 through), not the worker's
+    opts.update(extra)
+    return opts
+
+
+def _download_error(opts: dict, url: str) -> str:
+    with pytest.raises(yt_dlp.utils.DownloadError) as exc, egress.open_ydl(opts) as ydl:
+        ydl.extract_info(url, download=True)
+    return str(exc.value)
+
+
+# --- the judgement ---------------------------------------------------------------------------------
+
+
+def test_a_name_with_any_forbidden_answer_is_refused():
+    def resolve(answers):
+        return lambda host, port, *a: [(2, 1, 6, "", (ip, port)) for ip in answers]
+
+    assert egress.vetted_address(
+        "x", 80, resolve=resolve([PUBLIC]), forbidden=_forbid_internal
+    ) == (
+        PUBLIC,
+        80,
+    )
+    assert (
+        egress.vetted_address(
+            "x", 80, resolve=resolve([PUBLIC, INTERNAL]), forbidden=_forbid_internal
+        )
+        is None
+    )
+
+    def no_answer(*a):
+        raise socket.gaierror("no such name")
+
+    assert egress.vetted_address("x", 80, resolve=no_answer, forbidden=_forbid_internal) is None
+
+
+def test_the_real_judgement_refuses_our_own_networks():
+    # The worker's gate uses urlguard's judgement unchanged — the same one the enqueue check uses.
+    for addr in ("127.0.0.1", "10.1.2.3", "172.18.0.5", "192.168.1.1", "169.254.169.254"):
+        assert egress.vetted_address(addr, 80, forbidden=egress.urlguard.is_forbidden) is None, addr
+    assert egress.vetted_address("93.184.215.14", 80, forbidden=egress.urlguard.is_forbidden)
+
+
+def test_the_gate_dials_the_address_it_vetted_not_a_second_answer(gate):
+    # DNS rebinding: a name that answers public for the check and private for the fetch. The gate
+    # resolves once per connection and dials that answer.
+    public = _public_server({"/": (200, {}, b"public")})
+    internal = _Listener(INTERNAL, public.server_address[1])
+    calls = []
+
+    def rebinding(host, port, *a):
+        calls.append(host)
+        ip = PUBLIC if len(calls) == 1 else INTERNAL
+        return [(2, 1, 6, "", (ip, port))]
+
+    gate.resolve = rebinding
+    try:
+        with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
+            s.sendall(
+                f"GET http://rebind.test:{public.server_address[1]}/ HTTP/1.1\r\n"
+                f"Host: rebind.test\r\n\r\n".encode()
+            )
+            answer = b"".join(iter(lambda: s.recv(4096), b""))
+        assert answer.endswith(b"public")
+        assert calls == ["rebind.test"]
+        assert internal.hits == 0
+    finally:
+        public.shutdown()
+        internal.close()
+
+
+def test_a_tunnel_into_the_network_is_refused_before_it_is_dialled(gate):
+    internal = _Listener()
+    try:
+        with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
+            s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
+            answer = s.recv(4096)
+        assert answer.startswith(b"HTTP/1.1 403")
+        assert internal.hits == 0
+    finally:
+        internal.close()
+
+
+def test_a_client_that_hangs_up_mid_transfer_is_not_an_error(gate, monkeypatch):
+    # yt-dlp closes a connection once it has read what it needed; seen live as a traceback the gate
+    # printed for every such download.
+    errors_seen = []
+    monkeypatch.setattr(gate, "handle_error", lambda *a: errors_seen.append(a))
+    public = _public_server({"/big": (200, {}, b"x" * 4_000_000)})
+    try:
+        s = socket.create_connection(("127.0.0.1", gate.server_address[1]))
+        s.sendall(f"GET http://{PUBLIC}:{public.server_address[1]}/big HTTP/1.1\r\n\r\n".encode())
+        s.recv(1024)
+        s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
+        s.close()  # a reset, not a polite close
+        time.sleep(0.5)
+        assert errors_seen == []
+    finally:
+        public.shutdown()
+
+
+# --- a real yt-dlp through it ----------------------------------------------------------------------
+
+
+def test_a_redirect_into_the_network_reaches_it_unguarded_and_not_through_the_gate(tmp_path, gate):
+    internal = _Listener()
+    public = _public_server(
+        {"/v.mp4": (302, {"Location": f"http://{INTERNAL}:{internal.port}/v.mp4"}, b"")}
+    )
+    url = f"http://{PUBLIC}:{public.server_address[1]}/v.mp4"
+    try:
+        # The hole, as it was: yt-dlp follows the 3xx into our network.
+        bare = {"outtmpl": str(tmp_path / "bare" / "%(id)s.%(ext)s"), "quiet": True, "retries": 0}
+        with contextlib.suppress(yt_dlp.utils.DownloadError), yt_dlp.YoutubeDL(bare) as ydl:
+            ydl.extract_info(url, download=True)
+        assert internal.hits >= 1
+        before = internal.hits
+
+        _download_error(_guarded_opts(tmp_path, gate.url), url)
+        assert internal.hits == before
+    finally:
+        public.shutdown()
+        internal.close()
+
+
+def test_a_live_and_a_dead_internal_port_leave_the_same_text(tmp_path, gate):
+    live = _Listener()
+    dead = _Listener()
+    dead_port = dead.port
+    dead.close()
+    public = _public_server(
+        {
+            "/live.mp4": (302, {"Location": f"http://{INTERNAL}:{live.port}/x"}, b""),
+            "/dead.mp4": (302, {"Location": f"http://{INTERNAL}:{dead_port}/x"}, b""),
+        }
+    )
+    base = f"http://{PUBLIC}:{public.server_address[1]}"
+    try:
+        stored = []
+        for name in ("live", "dead"):
+            msg = _download_error(_guarded_opts(tmp_path / name, gate.url), f"{base}/{name}.mp4")
+            stored.append(errors.user_error_text("url", msg, errors.classify(msg)))
+        assert stored[0] == stored[1]
+        assert live.hits == 0
+    finally:
+        public.shutdown()
+        live.close()
+
+
+def test_a_format_fetched_past_the_gate_is_never_chosen(tmp_path):
+    # ffmpeg speaks rtmp, rtsp, tcp… directly, where no HTTP proxy is asked. A guarded download keeps
+    # only the formats fetched over http(s).
+    opts = _guarded_opts(tmp_path, "http://127.0.0.1:9")
+    info = {
+        "id": "x",
+        "title": "x",
+        "extractor": "generic",
+        "extractor_key": "Generic",
+        "webpage_url": f"http://{PUBLIC}/page",
+        "formats": [
+            {"format_id": "rtmp", "url": f"rtmp://{INTERNAL}/live", "ext": "flv", "height": 1080},
+            {"format_id": "web", "url": f"http://{PUBLIC}/a.mp4", "ext": "mp4", "height": 360},
+        ],
+    }
+    with egress.open_ydl(opts) as ydl:
+        chosen = ydl.process_ie_result(dict(info), download=False)
+    assert chosen["format_id"] == "web"
+
+    only_rtmp = dict(info, formats=[info["formats"][0]])
+    with (
+        pytest.raises(yt_dlp.utils.YoutubeDLError, match="Requested format is not available"),
+        egress.open_ydl(opts) as ydl,
+    ):
+        ydl.process_ie_result(only_rtmp, download=False)
+
+
+def test_a_format_yt_dlp_fetches_itself_is_kept():
+    # f4m, ism and the DASH generator fetch through yt-dlp's own urlopen; a bare m3u8 goes to the
+    # native HLS fetcher under hls_prefer_native (set for every guarded download). All ask the gate.
+    kept = [
+        {"format_id": p, "url": f"https://{PUBLIC}/{p}", "protocol": p}
+        for p in ("f4m", "ism", "m3u8", "m3u8_native", "http_dash_segments_generator", "https")
+    ]
+    raw = {"format_id": "rtmp", "url": f"rtmp://{PUBLIC}/live", "protocol": "rtmp"}
+    _, info = egress.HttpOnlyFormats().run({"formats": [*kept, raw]})
+    assert [f["format_id"] for f in info["formats"]] == [f["format_id"] for f in kept]
+
+
+def test_a_single_url_result_past_the_gate_loses_its_url():
+    _, info = egress.HttpOnlyFormats().run({"url": f"rtmp://{INTERNAL}/live"})
+    assert info["url"] is None
+    _, info = egress.HttpOnlyFormats().run({"url": f"https://{PUBLIC}/a.mp4"})
+    assert info["url"] == f"https://{PUBLIC}/a.mp4"
+
+
+def test_the_format_filter_says_no_phase_of_its_own(tmp_path):
+    # It is a post-processor only to run before the format choice; the job's postprocessor hook
+    # must not see it (it painted every guarded job "processing" before a byte was fetched).
+    calls: list[dict] = []
+    opts = _guarded_opts(tmp_path, "http://127.0.0.1:9", postprocessor_hooks=[calls.append])
+    with egress.open_ydl(opts) as ydl:
+        ydl.run_all_pps("pre_process", {"id": "x", "formats": []})
+    assert calls == []
+
+
+def test_an_allowed_tunnel_carries_bytes_both_ways(gate):
+    # The CONNECT path a site served over https takes: a 200, then the bytes the client sent after
+    # the head (in the same packet here) reach the far end, and its answer comes back.
+    far = socket.socket()
+    far.bind((PUBLIC, 0))
+    far.listen()
+
+    def echo():
+        conn, _ = far.accept()
+        with conn:
+            conn.sendall(b"PONG:" + conn.recv(64))
+
+    threading.Thread(target=echo, daemon=True).start()
+    try:
+        with socket.create_connection(("127.0.0.1", gate.server_address[1]), timeout=5) as s:
+            s.sendall(f"CONNECT {PUBLIC}:{far.getsockname()[1]} HTTP/1.1\r\n\r\nPING".encode())
+            got = b""
+            while b"PONG:PING" not in got:
+                chunk = s.recv(4096)
+                if not chunk:
+                    break
+                got += chunk
+        assert got.startswith(b"HTTP/1.1 200")
+        assert got.endswith(b"PONG:PING")
+    finally:
+        far.close()
+
+
+def test_the_workers_own_gate_refuses_our_network():
+    # Every other gate here is built with a test rule; this is the one `proxy_url()` starts.
+    internal = _Listener()
+    try:
+        port = int(egress.proxy_url().rsplit(":", 1)[1])
+        with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
+            s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
+            assert s.recv(4096).startswith(b"HTTP/1.1 403")
+        assert internal.hits == 0
+    finally:
+        internal.close()
+
+
+def _clip_hits(tmp_path: Path, segment: str, proxy: str) -> int:
+    """How many connections the internal listener takes while a clip (`download_ranges`, which
+    yt-dlp hands to ffmpeg even for an HLS format) is cut from a playlist whose one segment is
+    `segment` — formatted with the listener's address."""
+    internal = _Listener()
+    playlist = (
+        "#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:2\n#EXTINF:2.0,\n"
+        + segment.format(host=INTERNAL, port=internal.port)
+        + "\n#EXT-X-ENDLIST\n"
+    ).encode()
+    public = _public_server(
+        {"/p.m3u8": (200, {"Content-Type": "application/vnd.apple.mpegurl"}, playlist)}
+    )
+    try:
+        opts = _guarded_opts(tmp_path, proxy, download_ranges=download_range_func([], [(0, 1)]))
+        with contextlib.suppress(yt_dlp.utils.DownloadError), egress.open_ydl(opts) as ydl:
+            ydl.extract_info(f"http://{PUBLIC}:{public.server_address[1]}/p.m3u8", download=True)
+        return internal.hits
+    finally:
+        public.shutdown()
+        internal.close()
+
+
+def test_a_clip_cut_by_ffmpeg_reaches_a_segment_only_through_the_gate(tmp_path, gate):
+    # Without the gate, ffmpeg fetches a hostile playlist's segment from inside our network — the
+    # control that shows this path is real. Through the gate it is never dialled.
+    segment = "http://{host}:{port}/s.ts"
+    assert _clip_hits(tmp_path / "bare", segment, proxy="") >= 1
+    assert _clip_hits(tmp_path / "gated", segment, proxy=gate.url) == 0
+
+
+def test_ffmpeg_itself_refuses_a_raw_tcp_segment(tmp_path):
+    # A boundary, not our code: the HTTP proxy cannot see a tcp:// segment, and measured, ffmpeg's
+    # HLS reader does not open one from an http playlist. This fails if an ffmpeg upgrade starts to.
+    assert _clip_hits(tmp_path, "tcp://{host}:{port}/", proxy="") == 0
+
+
+# --- the wiring ------------------------------------------------------------------------------------
+
+
+def test_only_youtube_downloads_skip_the_gate():
+    assert adapters.SourceAdapter.egress_guarded is True
+    assert adapters.resolve_adapter("url", "https://www.facebook.com/x").egress_guarded is True
+    assert adapters.resolve_adapter("url", "https://vimeo.com/1").egress_guarded is True
+    assert adapters.resolve_adapter("youtube", "dQw4w9WgXcQ").egress_guarded is False
+
+
+def test_a_guarded_download_goes_through_the_gate_whatever_its_adapter_adds():
+    opts = formats.build_ydl_opts(
+        {}, "/tmp/x", lambda d: None, extra={"proxy": "", "hls_prefer_native": False}, egress=True
+    )
+    assert opts["proxy"] == egress.proxy_url()
+    assert opts["hls_prefer_native"] is True
+    assert opts[egress.GUARDED] is True
+    assert "proxy" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None, egress=False)
+
+
+def test_every_yt_dlp_this_app_opens_goes_through_open_ydl():
+    app = Path(egress.__file__).resolve().parents[1]
+    offenders = [
+        f"{p.relative_to(app)}:{n}"
+        for p in sorted(app.rglob("*.py"))
+        if p.name != "egress.py"
+        for n, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1)
+        if "YoutubeDL(" in line
+    ]
+    assert offenders == []

backend/tests/test_facebook_impersonation.py

@@ -18,7 +18,7 @@ import yt_dlp
 from yt_dlp.networking.exceptions import TransportError
 from yt_dlp.networking.impersonate import ImpersonateTarget
 
-from app.downloads import adapters, pipeline
+from app.downloads import adapters, egress, pipeline
 from app.downloads import errors as dlerrors
 
 _URL = "https://www.facebook.com/reel/806379462330494/?fs=e&fs=e"
@@ -73,6 +73,9 @@ def ydl(monkeypatch):
         def __enter__(self):
             return self
 
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
         def __exit__(self, *exc):
             return False
 
@@ -157,6 +160,16 @@ def test_a_cookie_group_post_downloads_the_posts_own_video(ydl, tmp_path):
     # Never the post link: that is where the feed video comes from.
     assert web["extracted"] == [watch, watch]
     assert [o.get("impersonate") for o in seen[1:]] == [None, _CHROME]
+    # The post page read and both attempts all go through the egress gate.
+    assert all(o.get(egress.GUARDED) and o["proxy"] == egress.proxy_url() for o in seen), seen
+
+
+def test_a_youtube_download_skips_the_gate(ydl, tmp_path):
+    seen, outcomes, _ = ydl
+    outcomes += [{"id": "dQw4w9WgXcQ"}]
+    yt = adapters.resolve_adapter("youtube", "dQw4w9WgXcQ")
+    pipeline._extract_with_fallback(1, 1, "https://youtu.be/dQw4w9WgXcQ", {}, tmp_path, yt)
+    assert seen and not any(o.get(egress.GUARDED) or o.get("proxy") for o in seen)
 
 
 def test_a_group_post_without_a_single_video_fails_saying_so(ydl, tmp_path):

backend/tests/test_facebook_titles.py

@@ -0,0 +1,303 @@
+"""A Facebook download's title when the signed-in read gives only a placeholder.
+
+Measured on prod (2026-11-01): every Facebook download since 2026-10-28 is titled "Facebook" — the
+signed-in read (the owner's cookie) now gives that placeholder and NO description, so the adapter had
+no post text to fall back on, and the download name copied the placeholder too. A signed-out read of
+the same reel (yt-dlp 2026.08.19, as on prod) still carries the post's text in its description.
+"""
+
+from sqlalchemy import select, update
+
+from app.config import settings
+from app.db import SessionLocal
+from app.downloads import adapters, backfill, egress, pipeline
+from app.downloads.models import DownloadJob, MediaAsset
+
+_URL = "https://www.facebook.com/reel/1860682468422402/"
+# The page yt-dlp says it read for it (`webpage_url`); a group post's link resolves the same way.
+_PAGE = "https://www.facebook.com/watch/?v=1860682468422402"
+# The measured signed-out read of that reel.
+_PUBLIC = {
+    "title": "163K reactions · 14K shares | sorry not helping #workhumor #officelife | Work dot exe",
+    "description": "sorry not helping\n\n#workhumor #officelife",
+    "uploader": "Work dot exe",
+}
+
+
+def _fake_ydl(monkeypatch, outcome):
+    """yt-dlp faked at its `YoutubeDL` boundary: records each read (its options and its `url`),
+    answers `outcome`."""
+    seen: list[dict] = []
+
+    class FakeYDL:
+        def __init__(self, opts):
+            self.opts = opts
+
+        def __enter__(self):
+            return self
+
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
+        def __exit__(self, *exc):
+            return False
+
+        def extract_info(self, url, download):
+            seen.append({**self.opts, "url": url})
+            return outcome
+
+    monkeypatch.setattr("yt_dlp.YoutubeDL", FakeYDL)
+    return seen
+
+
+def test_a_placeholder_with_nothing_behind_it_takes_the_signed_out_pages_text(monkeypatch):
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    fb = adapters.FacebookAdapter()
+
+    info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
+
+    assert fb.title(info) == "sorry not helping"
+    # One metadata-only read, and signed out: no cookie goes with it.
+    assert len(seen) == 1
+    assert seen[0]["skip_download"] is True
+    assert not seen[0].get("cookiefile")
+    assert seen[0][egress.GUARDED] is True and seen[0]["proxy"] == egress.proxy_url()
+
+
+def test_a_real_title_reads_nothing_more(monkeypatch):
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    info = {"title": "Funny Dark Humor Dad Joke!", "description": None}
+
+    assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
+    assert seen == []
+
+
+def test_a_placeholder_with_the_posts_text_behind_it_reads_nothing_more(monkeypatch):
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    info = {"title": "Video", "description": "Office jokes, part two"}
+
+    assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
+    assert seen == []
+
+
+def test_a_placeholder_the_signed_out_page_cannot_better_stays(monkeypatch):
+    _fake_ydl(monkeypatch, {"title": "Facebook", "description": None})
+    fb = adapters.FacebookAdapter()
+
+    info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
+
+    assert fb.title(info) == "Facebook"
+
+
+def test_another_source_is_never_read_again(monkeypatch):
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    info = {"title": "Video", "description": None}
+
+    assert backfill.public_title_info(info, _URL, adapters.SourceAdapter(), 7) is info
+    assert seen == []
+
+
+def test_the_backfill_retitles_a_placeholder_download_and_the_names_that_repeated_it(
+    db, make_user, monkeypatch
+):
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    owner, other = make_user(), make_user()
+    asset = MediaAsset(
+        source_kind="url",
+        source_ref=_URL,
+        format_sig="fb",
+        status="ready",
+        rel_path="facebook/x.mp4",
+        poster_path="facebook/x.jpg",
+        uploader_url="https://www.facebook.com/1",
+        title="Facebook",
+        ref_count=2,
+    )
+    db.add(asset)
+    db.flush()
+    copied = DownloadJob(
+        user_id=owner.id,
+        asset_id=asset.id,
+        source_kind="url",
+        source_ref=_URL,
+        status="done",
+        queue_pos=0,
+        profile_snapshot={},
+        display_name="Facebook",
+    )
+    renamed = DownloadJob(
+        user_id=other.id,
+        asset_id=asset.id,
+        source_kind="url",
+        source_ref=_URL,
+        status="done",
+        queue_pos=0,
+        profile_snapshot={},
+        display_name="Office jokes",
+    )
+    db.add_all([copied, renamed])
+    db.commit()
+
+    assert backfill.run()["retitled"] == 1
+    assert len(seen) == 1
+
+    db.expire_all()
+    assert db.get(MediaAsset, asset.id).title == "sorry not helping"
+    assert db.get(DownloadJob, copied.id).display_name == "sorry not helping"
+    assert db.get(DownloadJob, renamed.id).display_name == "Office jokes"
+    # Run again: nothing left to retitle, and no page read for it.
+    assert backfill.run()["retitled"] == 0
+    assert len(seen) == 1
+
+
+def test_a_name_given_while_the_backfill_runs_stays(db, make_user, monkeypatch):
+    # The owner renames the download while the backfill reads the page: the owner's name wins, it is
+    # not overwritten by what was read before it.
+    _fake_ydl(monkeypatch, _PUBLIC)
+    owner = make_user()
+    asset = MediaAsset(
+        source_kind="url",
+        source_ref=_URL,
+        format_sig="fb",
+        status="ready",
+        rel_path="facebook/x.mp4",
+        uploader_url="https://www.facebook.com/1",
+        title="Facebook",
+        ref_count=1,
+    )
+    db.add(asset)
+    db.flush()
+    job = DownloadJob(
+        user_id=owner.id,
+        asset_id=asset.id,
+        source_kind="url",
+        source_ref=_URL,
+        status="done",
+        queue_pos=0,
+        profile_snapshot={},
+        display_name="Facebook",
+    )
+    db.add(job)
+    db.commit()
+
+    read_page = backfill.public_title_info
+
+    def renamed_while_reading(*args):
+        with SessionLocal() as other:
+            other.execute(
+                update(DownloadJob)
+                .where(DownloadJob.id == job.id)
+                .values(display_name="Office jokes")
+            )
+            other.commit()
+        return read_page(*args)
+
+    monkeypatch.setattr(backfill, "public_title_info", renamed_while_reading)
+    monkeypatch.setattr(backfill.editmod, "ensure_poster", lambda a, root: None)
+
+    assert backfill.run()["retitled"] == 1
+
+    db.expire_all()
+    assert db.get(DownloadJob, job.id).display_name == "Office jokes"
+
+
+def test_the_poster_step_holds_no_row_lock(db, make_user, monkeypatch):
+    # It can cut a frame for 30 s. A delete locks the asset and then the job: with the job held here
+    # and the asset still to write, that is a deadlock, and a rename waits out the whole cut.
+    _fake_ydl(monkeypatch, _PUBLIC)
+    owner = make_user()
+    asset = MediaAsset(
+        source_kind="url",
+        source_ref=_URL,
+        format_sig="fb",
+        status="ready",
+        rel_path="facebook/x.mp4",
+        uploader_url="https://www.facebook.com/1",
+        title="Facebook",
+        ref_count=1,
+    )
+    db.add(asset)
+    db.flush()
+    job = DownloadJob(
+        user_id=owner.id,
+        asset_id=asset.id,
+        source_kind="url",
+        source_ref=_URL,
+        status="done",
+        queue_pos=0,
+        profile_snapshot={},
+        display_name="Facebook",
+    )
+    db.add(job)
+    db.commit()
+    asset_id, job_id = asset.id, job.id
+
+    def poster_finds_rows_free(a, root):
+        with SessionLocal() as other:
+            other.execute(
+                select(MediaAsset).where(MediaAsset.id == asset_id).with_for_update(nowait=True)
+            )
+            other.execute(
+                select(DownloadJob).where(DownloadJob.id == job_id).with_for_update(nowait=True)
+            )
+            other.rollback()
+        return None
+
+    monkeypatch.setattr(backfill.editmod, "ensure_poster", poster_finds_rows_free)
+
+    assert backfill.run()["retitled"] == 1
+
+
+def test_a_download_titled_facebook_is_named_from_the_signed_out_page(
+    db, make_user, monkeypatch, tmp_path
+):
+    # The whole download path with yt-dlp and the media tools faked: the signed-in read gives only
+    # the placeholder, and the file and the download's name get the post's text.
+    seen = _fake_ydl(monkeypatch, _PUBLIC)
+    monkeypatch.setattr(settings, "files_root", str(tmp_path))
+    monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
+    monkeypatch.setattr(pipeline, "_ensure_phone_safe", lambda _j, produced, *_a: produced)
+    monkeypatch.setattr(
+        pipeline.probe, "resolve_media_facts", lambda produced, info: ("h264", "aac", 720, 1280, 9)
+    )
+    monkeypatch.setattr(pipeline.editmod, "thumbnail_to_jpg", lambda *a: None)
+    monkeypatch.setattr(pipeline.editmod, "ensure_poster", lambda *a: None)
+    owner = make_user()
+    asset = MediaAsset(
+        source_kind="url", source_ref=_URL, format_sig="fb", status="downloading", ref_count=1
+    )
+    db.add(asset)
+    db.flush()
+    job = DownloadJob(
+        user_id=owner.id,
+        asset_id=asset.id,
+        source_kind="url",
+        source_ref=_URL,
+        status="running",
+        queue_pos=0,
+        profile_snapshot={},
+    )
+    db.add(job)
+    db.commit()
+
+    def signed_in_read(job_id, asset_id, url, spec, staging, adapter, **kw):
+        f = staging / "1860682468422402.mp4"
+        f.write_bytes(b"v" * 100)
+        return {
+            "id": "1860682468422402",
+            "title": "Facebook",
+            "uploader": "Work dot exe",
+            "webpage_url": _PAGE,
+            "requested_downloads": [{"filepath": str(f)}],
+        }
+
+    monkeypatch.setattr(pipeline, "_extract_with_fallback", signed_in_read)
+    pipeline._download(job.id, asset.id, "url", _URL, {}, owner.id)
+
+    # The signed-out read goes to the page yt-dlp read, not the link queued: a group post's link
+    # resolves to its video, and signed out the post itself is the login wall.
+    assert [r["url"] for r in seen] == [_PAGE]
+    db.expire_all()
+    assert db.get(DownloadJob, job.id).status == "done"
+    assert db.get(MediaAsset, asset.id).title == "sorry not helping"
+    assert db.get(DownloadJob, job.id).display_name == "sorry not helping"

backend/tests/test_failure_lines.py

@@ -359,6 +359,9 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
         def __enter__(self):
             return self
 
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
         def __exit__(self, *exc):
             return False
 
@@ -367,8 +370,8 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
                 f"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable ({target})"
             )
 
-    monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
-    assert backfill._extract_meta(url, adapters.YouTubeAdapter(), 11) is None
+    monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
+    assert backfill.extract_meta(url, adapters.YouTubeAdapter(), 11) is None
 
     lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
     assert lines and all("dQw4w9WgXcQ" not in json.dumps(o) for o in lines)
@@ -386,14 +389,17 @@ def test_the_metadata_backfill_takes_out_a_generic_name_whatever_it_holds(sink,
         def __enter__(self):
             return self
 
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
         def __exit__(self, *exc):
             return False
 
         def extract_info(self, target, download):
             raise yt_dlp.utils.DownloadError("ERROR: [generic] Our Trip: Day 1: Unable to download")
 
-    monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
-    assert backfill._extract_meta(url, adapters.SourceAdapter(), 11) is None
+    monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
+    assert backfill.extract_meta(url, adapters.SourceAdapter(), 11) is None
 
     lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
     assert lines and all("Day 1" not in json.dumps(o) for o in lines)
@@ -449,6 +455,9 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
         def __enter__(self):
             return self
 
+        def add_post_processor(self, pp, when="post_process"):
+            pass  # the egress format filter; nothing here to filter
+
         def __exit__(self, *exc):
             return False
 
@@ -463,6 +472,7 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
         resolve_url=lambda target, opts: target,
         ydl_opts_extra=lambda clients: {},
         retriable=lambda text: True,
+        egress_guarded=False,
     )
     monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", _YDL)
     monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)

backend/tests/test_folder_links.py

@@ -150,6 +150,7 @@ def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
     assert client.get(f"/api/public/watch/{token}").json() == {
         "needs_password": True,
         "kind": "folder",
+        "video": False,
     }
     bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
     assert client.get(bare).status_code == 403
@@ -312,7 +313,7 @@ def test_a_password_file_link_says_it_opens_a_file(shelf, client):
         .json()
     )
     meta = client.get(f"/api/public/watch/{made['token']}").json()
-    assert meta == {"needs_password": True, "kind": "file"}
+    assert meta == {"needs_password": True, "kind": "file", "video": True}
 
 
 def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):

backend/tests/test_formats.py

@@ -15,16 +15,20 @@ from app.downloads import formats
 
 
 def _opts(spec):
-    return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None)
+    return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None, egress=False)
 
 
 def test_adapter_extra_opts_are_merged_in():
     # The SourceAdapter contributes source-specific opts (e.g. YouTube's extractor_args) via `extra`.
     extra = {"extractor_args": {"youtube": {"player_client": ["web"]}}}
-    opts = formats.build_ydl_opts({}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra)
+    opts = formats.build_ydl_opts(
+        {}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra, egress=False
+    )
     assert opts["extractor_args"]["youtube"]["player_client"] == ["web"]
     # No extra → no extractor_args (a generic source contributes nothing).
-    assert "extractor_args" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None)
+    assert "extractor_args" not in formats.build_ydl_opts(
+        {}, "/tmp/x", lambda d: None, egress=False
+    )
 
 
 def test_mp4_default_selects_avc1_with_progressive_fallback():
@@ -113,7 +117,7 @@ def test_a_playlist_url_can_only_yield_one_item():
 
 
 def test_the_remaining_quota_becomes_a_hard_size_ceiling():
-    opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
+    opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000, egress=False)
     assert opts["max_filesize"] == 5_000_000
 
 
@@ -122,7 +126,7 @@ def test_no_budget_means_no_size_filter():
     # enqueue, not an impossible yt-dlp filter to express here.
     for budget in (None, 0, -1):
         assert "max_filesize" not in formats.build_ydl_opts(
-            {}, "/tmp/x", lambda d: None, max_bytes=budget
+            {}, "/tmp/x", lambda d: None, max_bytes=budget, egress=False
         )
 
 
@@ -130,7 +134,12 @@ def test_a_source_adapter_cannot_raise_the_size_ceiling():
     # The adapter merge happens after max_filesize is set, so this pins that no adapter option can
     # widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
     opts = formats.build_ydl_opts(
-        {}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
+        {},
+        "/tmp/x",
+        lambda d: None,
+        extra={"max_filesize": 999_999_999},
+        max_bytes=5_000_000,
+        egress=False,
     )
     assert opts["max_filesize"] == 5_000_000
 
@@ -197,7 +206,7 @@ def test_yt_dlp_refuses_a_file_over_the_cap_without_raising_and_the_watch_sees_i
         for cap, refused in ((1_000, True), (10_000_000, False)):
             out = tmp_path / f"out-{cap}"
             opts = formats.build_ydl_opts(
-                {}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap
+                {}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap, egress=False
             )
             with formats.watching_size_cap() as over, yt_dlp.YoutubeDL(opts) as ydl:
                 ydl.extract_info(url, download=True)

backend/tests/test_google_callback.py

@@ -7,6 +7,7 @@ the refused sign-in that doubles as an access request, and the link flow's confl
 """
 
 import pytest
+from authlib.integrations.starlette_client import OAuthError
 from sqlalchemy import select
 from starlette.responses import RedirectResponse
 
@@ -19,15 +20,20 @@ _PW = "correct horse battery"
 
 class _FakeGoogle:
     """authlib's client, reduced to the two calls the routes make: the redirect out to Google and
-    the token exchange when Google redirects back."""
+    the token exchange when Google redirects back. Like authlib, it honours only the newest flow's
+    state: starting a flow retires every earlier one."""
 
     def __init__(self, userinfo: dict):
         self.userinfo = userinfo
+        self.asked: dict = {}
 
-    async def authorize_redirect(self, _request, _redirect_uri, **_kw):
+    async def authorize_redirect(self, _request, _redirect_uri, **kw):
+        self.asked = kw
         return RedirectResponse(url="/fake-google")
 
-    async def authorize_access_token(self, _request):
+    async def authorize_access_token(self, request):
+        if request.query_params.get("state") != self.asked.get("state"):
+            raise OAuthError(error="mismatching_state")
         return {"userinfo": self.userinfo, "access_token": "at", "scope": "openid email profile"}
 
 
@@ -44,9 +50,13 @@ def google(monkeypatch):
     return fake
 
 
-def _sign_in(client, google, **userinfo):
+def _sign_in(client, google, state: str | None = None, **userinfo):
+    """Google redirecting back: by default with the state of the flow started last."""
     google.userinfo = {"email_verified": True, **userinfo}
-    return client.get("/auth/callback", follow_redirects=False)
+    state = state or google.asked.get("state")
+    return client.get(
+        "/auth/callback", params={"state": state} if state else {}, follow_redirects=False
+    )
 
 
 def _rows(db, model, column, value):
@@ -218,12 +228,20 @@ def test_a_refused_sign_in_records_a_request_in_the_reported_language(
     assert _rows(db, User, User.email, "new@example.com") == []
 
 
-def _start_link(client, email: str):
+def _password_login(client, email: str):
     assert (
         client.post("/auth/password-login", json={"email": email, "password": _PW}).status_code
         == 200
     )
-    assert client.get("/auth/link", follow_redirects=False).headers["location"] == "/fake-google"
+
+
+def _connect(client, access: str):
+    return client.get(f"/auth/google/connect?access={access}", follow_redirects=False)
+
+
+def _start_link(client, email: str):
+    _password_login(client, email)
+    assert _connect(client, "identity").headers["location"] == "/fake-google"
 
 
 def test_linking_an_identity_another_account_holds_is_a_conflict(client, db, google, make_user):
@@ -258,3 +276,151 @@ def test_linking_a_free_identity_attaches_it(client, db, google, make_user):
     assert r.headers["location"] == "/?link=ok"
     db.refresh(me)
     assert (me.google_sub, me.google_email) == ("sub-free", "me3.google@example.com")
+
+
+def test_a_connect_abandoned_at_google_leaves_the_next_sign_in_a_sign_in(
+    client, db, google, make_user, monkeypatch
+):
+    # A connect left at Google's screen (Back, a closed tab) never reaches the callback that pops
+    # its marker. "Sign in as someone else" afterwards must sign that person in, not link their
+    # Google identity onto the account still signed in here.
+    me = make_user(email="me4@example.com")
+    make_user(email="them4@example.com", google_sub="sub-them4", password=None)
+    monkeypatch.setattr(settings, "allowed_emails", "them4@example.com")
+    _start_link(client, "me4@example.com")
+    assert client.get("/auth/login", follow_redirects=False).headers["location"] == "/fake-google"
+
+    r = _sign_in(client, google, sub="sub-them4", email="them4@example.com")
+
+    assert r.headers["location"] == "/"
+    assert client.get("/api/me").json()["email"] == "them4@example.com"
+    db.refresh(me)
+    assert me.google_sub is None
+
+
+def test_a_link_whose_session_was_revoked_meanwhile_does_not_finish(client, db, google, make_user):
+    # "Log out everywhere" (or a password change) between the start and the callback: the cookie
+    # still names the account, but its session no longer holds, so it may not attach an identity.
+    me = make_user(email="me5@example.com")
+    _start_link(client, "me5@example.com")
+    me.session_epoch = (me.session_epoch or 0) + 1
+    db.commit()
+
+    r = _sign_in(client, google, sub="sub-late", email="late@example.com")
+
+    assert r.headers["location"] == "/?link=error"
+    db.refresh(me)
+    assert me.google_sub is None
+
+
+def test_an_older_flow_coming_back_leaves_the_connect_in_flight_a_link(
+    client, db, google, make_user
+):
+    # Two round-trips from one browser: an older one (another tab's, or Back to Google's screen)
+    # returns after a connect started. Its state is refused, and it must not spend the connect's
+    # marker: the connect itself, coming back, would then land as a plain sign-in.
+    me = make_user(email="me6@example.com")
+    _start_link(client, "me6@example.com")
+
+    older = _sign_in(client, google, state="an-older-flow", sub="sub-6", email="g6@example.com")
+    assert older.headers["location"] == "/?oauth=cancelled"
+    r = _sign_in(client, google, sub="sub-6", email="g6@example.com")
+
+    assert r.headers["location"] == "/?link=ok"
+    db.refresh(me)
+    assert me.google_sub == "sub-6"
+
+
+def test_a_connect_from_a_tab_showing_another_account_starts_nothing(client, google, make_user):
+    # A navigation carries no headers, so the tab names the account it was built for in the query.
+    # One still showing the account signed in before must not start a link onto the one signed in
+    # now: it lands on the app, which boots as that one and says why.
+    me = make_user(email="me7@example.com")
+    _password_login(client, "me7@example.com")
+
+    r = client.get(
+        f"/auth/google/connect?access=identity&expect={me.id + 1}", follow_redirects=False
+    )
+
+    assert r.headers["location"] == auth_routes.ACCOUNT_CHANGED_LANDING == "/?account=changed"
+    assert google.asked == {}
+    assert client.get("/api/me").json()["email"] == "me7@example.com"
+
+
+def test_a_connect_naming_the_signed_in_account_goes_to_google(client, google, make_user):
+    me = make_user(email="me8@example.com")
+    _password_login(client, "me8@example.com")
+
+    r = client.get(f"/auth/google/connect?access=identity&expect={me.id}", follow_redirects=False)
+
+    assert r.headers["location"] == "/fake-google"
+    assert google.asked["state"]
+
+
+@pytest.mark.parametrize(
+    ("access", "scope", "prompt"),
+    [
+        ("identity", "openid email profile", "select_account"),
+        ("read", auth_routes.READ_SCOPES, "consent"),
+        ("write", auth_routes.WRITE_SCOPES, "consent"),
+    ],
+)
+def test_one_connect_route_asks_google_for_what_the_access_needs(
+    client, google, make_user, access, scope, prompt
+):
+    make_user(email=f"ask-{access}@example.com")
+    _password_login(client, f"ask-{access}@example.com")
+
+    assert _connect(client, access).headers["location"] == "/fake-google"
+
+    assert (google.asked["scope"], google.asked["prompt"]) == (scope, prompt)
+    assert google.asked["access_type"] == "offline"
+
+
+@pytest.mark.parametrize("access", ["read", "write"])
+def test_a_youtube_grant_attaches_to_the_password_account_and_returns_to_the_app(
+    client, db, google, make_user, access
+):
+    me = make_user(email=f"grant-{access}@example.com")
+    _password_login(client, f"grant-{access}@example.com")
+    _connect(client, access)
+
+    r = _sign_in(client, google, sub=f"sub-grant-{access}", email=f"g-{access}@example.com")
+
+    # Back to the app, not the "linked" notice: the onboarding wizard picks up where it was.
+    assert r.headers["location"] == "/"
+    db.refresh(me)
+    assert me.google_sub == f"sub-grant-{access}"
+    assert _rows(db, User, User.google_sub, f"sub-grant-{access}") == [me]
+
+
+@pytest.mark.parametrize("access", ["", "admin", "READ"])
+def test_an_unknown_access_starts_no_connect(client, google, make_user, access):
+    make_user(email="odd@example.com")
+    _password_login(client, "odd@example.com")
+
+    assert _connect(client, access).status_code == 422
+    assert google.asked == {}
+
+
+def test_a_connect_needs_a_signed_in_account(client, google):
+    assert _connect(client, "identity").status_code == 401
+    assert google.asked == {}
+
+
+@pytest.mark.parametrize("access", ["identity", "read", "write"])
+def test_without_google_configured_a_connect_says_it_failed(client, make_user, monkeypatch, access):
+    monkeypatch.setattr(auth_routes, "google_oauth", lambda _db: None)
+    make_user(email=f"nog-{access}@example.com")
+    _password_login(client, f"nog-{access}@example.com")
+
+    assert _connect(client, access).headers["location"] == "/?link=error"
+
+
+@pytest.mark.parametrize("path", ["/auth/link", "/auth/upgrade", "/auth/upgrade?access=write"])
+def test_the_old_connect_routes_are_gone(client, google, make_user, path):
+    make_user(email="old@example.com")
+    _password_login(client, "old@example.com")
+
+    assert client.get(path, follow_redirects=False).status_code == 404
+    assert google.asked == {}