Compare commits

..
230 Commits
Author SHA1 Message Date
peter ca5e446fbd release: 0.85.0 (feature/e11-power-navigation)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m36s
2026-09-13 21:02:22 +02:00
peter 4c7ab12a3e release: 0.85.0 — E11 Power Navigation
Back, dialogs and menus keep your place, and the keyboard walks every list: five sprints plus three epic-end review rounds (19 → 14, 18 → 12, 14 → 14). Minor: j/k list navigation, s and g-chords, and Back-to-item are features.
2026-09-13 21:02:22 +02:00
peter c6b552e75f fix(keys): epic-end review round 3 — the part-way clip count shows, busy buttons keep focus, cards say their names
Targeted round over 3cd0c525's fix surfaces: 14 raised, 14 fixed.

- VideoEditor: the "only X of Y clips" toast went silent exactly on the job-cap/quota refusal it was
  written for (req had already reported the error); it now always shows. The cut list holds still while
  Create queues, and a retry continues the "(i/N)" series.
- ChannelPage: Subscribe/Unsubscribe use Button `loading`, "Load more" and ChannelDiscovery's Subscribe
  block the click with aria-disabled instead of native disabled; busy Block/Hide no longer light up on
  hover.
- listNav: an item can name itself (`navItemAttrs(i, name)`) — Plex posters, seasons, episodes, message
  rows and download rows pass their title; focus on a toast's Undo keeps the removed card's slot.
- AnchoredPopover: the hand-back only for focus-managed surfaces (a hover card no longer pulls focus
  back); the dead panel-ref check is gone.
- AccountPanel: no confirmation mail offered to the demo account; send_failed copy no longer blames
  the message for an unreachable server. Pager comment points at scrollParent.
2026-09-13 20:47:13 +02:00
peter 3cd0c525e2 fix(keys): epic-end review round 2 — focus comes back to the card, the right rail and the reason
Whole-epic deep-review of E11 (18 raised, 12 kept; B and G refuted, four latent test-hardening notes dropped),
plus the admin refusal copy gap round 1 surfaced.

- A context menu opened from a card the keyboard is on gives the focus back to the card. floating-ui returns
  focus through the opener's first tabbable descendant even when handed a ref, so for a non-tabbable opener the
  surface restores it itself (measured in Chrome: j, Shift+F10, Escape left it on the thumbnail link).
- Back finds a named list (a rail) by its name: a rail that appeared above it after the cache expired sent the
  focus into the wrong rail. An unnamed list is matched only when the page has the same number of lists.
- The slot kept for a removed item is forgotten once focus moves on to page content (search, a chip), so j after
  new results starts at the first visible item; a menu or dialog opened from the item keeps it.
- The list keys reach the cheat sheet only while a list has an item.
- A j-focused card root is a named group while it holds the focus.
- The channel page's Block and Hide toggles keep the focus while their request runs.
- A split export that fails part-way says how many clips were added and deselects them, so a retry adds only
  the rest.
- Config save, the one-click share link and "log out other sessions" report failures with their reason.
- VirtualGrid uses the one scroll-parent walk listNav measures against.
- Admin refusals (activate, resend, invite, set-password link) say why in HU and EN instead of "try again";
  the email-off special cases collapse into that one reason. The contract reads the codes from localizeDetail.
  HU copy points to Konfiguráció → E-mail / SMTP; HU "unreachable" matches the EN change of round 1.
2026-09-13 19:18:04 +02:00
peter f130a1c05a fix(keys): epic-end review round 1 — Back, menus and dialogs keep the focus where it belongs
Deep-review of E11 (19 raised, 14 confirmed; 2 refuted, 3 latent test-hardening notes dropped).

- Back from a sub-view (a Plex show page, a Messages thread) restores the item it was opened from
  even when focus was still inside the view: the first attempt now waits one task, past the
  sub-view's own popstate listener (measured in Chrome: it left focus on <body>).
- The restore listener is installed at load, so the first Back after a reload finds it; a push
  cancels a restore still waiting for the last landing's lists.
- Navigating over an overlay asks it again, a render apart, while its entry stays: the Plex
  player peels one panel per Back step, so Shift+W over info + stats closed one panel and dropped
  the navigation.
- j/k: a native <button> item keeps its own Tab stop; landing twice on the same item no longer
  leaks its tabindex; after the focused item is removed (a hidden card) the next j goes on from
  its slot when that slot is on screen.
- An abandoned `g` no longer swallows a modifier chord (g, then Mod+K opened the browser search).
- A hover card above a dialog no longer switches its Tab trap off (player, Modal, wizard,
  Lightbox): new layerStack `anyBlockingLayerAbove`.
- "Manage views…" closes its menu before opening the dialog, through a shared `closeThenAct`
  that ActionMenu now uses too.
- Copy: the EN welcome says "Feed"; the EN unreachable reason no longer claims a change;
  stale comments about PlayerModal's scroll lock and dialog contract.
2026-09-13 17:17:49 +02:00
peter bbd31c7143 feat(keys): Back returns focus to the item a view was opened from; "Feed" everywhere; every backend id named
Owner answers to the E11 S5 raises (2026-09-13): 1 → A, 2 → "Feed" in every language, 3 → translate the
proxy label and fix every config key missing in BOTH languages ("maintenance", "tmdb" — old bugs).

Focus across a view change (1-A). Opening a Plex title from Home's rail, a channel from the Channels
table or a title from a Plex grid switches the page or the sub-view, which unmounts the list; Back used
to hand focus to <body>, so the next `j` started over at the top. The YouTube player (an overlay)
already restored onto the card.
- `lib/listNav`: `navPosition()` (list's place in reading order, the page's list count, item index)
  and `restoreNavPosition()`: focuses the item once the page has as many lists on screen as when it was
  left, polls on a timer (a hidden tab runs no animation frames), gives up after 2 s or at once when
  something already holds focus, focuses without scrolling (the scroll restore owns the position).
- `lib/history`: `rememberNavPosition()` stamps `sfNav` onto the entry being left and returns the state
  without it (a new entry must not inherit it); one popstate listener restores and spends the stamp,
  so a later landing on the same entry (an overlay closing over it) cannot pull focus to an old item.
- Called before every view push: `setPage`, `openChannel`, `enterYtSearch`, `useHistorySubview.open`,
  the Channels tab push. `navPosition.wiring.contract.test` fails on any `history.pushState(` without
  it (the overlay stack's own push is the one exemption).

"Feed" (2). The module is "Feed" in both languages: every "hírfolyam" in the Hungarian UI (≈45 strings,
suffixed: Feedből, Feeded, Feedben…) and the prose "feed" in English that means the module ("your
Feed", "Hide from Feed"); "RSS feed" keeps its meaning. The welcome email says it too.

Backend ids without a name (3). The Configuration page showed `maintenance`, `tmdb`,
`maintenance_revalidate_batch`, `download_upload_max_bytes`, `audit_retention_days`, `plex_overlay_map`,
`tmdb_api_key` and `tmdb_language` as raw identifiers — the lookup falls back to the id, and the i18n
parity test stayed green because both languages missed the same keys. Added labels + hints (EN/HU),
the two groups (and their tab order), the missing `scheduler.jobDesc.demo_reset`, removed the stale
`plex_sync_interval_min` field, and "YouTube API kimenő proxy". `i18n/backendIds.contract.test` reads
`sysconfig.SPECS`, `scheduler.JOB_INTERVALS` and the audit/quota action enums from the backend sources
and fails on an id without its name in either language, or a name left for an id the backend dropped.
2026-09-13 07:36:41 +02:00
peter 8731508645 feat(keys): the conversation list walks with j/k; the cheat sheet names keys in the reader's language
E11 S5 — the HU/EN sweep and the keyboard-only pass over every module.

- The keyboard cheat sheet printed "Space" and "Menu" on a Hungarian sheet: a keycap that carries a
  WORD (not a legend every layout prints alike, like Esc, Ctrl or Enter) is now named by the caller in
  the reader's language (`formatChord(chord, words)`, `keys.cap.*`: Szóköz / Menü).
- The Plex player's ⌫/Esc line said "Stop & back to feed", but the player goes back to wherever it
  was opened from (Home, a Plex page): "Stop & go back" / "Leállítás és vissza".
- Messages: the conversation list is a keyboard list (NavGrid + navItemAttrs, the name button is the
  open element; the pop-out button keeps its own Enter). D1 covers every list whose rows open one
  thing; this one was missed in S4. Notification cards have no single open action and stay out, like
  the download queue.

Sweep, measured: every static `t("…")` key used in the source exists in both languages (the only
misses are template prefixes); the HU values equal to EN are proper nouns and loanwords (Admin, Plex,
Trakt, Shorts). Palette in Hungarian: "téma" makes "Téma: Midnight" the active row ahead of the channel
"Topik - mindig témában", live region "7 találat".

Keyboard-only pass in Chrome, Hungarian UI — 1024×768: every `g` chord lands on its module, `j` rings
the first item (solid 2px accent) on Home, Feed, Channels, Playlists, Plex and Downloads, `s` focuses
the search on Feed, Channels, Playlists and Plex, no page scrolls sideways. 3440×1440: `j` → → ↓
walks Home, Feed (8 columns: 0 → 1 → 9), Channels, Playlists, Plex and Downloads, no sideways scroll.
The help sheet lists every line translated; inside the Plex player "Szóköz / K" and "⌫ / Esc
Leállítás és vissza". Messages: a real `j` rings the row, Enter opens the thread.
2026-09-13 07:00:51 +02:00
peter b5d37ca142 feat(keys): j/k and the arrows walk the channel, playlist and download lists
Third and last slice of the E11 keyboard layer (S4c). S4a/S4b covered every card grid and rail; the
table-shaped pages — the channel manager, a playlist, the download center — were still Tab-stop
country: a channel row carries a priority stepper, four action buttons and a tag picker before the next
row's name.

- `ui/DataTable keyboardNav`: the rows are the items, Enter clicks the element a column marks
  `data-nav-open`. DataTable renders its rows twice (a table, and a card list below `md`); both register
  and `lib/listNav` now walks only lists on screen (`checkVisibility`), so `j` past the last row never
  carries on into the hidden copy.
- `lib/listNav` leaves a key something before it already handled (`defaultPrevented`): during a keyboard
  drag a playlist row's grip moves the row with the arrows, and dnd-kit takes the key on the document
  before listNav's window listener. Without the guard ↓ moved the focus off the grip and dropped the drag.
- `ui/NavGrid as="tbody"` for a table whose body is the list (the playlist, under dnd-kit's
  SortableContext).
- Wired: Subscriptions and Discovery tables, the card layout (`ChannelLayoutGrid` → VirtualGrid
  `keyboardNav`), Hidden/Blocked cards (the ChannelLink name is a channel's open element); a playlist's
  rows (the title plays; the grip keeps Space/Enter); the download queue, library and shared lists (the
  play button opens, `IconBtn open`; a row without one opens nothing).
- `listNav.wiring.contract.test`: `VirtualGrid` and `DataTable` must say `keyboardNav` like `Carousel`,
  `DownloadRow` is a card; the audit log's table is the one exemption (its rows open nothing), and an
  exemption that no longer matches a tag fails the test.

Measured in Chrome: Channels table j → row, ring round the whole row; a row walked down keeps 12px above
the viewport edge, walked up clears the sticky header by 35px; Enter opens the channel. Cards 3 columns:
→ 0→1, ↓ 1→4. Hidden and Discovery: j + Enter open the channel. Playlist: j + Enter plays, Escape returns
focus to the row; a keyboard drag (Enter on the grip, ↓) moves the row with focus kept on the grip, and
Escape cancels with the order unchanged. Downloads library: Enter plays a video row, does nothing on a
plain file row.
2026-09-13 06:08:05 +02:00
peter eb0507f942 fix(keys): a card reached by Tab rings the whole card, not just its poster
Owner report: on the Plex browse grid, Tab onto the first card ringed only the poster, and the first
arrow key then moved the ring round the whole card. The poster is the card's own role=button Tab stop,
so Tab gave it the app-wide button ring, while listNav's arrows focus the card root (D2: the ring goes
round the whole item).

The card's "open" element (`data-nav-open` — the Plex poster, a video card's title link) now stands
for the card: when it holds keyboard focus the card root wears the ring and the element's own ring is
transparent. Controls inside the card (watched toggle, actions) keep their own ring.

Measured in Chrome (jsdom evaluates neither :focus-visible nor :has): Plex browse, Tab from the sort
control → poster focused, card outline 2px accent, poster outline transparent; → moves to card 1 with
the same ring. Feed, Tab onto a card's title link → the card is ringed, the link is not.
2026-09-13 05:47:43 +02:00
peter 6d09b94bbb feat(keys): j/k and the arrows walk Home's rails and every Plex grid and rail
Second slice of the E11 keyboard layer (S4b). S4a gave the feed, the library and the channel page
`j`/`k`, the arrows and Enter; Home's shelves and the whole Plex module were still Tab-stop country —
a Plex poster is its own Tab stop, and a rail's cards sat behind every control of the cards before.

- `lib/listNav`: a list that mounts every item no longer declares a count or columns — both are
  measured off the DOM when a key is pressed, so an auto-fill grid's columns follow its width and a
  rail is one row. ↑ on a list's top row and ↓ on its bottom row go on to the neighbouring list, onto
  the item horizontally nearest the one left (owner decision D4: ←→ within a rail, ↑↓ to the rail
  above or below). The first `j` skips cards a rail has scrolled out to the left.
- `Carousel keyboardNav` registers the scroller; the scroller gets a 4px inset (against a -4px margin,
  layout unchanged) so the focus ring is not clipped, and `[data-nav-index]` a 2.5rem inline scroll
  margin so a card stepped into view clears the rail's edge fade and paging arrow.
- `ui/NavGrid`: the plain, fully mounted grid that registers itself.
- `navIndex` on PlexPosterCard / SeasonCard / EpisodeCard (the card root is the item, the poster the
  open element — the ring goes round poster and title, D2), PosterTile and Home's ShelfTile (the button
  is the item); cast, related and collection-strip buttons carry the attrs directly.
- Wired: Home's comfort and resume rails; Plex Browse titles + episodes, Discover rails + genres,
  collection view, playlists/collections browse, person known-for + filmography, show seasons, season
  episodes, cast/related strips, info-page cast + collection strips.
- `listNav.wiring.contract.test`: every `Carousel` says `keyboardNav`, every card says `navIndex`, and a
  card with `navIndex` sits inside a NavGrid / Carousel / VirtualGrid — both halves are opt-in, and
  either one missing leaves a list the keyboard silently skips.
2026-09-13 05:38:36 +02:00
peter 5274216981 feat(ui): PageUp/PageDown page through every dropdown list, not just the palette
The palette got PageUp/PageDown, and the same crawl stayed everywhere else a list opens under a
control: the sidebar's channel filter, the Plex genre/studio/collection pickers, the filter and sort
menus, submenus of every playlist you own. floating-ui's list navigation has only the arrows and
Home/End.

- `lib/listPaging`: the one definition of a page (the rows wholly in view minus one, a fixed 10 with
  no layout) and of where it lands (stops at both ends, never on a row it may not take, looking back
  toward the start rather than past it). The palette now uses it instead of its own copy, and
  `listNav` takes its scroll-parent helper from here.
- `AnchoredPopover`: one keydown handler on the reference AND the surface, so a combobox (keys on its
  input, `aria-activedescendant`) and a roving menu (keys on the focused row) both page. It runs after
  floating-ui's own, which has by then switched to keyboard modality and scrolls the new row in. A
  disabled row is never the target; a submenu's key does not also page the menu around it; with no
  mounted rows (closed, a hover card, a dialog) the key keeps its native meaning.

Measured in the browser: the channel filter shows 8 rows and pages by 7 with the active row in view
after the scroll; a menu pages to its last row and back, the page behind it does not scroll; the
palette still pages by 11 of its 12 visible rows.
2026-09-13 05:20:53 +02:00
peter 75d89613d0 feat(palette): PageUp/PageDown move a page of results at a time
The jump-to palette lists every module, channel, playlist, view and action — 461 rows on the
owner's account, ~11 of them in view — and the arrows were the only way through it: forty presses to
cross the channel group.

- PageDown/PageUp step by the rows that fit in the list right now, minus one, so the row you paged
  from stays in sight at the new page's edge. Measured per press (the list is 50vh and group headings
  eat into it); a fixed 10 when there is no layout to measure.
- Unlike the arrows they stop at the ends instead of wrapping: a page-sized jump that lands back at
  the top reads as having lost your place.
- Home/End stay the text field's own.

Measured in the browser: 0 → 10 → 20 rows, stops on row 461, PageUp from the end steps 9 (a group
heading is in view), the selected row stays visible throughout.
2026-09-13 05:03:35 +02:00
peter 8fb276aa90 feat(keys): j/k walk the feed, s finds the search box, g jumps to a module
A feed card cost ~10 Tab stops, so the tenth card sat ~135 Tabs from the top of the page, and the
only keyboard focus ring (a 60% accent outline on the thumbnail) all but vanished on the dark glass.
First slice of the E11 keyboard layer (S4a); Home, Plex and the tables follow in S4b/S4c.

- `lib/listNav`: one window listener over every registered list, walked in document order. `j`/`k`
  step item to item (on into the next list — owner decision D4), the arrows move around a grid, Enter
  clicks the item's own `data-nav-open` element, so it opens exactly like a click. An item root is
  never a Tab stop: it gets `tabindex=-1` only while the keyboard is on it, and keeps it until the next
  item takes it, so the player's focus restore lands back on the card. The first `j` takes the first
  card you can see, not the top of the list.
- `VirtualGrid` takes part with `keyboardNav` (column count + scrolling an unmounted row in);
  `VirtualFeed` turns it on, so the feed, the library and the channel page all have it.
- Keymap: two-step sequences (`g h`) whose pending step belongs to the scope that started it; the
  page-level scopes (`page`, `search`, `list`) are quiet while a layer that owns the keyboard is open,
  enforced in `dispatch`. A test pins the owner's rule that none of them needs a modifier on the
  Hungarian layout.
- `g h/f/c/p/x/n/m/d/s/,` jump to a module this account can reach; `s` focuses the header search box
  and selects the old query; Escape in it hands the keyboard back to the page (Plex's typeahead spends
  its first Escape on its panel).
- Hover popovers are passive layers: an open tooltip still takes Escape first but no longer silences
  the page keys.
- Focus ring (owner decision D2 = B): solid 2px accent around the whole item; the app-wide button/
  link/row ring goes from 60% to solid too. The item keeps a transparent resting outline because the
  video card transitions `all` — without it every `j` flashed a 3px white ring (measured in Chrome).
- Help sheet: the new keys, in a fixed order after the surface's own; a `g` chord only for a reachable
  module; opened over a player or dialog it leaves out the page-level keys and the context-menu key,
  which nothing can reach there (the latter pre-existing since E10).
- CLAUDE.md: a canonical-patterns row for list/grid keys (`useListNav` + `navItemAttrs` +
  `data-nav-open`, not a private arrow-key handler).
2026-09-13 04:16:20 +02:00
peter 78a9324381 fix(ui): the palette ranks what you typed, and lib stops leaning on components
Typing "theme" put the channel "The Meltzone Podcast" (matched only through its handle
`@themeltzone`) above every "Theme: …" row, and Enter opened the channel — the palette listed
matches in pure group order and always highlighted the first one.

- CommandPalette ranks each word: label starts with it > elsewhere in the label > only the hint,
  keywords or group name; an item ranks by its worst word, then the sum. The group order stays
  fixed, rows are ranked inside each group, and the best row overall starts highlighted. The
  highlight is held by id and follows the best match until the arrows or the pointer pick a row,
  so a channel list landing mid-typing no longer shifts it.
- A11y: result groups are `role="group"` named by their heading; `aria-expanded` is false with
  nothing to choose; the listbox stays mounted (hidden) so `aria-controls` never names a missing
  element, and "Nothing matches" sits outside it; a polite status announces the count once the
  typing pauses.
- `Column`/`FilterMap`/`filterRows`/`isActive` move from `ui/tableHeader` to `lib/tableColumns`,
  and the player's auto/loop modes from `playerModalControls` to `lib/playerModes` — three lib
  helpers imported the column contract from a ui component, and `playerModalHooks` its types from a
  feature component.
- eslint: `lib/` may not import from `components/` at any depth, type-only and re-exports included;
  the boundary contract test proves it fires. CLAUDE.md and the ui README say so.
- keymap: `dispatch`'s docblock sat stacked on `matchesShortcut`'s; moved onto `dispatch`.
2026-09-13 02:14:07 +02:00
peter 8a50538275 fix(ui): a write that fails says what failed and why, wherever it fails
`req` opens its blocking dialog for 400/409/422/5xx only. A 403/404/429 was left to the caller and a
502-504 or lost connection got just the self-clearing connection notice, so 34 mutations without a
reporting onError and 15 direct api writes with a bare or missing catch failed in silence: a hidden
channel that snapped back, a share link that never revoked, a confirmed account delete that ended
with nothing said.

- lib/queryClient: one MutationCache reporter. react-query awaits it BEFORE the mutation's own
  onError, so it waits a task and skips a failure the site already reported; the toast title comes
  from `meta.failTitle`, `meta.errorShownInline` opts out.
- notifyActionError: title names WHAT failed, the line under it says WHY (`failureReason`: a
  structured localized detail, else the status). It marks the error reported (`markReported`), so
  nothing reports one failure twice; `notifyKnownFailure` for a site's own specific sentence.
- Every silent site named (45 new HU/EN titles, 4 reasons). Stale "the global dialog shows it"
  comments corrected on the job actions, channel block, Plex sync, cookies, download enqueue,
  account delete and usePref.
- Channels passed the tag hook's channel id to the reporter as the error; the tag and priority hooks
  now report themselves, with the real error.
- A contract test keeps every useMutation naming its failure; CLAUDE.md gains the pattern row.
- history.dom.test: the "navigates only once the entries are back" assertion raced an async act's
  macrotask flush (1 in 5 runs); it now asserts from a later popstate listener.
2026-09-13 01:45:28 +02:00
peter b4aea7fa5b fix(ui): dialogs, menus and busy buttons keep the keyboard focus where it belongs
Measured in Chrome before each change:

- A dialog opened from a menu lost its focus: the menu row ran its action
  and closed in one batch, and floating-ui's focus return (a microtask)
  moved the focus to the row BEHIND the confirm dialog's scrim 7 ms after
  it opened, where Enter activated the row. MenuItemList now closes first
  and runs the action a task later; the dialog also captures the row, not
  a dying menu item, as its restore target.
- A side panel opened as a flyout collapsed the moment a dialog opened
  from inside it, unmounting the row the dialog hands the focus back to.
  useHoverFocusWithin holds while the focus is inside an aria-modal dialog.
- ui/Button's loading state set native disabled; Chrome moved the focus of
  the pressed button to <body> and never gave it back. Loading is now
  aria-disabled with a click guard that also cancels implicit submission.
- Escape/Enter in the playlist rail's rename field dropped the focus on
  <body>. The row refocuses from its ref callback when it re-attaches (a
  post-commit focus sticks; the earlier attempt ran before the button
  existed).
- PlayerModal was the one aria-modal shell without the dialog contract:
  it is now named by its title, traps Tab, restores the focus to its
  opener and shares one counted body scroll lock with every dialog.
2026-09-13 01:04:11 +02:00
peter 92c305bab9 fix(nav): a navigation closes every overlay before it pushes
The overlay history stack let a navigation land ABOVE open overlays and
"buried" their entries with a counter. Measured in Chrome: Shift+W over
About navigated under the still-open dialog, the next Back both undid the
navigation and closed About, and the Back after that did nothing; Shift+W
on Home with the player open did nothing at all. The counter could also be
spent by a different overlay closing first, and callers that could not
report whether they pushed were assumed to have.

setPage and openChannel now close every open overlay and push only once
the history is back in sync (one entry per live overlay, no programmatic
back() in flight, no reconcile queued), so nothing is ever buried and the
whole mechanism goes: navigateOverOverlays, closeThenNavigate, buryUnder,
setPage's boolean, and the three test files that pinned them. The palette
closes only itself before running a row, so a theme picked over the player
keeps the player. Shift+W on Home with the player open now closes it
(owner decision 2026-09-09).

The wait is on state, not on "a popstate": the old 250 ms fallback could
run with back()s still in flight. An overlay that refuses to close within
1.5 s drops the command instead of being navigated out from under.
2026-09-13 00:17:09 +02:00
peter 32fe0768b6 release: 0.84.0 (bug/hunt-2026-09)
gate / frontend (push) Failing after 2m45s
gate / backend (push) Successful in 4m5s
2026-09-12 16:44:01 +02:00
peter 415a6c16f9 release: 0.84.0 — the owner's sixteen-item bug hunt
Sixteen items written from real use: eleven fixed, one measured and closed without a code change
(the micro-stutter is a 23.976 fps film on a 120.0148 Hz display, reproduced identically outside
Siftlode), two parked for a reproducible case, one scoped out to its own sprint. Two of the eleven
were destroying user state — a YouTube video that rewound itself on reopen, and a "clear resume
point" that told Plex nothing about an in-progress item.

Then seven deep-review rounds over the fixes and over each other's fixes: 32 → 24 → 43 → 31 → 22 →
26 → 18 raised. Minor, not patch: item 14 added an admin knob, item 10 a sort control, and items
2/4/5/7 are features.
2026-09-12 16:43:54 +02:00
peter 8c51062e5b fix(api,ui): the demo refusal says why at every site, and the contract guard cannot fail open
Round 6 converted three of four `is_demo` 403s to a structured detail and wrote a docstring saying it
had converted all the sites there are. The account-delete route in `routes/me.py` still answered a
hand-written English sentence, which `notifyActionError` discards in favour of the caller's
translated fallback — so the demo user gets no hint that the demo account is the reason. It is
UI-gated today, so the string is unreachable; what was reachable was the false completeness claim,
which is precisely what stops the next reader from looking. The site is converted, and the list is
now DERIVED from the source: a new test walks `app/` for every `is_demo`-guarded 403, asserts each
raises a structured detail, and fails when the count moves — so this file cannot fall behind the code
the way it just did.

The same round's frontend contract guard was failing open. Its pattern required a bare identifier
before `.detail` and excluded `.detail` used as a last call argument, so `setErr(e.detail)`,
`(e as HttpError).detail`, `String(e.detail)` and four more idiomatic forms were invisible — and
because the file set is compared with `toEqual`, a file the pattern cannot see is simply never listed
and the gate stays green over a fresh copy of the contract. Widened to cover cast receivers, optional
chains and call-argument reads; the pinned set is unchanged, so the narrowing had removed no false
positive. A destructured `detail` stays out of reach (a pattern loose enough for it fires on
react-query's own `const { data: detail }`), and that hole is written into the test rather than
papered over with two innocent files.

Its anti-vacuity guard also did not guard anything: the comment says the glob must have matched
something, but the assertion under it read a sibling file by its own URL, which resolves whether or
not the glob enumerated anything. That only mattered while the pinned list was non-empty — the moment
the deferral sweep lands and it becomes [], the whole file would pass over a glob matching zero
files. The scan size is asserted now.
2026-09-12 16:22:27 +02:00
peter 63bf1bca78 fix(plex): an accepted reorder refetches the detail it just wrote
`commit`'s success handler writes the dragged list into the detail cache — round 6's fix, so a
re-entry inside the 30s staleTime renders the accepted order without a request — but then invalidated
only the LIST key, while the YouTube sibling it cites invalidates both. `next` is the membership of
the render that started the drag, and `reorder_playlist` only touches the rating_keys it is handed,
so it never ratifies that list: anything that landed meanwhile was clobbered by the write with
nothing scheduled to correct it. `commit` sets no busy flag either, so a removal really can complete
while a reorder is in flight, and with staleTime 30s and refetchOnWindowFocus off the phantom row
stayed on screen for as long as the view was mounted — and the next drag persisted an order built
from it.

Round 6's test had to be re-pointed, because it was written in a world with no refetch: its mock
answered the pre-reorder order forever, which a server that just accepted the write cannot do. It now
holds the refetch open instead, so it still pins what round 6 actually fixed — the cache is right
WITHOUT waiting for a request — and would fail if the write were deleted. Both halves are
mutation-checked against each other.
2026-09-12 16:22:27 +02:00
peter c7a377396b fix(ui): a closing dialog leaves the focus to the dialog standing above it
Round 6 widened dialog focus ownership to count focus inside ANY `aria-modal` subtree as ours, so a
dialog and the dialog stacked on it closing together would still hand the focus back. That clause has
no way to tell a layer closing WITH us from one that stays open, so a dialog closing on its own — an
in-flight mutation's `onSuccess`, not a click, since a scrim covers it — under a modal raised above it
pulled the focus onto its own pre-open trigger, behind that scrim, where Enter still activated it.
Three verifiers confirmed the mechanism and two corrected the severity down: it self-heals on the
next Tab, which is why this is a containment violation rather than a broken end state.

`lib/layerStack` cannot refine it. Measured against the real hook: at the instant the lower dialog's
focus cleanup runs, the both-closing case and the sibling-stays-open case give identical readings —
`anyLayerAbove` true, the upper layer topmost — because passive cleanups run parent-first and
`useDialogChrome` is declared before the effect that calls `removeLayer`. Gating on the stack turns
round 6's own test red. What separates them is the DOM: React detaches a deleted subtree before
passive cleanups, so a layer closing with us is already disconnected while one that stays open is
not. Both tests are green, and each was mutation-checked against the other.

Verified in Chrome on localdev: opening the cheat sheet from a sidebar button and closing it restores
that button; with the command palette stacked over the cheat sheet, closing the top one leaves the
focus neither on the background trigger nor on <body>, and closing both returns it to the trigger.

Welcome's Lightbox got the trap and the restore in round 6 but not the layer membership the other
shells have, so its Escape and Tab fired from a raw window listener whatever was on top — the shape
the wizard's comment records being bitten by, unreachable today only by a mount ordering nothing
enforces. It joins the stack. Its comment also read as though the four-shell sweep were finished:
`PlayerModal` is still an `aria-modal` surface with no name, no trap and no restore, which is on
ROADMAP.md now rather than implied to be done.
2026-09-12 16:22:06 +02:00
peter 8036251589 fix(plex): the reclaim window rolls, and a refused finish is not performed through a third route
Two ways round 6's own fixes did not bound what their comments said they bounded.

The reclaim budget's window anchor was written only where the window RESET, so it marked the window's
start rather than the last loss — a tumbling window, not the rolling one the constant is named for.
Any loss cadence slower than half the window satisfied the reset on every second loss, so the budget
never reached its bound: measured over 200 loss cycles, 61/70/90/119s apart all gave 200 reclaims and
never an error, while 5s and 30s apart correctly gave two and then the honest answer. The failing
band is exactly the case the ERROR handler names — a transcode that falls behind realtime — and both
committed tests spaced their losses 0s and 10min apart, i.e. outside it. The anchor now moves with
every loss.

And `finish`'s refusal handler lowered the play-through latch, which is the only gate on the 10s
interval, the page-hide beacon and `settle`. All three post `absRef`, which on the play-to-the-end
path IS the end of the media, and the backend reads a position within `_FINISH_MARGIN_S` of the
duration as a finish — so within ten seconds of the toast saying the mark had failed, the item was
marked watched, its position cleared and a scrobble sent to Plex. Round 6 closed the explicit
rollback write and left the other three open; the test it added set `upNext: false`, which tears the
effect down before any of them can fire. The refusal now takes back the CLAIM rather than the latch:
a second flag stops `settle` re-asserting watched, and playing again clears both.

`FINISH_MARGIN_S` duplicated the backend's constant in a second language with nothing relating them,
and its comment had the asymmetry backwards — if the BACKEND's margin grows, this guard starts
admitting a rollback the backend reads as a finish, i.e. re-opens the defect above. Pinned by a
contract test that reads the Python, and the comment corrected.
2026-09-12 16:21:51 +02:00
peter 71d7646bf1 fix(plex): a name this session cannot write fails at once, and the layout has one spelling
Round 6 turned `artifact_may_arrive` into a positive rule, but keyed it to the HLS vocabulary in
general rather than to what THIS session produces: the playlist regex accepted master/index/stream_*
for every session and the `seq == 0` branch accepted any variant token unconditionally. The two argv
branches are disjoint — a single-audio session is given `seg_%d.ts` + `index.m3u8` and no `%v` at
all, so `master.m3u8`, `stream_*.m3u8` and every `seg_<variant>_<n>.ts` are names it can never write,
and vice versa — so each of those was read as "an HLS name, so wait and see" and held a threadpool
worker, plus its pooled DB connection, for the full timeout. That is verbatim the round-5 outage the
route's own comment claims it prevents.

Measured through the real route against a live single-audio session, before: master.m3u8 404 after
30.0s, stream_q7z.m3u8 404 after 30.0s, seg_a0_0.ts / seg_zzzz_0.ts / seg_0_0.ts 404 after 20.1s
each. After, on the running localdev API: 404 in 44-316ms for all of them, while index.m3u8,
seg_0.ts and seg_1.ts still answer 200 — nothing is over-refused.

The session now carries the rendition names its ffmpeg was told to write, and `entry` is derived from
them instead of passed. And "one place" now means the argv BUILDS its patterns from the same
functions the parser reads back (`segment_name("%d", "%v")`, `variant_playlist_name("%v")`): the
layout was still spelled out independently in six places, with `_VIDEO_VARIANT` read by no consumer,
which is the drift that caused round 6's HIGH. A test asserts the argv's own patterns are what those
functions produce, and that every name they expand to is one the guard accepts.

The load-bearing MEASURED comment cited ffmpeg 8.1.1 — the Windows host's. The binary that writes
these files is the API container's 7.1.5; re-measured there, the conclusion holds.
2026-09-12 16:21:37 +02:00
peter 3eb0f41962 fix(plex): a Plex reorder the server accepted is the order the cache holds
The YouTube list got this rule in round 5 — one accepted write, one cache entry
that matches it — and the Plex list got only the rollback half of that fix. Its
success path invalidated `qk.plexPlaylists()`, a disjoint key root, and never
touched `qk.plexPlaylist(id)`: the key `putBack` rolls back to, and the one a
re-entry renders from without refetching.

So after an accepted drag the detail cache still held the pre-reorder order.
Leaving the playlist and coming back inside the 30 s staleTime showed the
reorder undone, and the next drag from there persisted that — silently undoing
an order the server had taken. Measured in jsdom: server 2,1,3, cache 1,2,3, no
refetch.

1 test, mutation-checked.
2026-09-12 05:50:40 +02:00
peter a0114deff1 fix(api,ui): a refused write says why, wherever the refusal happens
Round 5 taught `notifyActionError` to discard a plain-string `detail` in favour
of the caller's translated fallback, because `req` leaves 403/404/429 to the
caller and those routes hand-write English. Three places did not get the
message.

**The demo account lost its explanation.** `require_human` refused with a plain
string, so from that commit on the shared demo account got its caller's generic
"couldn't do that" and no hint that the demo account was the reason — and the
gate is not backed by a UI gate everywhere: the comfort shelf has no `is_demo`
check, so the pin button is live and simply fails. A structured `{code: "demo"}`
and an `errors.demo` string in both locales. The two auth routes that hand-roll
the same check were swept with it.

**`ChatThread` carried a second copy of the contract.** It filtered 404/429
itself and then did `e.detail || t(…)`, the pre-fix order, so the central fix
could not reach it and "That member hasn't set up messaging yet." stayed English
under a Hungarian UI. It goes through the shared helper now, and a contract test
pins the remaining raw-`detail` consumers as an explicit SET — never a count —
so a new copy fails the gate instead of shipping. The five on that list are
inline surfaces (a form field's error line, a setup banner) and are deferred
with the ~60 backend routes whose details they print.

**The Plex "Add to playlist" dialog was still silent.** Same shape as the
collection editor round 5 fixed and the YouTube twin that already had the fix:
no catch, so a playlist deleted on another device answered a structured 404 that
nothing reported — the tick never flipped, the gone row stayed, and the floating
promise became an unhandled rejection.

Also: the `ABORTED` sentinel's VALUE was asserted nowhere. The consumer test
hand-built the error it expected, so replacing `xhr.onabort`'s rejection with a
friendlier message left the lane green — and every upload the user cancels would
then raise a red "the upload failed" toast, which is what the sentinel prevents.
Both producer tests assert through `isAborted` now.

And two backend tests were pinning fewer sites than their own docstrings claimed:
the channel-404 loop said "all five" and issued three requests (explore and
subscribe unexercised, the excuse for subscribe naming a different call site
with a different reason slug), and the recipient/job test named three refusals
and requested two — `share_download`'s `not_found("recipient")` had no test in
the suite at all, nor did the locale keys added beside it. Every site named now
has a request behind it.

6 tests, mutation-checked against 7 mutations: the demo detail at both the gate
and an auth route, a re-hand-rolled ChatThread, the silent Plex dialog, the
abort sentinel, and each unpinned 404 — all red on the test that names them.
2026-09-12 05:50:31 +02:00
peter d89d887a9b fix(plex): the reclaim budget bounds what it says it bounds, and a refused finish is not performed anyway
`MAX_SESSION_RECLAIMS` was cleared by the first fragment of any session, and a
reclaimed session essentially always delivers one — `start_session` does not
return until its first segment exists. So the budget only ever bit where a
session parsed its manifest and then died before a single fragment, which is not
either case its own comment names. Measured: 40 loss cycles produced 41 sessions
and never an error. A transcode falling behind realtime, or two viewers of one
rating_key taking turns, respawned ffmpeg indefinitely — the round-5 behaviour,
still there behind the guard added to stop it.

It counts within a rolling window now. Two losses inside two minutes is a
session that keeps being lost; one an hour apart is not. And the window is reset
per item, because the player stays mounted across a queue advance — otherwise a
budget spent on one episode would deny the next its first legitimate reclaim.

Separately: `finish`'s rollback re-performed the write it was taking back. On
the play-to-the-end path the position it restores IS the end of the media, and
the backend reads a position within its finish margin as a finish — so the
branch named "take the finish back" set watched, cleared the position and
scrobbled to Plex through a second route, while the caller's error toast was
already on screen saying the mark had failed. There is no resume point at the
end of the media to restore; the cache drop is the whole of the rollback there.
The credits-skip caller captures a mid-episode position and still restores it,
which is the case that needs it.

7 tests, mutation-checked: the FRAG_LOADED reset, the missing window, and the
unguarded rollback each red on the test that names them.
2026-09-12 05:50:11 +02:00
peter f8a31d48cf fix(ui): a dialog keeps the focus it was given, and hands it back where it took it from
Round 5 extracted `ui/dialogFocus` so no shell would hand-roll the focus
contract, then introduced four ways to lose the focus anyway — all four measured
in jsdom against the real components.

**Ownership was narrower than the trap.** `ui/Modal` traps Tab over the whole
dialog on purpose, so a toast's Undo or ✕ stays reachable; ownership was tracked
over the panel alone. Tabbing to that ✕ therefore read as "the user left the
dialog", and since removing a toast fires no `focusin` to re-arm, closing
afterwards skipped the restore and left the focus on <body>. Before this hook
existed the restore was unconditional, so it was a regression.

**A dialog stacked above us is still ours.** A nested `ui/Modal` portals into its
own host, so it is outside the outer dialog's root however wide that scope is
drawn — widening alone could not fix it. Both closing in one commit left the
focus on <body>: the outer skipped its restore, and the child's own target (a
control inside the outer panel) was already detached, because passive cleanups
run parent-first. Ownership now means "inside my dialog, or inside a modal layer
above it", which is what the rule always intended.

**The wizard disagreed with `ui/Modal` on its first line.** `trapTab(e, panel)`,
with neither the dialog-root scope (so its own toast host was outside the cycle
— the connection notice never auto-dismisses, and its ✕ was unreachable from the
keyboard) nor the `container` argument the JSDoc exists for: `panel.contains(
panel)` is true, so focus sitting ON the tabIndex=-1 panel matched no branch and
a Shift+Tab walked into the app behind the scrim. Both call sites now pass the
same three arguments.

**The toast handover was a render-phase read-modify-write.** A `useMemo` read a
module global that the next line overwrote; React re-invokes a component body
and every `useMemo` factory in it on a StrictMode double render, and the LAST
invocation commits — so the second pass re-read what the first had just written,
the takeover host classified its OWN new toast as inherited, and the dev build
behaved exactly like the code round 5 replaced. Snapshotted through a ref keyed
on `isTop`, so a repeated invocation of one render is a no-op.

And the fourth `aria-modal` shell, `Welcome`'s screenshot lightbox, is finally
swept into the same contract: it declared itself a modal dialog while leaving
the focus on the thumbnail behind an 80%-black scrim and letting Tab walk the
landing page underneath.

The headline fix of the previous commit — StrictMode's re-mount taking the focus
off an `autoFocus` field — had no test at all; deleting it left all 1009 green.
It has one now.

9 tests, mutation-checked against 8 mutations, each red on the test that names it.
2026-09-12 05:49:58 +02:00
peter 7f17b17c13 fix(plex): an HLS session's file names are one vocabulary, and a name it cannot produce fails at once
Round 5 added a guard so a request for a segment beyond the produced region
answers instead of polling out the full timeout — the route is a SYNC one, so
every such wait parks a threadpool worker and, through `current_user`, a pooled
DB connection. Measured against real ffmpeg, the guard could not fire where it
mattered.

`-hls_segment_filename seg_%v_%d.ts` substitutes the VARIANT first: a dual-audio
session writes seg_0_0.ts (video) and seg_a0_0.ts / seg_a1_0.ts (the `name:aN`
renditions). The regex read those as sequence-first, so every video segment
parsed as segment 0 and every audio segment did not parse at all — the guard was
inert on every multi-audio item, which is most of this library. `_seg()` carried
the same inversion: the warm-up waited 8 s for `seg_1_0.ts`, a name ffmpeg never
writes, on every multi-audio session start AND every seek-restart.

The layout is now one constant plus one parser (`segment_name` /
`_SEG_NAME_RE`), with a test pinning them to each other, instead of three
comments that could drift — and they had drifted, all three ways.

The guard is also a positive rule now. It was a blocklist, so anything that was
not a single-audio segment name was waved through: `GET …/anything.m3u8` still
parked a worker for 30 s, while the route's comment claimed a flood of hopeless
names was what it prevented. A session produces its playlists and its segments
and nothing else. `.vtt` leaves `_HLS_CT` with it — ffmpeg runs with `-sn` and
subtitles are a separate route, so that extension could only ever name a file
that will not arrive.

Also here:
- A 5000-digit segment index reached the client as a 500: `int()` on an
  unbounded digit run raises past 4300 digits and only OSError was caught. The
  name is bounded on the route and the index in the regex — the same bug class
  the subtitle routes were bounded for.
- A session torn down between the wait and the read is a 404, not a 500. `_kill`
  rmtrees the directory and no lock spans those two steps; the player's recovery
  is keyed on the STATUS, so a 500 skipped the reclaim and ended playback with
  "couldn't play" instead of restarting at the playhead. The trigger is ordinary
  — `loadSession` awaits the new session before destroying the old hls instance.
- Two comments claimed an open player checkpoints every 10 s "whether or not it
  is playing", which is the stated reason the 30 s finished-idle reap is safe.
  `flush` sends nothing once the finish latch is up, or before a position is
  known. The states it does cover — playing or paused mid-item — are the ones
  the rule needs; the comments now say that instead of more.
- The route's tests leave the DB lane: the route reads nothing from the database
  since the auto-start was removed, and there those four regressions SKIP on a
  machine with no Postgres while pytest prints green.

14 tests, mutation-checked: the old parse, the old warm-up name, the blocklist,
both bounds off the absurd name, the unguarded read, and the route dropping
`proc` — each red on the test that names it.
2026-09-12 05:49:38 +02:00
peter 1f81684b6e fix(ui,plex): a dialog keeps the focus its child asked for, and a hopeless segment fails at once
Both found by UAT in a real browser, after the tests for them were already green — which is the
whole reason this repo settles focus and timing claims by running them.

- **The autoFocus fix worked, and then the dialog took the focus back.** React 18's StrictMode runs
  every effect mount→cleanup→mount in development: the cleanup handed focus back to the trigger, and
  the second mount, finding the focus outside the dialog, moved it to the first focusable — off the
  `autoFocus` field a beat after the browser had put it there. Chrome measured it: INPUT (commitMount)
  → BODY (cleanup) → ✕ (re-mount). The effect is idempotent now — it remembers what the dialog was
  focused on and restores THAT on a re-mount — and the close-time restore is conditional on the
  dialog still holding the focus, which it tracks as it changes rather than computing at cleanup,
  where the panel is already out of the document. A dialog no longer yanks focus back from wherever
  the user has since moved it.

- **Passing ffmpeg's handle to the segment wait only shortened the FINISHED case.** Measured against
  the running localdev API: a request for `seg_99999.ts` on a live session still took 20 195 ms,
  because that ffmpeg has not exited and the wait has no other reason to stop. ffmpeg writes segments
  in order, so a segment whose predecessor does not exist is not the next one and cannot arrive —
  and that is knowable at once. Same request now: 45 ms.

Verified on localdev against the real library (Westworld S01E06, remux, multi-audio): Mod+K opens the
command palette with the focus in its search field and typing reaches it; playback runs on the new
session-scoped HLS URL; a request naming a stale session id is a 404; and a viewer paused for longer
than the finished-session rule keeps their 2.4 GB session across two reaper passes — the progress
checkpoint touches it — and resumes cleanly.

vitest 1009, pytest 1127.
2026-09-12 04:31:51 +02:00
peter 33fddacee2 test(plex): the over-long-name guard is asserted the same way on every OS
Deep-review round 5. The assertion behind `subtitles._exists` was only red where pathlib re-raises
ENAMETOOLONG: on Linux it does, on Windows the same stat fails with ERROR_INVALID_NAME, which
pathlib's `exists()` already swallows — so the assertion passed with the fix reverted. It forces the
errno now instead of relying on the filesystem. (The gate's pytest lane is the Linux container, so
the branch was never actually unguarded; the test was just dishonest about why it passed.)
2026-09-12 04:18:35 +02:00
peter 12121255dd fix(playlists): a rejected reorder goes back to the order the server accepted
Deep-review round 5, over round 4's own fixes.

- Round 4 made a failed reorder roll the rows back to the CACHED detail, because a resync cannot do
  it (the server's list is unchanged, so the refetch is deep-equal, react-query keeps the same
  `data` object and the effect keyed on it never runs). But nothing wrote an ACCEPTED order into
  that cache — only the refetch that follows a successful reorder did. So a second reorder rejected
  while that refetch was still in flight rolled the rows back past the first one, which the server
  had accepted; the membership guard cannot correct an order (its key is the sorted id set), so the
  wrong order stayed on screen for good and the next drag persisted it. `onApply` now writes the
  accepted order into the cache before invalidating.

- `PlexPlaylistView.rename` had a `catch` no test could reach, and `commit` (reorder) was not gated
  by `busy` the way its siblings are — a reorder in flight while a removal completes is what makes a
  rollback ambiguous in the first place. The rename failure path is now covered; `commit` defers.

- `putBack` read `q.data` from the render that bound the failing handler instead of the cache at
  rollback time, unlike the sibling `restoreFromCache`. Aligned. Stated plainly: I could NOT drive
  that staleness in jsdom — a mutation that restores the old shape leaves every candidate test green
  — so this one ships as an alignment with the proven sibling, not as a fix with a red test behind
  it. The scenario and the harness limit are on ROADMAP.md.

Two regression tests, both mutation-checked red (the cache write, the rename catch). The
drag-reorder failure path still has no test: dnd-kit's pointer and keyboard sensors do not activate
under jsdom even with geometry and scrollIntoView stubbed, so a test there would have pinned the
harness, not the code. Logged, not faked.

vitest 1009.
2026-09-12 04:15:15 +02:00
peter 4a7b65d187 fix(api): a refused write is named, in the reader's language
Deep-review round 5, over round 4's own fixes.

- `notifyActionError` preferred the server's `detail` over the caller's translated fallback, and
  `reportFailure` deliberately leaves 403/404/429 to the caller — so for exactly those statuses the
  dozen call sites round 4 converted started printing whatever English a route hand-wrote into an
  otherwise Hungarian UI ("No user with that email.", "Not subscribed to this channel", "Unknown
  job"). A STRUCTURED detail is still preferred: `req` has already run it through `localizeDetail`,
  so it is both localized and more specific. A plain string is not.

- The other half of the same defect, on the backend: round 4's `get_or_404(what="channel")` sweep
  reached exactly ONE of the five identical sites in channels.py and shipped the HU/EN string with
  it, so the Block button said it in Hungarian and the four controls beside it in English. All five
  are converted, and the subscription / share-recipient / scheduler-job 404s with them (two new
  reasons in both locales).

- `PlexCollectionEditor`'s add/remove, take-over and delete were try/finally with no catch — in the
  very file round 4 edited to give `createNew` one. 403 and 404 are the caller's to report, so the
  tick simply never appeared and a CONFIRMED "delete collection" did nothing, silently.

- `UploadDialog` is the only `uploadReq` caller and kept an empty catch whose comment still claimed
  the global error modal had it. It had not since `reportFailure` became shared: a proxy 502 raised
  only the self-clearing connection notice and a 413/403/404 nothing at all, so the dialog snapped
  back to its pre-upload state with no error named. The user's own Cancel stays silent — and since
  a cancel and a network drop both carry status 0, that difference is now a named `isAborted`
  rather than a string literal at each end.

Regression tests for each (5 vitest, 2 pytest, plus two existing assertions corrected), all
mutation-checked: 6 mutations, each red on the assertion it is named after. vitest 1007,
pytest 1127.
2026-09-12 04:07:25 +02:00
peter 47af4d5c05 fix(plex): an HLS artifact names the session it belongs to, and a paused viewer says so
Deep-review round 5, over round 4's own fixes.

- Sessions are keyed by rating_key alone while a session's DIRECTORY is per (key, offset, tag), and
  the artifact URL carried no session identity — so once a seek-restart or a second viewer of the
  same item took the registry slot, the old viewer's request for `seg_12.ts` was answered 200 with
  the new session's twelfth segment: bytes from a different `-ss` offset under `-copyts`, on a
  timeline the client is not on. Wrong content with a success status is the one failure a player
  cannot notice, and round 4's reclaim-on-404 could never fire on it. The id is in the path now (so
  the playlist's relative names resolve back into the same session), a mismatch is a 404, and the
  player reclaims at the playhead. The entry playlist's auto-start is gone with it: it could only
  ever begin at 0, silently discarding the viewer's position.

- Round 4 exempted paused viewers from the reaper by keeping EVERY finished session for the full
  ten-minute idle window — one whole segmented copy of a film per item, on the volume the download
  center shares. The signal it needed already existed: an open player checkpoints its position every
  10 s whether or not it is playing, so that route now touches the session and the periodic pass is
  back on the short rule.

- POST /session's opportunistic reap was never scoped to the key being started: it walked the whole
  registry with the 30 s finished rule, so pressing play on ANY item — even a direct-play one, the
  call ran before that early return — killed another viewer's paused session. It is gone; the
  periodic pass owns reaping.

- A segment that can never arrive no longer polls out the full 20 s: the producer's handle is passed
  to `wait_for`, so a finished ffmpeg answers at once. This is a sync route, so each such wait parked
  a threadpool worker AND a pooled DB connection, and an ordinary seek past the produced region was
  enough to do it.

- The DataError handler's re-raise branch — the half that keeps a genuine DataError a 500 — is
  pinned, so simplifying it to an unconditional 422 cannot pass in silence.

Six regression tests (two route-level, in the DB lane), each mutation-checked red on the assertion
it is named after. pytest 1124, vitest 999.
2026-09-12 03:57:52 +02:00
peter 8a83bc0e9f fix(plex): a finished item is finished when the server says so, and a lost session is reclaimed a bounded number of times
Deep-review round 5, over round 4's own fixes.

- Round 4 added a `finish` latch so a played-through item records no checkpoint at the credits.
  On a NATURAL play-through it was raised a beat too late: per the HTML spec the browser sets
  `paused` and fires 'pause' BEFORE 'ended', so the pause checkpoint went out while the latch was
  still down and raced the watched mark — both requests read "not watched yet" from their own
  snapshot, so Plex counted one watch twice, and where Plex's reported duration runs more than the
  finish margin past the real media the checkpoint wrote a resume position AFTER the scrobble. The
  pause that ends the media no longer flushes. The regression test that was meant to cover this
  modelled the end as a bare 'ended' event — a browser that does not exist — and is now the real
  order.

- The watched mark was fired and forgotten (`.catch(() => {})`) while the latch patched the cached
  detail to watched / position 0 regardless, and round 4's once-per-id start guard meant the
  corrected detail arriving later was never used: an episode whose mark failed reopened at 0:00
  with a Watched badge the server disagreed with. `finish` now takes the in-flight write and takes
  itself back if the server refuses it — the real position is reported, the lying cache entry is
  dropped, and the failure is said out loud instead of swallowed. Both callers (the natural end
  and the credits auto-skip) hand it over.

- The reclaim of a 404'd HLS session had no bound. `parsed` cannot provide one — it belongs to one
  hls instance and every restart builds a fresh one — and the server answers 404 both for "the
  session is gone" and for "that segment is not there yet", so a transcode falling behind realtime,
  or two viewers of one item taking turns, respawned ffmpeg forever behind a spinner that never
  became an error. Two consecutive reclaims, reset by the first fragment that actually loads, then
  the honest failure.

Five regression tests, each mutation-checked red on the assertion it is named after.
2026-09-12 03:47:42 +02:00
peter 41b652f8fd fix(ui): a dialog opens focused where it said it would, and the wizard is a dialog
Deep-review round 5, over round 4's own fixes.

- Round 4 gave `ui/Modal` a persistent portal host so a fullscreen transition would stop tearing
  the dialog down. The host was attached in a `useLayoutEffect` — one commit phase AFTER React
  runs `autoFocus` in `commitMount`, and `focus()` on a disconnected node is a no-op. Every
  autoFocus in every dialog was dead: Mod+K opened the command palette with the focus on the ✕,
  so typing searched nothing and Enter closed it; the same for the download rename field and
  every non-danger confirm. The host is attached in a `useInsertionEffect` now, a whole phase
  earlier, and the whole re-parent — including the focus save/restore across the move, which
  `appendChild` needs because it is remove-then-insert — lives in one `useFullscreenPortalHost`.
  `ui/AnchoredPopover` had a second copy of that re-parent WITHOUT the focus restore, and a
  comment claiming focus survived it; both now come from the hook.

- `ui/Toaster` classified a toast raised in the SAME commit as a host takeover among the ones it
  inherits, because it read the store live at the takeover render. A dialog that notifies and
  closes in one handler (UploadDialog) therefore put its toast up with no role, into a region
  still `aria-live="off"` — announced by neither path. "Inherited" is now what the outgoing host
  last had on screen, and a toast that is NOT inherited is announced even inside the unarmed
  window.

- The onboarding wizard renders its own shell, and Escape ordering was all of the dialog contract
  it had: no role, no label, no initial focus, no trap, no restore. A keyboard user tabbed out of
  the first-login wizard into the app behind the scrim. The focus half of the contract is now one
  shared `ui/dialogFocus` that both `ui/Modal` and the wizard use.

Every fix has a regression test, each mutation-checked (6 mutations, all red on the assertion
they are named after).
2026-09-12 03:41:37 +02:00
peter 545996f807 fix(ui): a toast host ranks by when its overlay opened, and an overlay keeps its Escape, its dialog and its navigation straight
Deep-review round 4, the overlays.

- Toast hosts were ranked by mount time, and PlayerModal swaps its host on every fullscreen change:
  leaving true fullscreen with the download dialog open put the player's host back on top, so toasts
  were drawn under that dialog's scrim and a click on one closed the dialog. Each overlay takes a
  rank once, when it opens (`useOverlayOrder`), and hands it to every host it renders.
- A dialog open while fullscreen is left keeps a stable portal container (moved, not re-created):
  React tore the portal down when `fsRoot` changed, resetting a half-filled form and dropping focus
  to <body>. Focus is put back after the move.
- The onboarding wizard is a layer, not a raw document listener: one Escape used to dismiss the
  error dialog over it AND end onboarding, skipping the write-scope step.
- `navigateTo` (a toast's "View", clicked inside the player's own host) goes over the overlays it
  unmounts, so the player's Back-to-close cleanup no longer pops the page it just opened — the same
  fix Shift+W already had. The access-requests toast action too.
- The live region arms itself a task after it takes over inherited toasts: React flushes effects in
  the same task as that render, so the "silent inheritance" arming was a no-op.
- The overlay-host guard matches class tokens in any order (PlexPlayer's root was never checked),
  globs `ui/` too (ui/Modal was not checked either), and requires the rank.
2026-09-11 23:43:23 +02:00
peter 267fd7f70f fix(playlists): a refused edit is taken back on screen, and a gone list or tag is said, not swallowed
Deep-review round 4, the playlist surfaces.

- A failed move or reorder relied on the resync to put the rows back — but the server's list is
  unchanged, so the refetch comes back deep-equal, react-query keeps the same `data` object, and the
  re-seed effect keyed on it never ran: the moved row stayed gone, the rejected order stayed on
  screen. Both restore from the cached detail now.
- A failed removal puts its row back itself: when the server is unreachable the resync fails too,
  and round 3's "keep the list on a failed refetch" guard then kept showing it as removed.
- The Plex playlist page (remove / rename / delete / reorder), the add-to-playlist popover and the
  tag manager had no failure handling at all: a list or tag deleted elsewhere (a 404, now a
  structured `not_found` the request layer leaves to the caller) failed in silence and an
  optimistic removal stayed removed. Each says it and resyncs; deleting a Plex playlist that is
  already gone just leaves the page.
- Playlists' own loading/error branches use the shared `LoadingState` / `StateMessage` (the loading
  line had no live region).
- The "player stays mounted" tests count mounts: a remount ended with the same flag as staying up.
2026-09-11 23:34:33 +02:00
peter 6d49da17fe fix(api): a write that hit a connection failure is still reported, and every failure is reported once
Deep-review round 4, the "one report per failure" contract round 3 introduced.

- `req` marked gateway (502/503/504) and network failures "already reported" because they raise
  the connection notice. But that notice is about connectivity and removes itself on the very next
  answer — which every caller's own resync refetch fetches at once — so a delete that hit a
  keepalive reset flashed the notice and was then reported nowhere, the list still there. Only the
  durable reports (the error dialog, the 401 sign-out) count now; the caller says the action failed.
- A connection notice the user closed swallowed every later one for the rest of the visit; it is
  raised again.
- `req` and `uploadReq` share one failure policy (`reportFailure`). The upload copy had drifted: a
  502 opened the blocking "Server error (502)" dialog and cleared a notice that was still true.
- A taken Plex collection name is a structured 409, localized, and the create/rename calls are
  quiet: it was an English dialog plus a Hungarian toast. `notifyCollectionEditError` is gone.
- The mutation onErrors that toasted unconditionally after a non-quiet request (invite approve /
  deny / add, demo whitelist, audit clear, scheduler, share, full history, edit, config tests, tag
  attach) go through `notifyActionError`; the playlist delete's hand-rolled copy of it too (it takes
  a level now). The user role / suspend / delete actions had no onError at all: a user deleted in
  another tab failed silently and stayed listed — now said, and the list refetched.
2026-09-11 23:27:48 +02:00
peter 6a92585098 fix(plex): the player restarts a reaped session, finishes a played-through item as watched, and starts each item once
Deep-review round 4, the Plex player.

- A 404 on an HLS session that WAS playing means the server reaped it (a long pause, a deploy);
  hls.js never retries a 4xx, so playback ended with "couldn't play". A new session at the playhead
  now takes its place (the frontend half of the paused-viewer reap fix). A session that 404s before
  its manifest is still a plain failure, so it cannot restart itself in a loop.
- Leaving an item that ended, or that the credits auto-skip left, sent a final checkpoint at the
  credits (racing the watched mark, and pushing a resume point to Plex after the scrobble) and
  patched "new, at 49:55" into the cache — Prev reopened it at its last seconds. `finish` settles it
  as watched at 0, said before the pause whose own checkpoint could land first.
- An audio switch, an A/V offset change and every seek control read `abs`, which is 0 until
  playback ticks: a resume at 20:00 restarted from 0:00. They use the shared `playhead`.
- A seek while a restart is pending always restarts (the old media's `seekable` described the old
  session, and the pending answer overrode the newer seek).
- A superseded direct-play session's loadedmetadata/canplay are generation-gated and detached (they
  seeked the next item to the previous one's resume point).
- The session starts once per item: a reconnect refetch, the info panel's watched toggle or a
  language switch no longer restarts the film at the last checkpoint.
- Tests for the session generation's failure paths and the item-change bump, which had none.
2026-09-11 23:18:33 +02:00
peter 31dc4184f5 fix(api): a paused viewer keeps its HLS session, a refused subtitle publish backs off, and a NUL is a 422 everywhere
Deep-review round 4, backend half.

- The periodic HLS reaper (round 3) applied the "finished and unfetched for 30 s" rule that only a
  NEW session request may apply. A remux finishes long before the film, and a paused player with a
  full buffer fetches nothing, so a one-minute pause killed the session and playback died ~30 s after
  resuming. The periodic pass is `reap_abandoned` now (the long idle limit, finished or not).
- A subtitle `publish` that could not keep its output (full / read-only cache volume) returned
  quietly, so every poll queued the window and the whole-container pass again. It reports failure
  and the pass is backed off like a failed read; a cached VTT that cannot be READ is answered
  `unavailable` (`cache_unreadable`) instead of scheduling reads that can never land.
- The subtitle routes bound `rating_key` / `sub_ord` (they become a cache file name: a 250-digit
  one was ENAMETOOLONG, which pathlib's exists() re-raises -> a 500), and the cache helpers no longer
  throw on such a name.
- One app-wide handler turns psycopg's "text fields cannot contain NUL" DataError into a 422: round 3
  guarded only the by-rating_key lookup, and `contains=`, `contains_group=`, reorder/bulk key lists
  and the collection search still 500'd.
- The hand-written `db.get(...) is None -> 404 "Unknown ..."` siblings use `get_or_404` with a
  `what` (channel / video / user), so the Hungarian UI no longer shows English for a gone row.
2026-09-11 23:08:11 +02:00
peter 86971c574d fix(api): progress checkpoints past the int4 column are a 422, and idle HLS sessions are reaped on a timer
Deep-review round 3 of bug hunt 2026-09.

- The YouTube and Plex progress bodies are bounded by POSITION_MAX_S (the int4 position columns): a
  finite position of 2^31 or more was a DataError, i.e. a 500 — the sibling of round 2's non-finite
  fix.
- plex/stream.py promised "a periodic reaper", but reap_idle ran only inside the NEXT session
  request: after the last playback an abandoned session (its ffmpeg still working, its directory a
  whole segmented copy of the film) lived until someone pressed play again. A lifespan task reaps
  every 60 s.
- Tests for what round 2 left untested: the 422's body keeps the list shape the UI localizes, and
  stream_session refuses non-finite start/aoff.
2026-09-11 18:05:30 +02:00
peter 785080f3da fix(plex): a Surprise card whose watched toggle failed is put back
Deep-review round 3 of bug hunt 2026-09 — the sibling of the collection page's fix (d349bc90).
Discover's toggle flips the Surprise draw too, but its resync refetches only the rails (refetching the
draw would reshuffle it under the viewer), so a refused toggle stayed flipped there. toggleWatchState
takes an optional `revert`, called on a failure only; the queryKeys comment that claimed the watched
toggle invalidates the Surprise key now says what the code does.
2026-09-11 18:05:30 +02:00
peter 76b5f7c93e fix(playlists): a failed background refetch keeps the list on screen, and the player outlives every branch
Deep-review round 3 of bug hunt 2026-09.

- The detail's error branch applies only when there is nothing to show (`isError && <no data>`, the
  QueryState convention): one failed refetch — on focus, after a move or a removal — replaced a
  loaded list with "couldn't load". The error line is announced (role=alert).
- Tests for the two branches of "the player survives a change of selection" that round 2 left
  untested (a list that fails to load, no selection).
2026-09-11 18:05:19 +02:00
peter 1cfe6a7f59 fix(api): a gone row and a refused playlist action are said in the reader's language, and a failure is reported once
Deep-review round 3 of bug hunt 2026-09.

- get_or_404 / owned_or_404 answer a structured {code: "not_found", reason: <what>}
  (errors.notFound.* in HU+EN, generic fallback). The structured-404 fix of round 2 had gone into the
  ad-hoc `_own_playlist` (CLAUDE.md's "Not" pattern) while the shared helpers stayed string-only, so
  views, tags, pins, Plex playlists, invites and users still 404'd in English. `_own_playlist` now
  goes through owned_or_404; admin's hand-rolled user lookups use get_or_404.
- A move into a list that is gone names the TARGET (reason "move_target") — "this playlist doesn't
  exist any more" was read as the list on screen, which still exists.
- Every refusal in the playlists router is a {code: "playlist", reason} (playlists.errors.* HU+EN)
  instead of a bare English string shown verbatim in the Hungarian UI.
- `req` / `uploadReq` mark the errors they already put on screen (the error dialog, the connection
  notice, the 401 sign-out), and notifyActionError skips those. The watched toggles, the saved views
  and a playlist delete toasted a fault the dialog was already showing; Playlists' two hand-written
  status filters now go through the same check.
2026-09-11 18:05:19 +02:00
peter 33fe895539 fix(ui): every full-screen overlay hosts its own toasts, above the players' controls, without re-announcing them
Deep-review round 3 of bug hunt 2026-09 (the round-2 overlay toast host, finished).

- Inside an overlay the toast stack sits above the bottom band (bottom-24, the UpNextCard rung) and
  lets clicks between cards through: at the app's own spot it covered the Plex transport row and
  PlayerModal's docked bar and took their clicks while a toast was up (measured: play/pause, ±10,
  stop, prev/next, mute all hit the toast).
- ui/Modal and the onboarding wizard mount an overlay host too, so a toast raised under any dialog
  is readable and a click on it no longer closes the dialog; the Modal's Tab trap spans the whole
  dialog, so a toast's action and dismiss are keyboard-reachable. A test now enforces the
  convention: a component that covers the viewport at z-overlay must mount <Toaster overlay />.
- PlayerModal's host moves into the stage in true fullscreen (outside the element holding the lock it
  was painted under it).
- A host that takes over shows the toasts it inherits silently (no second role=alert, its region
  aria-live=off until they are in), and each countdown bar starts as far in as its toast already is
  instead of restarting while the real dismiss timer does not.
- The overlay-host registry uses lib/store instead of a hand-rolled listener triad.
2026-09-11 18:04:54 +02:00
peter 5c5cabbe5a fix(plex): the player drops a session answer that lands late, and Next neither loses nor lends a position
Deep-review round 3 of bug hunt 2026-09.

- loadSession checks, after its await, that it is still the newest call and the player is still
  mounted. A late answer used to replace what was playing (Next twice quickly: the slower transcode
  landed last and played the middle item under the last one's title, recording its progress there)
  or attach a stream to the detached <video> after Back, never destroyed.
- The <video>'s clock is trusted only once the new session has answered. go()'s pause() fires a
  timeupdate AFTER the reset to 0 (measured in Chrome), so the previous item's position was sent as
  the next item's subtitle position and saved as its progress (final, pushed to Plex).
- Next settles the item being left (final checkpoint + reopen-cache patch) before switching: the
  progress cleanup ran with the position already reset and skipped, so every Next dropped the last
  stretch watched.
- A seek-restart puts its target into the position state too; the ref alone was re-synced back to
  the pre-seek spot by the restart's own render.
- The unmount teardown captures the <video> at mount: React detaches refs before an unmount's
  cleanups, so the stop/detach never ran.
- The tracks panel exposes the selected audio/subtitle track via aria-pressed.

Verified in real Chrome on localdev (Westworld S1E6 → Next): the item left was settled at 783 s, the
next asked for subtitles at its own start (0), and no progress was written for it.
2026-09-11 18:04:41 +02:00
peter 4f37c41021 fix(plex): a subtitle failure about one position no longer ends the track, and windows are read ahead and never wait behind a full pass
Deep-review round 3 of bug hunt 2026-09.

- `failed` is now final only for a track that can never be text (bitmap, no item/track/file). A Plex
  hiccup or a stretch ffmpeg could not read answers `unavailable`: said, polled again every 5 s, and
  the <track> stays mounted when cues exist elsewhere (gen > 0). Re-picking the selected track asks
  the server again at once.
- Within LOOKAHEAD_S of a covered stretch's end the next window is read ahead, instead of only once
  the playhead has fallen off it (subtitles dropped out at every edge while the full pass was backed
  off or slower than a window lasts).
- Windows and full passes run on separate pools with separate in-flight caps, so a window never
  queues behind (or is refused by) minute-long full passes — the pool comment promised that and
  nothing enforced it.
- A pass that times out is TERMinated and keeps what it reached (published as partial) instead of
  being killed and discarded.
- Lapsed failure memos are swept on every write (they were only pruned when the same key was read
  again, so polling fresh keys grew the process without bound).
- `publish` no longer merges into a sidecar with no VTT behind it, or claims a region whose VTT write
  failed, or overwrites an unreadable track with one window.
- A NUL in a rating_key names no row (it was a DataError, a 500, on every Plex route) and no longer
  500s the status poll.

Tests: 16 new, each mutation-checked (18 mutations, all red on the test that names them).
2026-09-11 18:04:30 +02:00
peter f7faa19ee2 fix(plex): the player starts, and resumes after a session-restart seek, when React mounts it twice
The player's "still alive" flag — every play() is gated on it, so a late one cannot start a detached
<video> after Back — was only ever LOWERED, in an effect's cleanup. React may run a cleanup and mount
again (StrictMode does exactly that in development), and then the flag stayed false for the life of
the player: it never started by itself, and a seek back past the loaded region, which restarts the
session, left it paused. Found on localdev (Westworld S01E06: playing → seek back → the picture
settles, paused). Production does not double-mount, which is why only dev showed it — and why
several earlier localdev measurements "stayed paused".

The effect now raises the flag in its setup. A new test renders the real player under StrictMode and
checks it plays once the media is ready; red without the fix. Verified on localdev: it starts by
itself, and keeps playing through a backward session-restart seek.
2026-09-11 03:47:36 +02:00
peter 0b4b268678 test(feed): pin the sort options' default directions as literals
The feedSortOptions test compared each option's default direction against SORT_DEFAULT_DIR — the
implementation's own table, so a wrong entry passed — and that import was the export's only user
outside its module, which kept knip from seeing it had become dead after F3. The test now states the
directions, the table is private again.
2026-09-11 03:19:17 +02:00
peter d349bc9084 fix(plex): a collection's failed watched toggle is put back the way it was
C1 moved the optimistic flip onto the key the page reads — and so made visible that nothing corrects
it: the resync only invalidated `plex-library`, so a failed toggle stayed "watched" beside its own
error toast, and the collection's other cached sort orders stayed stale after a good one. The
resync now takes this page's `detail` prefix too, as the Discover and person pages do for theirs.
The C1 test holds the request open, so it still pins the optimistic write on its own.
2026-09-11 03:19:17 +02:00
peter dcd8a9566a fix(playlists): the player's Remove follows the list it was launched from, the player survives a change of selection, and a gone list is said in the reader's language
- D3 snapshotted the removal's TARGET, but whether the player offered Remove still followed the
  selected list: launched from Watch later (no Remove, by design), selecting another list under it
  made Remove appear — and delete from Watch later. The snapshot now carries the launch list's
  editability too.
- Selecting another list under an open player (the command palette works over it) put the page into
  "Loading…", and the player was rendered only in the loaded branch — it vanished mid-playback and
  came back at its launch video. It now sits at one place in the tree whatever the page is doing.
- A removal takes the video out of the CACHED list as well as the rows: the player's progress
  checkpoint rewrote that entry before the refetch landed, handing the membership guard a list that
  still held the video — the row came back and the undo history went.
- The watched-state callback refreshed the selected list, not the launch list (D3's sibling).
- `_own_playlist`'s 404 is structured (`{"code": "playlist", "reason": "not_found"}`) and localised:
  the bare "Unknown playlist" reached the Hungarian UI in English, which also made round 1's new
  `removeFailed` string unreachable. Both Remove and Move report through `notifyActionError` now.
- The "viewer moved on under the confirm" test stepped back only once, which lands on the same video
  whether or not the removed one left the queue; it now takes the second step that tells them apart.
2026-09-11 03:19:07 +02:00
peter 6de2cd950d fix(ui): toasts raised while a full-screen player is open show inside it, not under its scrim
The app's Toaster lives in the content column, which is a stacking context (its `@container` implies
layout containment), so both full-screen players — portalled to <body> at the same z-index — painted
over every toast. Round 1's "a failed Remove is reported" fix was therefore still silent inside the
player: its toast sat under an 80% black, blurred scrim (elementFromPoint returned the scrim), and a
click on it closed the player.

An overlay now mounts its own `<Toaster overlay />`, and exactly one host shows the toasts: the most
recently mounted overlay, else the app's. Same component, same look — and inside the aria-modal
dialog, where a screen reader can still reach it. A click on a toast inside an overlay stays there.
Both players get one (the Plex player's also works in native fullscreen, where nothing outside the
fullscreen element is drawn). Verified on localdev: the removal failure reads above the scrim,
bottom-left, and clicking it leaves the player open.
2026-09-11 03:18:55 +02:00
peter 645d14f45d fix(plex): a subtitle track's failed load is heard and retried with a pause, and the poll asks about the right item, position and failure
- The <track>'s `error` never reached `onSubTrackError`: it does not bubble, and React 18 attaches no
  listener of its own to a <track> (measured in Chrome and jsdom — `onError` 0 calls, `onErrorCapture`
  1). Round 1's remount-and-give-up was dead code behind it. The hook now hands the player
  `onErrorCapture`, and the test fires a real non-bubbling `error` at a rendered <track>.
- Once reachable, the retry burned its whole budget within milliseconds of an outage (a remount
  fetches at once) and the give-up outlived turning subtitles off and on. Retries now wait (2 s,
  doubling), running out is reported at once, and picking the track again starts a fresh budget.
- Until the first timeupdate the poll sent `sessionK` — still 0 while the session request was out,
  and the PREVIOUS item's start after a move to the next episode. It now sends where the session was
  asked to start, and the track choice belongs to the item it was made for, so the next item is not
  asked about the old ordinal at the old position.
- A failing poll over a stale `partial` answer was silent: react-query keeps the last good data, the
  track stayed "showable", and the notice only rendered for an unmounted track. The hook now decides
  what the player says (`subNotice`), independently of whether a window is mounted.
2026-09-11 03:18:42 +02:00
peter 7360339039 fix(api): a non-finite number in a JSON body is refused with a 422, not a 500
The JSON parser under FastAPI accepts `Infinity`/`NaN`, and both progress checkpoints truncate the
position with `int()` — a hand-written checkpoint came back as a 500 with a traceback. Their bodies
now refuse non-finite floats, and the app's validation handler — FastAPI's own shape — echoes a
non-finite input by name: the default one failed to serialise its own error (JSON has no NaN), which
was a 500 by another route. Sibling of the subtitle routes' non-finite query parameters.
2026-09-11 03:18:17 +02:00
peter 473d9f02f2 fix(plex): subtitle windows land where the full pass does, and the extractor stops re-reading, respawning and outliving the process
Deep-review round 2 of the 2026-09 bug hunt, over the round-1 fix diff.

- A window was LATE by the distance to the keyframe its input seek landed on, and then doubled for
  good once the full pass merged in: a cue already running at the window's start came out at a
  negative time, and the webvtt muxer's default shifted the whole output so it sat at 0. Measured on
  the real Westworld S01E06 at 25:00: every window cue +5.005 s, 682 cues merged instead of 608.
  `-avoid_negative_ts disabled` keeps the times exact (the parser reads ffmpeg's signed-field
  spelling of a negative time); now 608/608 on that file, and a real-ffmpeg test pins it.
- A full pass that failed was re-armed every minute while the viewer watched what it did publish —
  a whole-container read per minute. It now backs off for an hour; windows keep running meanwhile.
- A failed WINDOW set the track's failure memo, so any uncovered position read "failed" — a final
  answer that ends the player's poll, even with the full pass about to deliver, and one any account
  could plant with an absurd `at`. It now marks only the span it failed to read; "failed" is given
  once the full pass AND a window there have both failed.
- Windows join by what they COVER, not a bucketed identity: a backward seek no longer waits behind a
  window that cannot cover it, and a playhead crossing a bucket edge no longer queues a second read.
  The queue is bounded per track and overall.
- shutdown() sets a flag a worker checks before it spawns ffmpeg (one still in its Plex lookup used to
  start a read after the kill sweep); queued-job cancellation and the lifespan call are now tested.
- A window starts where ffmpeg is told to seek (rounded to the ms it gets), a rounded-up millisecond
  carries into the minute (no more `00:00:60.000`), a key's first publish claims nothing before its
  VTT lands, and `at`/`offset`/`start`/`aoff` refuse non-finite values (`at` also an absurd one).
2026-09-11 03:18:08 +02:00
peter d2e225666b test(playlists): let react-query's notify tick run inside act before the reopen click
The D1 regression test patched the cache and clicked at once; react-query delivers a cache write to
its observers on a setTimeout(0), so under the gate's parallel load the click could beat the page's
re-render and read the old row (748, not 830). Test-only race — a human reopen comes seconds later.
Stable across three full-suite runs; the D1 mutation still turns it red.
2026-09-11 01:45:53 +02:00
peter efd95fca01 fix(ui): the card's channel name is a real link too; call sites the bug hunt left untested now have tests
Deep-review R6 — the small items:

- The video card's channel name was still a <button> (pre-existing): ctrl/middle-click did nothing
  there while the title above it and the player's channel name honoured them. It renders through
  ChannelNameLink now, which grew a forwarded ref (the clamp measurement), pass-through attributes,
  a `tooltip`, and an opt-in `stopPropagation` so no click reaches the card's row.
- ChannelNameLink rendered an empty, still-tabbable <a> for a channel with no title yet (a stub first
  seen through a live search; pre-existing at this site). It falls back to the translated "Channel".
- The feed's SortKey→SortOption mapping existed twice in Feed.tsx; one `feedSortOptions` in
  feedSort.ts serves both modes (DIRECTIONLESS is module-private again).
- The backfill refresh's comment claimed a one-page cost. It refetches every loaded page of the
  grid, and that is the right price: backfilled videos are older, so they land below the head.

Tests for call sites that could be deleted with every lane green: the channel page running the
backfill refresh (knip had been blinded by the hook test's import), Feed handing the grid the live
results in the chosen key AND direction, the card using the link. Plus the alt-click arm of the shared
click predicate, and a plex filter-chip test that claimed it made no request while firing a real one.
Verified on localdev: the feed's channel names are anchors (same 14 px muted look), a plain click
opens the channel in-app. 7 mutations, each red on its assertion.
2026-09-11 01:38:31 +02:00
peter 060deb89d2 fix(playlists): "remove" inside the player targets the list it came from, reports failure, and keeps the cursor and focus honest
Deep-review R5 — five defects in the Remove button f76a2afb put inside the YT player:

- It removed from whichever playlist was selected by the time it was pressed, not the one the player
  was launched from; the selection can change under an open player (the command palette opens over
  it). The launch list is now part of the player snapshot, and the optimistic row removal only
  touches the list actually on screen.
- A 403/404 failed in total silence: the catch claimed the global dialog had reported it, but `req`
  deliberately leaves those to the caller. It now notifies, as moveItemTo already did (the silent
  catch predates this round; the full-screen consumer did not). New key playlists.removeFailed, HU+EN.
- It computed the next item from the cursor as it was when the button was pressed, but prev/next and
  auto-advance keep working under the confirm dialog: the viewer was thrown back onto a video they had
  moved past. It reads the live cursor, and if they moved on it just drops the item from the queue.
- The button was disabled for the confirm's lifetime, so the dialog's focus restore hit a disabled
  element and a keyboard user who cancelled was left on <body>, outside the aria-modal player. A ref
  guards the double press instead.
- Its accessible name was a bare "Remove"; it now names its object, like the row control.

The call-site wiring (step on, drop from the queue, close on the last item, leave everything alone
when the removal did not happen) finally has tests. Verified on localdev in the real YT player: a row
reopened at the 136 s it had just stored; Cancel returned focus to the Remove button; a confirmed
removal stepped 1/3 -> 1/2. 10 mutations, each red on its assertion. Scratch list and positions reset.
2026-09-11 01:28:27 +02:00
peter 4869d6c108 fix(player): the resume position reaches the playlist page and live search, not just the feed
Deep-review R4. 1630a3df wrote the YT player's stored position into the react-query cache so a
reopen could not rewind it — but only the feed actually read it back:

- Playlists renders from its own undoable copy of the list, re-seeded only when the SET of ids
  changes, so the patched position never reached it: close a video at 830 s, reopen it from the
  playlist, and it resumed at the old spot and the next checkpoint wrote that over the 830. A detail
  with the same membership now rebases its fresh rows into every snapshot, keeping order and undo
  history. (The ids-only guard is older than this round; the write that depended on it was new.)
- Live YouTube search results carry the same per-user position and open the same player, but their
  cache was never patched. It is now — patched only, never invalidated, since refetching an
  API-sourced search page spends 100 quota units.

Tests for what had none: a Playlists harness (PlayerModal stubbed to record its launch props) and a
PlayerModal harness with a fake IFrame API that records where each item is started — the latter pins
that stepping back to an item watched earlier in the same session resumes at THIS session's position.
3 mutations, each red on its assertion.
2026-09-11 01:19:14 +02:00
peter d20317e0f1 fix(plex): the collection page's watched toggle and Sort control survive the sort joining the query key
Deep-review R3. When 889d2acf put the sort into the collection detail's query key, two things on
the page did not follow it:

- The watched toggle's optimistic write kept the key's old spelling. setQueryData matches exactly,
  so it landed on an entry nothing reads, and its resync only invalidates the library root — the
  click succeeded server-side and changed nothing on screen (badge, icon, aria-label) until a
  reload; a second click toggled it back. One `detailKey` now serves the read and the write.
- Every sort change was a key with no data yet, so the whole page — hero, grid, and the Sort control
  with its open panel — was swapped for a spinner, and focus fell to <body>: key and direction could
  not be set in one visit. The previous page now stays on screen while the new order loads, for the
  SAME collection only (stepping to another must not show this one's hero meanwhile).

Verified on localdev (Rocky Collection): the toggle flips the poster; IMDb rating then Descending set
in one open panel, the hero never blanked, order reversed. 3 mutations, each red on its test.
2026-09-11 01:12:51 +02:00
peter d2ba65e383 fix(plex): the subtitle poll backs off on errors, a failed <track> is replaced, and a resume asks for the right window
Deep-review R2, plus one defect found while verifying it on localdev.

- The status poll re-armed every second after a FAILED request: `data` stays empty on an error,
  so it read as "still preparing" for as long as the player was open — and the request was not
  quiet, so a persistent 500 re-opened the global error dialog every second. It now backs off to
  10 s (a blip still heals on its own) and is quiet; the player already shows the failure inline.
- A <track> whose own VTT fetch failed never fetched again: with the server's answer unchanged its
  key was unchanged, so React kept the dead element — silently blank subtitles. The key (now built
  in the hook, where it is tested) carries a failed-load count; after two remounts the track is
  reported as unavailable instead.
- Found in UAT: until the first timeupdate `abs` is 0, so after a resume every poll said at=0 and
  the first window was cut around the head of the film (a resume at 20:00 polled at=0 for as long
  as it stayed paused). The poll reports the session's start until playback ticks. And the window's
  dedup identity is the track plus a window-length bucket of its start — keyed on the track alone
  (R1), the right window queued behind that wrong one.

The poll's start/stop rules finally have tests (every earlier one resolved a single fixed answer).
Verified on localdev against Westworld S1E6 resumed at 20:00: the polls carried at=1197, one
subtitle line on screen at 21:02, the poll stopped at `ready`; the real file's window (56 cues) +
full pass (608) merged to 608 — the old merge would have served 664. 11 mutations, all red.
2026-09-11 01:08:27 +02:00
peter cf7ec5e9ec fix(plex): the subtitle extractor's seams — doubled cues, a truncated track cached as whole, a poll that respawned ffmpeg
Deep-review of the bug-hunt round (R1). The background extractor from ed4ab4e1 held, but its seams
did not:

- Window + full pass doubled every cue in the first hour. merge_vtt de-duplicated on the raw timing
  text, and the two passes spell a timestamp differently (shift_vtt writes HH:MM:SS.mmm, ffmpeg's
  webvtt muxer drops the hour below 1 h; shift_vtt(-0.0) never normalised the full pass). Timing is
  now parsed and re-emitted canonically, keyed with the cue text so two cues sharing a time survive.
- A full pass that exited non-zero was published as complete: "ready" forever, subtitles ending
  where ffmpeg died, and no eviction to ever retry. It now covers only up to its last cue, is never
  complete, and records the failure memo.
- The window's dedup identity held the live playhead, so every 1 s poll queued another read of the
  file. It is keyed on the track now, and a queued job re-checks coverage before reading anything.
- A sidecar track waited behind an unrelated embedded full pass (the full job is per item). Its
  window job now fetches it whole — one small request.
- The <track> fetch ran the scheduler at position 0: an unrequested window over the head of the file,
  and a refusal of a window that was on disk. It is a pure read of the cache now.
- publish() is serialised (a lost update became permanent once complete was set); the pool is shut
  down and running ffmpeg children killed at app shutdown (they held interpreter exit for minutes);
  ?at=inf is a 422, not a 500; and both subtitle routes use current_user instead of the cookie-only
  poster gate, which let a suspended or epoch-revoked session keep scheduling reads (pre-existing).

_extract_and_publish finally has tests of its own — every earlier test stubbed it out. 19 mutations,
each red on the assertion that names it.
2026-09-11 00:52:59 +02:00
peter f76a2afbf1 feat(ui): the four small gaps — player delete, repeat-move, a real channel link, YT search sorting
Items 2, 4, 5 and 7 of the bug hunt, one round. They are independent, but they are all the same
kind of thing: an action that exists somewhere else in the app and is missing where you actually
want it.

**5 — the channel name in the YT player is a link again.** It was a `<button>`, so middle-click and
Ctrl/Cmd-click did nothing on a thing that looks exactly like a link (the YouTube icon beside it IS
an anchor and opens a tab). It is now a real `<a>` whose href is the same `?chan=<id>` URL
`openChannel` pushes, with a plain left click still taking the in-app path — that path does more
than navigate, since the player must close and reclaim its own Back-to-close history entry first.
The "is this click ours or the browser's" test is now ONE predicate (`isPlainLeftClick`), which
VideoCard's card link also uses, so a second such link cannot forget the middle-click arm.

**2 — "Remove" from inside the player.** The parent owns the removal (the SAME `removeItem` the row
menu and the inline ✕ use: one confirm, one optimistic update, one undo-stack rebase), and the
player owns what happens next, because it is the only side that knows which item is playing — its
launch queue is frozen at mount. It drops the item, plays what followed, and closes when the list
runs out. That forced a real fix in Playlists: the player's mount was derived from `shown` every
render, so anything that dropped the launch row yanked the player off the screen mid-playback. That
was a latent edge (change the view and it happens) and the normal case for this feature. The parent
now snapshots what it launched with, and only onClose takes the player down.

**4 — "Move to <the last list you moved to>".** The submenu is two levels deep and moving a run of
videos into the same list is the shape this gets used in. The remembered id is a per-account pref;
the row is derived from `moveTargets`, which already excludes the list you are looking at, so it
cannot offer a move to where the video already is. Remembered only on a SUCCESSFUL move — a
shortcut to a move that failed is a shortcut to failing again. Absent, not disabled, when there is
nothing to repeat: there is nothing to explain.

**7 — the live-YouTube results can be sorted.** Client-side, over the page already loaded. Not a
new `order=` param, and this is the decision worth recording: the owner's case is "even one page of
results", i.e. reorder what is on screen; a re-query would REPLACE the results, which is not what
"sort these" means; and the zero-quota scrape source has no order parameter at all, so a
server-side control would silently do nothing (or quietly cost 100 quota units) depending on which
source served the page. Only keys a search hit carries are offered — `subscribers`/`priority`
describe a subscription, which a search hit is not, and `shuffle` on a ranked list is noise. Its
own pref, so sorting a search page never re-sorts the library behind it.

Verified in the browser on localdev against the real account: the channel name renders as
`<a href="…&chan=…">` and a ctrl-click / middle-click is left to the browser while a plain click
still opens the channel in-app; removing from the player stepped 1/3 → 1/2 onto the next video and
closed on the last one; the row menu grew "Move to “…”" only after a move and only on a list that
is not the target; and the results sort reordered by views 41.7M → 29.8M → 24.2M → 15.8M. (The
pane's own ctrl-click arrives without the modifier — a harness limit, so the modifier cases were
driven as real MouseEvents.) Scratch playlists and the touched prefs were cleaned up afterwards.

22 vitest, mutation-checked against 14 mutations (no href at all · href without the rest of the
query · every click hijacked in-app · the middle-click arm dropped from the predicate · plain click
no longer in-app · always jumping to the top of the list · the removed item left in the queue · no
clamp when the last item goes · an empty queue not reported as "nothing left" · sort-then-reverse
flipping ties · unknown values first · titles compared numerically · shuffle offered on search
results · direction ignored) — each red on exactly the test that names it. Item 4 has no test of
its own: its rule is enforced by construction, not by a check.
2026-09-10 22:34:12 +02:00
peter 889d2acf52 feat(plex): the collection page sorts, and a linked collection has a name
Two halves of the same report ("Collectionben nem működik a rendezés?"). The pasted browse URL
sorts correctly — that part was working. What did not:

The Collection INFO page had no sort control at all and a hardcoded release-year-ascending order.
It now shows the browse grid's Sort control, with the browse grid's vocabulary, because a second
sort vocabulary on a neighbouring surface is exactly the kind of drift this repo keeps paying for:
`PLEX_SORT_KEYS` is now one tuple in `plex/query.py`, and a test pins BOTH implementations to it —
browse's key→column map (extracted from the handler to be reachable) and the collection page's
key→member map. A key added to the vocabulary without teaching one of them fails the gate instead
of putting a dead option in the menu.

Ordering a collection cannot be a single ORDER BY: its two legs are different tables, so the merged
member list is sorted in Python. Members with no value for the key sink to the END in both
directions (a movie with no rating is not "the worst rated" — the browse grid's `nullslast()`), and
ties always resolve title-ascending, so flipping the direction reverses the ranking rather than the
alphabet inside it. `lastwatched` needed the show leg's `resumed_at`, so the page now takes ONE
rollup pass that yields both the badge status and the recency instead of the status alone — without
it that menu entry would have ordered every show as never-watched. The default stays release-year
ascending: a franchise reads in release order, and that is what the page did before.

The other half: a shared Plex link carries the collection filter as bare rating_keys, so a link
opened cold had no name for them — the sidebar token rendered `17045` and the chip could only say
"Collection". `GET /api/plex/collections` takes a `keys=` CSV now, and a display-only hook resolves
just the missing ids. It deliberately does NOT write them into `collectionTitles`: that map is
hashed into the Plex query keys, so storing a resolved label there would split the cache from the
identical state and refetch the grid for a piece of text.

Verified in the browser on localdev against the real Plex library: the movie collection's menu
offers Length and the show collection's does not (the same scope rule as browse); rating-ascending
reorders 8.1/7.4/7.2/6.6/6.5 correctly; `lastwatched` on a show collection brings the in-progress
shows up; the choice survives a reload (it is a server pref, on its own key so opening a collection
never re-sorts the library behind it); and the reported URL now reads "Pirates of the Caribbean
Collection" in both the sidebar and the chip.

12 pytest + 8 vitest, mutation-checked against 14 mutations (direction ignored, nulls first, no
stable tiebreak, unknown key untouched, wrong default, show recency dropped, empty `keys=` falling
through to everything, a key missing from either sort map, every id refetched, ids unsorted,
fetched titles unmerged, the chip ignoring them, the hook's result never reaching the builder) —
each red on exactly the test that names it.
2026-09-10 22:05:39 +02:00
peter 782362b808 fix(channels): the video grid follows a running backfill instead of freezing until F5
"Get full history" on a channel page left the list on the cards the channel already had. The
count above it climbed into the thousands, the grid did not move, and only a browser F5 brought
the rest in.

The mechanism: the backfill runs in a BackgroundTask, so the PATCH that starts it returns before
a single row is written — and the mutation invalidates `qk.feed()` on that RESPONSE, which just
refetches the OLD catalogue. Nothing afterwards refetches it: the feed's infinite query has no
interval, and the app-wide defaults are `staleTime: 30s` + `refetchOnWindowFocus: false`. The job
emits no event either (the realtime layer only carries messages today), so the channel detail's
4s live poll is the only thing in the app that sees the job progress — and it refreshed only
`qk.channel`.

So the poll now drives the grid too: on a tick where the stored count moved (or the backfill
finished), it invalidates `qk.feed()` and the `qk.feedSummary()` behind the count pill. The window
is the poll's own (`deep_requested && !backfill_done` on the previous reading), so a detail
refetch caused by some other mutation — each of which invalidates the feed itself — cannot come
through here twice.

Measured on localdev against the real account, same instrument, `document.visibilityState`
forced visible (the poll deliberately pauses on a hidden tab, and a headless pane reads hidden —
without that the first measurement was about the harness, not the fix):

  before — CASCADA 19 -> 176 stored: the header updated live to "162 normal", the grid and its
           count pill sat on "5 items" for 50s.
  after  — fern 54 -> 139 stored: "54 items" -> "139 items" by ~12s, no reload.

6 vitest, mutation-checked against six mutations (no invalidation at all; grid only, count pill
forgotten; predicate ignored; no channel-identity guard; completion with an unchanged count
dropped; poll-window guard dropped) — each one red on exactly the test that names it.
2026-09-10 20:56:16 +02:00
peter 5a42d76e8a feat(plex): an off-by-default loudness normalisation toggle for playback
Bug hunt 2026-09, item 14 — "how does the multichannel downmix work, and is it
normalised?" It isn't: `plex/stream.py` has always encoded `-c:a aac -ac 2 -b:a
192k`, i.e. fixed stereo with ffmpeg's default downmix matrix and no loudness
handling at all, which is why a film mastered for a cinema arrives with whispered
dialogue and deafening action.

New admin knob `plex_audio_loudnorm` (Configuration → Plex), DEFAULT OFF: it makes
a film markedly louder and narrows its dynamic range, which is a taste call rather
than a fix. It sits at the end of the Plex group, after the software-transcode
knobs — it is an audio concern, not one of the libx264 ones.

  -af aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000

Three things about that chain are load-bearing and each has a test:

  * the downmix is FORCED FIRST rather than left to `-ac 2` — measuring 5.1 and
    folding it down afterwards would normalise something other than what is heard;
  * the trailing resample is not decoration: `loudnorm` runs its own resampler at
    192 kHz internally, which the AAC encoder will not take;
  * one `-af`, no stream index, so EVERY audio rendition of a multi-audio file is
    normalised — not just the first.

The knob is also part of the session directory name (`…_multi-ln_+0.00`), because
it changes the audio the segments contain, and a session left over from before the
admin flipped it must not be mistaken for one that honours it. The suffix is
appended with a DASH: `_SESSION_DIR_RE` treats the tag as one underscore-free
field, and that regex is the only thing keeping the orphan sweep off what may be a
shared scratch path. A test pins both halves — the name changes, and the sweep
still recognises it.

Measured end-to-end on the app's own HLS segments (a real 5.1 AC3 file, same item,
same 160 s offset, same ten segments): OFF -35.2 LUFS, ON -17.2 LUFS. On a 60 s
bench: I -27.6 → -16.6 LUFS with LRA 13.3 → 9.8 LU. A/V sync is untouched — under
this pipeline's own `-copyts` the audio start PTS is identical with and without the
filter, so the 3 s lookahead desyncs nothing.

Direct-played files are served untouched and stay so; a 5.1 mix is always `remux`,
so the reported case is fully covered.
2026-09-10 20:02:06 +02:00
peter ed4ab4e13c fix(plex): subtitles stop blocking the request, and stop re-reading the file
Bug hunt 2026-09, items 8 and 12 — the two things the Plex player was measurably
lying about.

Item 8: picking a subtitle track showed nothing for a long time, and the reported
workaround was to stop/start until it suddenly worked. Measured on prod against a
17.06 GB MKV with two embedded SubRip tracks:

  full pass, one track                154 s
  full pass, both tracks in ONE run   111 s
  -c:s copy (no WebVTT conversion)    143 s
  from 1:00:00 to the end of file      66 s
  a 630 s window at 1:00:00, both       8 s

So the conversion was never the cost: ffmpeg streams every byte of the container
between its seek point and its stop point, and the array delivers ~140 MB/s. The
only lever is reading less. Three changes follow from that:

  * a window around the playhead is extracted and published FIRST (seconds), and
    the full pass completes behind it;
  * every embedded text track rides that same single read — picking Hungarian used
    to cost a full pass and then switching to English cost another one;
  * none of it happens inside a request any more. The endpoint answers 202 while a
    background worker extracts, the player polls /status (carrying its position, so
    the window follows the viewer), and one ffmpeg run finally has a timeout.

-copyts is deliberately absent from a windowed pass: with it ffmpeg ignores -t and
reads the whole container anyway (measured — a [0, 600 s] window still took the
full 145 s), so a window comes back on a zero-based clock and is shifted to
absolute time here. Cached tracks written by older builds have no coverage sidecar
and count as complete, so nothing is re-extracted after the upgrade.

The <track> is no longer mounted before the server has something to put in it; a
"Preparing subtitles: …" notice stands in, and a track that can never be shown says
why instead of leaving the viewer to conclude the picker does nothing.

Item 12: "Bitrate" read a constant 211 kbps on every remux, because it was the
master playlist's BANDWIDTH — which ffmpeg derives from the one stream whose rate
it knows, the -b:a 192k AAC track, plus 10%. It is a measurement now: bytes per
second of MEDIA from hls.js FRAG_LOADED, accumulated per fragment type and summed
(a multi-audio session's video and audio renditions cover the same media seconds,
so pooling them would report half the truth), and an honest dash until the first
fragment lands. The panel also finally names the audio and subtitle track in play.

Verified in the browser against the real Plex library, and every new test
mutation-checked.
2026-09-10 19:31:05 +02:00
peter b8bff504d8 fix(plex): "Clear resume point" now tells Plex about an in-progress item
The button deletes the local row and mirrors the un-mark to Plex, but it decided whether
to push from the previous STATUS alone (prev_status != "new") — and an in-progress item's
row is status="new" carrying a position, which is exactly how the importer maps a bare
viewOffset. So for the single case the button exists for, nothing was ever sent to Plex:
the offset stayed, and the next pull imported it straight back. The user cleared Protector
twice and it returned to Continue watching at 92s both times.

Push now when Plex holds anything to clear — a view count, or a resume point. "hidden"
stays Siftlode-only and pushes nothing, which is what this endpoint's docstring always
claimed and the status-only guard contradicted.

One unscrobble clears BOTH viewCount and viewOffset — measured against the live server from
a watched+offset state and an offset-only one, so the push itself needed no change. The
client docstring that claimed "clears viewCount" is what sent the first diagnosis down a
wrong path (a timeline write with time=0, which does not clear an offset at all); it now
records what the server actually does.

Verified end to end on the real Plex server: the button leaves viewCount=None viewOffset=None,
and a full 44,885-item re-import brings back no row for it.
2026-09-09 23:55:51 +02:00
peter 1630a3df3c fix(player): the YT progress checkpoint reaches the cache, so a reopen cannot rewind it
The player checkpointed every 5s and the server stored it, but nothing wrote the new
position back into react-query: the close path only marked qk.feed() stale with
refetchType "none", and nothing refetches the feed until it remounts (staleTime 30s,
refetchOnWindowFocus off). So the resume read the position the cache had been FETCHED
with, and the next checkpoint then persisted that older value over the newer one.

Measured on localdev: played to 830s, closed (server had 830), reopened the same card
without a reload -> the row was back to 751s. 79 seconds of progress destroyed by a
reopen, and the card's progress bar never moved either.

The Plex player already solved this for its detail query, with a comment describing the
identical failure; this is the feed-side twin that was missed in that sweep. patchCachedProgress
writes the position the SERVER stored (the endpoint clears trivially-early and near-finished
checkpoints to 0, so mirroring our own reading would disagree with the row) into every cached
feed page and playlist detail, keeping array identity where nothing changed. A session map
covers the second half: the launch queue is frozen at mount, so stepping next->prev back to an
item played earlier in the same session resumed from that snapshot.

Verified in the browser: 811 -> close 816 -> reopen 825, and the card now shows Continue/Restart.
2026-09-09 23:55:51 +02:00
peter 14b5fe2e31 release: 0.83.0 (feature/e24-anti-paralysis)
gate / frontend (push) Failing after 2m5s
gate / backend (push) Successful in 7m0s
2026-09-09 22:04:14 +02:00
peter d260aabfd7 chore(release): 0.83.0 — the Home surface 2026-09-09 22:01:33 +02:00
peter d832bb19ce fix(e24): review round 4 — a test that pinned nothing, and prose still ahead of the code
Round 4 reviewed round 3's commit. Nothing in production behaviour was wrong; what was wrong
was the round's own test and its comments, which is the failure class this loop keeps
finding.

- The new NavigationProvider case named after the `&& !channelViewRef.current` clause never
  reached it. With storage cleared and a bare URL the provider lands on "home" (E24's own
  default), not "feed", so asking for "feed" was an ordinary page change and the guard's
  first conjunct short-circuited. Deleting the clause left all 828 tests green — a dead nav
  control (the rail's own module stops closing an open channel page) with no red anywhere.
  The test now asserts the precondition and calls `setPage` with the page that IS current.
- Both new Home cases rejected a promise and then asserted after one fixed macrotask.
  React-query notifies through `notifyManager`'s own `setTimeout(0)`, registered after any
  timer the test schedules, so the assertion raced it; two reviewers measured intermittent
  reds. They poll for a signal that the rejection LANDED instead — the reroll button
  releasing, which is also the honest one for the empty-answer case, where the empty state is
  already on screen while the press is in flight and asserting it early would pass either way.
- `buryUnder`'s comment still opened with "one rule for both entry points" ten lines above the
  paragraph saying the rule is complete only for one of them, listed the non-reporting palette
  rows without `openChannel` — which has its own `replaceState` no-push branch and is reached
  from PlayerModal too — and pointed at a ROADMAP.md this repository does not contain.
- `history.dom.test.ts`'s header claimed its layers close by genuine Backs only, which two of
  its own tests contradict; and `CommandPalette.ordering.dom.test.tsx` still claimed to be
  "the only assertion in the suite" doing what the file corrected in round 3 also does. Round
  3's message said that pair was handled and only one side had been.

The provider test was re-mutation-checked (dropping the clause now goes red), and both
hardened Home tests still go red against their own reverts. Seven consecutive full runs green.
Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 828, pytest 1005.
2026-09-09 01:02:22 +02:00
peter 59efc8f2e8 fix(e24): review round 3 — the holes round 2's own fixes left
Round 3 reviewed the round-2 commit across six dimensions. Fifteen findings, five survived
three adversarial refuters each; three died on 2+ refutations (a keyboard-focus claim that
is a property of `ui/Button` at all six of its `loading` call sites, a contract-test
false-pass that needs a deliberate revert plus a self-incriminating comment, and a
MAX_EXCLUDE coupling whose slice is unreachable from the real client).

- `rerolling` was still wrong for a FAILED press. `data === undefined` is true forever for a
  key whose request failed (`retry: false`), so the next unrelated mutation re-disabled the
  reroll button — the same defect round 2 reported, one branch over. `!isFetched` is the
  flag that remembers a key settled; `!isError` would not have worked, because query-core
  resets a data-less query to "pending" for the duration of a refetch, hiding the error
  exactly when it matters.
- The hero fallback used `??`, which cannot tell "this budget's last pick resolved to NO
  hero" from "nothing remembered". So a press that failed after an empty one resurrected the
  card the page had already excluded as seen and just said it had nothing beyond. The tagged
  slot is now authoritative for its own budget.
- `setPage`'s boolean — the input the whole burial rule reads — was asserted by nothing.
  The type fences the SHAPE, so deleting a `return` fails tsc, but swapping the two values
  passed typecheck and all 822 tests. NavigationProvider gets its first render test, on the
  value AND on the effects the early return skips.
- `closeThenNavigate`'s door into `buryUnder` was equally unpinned: reverting that one call
  site to the old unconditional burial left the suite green. It has the mirror case now.
- The comment claiming one rule serves both entry points was overstated. `buryUnder` fails
  OPEN, and `CommandItem.run` is typed `() => void`, so only the palette's bare
  `() => setPage(page)` rows forward the boolean at all. Nothing regressed — that path buried
  unconditionally before — but the prose said more than the code does, so it now says what is
  true and points at the deferred fix. Same for a "the only assertion in the suite" claim
  that the file it cites as precedent also makes.

Every new test mutation-checked red for the right reason, both edges where there are two.
Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 828, pytest 1005.
2026-09-09 00:39:37 +02:00
peter e4ffd59ca7 fix(e24): review round 2 — a buried history entry, a stale hero, and a spinner that was not ours
Nine of the round's ten surviving findings. The tenth (a failed shelf reorder leaving an
order the server rejected on screen) is on ROADMAP.md: its obvious remedy reintroduces the
overlapping-move defect the code already documents, so it wants a design, not a review edit.

- `navigateOverOverlays` buried the live overlay entries before knowing whether the
  navigation pushed one. `setPage` no-ops when the page is already current and Home is the
  default landing page, so Shift+W over a player ON Home marked its entry buried under a
  push that never happened: the unmount consumed the mark instead of reclaiming the entry,
  and the next browser Back was a press that did nothing. `setPage` now reports whether it
  navigated, and one `buryUnder` rule serves both entry points — `closeThenNavigate` takes
  the same no-op navigation from a palette command for the page you are already on.
- The hero could answer a new time budget with the previous one's pick. The chip row resets
  seed/seen/press but cannot reset an answer that already resolved, and `keepPreviousData`
  chained through the disabled seed-0 query — so a reroll under "10 perc" showed the film
  picked for "Bármennyi" while it loaded, and kept it if the request failed. `lastPick` now
  carries the server's own budget echo, and the placeholder is scoped to the budget.
- The reroll button spun and swallowed clicks after mark-watched, bookmark, pin and shelf
  moves: the pick key sits under the invalidated `discover` prefix on purpose, so raw
  `isFetching` was true for background refetches. It now means "a press of mine is
  outstanding" — which still covers the first press, where there is no placeholder.
- Failed writes on Home toasted the page's LOAD-failure string over a fully loaded page;
  they get their own key (HU+EN). The empty-state reroll had no `loading`, so the press was
  unacknowledged and a second click skipped a tier. `danger` moved into the shared unpin row
  so the same command is not red on the shelf and neutral in the rail below it.
- Two comments claimed things the code does not do: the `exclude` cap guards no "unbounded
  scan" (it never reaches SQL), and the route test that named the cap sends three keys.

Tests: the burial rule gets its own file (it is the one assertion that lets a reconcile
issue a real programmatic back()); App's keymap wiring gets a source contract, because
dropping it left all 818 tests green; `add_pin`'s IntegrityError arm was reached by nothing.
Every new test mutation-checked red, both edges where the predicate has two.

Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 822, pytest 1005.
2026-09-09 00:12:53 +02:00
peter 3709b71a94 fix(e24): the review's low findings — dead constants, honest labels, a 500 that should be a 404
`add_pin`'s IntegrityError handler assumed the unique constraint was the only way in and
re-raised otherwise, so a target row pruned between `resolve_ref` confirming it and the
INSERT (a Plex sync dropping a removed item) tripped the FK and handed the user a 500.
Semantically that is the same "no such thing" the pre-check already returns. Untested: the
race needs two concurrent transactions.

The five per-kind unique constraints had no test at all — the route's own dedupe returns
before reaching the DB, and the migration drift check filters constraint diffs out as noise,
so they could go missing with every lane green.

Frontend, all from the same review: the shelf's unpin row is now `danger` like every other
destructive row in the app; the hero's artwork button says what it will actually do and
names the item instead of always announcing "Play"; the empty-budget message offers a reroll
rather than `StateMessage`'s "Retry", which promised a re-run of something that had not
failed; `setState` and the watch-later toggle catch their rejections instead of leaving an
unhandled promise and a click that silently does nothing.

`StateMessage` gained a `compact` rung, which retires the frontend's only two `!important`
overrides — they existed to beat the primitive's own padding from the call site, and were
the copyable idiom for the next surface that wanted a compact empty state.

Dead vocabulary removed or made load-bearing: `DISCOVER_BUDGETS` was exported, re-exported
and consumed by nothing while `Home.tsx` re-typed the same four bands (a `readonly
DiscoverBudget[]` accepts any subset, so a fifth band would have typechecked green and
rendered no chip); `PIN_KINDS`' value export had no consumer at all; `useComfortPin` no
longer publishes `pinIdOf`/`toggle`, which nothing outside the module calls.

Three comments corrected against the code they describe: a pointer to a `resolve.py` that
was never written, the file header's claim that Home defines no card component of its own,
and the `seen` window still quoting the 8 that was measured and abandoned for 32.

The rest of the review's survivors are logged on ROADMAP.md under "Deferred — E24 epic-end
deep-review (2026-09-08)" — the ones whose dissent was substantive, and the ones whose fix
is wider than the finding.
2026-09-08 03:23:59 +02:00
peter cedebe0ffb test(e24): close the review's coverage gaps, and correct a comment that was untrue
`test_only_a_show_row_pays_for_a_resolution` seeded no watch state, so every rail query
inner-joined its way to zero rows and the loop body — with it the assertion — never ran. It
now seeds a video, a movie and an episode, and asserts all three kinds are present before
the loop, so it cannot go vacuous again.

The Plex half of the surprise rail had no test for either of its promises: every test that
reached `plex_surprise` passed budget="any" against a library with no state rows, so the
duration bounds and the already-started exclusion could both be deleted with the suite
green. The YouTube half has had tests for the same two rules since S1 — the asymmetric half
of one principle.

`plex_item` and `plex_show` — two of the five pinnable kinds — were pinned by no test at
all, leaving the rating_key → internal-id translation and both shelf arms uncovered,
including the self-reference cycle that once 500'd the shelf. And `GET /api/discover/pick`
was exercised only through its domain function, so the route's own query-string translation
could regress silently; the new test excludes all three candidates, which is what proves the
whole list parsed rather than one unsplit string.

`PIN_COLUMNS`/`PIN_KINDS` claimed in a comment to be the one place the pin vocabulary is
spoken and had no readers anywhere: `resolve_ref` re-spelled the column names and the route
re-typed the kind list. Both now drive off the constants.

The `_NO_TS` sentinel was introduced as a crash fix and is not one. The key it replaced,
`(ts is not None, ts)`, cannot raise — tuple comparison finds the first UNEQUAL element via
`==`, so two null keys compare equal and `None < None` is never reached; three independent
runs confirmed it over every permutation. The rationale now says what the sentinel actually
buys, lives once in `plex.rollup`, and the third rail that was still on the older spelling
has been swept to match. Its test is renamed and re-scoped to the mutation it does catch.

Frontend: the reroll's second press asserted `expect.any(Array)`, which cannot tell an
accumulating exclusion window from a replacing one — both that and SEEN_WINDOW = 1 (each
reinstating the measured nine-card cycle) left the suite green. And `onInfo`'s jump type was
untested while `onPlay`'s was pinned twice, so pointing the ⓘ affordance at the player
typechecked and shipped green.

All eight fixes mutation-checked; every mutant died on its intended test.
2026-09-08 03:14:21 +02:00
peter fdc1eabbe0 fix(e24): the review's seven frontend defects on the Home surface
Shift+W was dead whenever anything was open. The global handler called `setPage` directly,
so the page change unmounted the player (it lives inside the module column) and that
unmount's reconcile reclaimed the entry the navigation had just pushed — the player closed
and the page snapped back. `closeThenNavigate` cannot serve here: a global key does not
know which overlay is open, or whether one is. `navigateOverOverlays` is the same fix one
step shorter — bury the live overlay entries first, so the unmount consumes the marks
instead of issuing `history.back()`.

`q.isError` had no `&& !data` guard, alone among 33 call sites and against the convention
`ui/QueryState` documents: react-query flips status to error on a failed REFETCH while
keeping the payload, and every mutation here invalidates the discover prefix, so one blip
while marking a card watched blanked a fully populated landing page.

The reroll's key rooted at "discover-pick", outside the prefix every mutation invalidates,
so a rerolled hero outlived the page: play it, finish it, and it was still the answer with
a zeroed progress bar. It now sits under the same root. A FAILED reroll also rewound the
hero to the card the page loaded with, several presses back, because a new key per press
means `keepPreviousData` (pending-only) cannot cover an error; the last resolved pick is
held instead.

The bookmark on a rail tile never updated — `VideoCard` reads `video.saved` from this
payload and keeps no optimistic state — so the star stayed empty and the next click
re-sent the same call.

The shelf reorder wrote a captured snapshot back on failure rather than re-asking the
server, which its own comment promised: two overlapping moves each reverted to their own
snapshot and settled on an order neither side held, and since `before` is the whole
payload a rollback also discarded any rail refresh that had landed meanwhile. It now
invalidates the discover PREFIX on settle, which also fixes the sibling budgets keeping
the pre-move order.

The hero/rail de-duplication matched one key, so for a Plex series — the rail row is the
show, the hero is the episode it resolves to — the same series printed twice; it now
matches both keys, as `hero.pick`'s own exclusion already did. And when the hero empties
the resume rail, the rail is hidden rather than announcing "nothing half-finished right
now" under a hero captioned "where you left off".

Ten regression tests, all mutation-checked. Three of them were rewritten after the first
mutation round showed they could not go red: two asserted before the state under test had
settled, one matched a title the Plex tile does not render.
2026-09-08 03:03:45 +02:00
peter 84f4b20e5a fix(e24): the review's three backend defects — blocklist, shelf order, next episode
The resume rail built its own SELECT instead of going through `filtered_query` and copied
only three of its predicates, so it dropped the per-user channel blocklist the module
docstring claims it inherits. Blocking a channel never touches `video_states`, so a
half-watched video from it kept its resume position and headed the rail — and, at the
default budget with the resume tier leading, became the landing page's hero: the app
answering "what should I watch" with the thing the user had just banned. The clause is now
`feedquery.not_blocked_cond`, shared with `filtered_query` rather than copied a second
time; the feed's SCOPE clauses stay deliberately unapplied, for the reason `yt_video_row`
already gives.

`next_position` returned the shelf's row COUNT while nothing ever renumbers positions —
`remove_pin` deletes, and every FK cascade removes a pin when its target disappears. Past
the first hole the count under-counts, so a new pin landed in the middle of the shelf and,
past the second, on a position another pin already held. `max(position) + 1`, the idiom
`routes/playlists` already uses.

`next_episode` took the FIRST in-progress episode in broadcast order while `plex.rollup`,
which drives the show page's Resume button, takes the last. One show therefore had two
different "continue watching" answers, with the Home tile ranking it by its most recent
touch and then offering the episode abandoned weeks earlier. The new test asserts the two
implementations against each other rather than against a literal.

All three mutation-checked: each new test goes red for the defect it names.
2026-09-08 02:32:08 +02:00
peter 828dad015c feat(e24): reach the Home surface — the landing toggle, a palette command, Shift+W
S4's reachability half. Home already led the nav order; what was missing was every
way IN other than the rail.

- Settings → Appearance → Startup: "Start on Home", the documented escape hatch for
  the landing override. Deliberately still localStorage and not a server pref —
  `loadInitialPage` answers before /api/me lands, and a landing module decided after
  the first paint is a visible jump. Its two stored values are now ONE exported
  constant, so the writer and the boot-time reader cannot drift on a magic string.
- The command palette gets "Meglepetés" / "Surprise me": it navigates to Home and
  rerolls the hero on arrival. No second "Mit nézzek?" row beside it — the Home
  module row already matches that phrase through its descriptive title.
- A global Shift+W jumps to Home. A chord, not a bare `w`: the Plex player binds bare
  letters and `w` sits on `s`, so a blind reach for the subtitle key would have thrown
  a viewer out of the film.

The palette's request crosses a lazy module boundary, so it is a token that waits
(lib/homeIntent) rather than an event fired into an empty room. Browser-measured:
taken with a compare-and-clear, because StrictMode's double-invoked mount effect
otherwise rerolled twice and landed on the page two picks deep.

HU/EN parity kept; UAT on localdev in both languages, including the empty shelf.
2026-09-08 00:06:34 +02:00
peter 82914b56ca fix(e24): the Home surface's three UAT findings — Back, the reroll, and a show card
All three were measured before they were touched, and the diagnosis moved twice.

**Back from a Plex title played off Home landed on the Plex module's root.** One
click pushed three history entries: the page navigation, the sub-view opened by
the `jumpTo` hand-off, and — under StrictMode, whose effect body runs twice
before `setJumpTo(null)` re-renders — that same sub-view a second time. So the
first Back visibly did nothing and the second stranded the user on a Discover
page they had never visited.

A cross-module hand-off is the CONTINUATION of the navigation that brought you
there, not a second navigation, so it now REPLACES the page entry instead of
stacking on it (`useHistorySubview.open(view, {replace})`, asked for by the jump
target). Replacing is also idempotent, which is what disarms the StrictMode
double-run. The intent rides on `PlexJumpTarget` rather than being inferred,
because the header typeahead jumps from INSIDE the module, where the entry being
replaced would be a sub-view the user really was looking at.

**"Mást kérek" cycled nine cards and never left the resume tier.** Measured:
twelve presses, nine distinct titles, period nine, every one `tier: "resume"` —
the shelf and the surprise tier were unreachable by pressing the button. Two
causes multiplied: the tiers were tried in a fixed priority order over a 20-row
resume rail, so the button just walked down it; and the client's rolling
exclusion window of 8 guaranteed the wrap.

The press count now rotates which tier answers first (`start`), so each press
offers a different KIND of answer — the user's call, given the choice between
that, a wider window, and a surprise-only reroll. The fallthrough is kept, so a
press does not come back empty because the tier it led with had nothing. The
window is matched to the server's own cap rather than a smaller number of ours.

**A half-watched series on Home opened the show page.** A movie already went
straight to the player; only shows did not — and the comfort shelf and the hero
both resolved a show to its next episode while this rail alone did not, handing
back six seasons to choose between. The rail row now carries `next`, and the
client already played `next ?? card`, so this is a backend fix.

Affording it needed `queries.next_episode` to stop loading every episode of the
show into Python to pick one of them (236 rows for Friends, 227 for JAG) — three
ordered LIMIT 1 probes instead. The show→episode rule now lives in ONE function
(`rails.playable`), which is what the shelf, the hero and the rail all call; two
implementations are how they came to disagree.

Ten mutations run, all ten red for the right test — including a first pass where
the press count reached `api.discoverPick` and stopped there, invisible because
every Home test mocks that module. The slice now has its own URL test.

Verified in a clean browser tab against the live account: one Back returns to
Home, the rotation gives resume/comfort/surprise across nine presses with no
repeat, and Westworld opens at S1E1 rather than at its show page.
2026-09-06 05:38:28 +02:00
peter ccd5f3739a feat(e24): pin to the comfort shelf from every card and row
S3 wires the shelf's editor. The gesture is the E10 right-click menu, so the
decision "this is comfort media" can be made where the thing is — the feed, a
Plex grid, a channel row, a playlist — rather than only on the page that lists
the shelf.

One shared piece, `components/comfortPin.tsx`: the five pin-kind constructors,
the shelf query, and ONE `ActionMenuItem` that switches Pin/Unpin honestly (and
says "not known yet" with a disabled row until the shelf has answered, since a
confident "Pin" over something already pinned is exactly the quiet lie the house
rules forbid). Surfaces that already have a row menu compose the item in
(`Playlists`, `PlaylistsRail`); the rest mount `ComfortPinMenu`.

Two structural notes:

- The feed's menu lives in `VirtualFeed`, not in `Feed` — that is the list, and
  it is also what gives the channel page the gesture for free, since it renders
  `<Feed channelScoped>` all the way down.
- Cards take `menuProps` as a FUNCTION of the row, not the built props object.
  `triggerProps` is stable across renders while its return value is not, and a
  fresh object prop would blow past `VideoCard`'s memo for every card in the
  virtualized feed on every parent render.

`DataTable` and `ChannelLayoutGrid` grow a generic `rowProps` so a channel row
answers the same gesture in both of the manager's layouts; the table stays
domain-free and never learns what a menu is.

On Home the shelf tile's menu is the whole editor: four move rows (front / left
/ right / end, reusing the playlist menu's ordering idiom) plus the unpin, over
menu rows rather than drag & drop — dragging inside a horizontal, scrolling
Carousel fights the scroll, and menu rows are keyboard-reachable for free. The
reorder is optimistic on the cached payload and re-asks the server on failure
rather than showing an order that was never accepted.

The exhausted shelf tile (a pinned channel with nothing unwatched left) is
`aria-disabled`, not `disabled`: measured in Chromium, a disabled button still
receives contextmenu and pointerdown, but it loses focus — and with it Shift+F10,
the only keyboard route to the menu it most needs.

Verified end to end in the browser against the live account: all five pin kinds,
the reorder (persisted across a reload), the unpin, and the honest Unpin label on
a resume tile that is already on the shelf.
2026-09-06 05:07:06 +02:00
peter f3c0b7be23 feat(e24): the Home surface — one question, one answer, both sources
E24 S2. A new `home` module in layout C+ (the mockup the owner picked): the
time-budget chips on top, one large hero that answers them, then the comfort
shelf and a cross-source resume rail. Home is the landing — deliberately OVER
the remembered last module, since landing on whatever you happened to close the
app on is the habit this epic exists to replace; `LS.startOnHome = "0"` opts an
account back out (Settings toggle lands in S4).

No third card component: each unified card carries the native feed / Plex
payload, so the rails render through the existing VideoCard (`cardsSmall`) and
PlexPosterCard, and `aspect` decides the tile shape without branching on source.
Plex playback hands off to the Plex module via a new `jumpTo` type — the player
lives there with its lazy hls.js and its provider tree, and mounting a second one
on the landing page to save one hand-off would be the expensive way round.

Four defects found by driving it in a real browser against the live account, each
now pinned by a mutation-checked test:

- **The reroll gave the same card back.** A new seed only re-draws the SURPRISE
  tier; the resume tier deterministically returns the most recently progressed
  thing that fits, so anyone with something half-finished pressed a dead button.
  `/pick` now takes an `exclude` list of `<source>:<id>` keys and the client keeps
  a rolling window of 8. Excluding a SHOW also excludes the episode it resolves
  to — otherwise the same suggestion comes back wearing a hat.
- **"Nothing to suggest yet" flashed over a full library** whenever a reroll was
  in flight: the new query key has no data for a beat. An empty state standing in
  for a loading one is the dishonesty the house rules name outright. The hero now
  falls back to the loaded payload while a pick is pending, and only trusts a null
  once the pick has RESOLVED.
- **A budget change replaced the page — chips included — with a spinner**, so the
  control you just clicked vanished under your cursor. `keepPreviousData`, the
  same reason the feed uses it.
- **The hero printed twice**, once as itself and once as the first resume tile.

Two smaller UI calls made from the same session: the budget caption sits NEXT to
its chips rather than in SegmentedToggle's justify-between mode, which on an 80rem
page throws the answers to the far edge; and the feed takes the YouTube mark its
own toolbar already uses, since Home now owns the house icon.

Gate green: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 780,
pytest 989. Verified end to end on localdev against the real account: budget
chips re-answer (matched on REMAINING time — a 51-minute video with 2:46 left is
a ten-minute answer), the reroll walks three distinct heroes, and a Plex show
tile opens its drill-down in the Plex module.
2026-09-06 04:14:14 +02:00
peter 53b50231ff feat(e24): the Home surface read-model — hero, resume rail, comfort shelf
E24 S1. The anti-paralysis answer layer, served from our own Postgres with no
upstream call: one hero suggestion, one CROSS-SOURCE resume rail, and the user's
own comfort shelf.

Survey first, because the roadmap line was wrong about the starting point: Plex
already ships the whole trio (E5's Discover rails, including a reshuffling
Surprise me), the YouTube side ships none of it, and nothing anywhere answers
the question across both. The data already existed on both sides — VideoState
and PlexState are structurally identical, the feed has an `in_progress` filter
and a seeded shuffle, and the YT player already resumes position.

- app/discover/budget.py — the time bands. "How much time do you have"
  (10 perc / 30 perc / egy este / mindegy) replaces 15/D's mood questionnaire:
  deterministic, answerable from a column we have, and it picks the source for
  free. Matched against REMAINING time, so a film with 8 minutes left is a
  ten-minute answer. An unknown runtime fits only the unbounded band.
- app/discover/cards.py — a thin unified card that carries the NATIVE feed /
  Plex payload, so the page renders with the existing card components instead of
  a third one. `aspect` is the one new field: a rail needs to know poster from
  16:9, and that is a property of the artwork, not of the module.
- app/discover/queries.py — the SQL, reusing feedquery and plex.rollup rather
  than the Plex Discover ROUTE, whose LibraryFilters narrowing belongs to a
  surface that has a sidebar. One seed drives both halves so a reroll moves
  them together.
- app/discover/hero.py — resume -> comfort -> surprise, with the chosen tier in
  the payload so the client labels it honestly instead of guessing.
- comfort_pins (0083): five nullable CASCADE-ing FKs under num_nonnulls(...) = 1
  rather than a flag or a polymorphic string ref. A flag cannot order a shelf or
  hold a channel; a string ref would let a deleted library item leave a dead tile
  on the one surface whose whole job is to be reliable.

Two honest-state rules the tests pin: the surprise draw excludes anything
already started (the resume tier may have skipped it for not fitting, and
offering it back as "Meglepetés" would contradict the question), and which
source leads the draw follows the seed, so the Plex library can actually answer
"egy este".

Gate green: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 770,
pytest 987.
2026-09-06 03:31:26 +02:00
peter 5703c4d713 fix(plex): a null progress timestamp must sort a rail row last, not raise
The Continue-watching and Surprise rails sorted on `(ts is not None, ts)`. That
key LOOKS null-safe and is not: two rows that both lack a timestamp compare
equal on the flag and then compare `None < None`, which raises — so a show whose
rollup has no `resumed_at`, or a movie state written before
`progress_updated_at` existed, could 500 the rail instead of sorting last.

Replaced with an explicit epoch sentinel, and named it beside the other rail
constants so the sibling rail E24 adds cannot re-derive the same broken key.
2026-09-06 03:31:09 +02:00
peter 013f19faca refactor(plex): move content_libs down into the query layer
The browse scope is a pure DB query with no request state, but it lived in
routes/plex/_shared.py. E24's Home surface needs the same scope from a domain
package, and a domain package importing from routes/ inverts the one-way
dependency CLAUDE.md draws.

Moved to app/plex/query.py and renamed to content_libs (public — it is a domain
helper now). Swept all five call sites and both tests in one go rather than
leaving a re-export shim: the canonical-patterns table already names the
routes.admin re-export as the anti-pattern.
2026-09-06 03:31:02 +02:00
peter 8d9a99ce54 release: 0.82.0 (feature/e10-platform-ergonomics)
gate / frontend (push) Failing after 2m7s
gate / backend (push) Successful in 3m36s
2026-09-06 02:37:10 +02:00
peter 45dd4bdda7 chore(release): 0.82.0 — E10 Platform & Ergonomics 2026-09-06 02:37:07 +02:00
peter dac391a890 fix(e10): review round 3 — settle the focus race in a browser instead of guessing again
Round 3 reviewed round 2's diff. Trend across the loop: 32 findings over
the whole epic, 15 over round 1's fixes, 11 over round 2's. What it caught
is mostly the same shape as before — a guard narrowed until it was wrong
in the other direction — plus one item this branch had now got wrong
three different ways.

Backspace ownership over the player, third and final answer. Round 1
deferred to nothing (the cheat sheet took the key and the player stopped
underneath it). Round 1's fix deferred to `anyLayerAbove` (the player's
own gear/tracks popovers push layers too, so it stopped closing them).
Round 2's fix named the sheet and palette (the app-wide ErrorDialog is
neither, so the player tore down under it). The rule that actually holds
is not a list of overlays: the player's own panels answer first, because
handleBack's cascade is what closes them, and anything else layered above
takes the key.

The rail's inline rename is settled by measurement rather than argument.
Two rounds reasoned about whether a React passive effect lands before or
after floating-ui's return-focus microtask and reached opposite
conclusions; a jsdom harness built to decide it reproduced no failure for
ANY shape, because focus never enters the menu there — so it was deleted
rather than kept as a test that discriminates nothing. Checked in a real
browser instead: the field opens and holds focus. The shipped fix works,
and the comment now says what was measured rather than what was inferred.

The same check found that round 1's OTHER rail fix never worked at all.
Escape does run its own handler (typing then Escape leaves the name
unchanged, so it cancels rather than blur-commits) and the row button is
focusable when focused from outside — yet focus still lands on <body>.
Three explanations, none of which survived. So the refocus machinery is
reverted to the shape that shipped before this epic and the gap is
documented where the next person will hit it, with the evidence and an
instruction to instrument before theorising. A fourth guess would have
been worse than a known bug.

Also: the two comments in Playlists.tsx that still claimed the disabled
menu rows explain themselves — untrue since round 2 reverted `inert`, and
exactly the comment-vs-code drift CLAUDE.md calls the source of the worst
bugs here. And the move-failure toast printed HttpError.detail verbatim,
which for the reachable 404 is the backend's bare English "Unknown
playlist"; only a structured detail has been through localizeDetail, so
only a structured detail reaches the toast now.
2026-09-06 02:22:23 +02:00
peter 4b33036e7c fix(e10): review round 2 — revert what round 1 got wrong, narrow what it got too broad
Round 2 reviewed round 1's own fix diff, which is where a fix round does
its damage. It found 15 issues in 792 lines — a much higher density than
round 1 found in the whole epic — and most were regressions round 1
introduced. So this follows the rule rather than fighting it: revert the
fixes whose mechanism was wrong, narrow the ones whose guard was too
broad, and stop reaching for a third variant.

REVERTED, because the mechanism does not work:

- The `inert` menu item. Round 1 swapped a native `disabled` for
  `aria-disabled` so a row's "why is this off" tooltip could be reached.
  floating-ui's list navigation skips `aria-disabled="true"` exactly as it
  skips `disabled` (isListIndexDisabled) — and AnchoredPopover's own
  comment, written before this branch, already said so. The swap fixed
  nothing and made the mouse worse: without native `disabled` the row took
  pointer events, the full active highlight and DOM focus, then no-opped
  on click. The underlying finding is logged where the prop was, unfixed
  and honest: the reason has to go somewhere a disabled row is not.

- `useContextMenu` owning the focus restore. Turning floating-ui's
  returnFocus off platform-wide traded one bug for two: Playlists spreads
  triggerProps on a bare <tr>, so restoring to it is a spec no-op and
  Escape stranded focus on <body>; and with dismissOnScroll now on, the
  restore's plain focus() scrolled the row back into view, undoing the
  scroll that closed the menu. Patching that needs three more guards on
  one fix. The original bug — the rail's rename field dying on arrival —
  is fixed at the field instead: it focuses from an effect, which is
  scheduled after the microtask that was blurring it.

NARROWED, because the fix was right and the guard was not:

- PlexPlayer's Backspace deferred to `anyLayerAbove`, but the player's own
  gear and tracks menus register layers via useDismiss — so Backspace
  stopped closing them and did nothing at all. Only the cheat sheet and
  the palette own that key, so it names them.
- `historyDepth = 0` also desynced the counter from overlays the
  navigation does NOT unmount (About and ReleaseNotes live outside the
  page column), leaving one Back to both navigate and close them. The
  entries are marked buried and consumed silently by reconcile instead.
- The move's error toast fired for every status, so a transport fault
  surfaced twice on top of req's own dialog. Scoped to 403/404.
- The palette's moved-pointer guard could not judge the FIRST move of a
  session and went stale across a close. `e.movementX/Y` asks the browser
  directly and needs no state.
- `useKeyScope("contextMenu")` runs once per hook, and the Playlists page
  mounts two — so the cheat sheet rendered the chords twice under a
  duplicated React key. The scope stack dedupes.
- The `enabled:hover:` sweep missed the half that lives in CSS, so a
  disabled glass button still lifted. `.glass-hover:not(:disabled):hover`.
- ShareDialog's migrated buttons passed a colour className that the
  variant's own colour class outranks under clsx, so Revoke hovered
  neutral and a password-protected link looked unprotected. The colour
  goes on the icon, as the Copy button beside them already did.
- `matchesShortcut` took an untyped id and ids are only unique within a
  scope, so `back` would have answered for the wrong player. Keyed on
  scope + id, both literal.

And the systemic one: round 2 reverted eight of round 1's behaviour fixes
as a single mutation and all 763 tests still passed. Round 1 tested the
three fixes that were easy to test and none of the rest. So the guards
with the widest blast radius are pinned now, each mutation-checked to
fail for its own reason: the global keymap's repeat/allowInInput/disabled
handling (a new dom test), the cheat sheet bound to the real
useContextMenu rather than a scope stub, the scope dedupe, the history
burial, and the Button accessible-name union — whose @ts-expect-error
fixtures turn a widened type into a red typecheck lane.

The chipLane contract test that round 1 rewrote to count per lane counted
the import line as a lane, giving every file one free mirror — so the
exact hole it was rewritten to close was still open. Mutation-checked
both ways this time.
2026-09-06 01:57:32 +02:00
peter 80dea4f307 fix(e10): review round 1, part 2 — the docs the epic invalidated, and three contracts made real
Finishes round 1. Part 1 took the runtime bugs; this takes the prose that
had gone false and the three "the comment says so" gaps behind it.

CLAUDE.md is auto-loaded into every agent session and ui/README.md is
not, so an architecture paragraph that still described a flat components
bag was the most expensive stale sentence in the repo: the next agent
adding a popover reads it, sees no platform layer, and writes the generic
component into components/ — the invisible-layer failure this epic exists
to end. It now names components/ui/ and says the boundary is lint-enforced.
The canonical-patterns table gained the three rows the epic established:
ui/Button over a hand-rolled <button>, a keymap row over a private keydown
listener, and disabled-when-busy vs inert-when-inapplicable.

ui/README.md's inventory was written in this branch's first commit and
never learned about CommandPalette, ContextMenu/useContextMenu,
KeyboardHelp or gridColumns — in a section whose own closing rule is that
a second component painting the same thing is how the last inconsistency
happened. It also claimed "everything is rem-derived" while VirtualGrid's
minCol/GRID_GAP are deliberately px; that is now stated as the exception
it is, and a new section records which promises here are tests rather
than sentences.

Three contracts stopped being prose:

- A text-less Button must carry a name. "Required for iconOnly" was a doc
  comment, which is why the migration shipped two nameless icon buttons
  through eight green lanes. It is a union now, covering both spellings
  (iconOnly, and size="icon"/"icon-sm"), satisfied by label, aria-label or
  title. It caught one more on the way in: Banner's dismiss took an
  optional dismissTitle, so it could render unnamed — it falls back to
  common.close now.
- gridColumns' gutter arithmetic. Every existing assertion survived
  replacing floor((w + GAP) / (minCol + GAP)) with the naive
  floor(w / (minCol + GAP)): the fixed widths agree, the capped cases are
  cap-bound, and the loops compare columnsFor against itself. Pinned at
  the exact-fit boundaries where the two first disagree (536 and 828).
- ShareDialog's link row. Only Copy was migrated in S1, leaving one 2rem
  button between two 1.75rem ones — and on a coarse pointer .touch-target
  grew only the migrated one to 2.75rem while the destructive Revoke
  stayed small.

Also: the maxCols docstring said 8 columns of 384px where the test it
shipped with asserts 369.
2026-09-06 01:30:40 +02:00
peter 20e17c0ccb fix(e10): review round 1 — the boundary that did not bind, and 14 more
The epic-end review found 32 verified issues. This is the first fix round:
the three highs, the user-facing mediums, and the tests that were passing
for the wrong reason. Docs and the remaining low-severity tail are next.

The boundary rule was the worst of them. Both `no-restricted-imports`
regexes were written in plain "..." literals, where `\.` is an unknown
escape that collapses to the any-char `.` — so the rule read like a
boundary and matched far less than one. `../../lib/api` (the barrel every
feature component imports) and `../../components/Feed` (the same file
`../Feed` names) both sailed through a green gate. String.raw, a third
pattern for the second spelling, and a vitest over ESLint.lintText so a
rule that silently matches nothing can never look like a clean codebase
again.

closeThenNavigate waited for exactly one reclaim popstate. With a second
Back-closable layer still mounted — a palette opened over the player —
the navigation unmounted that layer too, and its reconcile backed over
the entry the navigation had just pushed: "go to Channels" closed the
player and landed on the feed. It now waits for the LAST programmatic
back, then adopts the entries the navigation buried, and falls back after
250ms rather than hanging if no popstate is ever coming.

The rail's inline rename died on arrival. With no domReference, floating-
ui's returnFocus falls through to a module-global stack and restores some
unrelated element from an earlier popover — blurring the field into its
own commit-and-close, from the second popover of a session onward. The
context menu now owns the restore: back to the row, but only if the row
is still there and nothing else has claimed focus.

Also:
- Backspace under the cheat sheet stopped Plex playback and dropped to
  the grid; `back` now defers to any layer above, like `escape` beside it.
- A move invalidated only the source list, so the target kept serving
  pre-move contents for 30s while the rail already counted N+1.
- Every move error a user can reach is a 404, which `req` leaves to the
  caller — so the row snapped back in silence and the shipped moveErrors
  strings were unreachable.
- swallowClick armed on long-press but was disarmed only by the click it
  was armed for; any press that produced no click ate the next tap on the
  whole list.
- The context menu stayed parked over whatever row scrolled under it.
- overflow-clip on the chip lanes ate the focus ring's top and bottom.
- "Why is this off" rode on disabled rows, which show no tooltip and take
  no focus: a new `inert` item keeps the reason readable.
- Shift+F10 was advertised app-wide but wired only on Playlists; it has
  its own scope now, and useContextMenu matches through the shared parser
  instead of its own literals.
- Mod+K could not close the palette, `h` machine-gunned on auto-repeat,
  and both were bound above the branch that mounts their overlays.
- Palette options were in the tab order under aria-activedescendant, and
  a scroll-synthesised mousemove stole the keyboard selection.
- The OSD live region was created together with its own text.
- Ending a rename stranded focus on <body>.
- The Button variants' hover halves are `enabled:` — the one migrated
  site that had guarded its own lost the guard on the way in.
- HU builtinLocked named the list "Watch later" in a Hungarian sentence.

Tests: the chip-lane contract counted per FILE, so a second hand-rolled
lane in an already-compliant file was invisible; the closeThenNavigate
ordering test only passed on a popstate left behind by its neighbours and
failed alone, so it moved to its own file.
2026-09-05 23:52:29 +02:00
peter 7d302f71b7 feat(e10): let an ultrawide buy bigger cards instead of more of them
The viewport contract has two ends, and only the floor was enforced. An auto-fill grid
answers "more room" with more columns, never bigger ones, so it settles just above its
minimum column at any width: measured at 3440px, the feed laid out 11 columns of 264px.
A feed diluted across an ultrawide fails the contract as surely as a broken one at the
floor.

Multi-column feed views now carry a `maxCols`; past it the leftover width flows into
the items. At 3440: cards 11x264 -> 8x370, cardsSmall 15 -> 11x264 (still visibly the
dense card, and never as wide as cards gets), tiles 7 -> 6x498. Each cap is set so the
item tops out near 1.5x its own minCol, which is what keeps a view recognisably itself.

A cap can only bind ABOVE the width at which that many columns already fit, so it is a
pure ultrawide affordance — 1024, 1920 and 2560 lay out exactly as before. Verified in
the browser at all three widths and across all five views.

The Channels grid needs no cap: the page's own 96rem --content-mw already bounds it.

Also renames VirtualGrid's `maxCol` to `colMaxWidth`. It was never a column count — it
is the single-column reading-width cap — and `maxCol`/`maxCols` one letter apart is the
kind of drift that has bitten this repo before.

The column maths moves to ui/gridColumns.ts as a pure function so both ends of the
contract are unit-testable without a layout engine; the tests assert the cap's numbers,
that it never binds at the floor, and that the dense card cannot grow into the standard
one.
2026-09-05 22:22:28 +02:00
peter f99240d8ca fix(e10): stop the hidden chip measurer scrolling the whole app sideways
The four chip lanes render an intentionally unbounded hidden "mirror" so the shared
fit core can read each chip's natural width. An absolutely positioned box still
contributes to its ancestors' SCROLLABLE overflow, and `visibility: hidden` does not
exempt it — and nothing between the toolbar band and <html> clipped it.

At the 1024x768 floor that meant the channel manager's document scroll width was
1258px: the whole app scrolled 249px sideways into dead space and dragged the fixed
header to x=-177. The same mirror was also 228px wide inside the Tags column's 144px
cell, which pushed the table to 990px inside its 906px container — i.e. it broke
DataTable's "the table never exceeds its container" contract and truncated channel
names to "The...". One cause, two symptoms.

The lane now carries a shared `chipLaneClass` = `relative overflow-clip`, so the
contract lives in one constant instead of four hand-rolled class strings. `clip`
rather than `hidden` because it creates no scrollport, so nothing can ever scroll the
mirror into view. Measurement is untouched: offsetLeft/offsetWidth are layout values
and a clip does not change layout.

Measured after: document 1258 -> 1024, sideways scroll 249 -> 0, table 990 -> 921
with no cell overflow, and the "+N" collapse still fits exactly as before.

A source-level test asserts every component rendering a mirror uses the lane class —
the drift a fifth hand-rolled lane would introduce, which no DOM test can catch since
jsdom has no layout engine to overflow.
2026-09-05 22:22:14 +02:00
peter cc8cf96c19 feat(e10): a jump-to palette on Ctrl/Cmd+K
Reaching a channel three modules away cost a rail, a page and a scroll.
Now it costs a keystroke and half a word: Mod+K searches modules,
channels, playlists, the saved views applicable where you stand, and the
app-wide actions.

Scope agreed with the product owner: navigation + actions, but only the
reversible local ones (the shortcuts sheet, the theme). "Sync from
YouTube" is deliberately absent — it is a long-running job against a
quota, it wants the progress and error surface the Channels page already
gives it, and one mistyped Enter is a worse failure than the walk it
would have saved.

Matching is word-wise and accent-folded, because this is a bilingual app
and nobody reaches for the accent keys in a hurry: "pod thevr" finds
TheVR Podcast and "penzugy" finds pénzügy. The arrows walk one flat
result list so crossing a group heading is invisible, and the field
carries the combobox/listbox aria tree that virtual selection needs.

ui/CommandPalette is domain-free (items are a prop, per the ui/ import
rule); AppCommandPalette supplies them. The shell is Modal again, with a
new `top` placement so a search-first dialog grows downwards instead of
jumping as results change.

Also fixes a bug the palette hit on its first run, and that PlayerModal
had already hit once: closing a dialog reclaims its Back-to-close
history entry with a programmatic history.back(), which undoes any
pushState made before it lands — so a command that navigates only closed
the palette. The rule is now closeThenNavigate() in history.ts with a
test, and PlayerModal's inline copy is gone.

The global `h` moves out of KeyboardHelp into useGlobalKeymap beside
Mod+K, so one listener owns the whole `global` scope and dispatch's
exhaustiveness keeps meaning what it says.
2026-09-05 21:33:29 +02:00
peter 86163958d6 feat(e10): one keyboard cheat sheet for the whole app
The Plex player had a shortcuts overlay nobody else did, so the YouTube
player's Space/F/arrows and the Plex browser's Backspace were documented
nowhere. ui/KeyboardHelp is that sheet for every surface, opened by a
global `h` - not the conventional `?`, which is a Shift chord on a
Hungarian layout, while `h` is the key this app's reader already has in
their fingers from the player.

It renders the shared table filtered to the scopes that are LIVE: a
surface registers its scope with useKeyScope while mounted, and only
while its keys really do something - PlexBrowse deactivates its own
scope inside the player, where its Backspace no-ops, so the sheet cannot
promise a "Back one level" the handler declines to perform. Same rule
for the `when` rows: ⇧←/⇧→ are listed only inside a series, an answer
the player publishes because the sheet cannot ask.

No new overlay primitive, after checking: Modal already portals into
document.fullscreenElement ?? document.body (so this works over a
fullscreen player), registers an Escape layer, traps focus and handles
browser Back. Third time this epic that the thing was already built.

Shift+F10 is in the table as `handledLocally` - documented and sharing
the chord parser, but matched by useContextMenu on the focused row.
dispatch neither routes it nor demands a handler for it, so the
exhaustiveness check stays honest about what it actually covers.

Also: A/S have cycled the Plex audio and subtitle tracks since E5, but
silently - the only way to see where you landed was to open the tracks
menu, the exact round-trip the shortcut exists to avoid. cycleAudio and
cycleSubtitle now return the track they moved to, and the player flashes
it over the video.
2026-09-05 21:15:42 +02:00
peter 12822a9e04 feat(e10): put every keyboard shortcut in one table
Six surfaces each owned a private keydown listener with its own key
literals and its own idea of "am I typing?", and they had drifted:
PlexPlayer matched `case "a"` without checking modifiers, so Ctrl+A
(select all) cycled the audio track, and its typing guard missed
<select> and contentEditable where the other two surfaces covered both.

lib/keymap.ts is a TABLE, not a dispatcher. Each surface keeps its own
listener and closures - PlexPlayer's refs are not worth re-plumbing -
but the chords, the modifier rules and the typing guard live in one
place, and dispatch() binds a scope's rows to handlers through a mapped
type, so a row without a handler (or a handler without a row) is a tsc
error rather than a comment someone forgets.

Matching is now exact on modifiers, which is what kills the Ctrl+A bug
and lets Shift+arrow be its own row beside the bare arrow. Escape opts
out of the typing guard (allowInInput) so it stays the way out of a
player's own subtitle-offset field.

The player's shortcut sheet renders helpLines() instead of a hand-copied
array beside the handler - the two used to be edited separately, which
is how a sheet ends up documenting a key that no longer exists. Labels
move to the new `keys` i18n namespace (EN + HU), and a test asserts
every row is named in both.
2026-09-05 20:58:17 +02:00
peter 928eac4c89 feat(e10): open the app's menu with a right-click, starting with playlist rows
S2's headline was never a new Menu: `ActionMenu` already sits on `AnchoredPopover`
(arrows, typeahead, Escape, anchor-flip). What was missing is an INPUT PATH into it, so
this adds `useContextMenu` + `<ContextMenu>` over the same `ActionMenuItem[]` and the same
`MenuItemList` — a row cannot grow a right-click menu that looks or behaves unlike its own
⋯ menu. Three ways in: right-click anywhere on the row, long-press on touch/pen, and
Shift+F10 / the Menu key. The LEFT button is untouched and keeps the row's own action.

One popover per list, anchored to a virtual element at the pointer (native behaviour, and
a table of 200 rows would otherwise mount 200 floating contexts to keep 199 shut).

Platform bits it needed: `hoverAlso` on AnchoredPopover (a submenu falls open under a
mouse and is tapped by a finger — mouse-only, so a tap's synthetic mouseenter can't race
the click toggle) and `elementRef` on PopoverItem (the submenu anchors to the very
menuitem that opens it, where aria-haspopup/expanded belong). `ActionMenuItem` becomes a
discriminated union — a row either fires or opens a submenu — plus `danger` and
`separatorBefore`.

First consumers, both in Playlists:
- item rows: play · move to another list (a submenu of every other playlist, one gesture)
  · to top / up / down / bottom · remove. Reordering goes through the same undoable store
  as the drag handle and is disabled under a view for the same reason, with the same
  explanation rather than silence.
- rail rows: rename (in the row that was right-clicked, keeping its cover so the rail
  doesn't jump) · delete. Both now run through `usePlaylistActions`, which also takes over
  the detail header's pencil and trash — the delete dialog's YouTube branch and the
  "what gets selected next" rule exist once, not twice.
- the rail gains a "recently changed" sort key off the new `updated_at`. NOT the default:
  a rail that reshuffles itself while you look at it costs more than it saves.
2026-09-05 19:59:56 +02:00
peter b28be682fe feat(e10): make "move a video to another list" one atomic request
The one-step move S2 needs has no honest client-side shape: "add to the target, then
remove from the source" leaves the video in BOTH lists whenever the second call fails,
and the reverse order loses it entirely. So it is one endpoint in one transaction —
POST /api/playlists/{id}/items/{video_id}/move — which re-parents the existing item row
(so `added_at` survives), recomputes `dirty` on both sides, and is idempotent when the
target already holds the video.

Also publishes `updated_at` on the playlist summary, and bumps it explicitly for the
edits that only write `playlist_items` (add / remove / reorder / move). UpdatedAtMixin's
`onupdate` fires on an UPDATE of the playlist row, so without `_touch` the rail's coming
"recently changed" sort would have seen renames and nothing else.
2026-09-05 19:59:41 +02:00
peter 7e97559615 feat(e10): put the 1024x768 size contract inside the primitives
S1c, the last part of S1. The contract now lives in components/ui/ and
in one CSS recipe instead of in ~160 call sites:

- Minimum target 2.75rem, on coarse pointers only (.touch-target). A
  32px icon button is fine with a mouse -- precise pointer, hover to
  confirm the aim -- and not fine with a thumb, so it grows exactly where
  there is no precise pointer, which on this app's real second device is
  the tablet. It grows the real box rather than painting an invisible
  ::after hit area, because two expanded hit areas on icon buttons 4px
  apart would overlap and the overlap fires the WRONG button. `link` is
  excluded: it is text inside a sentence.
- 2.75rem rather than 44px so the target obeys the text-size slider like
  everything else; verified that the type/geometry ladder was already
  rem-derived off --font-scale, so nothing else was needed there.
- The ladder is documented as a closed set in Button.tsx, with the site
  counts it was derived from, and ui/README.md states the contract.

Measured in the browser: fine pointer 36x36 and 34px-tall buttons
(unchanged), coarse pointer the same buttons at 44x44 and 44px tall.
1024x768 renders identically. 3x tsc, eslint, knip, prettier and 690
vitest green.
2026-09-05 18:49:48 +02:00
peter 721467ef69 feat(e10): finish the mappable Button migration, including the ui/ layer itself
Third sweep, 20 more sites: the rest of AdminUsers (the access-request
and invite clusters), Scheduler, SettingsPanel, SetupWizard, About, Feed,
KeyGate, OnboardingWizard -- and, importantly, four platform components
that were still hand-rolling their own buttons: Modal's close, Banner's
dismiss, SidePanel's two toggles and QueryState's retry. A platform layer
that bypasses its own primitive is the same defect one level up.

Hand-rolled buttons with their own geometry: 110 -> 39; Button call sites
37 -> 106. What is left is deliberate, and it is the list Button's own
docstring excludes plus two families this pass refused to force: video
overlay controls painted on white/N, drag handles, chip removers,
accent-solid pills, the bg-card family on the signed-out screen, and
sites whose paint is conditional (`suspended ? warn : neutral`) or
otherwise has no variant yet. Forcing those through a className override
would be unsafe without tailwind-merge, which is exactly the hazard this
sprint documented.

3x tsc, eslint, knip, prettier, 689 vitest green; UAT on Settings and the
About dialog (its close button and full-width CTA are both migrated),
clean console.
2026-09-05 18:41:41 +02:00
peter 47e2293de3 feat(e10): migrate the notification, message, auth and Plex action buttons
Second sweep, 34 more sites: NotificationsPanel (14 -- eight identical
accent text links, three ghost icon buttons, two ghost text actions),
Welcome's five auth links, KeyGate, Messages, NavSidebar, ChatThread,
ShareDialog, ChannelPage, PlexPlaylistView, PlexPlaylistAdd,
PlexCollectionEditor and plexViews.

Moves the font-weight out of Button's BASE and into each variant, so the
`link` variant can be semibold (which is what every text link in this app
looks like) while the action variants stay medium, and no element can end
up with two font-* utilities.

Welcome is the one signed-out surface this session cannot see, so only
its byte-identical link swaps were taken; its primary submit and the
bg-card button family are left for a pass that can UAT the screen.

3x tsc, eslint, knip, prettier, 689 vitest green; UAT on Notifications,
Messages and the nav rail, clean console in a fresh tab.
2026-09-05 18:39:22 +02:00
peter 29bb914a95 feat(e10): re-aim the Button variants at the paint the app actually uses, and migrate the first 15 sites
Counting the un-migrated call sites showed the primitive and the app had
drifted into two different looks: the majority of hand-rolled buttons
hover to accent (outline) and fill with card (ghost), while Button
hovered to fg and filled with surface. That split is the inconsistency
this epic exists to remove, so the primitive follows the majority rather
than the other way round -- 16 existing call sites shift slightly, ~90
hand-rolled ones stop being special.

Adds the variants the call sites prove are missing (outline-accent,
outline-danger, outline-warn, link), an icon-sm size for the p-1.5 icon
family, and a size=none so a link owns no button padding; renames
danger-outline to danger-soft so it cannot be confused with
outline-danger (red at rest vs red on hover). Two new tests lock the
link-size default and the rule that a variant never adds a second
font-weight utility -- without tailwind-merge, two font-* utilities are
decided by stylesheet order, not class order, so that is a real hazard.

Migrates AdminUsers, AuditLog, Channels, TagManager, Playlists,
PlaylistsRail and SettingsPanel (Button call sites 37 -> 52). Left alone
deliberately: conditional-paint sites, video-overlay white buttons, drag
handles, chip removers and card-click targets -- the families Button's
own docstring excludes.

3x tsc, eslint, knip, prettier and 689 vitest green; visual UAT on Users,
Access requests and Playlists at 1280x900, no console errors.
2026-09-05 18:31:00 +02:00
peter 5bfcf5d83a refactor(e10): move the platform components into components/ui/ and enforce the boundary
The app HAS a platform layer -- AnchoredPopover (22 importers, with
arrows/typeahead/Escape/anchor-flip), QueryState (21), Modal (17),
PageShell, DataTable, Tabs, Tooltip and ~26 more -- but it sat flat among
~160 feature components with nothing marking it as one. Several of those
files describe themselves as 'the one place' or 'canonical' and were
still bypassed by hand-rolled markup, because a layer nobody can see is a
layer nobody uses; three separate inventory passes of this codebase
misread it the same way.

Moves the 33 domain-free modules into components/ui/, rewrites the
imports in 89 files, and makes the boundary a lint rule rather than a
convention: inside ui/ a one-level-up import (a feature component) and
any lib/api slice are errors. The rule was verified to fire on both and
to leave generic lib/ helpers alone. README.md states which side of the
line a component belongs on and why.

No behaviour change: 3x tsc, eslint, knip, prettier and 687 vitest tests
all green; no backend file touched.
2026-09-05 16:58:29 +02:00
peter 31fa554b87 release: 0.81.0 (improvement/e12-security-ops)
gate / frontend (push) Failing after 1m57s
gate / backend (push) Successful in 3m25s
2026-09-05 04:54:26 +02:00
peter 6a0efba477 chore(release): 0.81.0 — E12 Security & Ops Hardening 2026-09-05 04:54:15 +02:00
peter 9f28f46839 fix(e12): unbreak the gate, and stop printing a command that can strand the restore
Round five. 47 -> 34 -> 15 -> 7 -> 6 raw findings, and this round's sharpest was the
loop's own blind spot: **the branch did not pass the gate.** `prettier --check` is one of
its eight lanes and I had been running the other seven, so an unformatted `onError` from
round one sat on the branch for four rounds — `siftlode publish` would have refused it,
with no --skip flag to get past.

**The restart hint could still strand the restored data.** Round four said the scripts no
longer print a command they cannot resolve; they did. The COMPOSE_FILE fallback only fires
when `docker-compose.yml` is ABSENT, so in a source checkout it never fires and the hint
expanded to a bare, pasteable `docker compose up -d` — which reads the other file, whose
volumes are `pgdata`/`downloads` rather than `siftlode_pgdata`/`siftlode_downloads`, and
recreates the database on an empty one. When the file is unresolved the scripts now print
`-f <the compose file you started with>`: not runnable, which is the point.

**A comment naming the wrong owner.** `formats.py` still said a zero budget is "a refusal
for enqueue to make" — the premise round one disproved, three files away from the two
docstrings that now say the opposite. Anyone acting on it would delete the worker's refusal
and re-open the original inversion, so it now says where the refusal actually lives.

Also: the reset-flag rule was rewritten in three consecutive rounds and pinned by nothing —
two tests now cover a fragment losing its token, both by a pasted verify link and by
navigating away entirely; `backup.ps1` deletes a partially-copied dump before throwing, the
guard round four gave the docs but not the script it mirrors; and one comment that read as
if the asset were untouched by a line four below it.

Gate green, all eight lanes: ruff, ruff format, 3x tsc, eslint, prettier, knip,
vitest 687, pytest 950.
2026-09-05 04:41:21 +02:00
peter 14f1c97c0a fix(e12): a bare up -d picks the wrong compose file, and recreates the database
Round four, on round three's diff. 47 -> 34 -> 15 -> 7 raw findings.

**The printed restart command could strand the restored data.** Round three replaced
`start` with `up -d` because `start` ignores a newly pinned tag — but printed it bare. In
the git-clone layout BOTH compose files exist, so the scripts' COMPOSE_FILE fallback never
fires and a bare `docker compose up -d` reads `docker-compose.yml`: different volume names,
locally built images, no IMAGE_TAG at all. `stop` and `exec` had been fine there (they
resolve running containers by project and service), but `up -d` RECREATES — so the operator
pastes the hint straight after a successful restore and gets `db` on a new empty volume and
a first-run setup wizard, with their data in an orphaned volume. Verified: the two files
resolve `siftlode_pgdata` vs `pgdata` and different images.

The scripts no longer print a command that recreates from a file they did not resolve. They
name the compose file when they resolved one, and otherwise say to start it the way the
operator normally does — with the caveats on their own lines rather than in a parenthetical
at the end of a very long one.

**The reset flag is now decided where the live hash is already parsed.** Round three cleared
it in the token effect, which only runs when a NEW verify token arrives — so the back
button, or re-pasting an already-consumed verify link, still left the banner promising a
recovery whose token was gone. Setting it from the hash on every fragment change is both
smaller and closes those.

Also: the doc said `start` "still carries the old tag" four lines after "pin the older
IMAGE_TAG", where "old" then reads as the tag being rolled back TO — it now reuses the
scripts' unambiguous wording; the Windows recipe removes a partially-copied dump before
throwing, so a mid-stream `cp` failure cannot leave a truncated file under the final name;
and `$WarningPreference` is pinned beside `$ErrorActionPreference` so an inherited
SilentlyContinue cannot drop the half-restored warning, with the comment corrected to say
what the warning stream actually is.

Gate green: vitest 685, pytest 950.
2026-09-05 04:31:13 +02:00
peter fc56f514b3 fix(e12): the restart hint pointed at the one command that undoes a rollback
Round three, on round two's diff. The trend across the loop is 47 -> 34 -> 15 raw
findings; the ops half kept producing the sharpest ones because the gate cannot run it.

**`docker compose start` ignores a newly pinned IMAGE_TAG.** Round two removed the
automatic restart because it re-ran `alembic upgrade head` — and then told the operator to
run that exact command by hand, in both scripts and the docs. `start` relaunches the
container that already exists, which still carries the tag being rolled back FROM, so
pinning 0.79.0 and following the printed instruction brings 0.80.0 back up and re-applies
its migrations. Verified against a real daemon: after changing the pinned tag, `start` kept
the old image and only `up -d` picked up the new one. Both scripts and the doc now say
`up -d`, with the pin-first caveat where the operator reads it.

**The reset flag outlived its token.** Reverting round one's "each link clears the other"
went one step too far: a `#verify=` link pasted over a standing reset notice replaces the
whole fragment, so the reset token is gone — but `resetPending` stayed true, and being
checked first it hid the confirmation while promising a recovery that signing out could no
longer complete. The flag is now cleared when the fragment it describes is gone, which is
narrower than what was reverted and does not blank the signed-out card.

**A test that could not fail.** The shared-asset regression test seeded the asset as
`pending`, and `_reset_asset_pending` only acts on `downloading` — so the release half was a
no-op and deleting it kept the suite green. Seeded as the worker actually leaves it now.

Also: the malformed-URL guard answers with urlguard's structured detail rather than a bare
English string, so both halves of one typo class get the same localized copy, and the third
site in that module handing user input to `urlparse` was swept with it; the asset is
released after the job is terminal, not before; the Stats 403 toast gets the wizard handle
its own comment claimed it had; the pasted Windows recipe sets `$ErrorActionPreference`
inside the block (a scriptblock inherits the caller's, so a `SilentlyContinue` profile
swallowed every throw) and cleans up the container-side dump on the failure path too; the
restore warning moved to the warning stream so a `> log` redirect cannot file it away; and
two comments narrowed to what the code does.

Gate green: vitest 685, pytest 950.
2026-09-05 04:18:33 +02:00
peter 358d722ce9 fix(e12): close what round one broke, and revert two fixes that overreached
Round two of the review loop, on round one's own diff. Two of its findings were
regressions the fixes themselves introduced, which is exactly what the loop is for:

**A per-user refusal was writing onto a shared row.** MediaAssets are deduped across
users, and the generic failure handler puts the message on the asset — right for an
upstream failure, wrong for "your storage is full". Two users queuing the same video share
one asset, so the first user hitting their cap failed the second user's job with the first
one's private reason, telling someone with an empty library to free up space. The refusal
now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to
`pending` for whoever else holds it. Pinned by a test that drives `_download` directly —
the refusal happens before extraction, so nothing needs stubbing.

**The restore scripts restarted the app, undoing the restore.** `docker compose start`
re-runs the container command, and that runs `alembic upgrade head` — so on the rollback
path this documents as recommended (restore the dump, THEN pin the older tag) the newer
image immediately re-applied the migrations the restore had just undone, and on a FAILED
restore it put the app back on a half-restored schema. Both are the opposite of why the
stop was added. Nothing is restarted automatically now; the scripts say what to run and
when, and the docs' snippets say the same.

**Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders
that failure inline, next to a Retry button — three recoveries for one failure), and
`useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out
card's only banner, where nothing renders the reset flag to replace it. The two states are
independent again; what changes is that `dismiss` peels ONE layer, which is what the
original finding was actually about.

Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403
detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded
alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed`
since "bad port" misdescribes it; three docstrings narrowed to what the code actually does
(the worker's byte check is per-job, not an aggregate ledger, and not on the edit path;
the redirect residual leaks a reachability oracle, which the note had denied; the
invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw`
can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were
missing for the round-one fixes — the account panel's error paths, the invalidation set,
and the shared-asset case above.

Gate green: vitest 685, pytest 950.
2026-09-05 04:02:57 +02:00
peter fe38dfbcca fix(e12): close the eighteen findings the E12 review round left standing
The deep review of the branch raised 47 findings, 18 of which survived adversarial
verification. Grouped by what they actually were:

**The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user
already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one
user over quota was the only one downloading with no ceiling at all. `check_enqueue`
could not catch it: it ran while the job was queued and the user was still under the cap.
The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision
lives, and where it can be tested), and the worker now fails the job outright rather than
proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help
until space is freed.

**Silent failures in the new account panel.** All three actions went through `quiet: true`
endpoints, which suppress the global error dialog on the promise that the caller reports
inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On
the broken-SMTP instance this panel exists for, an admin would read silence as "the mail
went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`,
shared with the pending-access queue, so the two copies of these mutations cannot drift
apart again — which is how half of this was lost in the first place.

**A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps
back one migration, but a release routinely ships several (0.80.0 shipped four): it drops
the newest one's columns and leaves the schema at a revision the older image also cannot
resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is
now the recommended path, and the alembic route asks the target version for its own head
revision instead of counting backwards.

Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they
share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as
a 500; the pull cooldown extended to the sibling route spending the same QuotaAction;
restore stops the app before dropping its tables; the PowerShell cleanup can no longer
decide the script's verdict on 5.1, nor report the wrong exit code; the documented
one-liners hardened to match the scripts; the registry's retention window stated where
pinning is advertised; the two banners made mutually exclusive so neither can shadow the
other; `hover:bg-hover` (a utility this project never generated) replaced by the
`selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to
the two callers that discarded it.

Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset
banner UAT'd on localdev.
2026-09-05 03:41:58 +02:00
peter f98b060e2b feat(admin): an account panel, and a set-password link that needs no email
Design option C, chosen by the user 2026-09-05. Nothing in the admin UI
ever showed an account's GATE STATE: an admin could see a "Pending
approval" chip but not whether the address was confirmed, whether SEC-10
had cleared the password, or which Google account was attached — so
every support conversation started by guessing. Clicking a row now opens
a drawer that says which gate the person is actually stuck behind, with
the recovery actions under that readout rather than in the row.

The drawer is `Modal placement="drawer"`, not a second dialog: the layer
stack, ESC, the focus trap, Back-to-close and the scroll lock are the
hard parts, and duplicating them is how two shells drift apart. It
slides in, and honours prefers-reduced-motion like the rest.

The link itself (SEC-15, SEC-16, and the everyday "no SMTP and someone
forgot theirs") reuses the existing token machinery and the SAME
`#reset=` entry the mailed link uses, so the signed-out card consumes it
with no new flow. It is a distinct token KIND for one reason: a mailed
reset arrives in the mailbox, so completing it proves ownership and
confirms the address; an admin-issued link proves nothing about the
mailbox — the admin hands it over by whatever means they choose — so it
must not satisfy the email gate. Claiming otherwise would be the same
false attestation SEC-15 is about.

Issuing does not revoke the current password: a misclick must not lock
anyone out. USING it rewrites the hash and bumps the session epoch,
which is what evicts a squatter — SEC-16's actual remedy. Spending
either kind burns the siblings of both, so a completed reset cannot
leave a live admin link behind. The non-consuming probe learned the new
kind too, or every admin link would have opened onto "this link is
spent". The audit log records that a link was issued, by whom, for whom
— never the link, which is a live credential-setting secret and the last
thing that belongs in a long-lived log a human reads.

Not gated on account state on purpose: it is useful for a pre-empted
address, a revoked password, a Google-only account that wants one, and a
plain forgotten password. Narrowing it would only make it refuse exactly
when the admin needs it. The shared demo account is the one exception.

Verified on localdev, both languages, end to end: issue from the panel
-> the signed-in tab shows the honest "reset links open in a signed-out
window" banner with the token still in the address bar -> signing out
draws the "Set a new password" form, which is only possible because the
probe recognises the new kind. Gate: pytest 925 -> 942, vitest 676.
2026-09-05 01:46:58 +02:00
peter bc9e34d0ae fix(auth): close the three leftovers E9.5 logged but kept out of its diff
DX-17, three one-liners the epic's canonical sweep did not reach:

`routes/setup.py::setup_admin` still resolved the wizard's admin by
literal `User.email ==`. The wizard can be re-run, and E9.5's premise is
that two spellings of one Gmail mailbox are one person — so a second run
under the other spelling would have created exactly the duplicate admin
row the epic exists to prevent.

`routes/admin.py::_decide` hand-rolled `db.get` + 404 where the file's
newer endpoints use `get_or_404`. There were TWO such sites, not the one
the review named; both are converted, per the house rule that a
principle gets fixed everywhere in one sweep.

`useVerifyEntry`'s `hashchange` listener recognised `#verify=` but not
`#reset=`, so pasting a reset link into a tab already running the app
was silently ignored — the same silence that made the verify half a bug.
It now notices, and says out loud that reset links need a signed-out
window; the fragment is still deliberately left in the address bar
(review round 4), so signing out from that tab recovers it.

DX-16: `.catch(() => setOutcome("invalid"))` could not tell a spent
token from a 502 mid-deploy or a dropped connection, and by then the
fragment is cleared — so a network blip was reported as "that link is no
longer valid", pushing the user to request a fresh one on the strength
of a false statement. Only a 400 carrying {code:"verify"} is the
server's verdict now; everything else gets its own outcome, with no
resend CTA, because the token is untouched and retrying it works. The
rule was already written one file away for the reset-link probe: "a
network blip must never be cached as 'this link is dead'".

Two existing tests asserted the old conflation by rejecting with a bare
Error("400"); they now reject with the real HttpError. The errors are
built inside the rejection, not hoisted: mountAt calls vi.resetModules(),
so an HttpError imported earlier is a different class identity and
`instanceof` would quietly miss.

Gate: vitest 671 -> 676, pytest 924 -> 925.
2026-09-05 01:35:09 +02:00
peter 0cc450d1c9 fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches
it from the WORKER, so any approved user could aim the server at the compose
network, the LAN or a cloud metadata endpoint — and read the result off
job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a
port scanner. downloads/urlguard.py refuses a URL whose host resolves to a
loopback/private/link-local/reserved address, at enqueue AND again in the
worker (DNS can move in between, and the worker is where the socket opens).
A network-class failure on a `url` job now reports one fixed sentence instead
of the raw transport text; the error CODE is still classified from the
original, so the localized explanation is unchanged. DNS rebinding is
explicitly NOT closed — that needs the connection gated inside yt-dlp — and
the module says so rather than implying more than it does.

SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video,
so a URL that is purely a playlist or channel still expanded into every
entry — one enqueue, unbounded downloads. `playlist_items` pins it to one.
Size was unbounded too: `check_enqueue` compares the CURRENT footprint against
the cap, and a download's size is unknown until it lands, so one job could
overshoot by any amount; the owner's remaining budget is now handed to yt-dlp
as `max_filesize`, applied AFTER the adapter merge so no source adapter can
raise a limit the quota layer set. A join's cut-list is capped at 100: an
accurate join emits two filters per segment into one ffmpeg filter_complex.

SEC-14: the YouTube budget is instance-wide, so one user holding down a
manual sync spent the day for everyone. Per-user (not per-IP — the cost
belongs to the account) 3-per-10-minutes limiters on the two bulk pulls,
subscriptions and playlists, with separate budgets so one does not close the
other. The review also listed /rss, /backfill and /enrich; E3 deleted them,
and search stays unlimited on purpose — it is interactive, not a bulk pull.

Gate: pytest 896 -> 924 (none skipped), vitest 671.
2026-09-05 01:03:24 +02:00
peter 688efb9ed6 feat(selfhost): license the repo, and make Plex actually usable on the public compose files
DX-10: the repo invited people to clone, build and self-host while carrying
no LICENSE at all — which legally means all rights reserved, so none of that
was permitted. Adds the verbatim GNU AGPL-3.0 text, the SPDX id in
package.json and a README section. AGPL rather than MIT deliberately: this is
a server application, so the case worth covering is someone running a
modified copy as a network service.

DX-12: .env.example documented PLEX_MEDIA_HOST_PATH -> /plex-media and the
README advertised the Plex library, but only the localdev and home compose
files carried the mount — on both PUBLIC files the module could read Plex's
catalogue and then fail to open a single file, because it streams the media
itself rather than proxying Plex. Both api and worker now mount it read-only
with an empty named volume as the fallback, and self-hosting.md gains the
section that explains the server URL/token, the path map and the mount
together (it had never mentioned Plex).

DX-15: the worker had no init: true on either public file, though localdev
and home both carry tini with the reason written out — without PID 1 reaping,
a watchdog hard-kill of a download's process group orphans its ffmpeg/yt-dlp
grandchildren.

Verified: 'compose config' resolves on both files, the Plex mount comes out
read_only: true on api and worker, and init lands on worker (not just the
bgutil-pot sidecar). Gate green: vitest 671, pytest 896.
2026-09-05 00:53:26 +02:00
peter ca4e559cc7 fix(ops): make backup/restore work, and give upgrades a pin and a rollback
DX-13: both shell scripts ran bare 'docker compose', which has no config
file to find in the layout docs/self-hosting.md tells people to use (only
docker-compose.selfhost.yml is downloaded) — they now fall back to it, while
a source checkout keeps resolving docker-compose.yml as before. The two
PowerShell scripts moved the dump through the PS pipeline, which corrupts
binary on 5.1 and cannot express -Encoding Byte at all on 7 ('Byte' is not a
supported encoding name), so they simply failed there; they now write inside
the container and 'docker compose cp' it across. backup.sh also wrote its
output straight into backups/, leaving a truncated file that looks like a
real backup when pg_dump fails — it writes a .part and promotes on success.
Both restores report a non-zero pg_restore rather than printing 'Restored'
over it. Docs now state that a backup is the dump PLUS .env: without
TOKEN_ENCRYPTION_KEY every encrypted secret in the restored database is
unreadable, which the old one-line 'back up the volume' never said.

DX-14: 'latest' moves with every release and migrations run on every boot,
so self-hosters had no way to choose when they move and no way back. Docs
gain IMAGE_TAG pinning, a rollback recipe (image-only vs schema-bearing) and
the migration-failed-at-boot case; CHANGELOG.md is generated from
releaseNotes.ts so the file cannot drift from what the app shows, and
'siftlode release' refuses on a stale one rather than regenerating it into
the clean tree it just demanded.

Verified: dump + restore round-trip on both platforms against localdev into
a scratch database (8 users, 262822 videos, schema 0082_invite_lang), and
IMAGE_TAG=0.80.0 resolves on both services in 'compose config'.
2026-09-05 00:50:07 +02:00
peter e23128b4a1 release: 0.80.0 (improvement/e9.5-onboarding-access)
gate / frontend (push) Failing after 2m26s
gate / backend (push) Successful in 3m42s
2026-09-04 23:28:30 +02:00
peter 0d5f8f0708 chore(release): 0.80.0 — E9.5 Onboarding & Access flow
Release notes for the epic: bilingual mail, one-mail-one-click approval, order-independent gates
with named states, canonical Gmail identity, the admin lifeline for sign-ups stuck outside, and the
four-round closing review that converged 12 -> 4 -> 3 -> 2.
2026-09-04 23:28:23 +02:00
peter a03bf8d371 fix(auth): stop erasing a reset link the signed-in tree never reads (E9.5 review round 4)
Round 4 came back converged — nothing above low, and both findings in one family. This is the half
worth fixing, because the branch introduced it.

Only the signed-out auth card consumes a `#reset=` token, and S2 moved the address-bar cleanup up to
a hook App mounts unconditionally. So opening a password-reset link in a browser that already holds a
session erased the token from the URL while nothing had read it: the page loads, nothing is said, and
clicking the link again does exactly the same, because the token is untouched and the session is
still there. That is the dead end this epic exists to remove, and `main` did not have it — the strip
used to live inside the card, which a signed-in visitor never renders.

An unspent reset fragment now stays, which restores the recovery that shape always had: signing out
reloads with the fragment intact and the form is waiting. Once the card has spent the token a later
reload meets S4's named dead-link screen, which is true. The affirmative version — a banner telling a
signed-in visitor that reset links open in a signed-out window — is logged with DX-15, along with the
`hashchange` half that still only recognises `#verify=`.
2026-09-04 23:14:46 +02:00
peter 3dc14402b0 test(auth): pin what rounds 1 and 2 left unpinned (E9.5 review round 3)
Three low findings, all in what the review rounds themselves wrote.

The comment on `test_approving_an_unverified_account_keeps_its_password` still asserted the argument
rounds 1 and 2 refuted — that dropping the approval-time revoke was safe because `password_login`
reads `email_verified` first — while the docstrings on the two functions it describes now say the
opposite. Two contradictory "why" records for one rule is how the round-1 takeover primitive got
reasoned into existence in the first place, so the test comment now carries the corrected account
and names SEC-16.

`test_a_later_registration_leaves_a_link_already_in_flight_alive` asserted a subset relation that is
vacuously true on an empty set, and the guard that made it non-empty had gone out with the revert.
It now pins both halves: the old link survives, and exactly one new one appears.

Round 2's own fix — activation moved to after a successful send — was unpinned; putting it back
above the send left the suite green. The failed-invitation test now also asserts the account was not
activated by a request the UI reported as a failure.
2026-09-04 23:04:09 +02:00
peter 540ca244c8 revert(auth): the newest registration does not own an unverified account (E9.5 review round 2)
Round 1 fixed the address pre-emption chain by letting a later registration replace the password on
an unverified account and delete its outstanding verify tokens. Round 2 read that code and showed it
was worse than what it fixed, so it goes back out.

It closes nothing. The rule is symmetric and an attacker can simply register last — on a timer,
inside the five-per-five-minutes budget — while the owner registers once. What it does add is two
primitives an unauthenticated caller did not have: overwrite a real registrant's chosen password,
and delete the confirmation link already sitting in their inbox (the approval mail's 7-day token is
an ordinary verify token, so the blanket delete took that too). Worst of all it fired on the rows
SEC-10 left stripped — password_hash NULL, password_revoked_at set, active — installing an
attacker's password where the entire state means "there is none", and clearing the stamp that is the
only reason password-reset still accepts them. That is a straight takeover which did not exist
before, in the code written to prevent one.

The owner's recourse needed no new primitive and already worked: password-reset accepts a
pre-empted account, proves the mailbox, rewrites the hash and bumps the session epoch. What remains
is the bounded window between a confirmation click and that reset, which is SEC-16 — logged with
this attempt written down, so the next person does not reach for it again.

Two smaller round-2 findings go with it. `upsert_pending_invite`'s new exact-spelling tie-break ran
before any status preference, so it could deterministically pick a DENIED sibling over a pending one
and reopen it — the very outcome its comment claimed to prevent; the status preference now comes
first. And `send_invitation` activated the account before the send it might have to report as
failed, so a request the UI called a failure had quietly let someone in; activation now follows a
mail that actually left. The verify token minted to render that mail still commits either way — it
is single-use, expires on its own and was mailed nowhere, and the note says so rather than leaving
the next reader to wonder whether it was missed.
2026-09-04 22:54:52 +02:00
peter 563fa0513d fix(auth): close what the epic's own review found (E9.5 review round 1)
Eleven findings survived adversarial verification; these are the fixes.

**Address pre-emption.** S1 dropped the approval-time password revoke, arguing `password_login`
refuses an unverified account before it reads `is_active`. Two verifiers showed that holds only
while the address stays unverified — and S6's approval mail exists to get the real owner to end that
state, at which point the squatter's password works. While an address is unverified the NEWEST
registration now owns it: the password is replaced and every outstanding verify token deleted, so
the owner takes the row back simply by registering, and a link already in flight cannot confirm it
for anyone else. The residual (an owner who confirms and neither registers nor resets) is logged as
SEC-16 beside SEC-15, where the re-attestation path it needs already lives. The comments that
asserted the old, false invariant now say what actually guards this.

**The entry URL.** S2 moved `clearEntryUrl()` out of the signed-out auth card into a hook App mounts
unconditionally, and it blanket-stripped the whole query string, the fragment and `history.state` on
the FIRST commit. That broke first-run setup outright — the wizard mounts only after its query
resolves and then finds no `?token=` — and silently ate `?link=`/`?oauth=` (no toast, no refetch
after connecting Google), `?v=`/`?chan=` deep links, and scroll restoration. It now clears only the
fragment and the two one-time flags it owns, and preserves `history.state`.

**The env allow-list.** S3 canonicalised the Invite branch of `is_allowed` and left the
ALLOWED_EMAILS/ADMIN_EMAILS fallback matching literally — the 2026-09-03 lockout with a different
source of truth, plus a shadowing trick where an unauthenticated request under a sibling spelling
permanently masked an env grant. Canonical on both sides now, and the same for the delete dialog's
warning and the ADMIN_EMAILS role bootstrap.

**Honest states.** `send_invitation` stamped "I already wrote to them" before a fail-soft background
send; it sends inline and records only what left. The admin-side resend answered `{ok: true}` for a
dropped mail; it now says so. Both also activate/serialize consistently with the other decide paths.

**Correctness sweep.** The 7-day approval link said "expires in 168 hours" — durations render in
days once there is more than one. `_register_account`, `upsert_pending_invite` and `add_invite`
picked among canonical siblings with no ORDER BY and no exact-spelling tie-break, so which account
got the re-send, and whether a denied row was reopened over a pending one, was up to Postgres.
`purge_user` deleted every canonically-equal invite, revoking a surviving sibling account's access.
`ConfirmProvider` wrapped its ReactNode message in a `<p>`. Approving no longer leaves the
stuck-sign-ups list stale. The refusal panels announce and take focus.

**Tests.** Three mailers rewritten this epic never rendered in any test; every `send_*` now renders
in both languages. The orphan-key guard could not fail (it scanned the catalog that defines the
keys) and had no reverse direction. Newly pinned: canonical activation, the surviving sibling's
grant, a failed invitation and resend, an expired reset link, a throttled login's silence, the
per-language admin fan-out, and that the entry URL keeps what it does not own.
2026-09-04 22:42:15 +02:00
peter 10c0ca4cf7 feat(email): write to people in their own language (E9.5 S7)
Twelve mails carried hardcoded English while the app itself has enforced HU/EN parity since E5.
A mail is the one user-facing surface nobody can re-render by switching the UI language: it arrives
once, in whatever language we picked, and stays that way. That made the epic's own headline mail —
"one mail, one click" — English for a Hungarian reader.

The text moved to a catalog (`mailtext.py`) with a `text(key, lang, **vars)` renderer, and the parity
rule is now a test rather than a habit: every key must carry both languages AND the same
`{placeholders}`. A translation that drops `{link}` sends a mail with no link in it; one that invents
a placeholder raises inside a background task nobody is watching. Both are silent in production.

Every `send_*` takes the language as its second argument, resolved by the REQUEST handler and never
inside the mail — these all run as background tasks, after the request's session is gone, which is
the same reason the approval mail computes its link in the handler.

Where the language comes from, in order: the account's own preference; failing that, the language the
person asked us in; failing that, English. The middle one is new — `invites.lang`, migration `0082` —
because the mails an access request produces go to an address that usually has no account yet, and
the signal does exist: the page their form was submitted from, or the locale Google reported for a
sign-in we refused. Captured rather than derived, for the third time in this epic (`google_email` in
S3, `preferences.language` in S6 taught the same lesson): a signal that exists only in one moment
cannot be backfilled later. So the admin is asked in exactly one place — the invitation to an address
they typed in themselves, where nobody has ever told us anything.

That send is now a dialog rather than a bare click, which is also where the "you already wrote to
them" warning belongs — it used to live only in a tooltip the admin may never hover, guarding the one
action whose failure mode is mailing a real person twice.

S6's registration `lang` field turned out to be dead on arrival: the backend accepted it, the
frontend never sent it, so every account has been storing nothing. The request builders now stamp the
UI language themselves — a signal that has to be passed in by every caller is one every caller can
forget, which is exactly how it shipped unset.

Admins are notified one mail per language group instead of one blast to everyone. They are people
with their own UI language; a single message would have to pick one of them and be wrong for the rest.

Also removes a duplicated `invited_at` column declaration on `Invite` (two identical definitions
landed in S6).
2026-09-04 20:29:33 +02:00
peter a76bab9b34 feat(auth): one mail, one click — the approval mail carries the confirmation link (E9.5 S6)
The funnel's common ordering made a person open TWO messages: one saying their access was approved,
another asking them to confirm their address. The approval mail now branches on what they actually
still have to do — carry the confirmation link, point at sign-in, or say a confirmation mail follows
once they register — so the usual case is one mail and one click. A mail that guesses wrong is worse
than a plain one: telling someone to sign in while their address is unconfirmed contradicts the
screen that will refuse them.

That embedded link lives 7 days rather than the usual 24 hours. Nobody sits waiting for an approval —
it can easily be read on Monday after a weekend, and an expired link spends the one-mail promise and
puts them back on two. The risk is small: the token only ever confirms an address; it signs nobody in
and sets no password.

The most likely root cause of the 2026-09-03 report that a confirmation mail "never arrived" is here
too, and it was never a delivery failure: re-registering an address that already had an unconfirmed
account returned in SILENCE — same cheerful "check your inbox", no mail sent at all. It now re-sends
the link. Nothing new leaks: the response and its timing are unchanged, and the public verify/resend
already does exactly this for the same input.

Whitelisting stays silent, and that is a decision, not an omission. The operator opens a door as a
possibility — usually already agreed with the person — and a row can sit approved for weeks with
nobody registering; a mail fired automatically there would be nudging someone who never asked.
Telling them is its own act on its own button, `invited_at` records it so a second click is a
decision rather than an accident, and the button only appears where nobody is behind the address yet:
"the door is open, come in" is meaningless to someone already inside.

Two things UAT caught that no test would have. Every fresh registration appeared in BOTH admin lists
at once, because the stuck-sign-ups filter keyed on "unconfirmed" — the rule is now "the address may
sign in, and this account still can't", which leaves an undecided request with the queue that owns
it. And the invitation button was offered on every approved row, half of which are existing users.

Also: the confirmation mail finally says what its link's rules are (once, 24 hours) — the reset mail
always has, while this one left the reader to discover both by meeting a dead link. The card's
"didn't get the email?" moves up under the form, and the three waiting messages name the spam folder,
which is where a first message to a fresh recipient sometimes lands.

Registration now records the language the sign-up page was in. Nothing reads it yet — the bilingual
templates are S7 — but it is the only moment that choice exists anywhere the server can see, and a
preference nobody recorded cannot be backfilled later.

UAT on localdev: re-registering re-sent the link (one token became two, one account); approval issued
a 7-day token and moved the account into the stuck list with only a re-send; whitelisting mailed
nobody; the invitation recorded itself and the row then warned that sending again would write twice.
Test data removed.
2026-09-04 04:05:28 +02:00
peter 7acdf5d577 feat(admin): surface the sign-ups that are stuck outside, and let an admin finish letting them in (E9.5 S5)
Approve an address BEFORE the person registers and their sign-up produces no queue row at all —
`_decide` only takes a still-pending invite — so the Access-requests screen sat empty, the tab badge
read zero, and the only working move was re-typing the address into the whitelist box. Nobody would
guess that, and on 2026-09-03 nobody did.

This is roadmap UX-25, where two earlier attempts were reverted after review. Both moved AUTHORITY:
auto-activating on registration turns a standing whitelist entry into a bearer grant for whoever
registers the address first, and reopening the approved invite to `pending` lets an anonymous sign-up
REVOKE a standing approval. What was actually missing was visibility, so this changes nothing about
what an unauthenticated caller can cause — the admin could already do exactly this by re-entering the
address; they just had no way to know they had to.

The section lists every account that cannot sign in yet and says WHICH gate is shut, because the two
dead ends look the same from the admin's chair: waiting for activation, or waiting on a confirmation
mail. Naming the unconfirmed ones is not decoration — on a no-SMTP instance that is the difference
between the owner and whoever registered the address first (SEC-15), and an admin about to let
someone in should see it. A person the queue above already covers is deliberately NOT listed here;
two buttons for one person is worse than one.

Activate is gated on `is_allowed` — the same function the sign-in paths consult, in core, so the
button that appears and the badge that counts cannot disagree. Without that gate it would
manufacture an inconsistent account: password sign-in only checks `is_active` and would let them in,
while their Google sign-in would be turned away and silently reopened as a fresh access request.
Letting in someone with no approved invite stays the whitelist box's job, which says so out loud.

The badge counts activations SEPARATELY from pending invites rather than widening an existing
field's meaning, and only counts what genuinely waits on the admin: an unconfirmed account waits on
its owner, so it is listed with a re-send but does not nag. The admin-side resend says plainly what
happened, including refusing when the instance has no working SMTP — mail is fail-soft everywhere
else, and that silence is exactly what left the 0.78.5 sign-ups with nothing to fall back on.

`send_verification_link` is now one function with three callers (registration, the public resend,
this one), so the link's shape and TTL cannot drift by who triggered it.

UAT on localdev, both languages: the UX-25 shape reproduced end to end (approve the address, then
register), the badge counting it, Activate letting them in and dropping the badge, the row losing
its Activate but keeping the re-send, a fresh token issued by the re-send, and both actions in the
audit log. Test data removed.
2026-09-04 03:27:48 +02:00
peter b1086e05dc fix(auth): stop drawing a reset form the link can no longer fill (E9.5 S4)
Reported from UAT: delete an account, then open its password-reset link. The card drew the full "set
a new password" form — because a `#reset=` fragment was present, and nothing more — and only after
the user had typed a password did the server answer 400. Measured first: no security hole. A user's
tokens cascade away with the row, so nothing can be set on a deleted or a re-created account. What
was left is a dead end of exactly the kind this epic exists to remove, and it is not rare: a reset
link lives one hour, mail is slow or lands in spam, and whoever got two mails clicks the older one.

password-reset/check answers whether a link is still usable WITHOUT spending it, so asking can never
cost the user their one real attempt — the property the tests pin hardest. It gets its own limiter,
because opening a link must not eat the budget for requesting one, and a throttled caller is told
"ok", never "dead": this check may only ever save someone a wasted form, so when it cannot answer it
degrades to the old behaviour instead of declaring a live link finished. The confirm endpoint stays
the authority, and its failure raises the same named screen, since the token can die between the two
calls or the probe itself may have failed.

A too-short password is deliberately NOT that screen. The length check runs before the token is
spent, so a typo costs the link nothing, and the card tells the two 400s apart by `reason` so only
one of them takes the form down.

Both email-link confirmations (verify, password-reset/confirm) now answer in the {code, reason}
shape S4 introduced. They were the last refusals a user meets by simply clicking an old link while
still being answered in hardcoded English under a bilingual UI. The seven remaining bare strings in
this module (register, request-access, the demo guards, set-password) sit behind client-side
validation the UI does first; they are logged as a follow-up rather than swept in silently here.

One bug found by the UAT itself, in the new code: the check went out FOUR times per opened link —
StrictMode double-invokes effects and the shell remounts the card as the session probe settles.
Harmless per request, but against a per-IP limit it would exhaust the budget in two reloads and then
silently fall back to drawing the form unchecked: nothing errors, the feature just stops working.
Deduped in a module-level map holding the in-flight promise (a ref dies with the remount that causes
it); a failed probe drops itself so a blip is never cached as "dead". The test renders the card
twice, because with one render the guard would rot undetected.

UAT on localdev: the reported scenario end to end (delete the account, open the link) shows the named
screen and no form; a live link still opens the form and completes; the CTA lands on the request-a-
link form; one probe per opened link. Throwaway accounts removed.
2026-09-04 02:38:01 +02:00
peter ed91c83dd7 fix(auth): name the state and the address behind a refused sign-in (E9.5 S4)
"Your account is awaiting admin approval." was true and useless. It never said WHICH mailbox the
account was on, and on 2026-09-03 that silence hid a Gmail alias mismatch for a day: a registration
as np.temu@gmail.com met a Google claim of nptemu@gmail.com, and every screen in the funnel agreed
to say nothing about it.

Every refusal password-login can answer with now uses the codebase's structured {code, reason}
detail, built in one helper. The three that arrive AFTER the password is proven carry the address on
the account plus both gate flags; the two that answer before it (invalid, rate_limited) carry
nothing, so a wrong password and an unknown account stay byte-identical. The frontend renders the
first three as a named screen in place of the form — there is nothing to retype once the password is
correct — with an icon, the two-gate checklist, the address in monospace, and a resend CTA where one
helps. Reporting each gate from its own flag is the payoff of S1: approved-but-unverified is a real
state now, and a screen that assumed a sequence would send that user off to wait for an admin who
has already decided.

The Google access-request notice names the address too, and says so when the request was already on
file under ANOTHER spelling of the same mailbox — the honest version of "we didn't open a second
one", stated only when it applies. It moved from the query string to the fragment: the notice now
carries an address, and a query string lands in every reverse-proxy access log. Same reasoning that
put the verify/reset tokens there, and the same owner already clears it. urlencode, not an f-string,
because URLSearchParams reads a bare + in a fragment as a space and would mangle exactly the
+tag addresses this epic exists to protect.

upsert_pending_invite returns the row it settled on beside the is-new flag, since the row's stored
address is what the notice has to name and what mail will go to.

Deviation from the PendingScreen mockup, deliberately: no "Sign out" button and no separate "Not
your address?" link. The mockup drew the panel in the app shell; the accepted decision (a) keeps the
pending user session-less, so there is no session to end, and the address is named directly above a
"Back to sign in" that answers the same question.

UAT on localdev, both languages: all three refusals with the typed spelling deliberately differing
from the stored one (the panel named the stored one), the resend CTA carrying the typed address into
the resend form, and both access-request banners with a +tag address surviving the fragment round
trip. Throwaway accounts and the language switch reverted.
2026-09-04 01:56:55 +02:00
peter a614ee2435 feat(auth): show the Google address beside the one the account registered with (E9.5 S3)
An account can now hold two true addresses: the one its owner typed, which a Google
sign-in no longer overwrites, and the one Google answers with. Only the first was
stored, so Settings could say 'Connected' without saying connected AS WHOM -- exactly
what someone who registered with a +tag needs to know.

users.google_email (migration 0080) records Google's answer at sign-in and at an
in-app link. /api/me exposes it ONLY when it differs from the account address, so the
comparison lives in one place and the majority, whose two addresses are the same
string, see no change at all. Rendered in the account menu as a muted third line under
the identity, and in the Settings sign-in row under the existing hint.

A short 'Google' label rather than a brand glyph: the app draws one icon set (lucide),
and a generic mail icon next to an email address says nothing. It also reads to a
screen reader, which an unlabelled logo would not.

No backfill is possible -- Google's answer is only known at sign-in -- so accounts
linked earlier stay NULL until their owner next signs in with Google.
2026-09-04 01:23:11 +02:00
peter c74487fbb3 fix(auth): keep the address the user registered with, and let it sign them in (E9.5 S3)
Two halves of one rule: the address someone typed is theirs to keep, and it must be
the one that gets them back in.

A plus-tagged or dotted Gmail is often a deliberate choice -- registering as
me+siftlode@gmail.com is how you find out later which site leaked your address. The
Google callback overwrote it with the account's primary, which can never contain a
'+', so the tag vanished on the first Google sign-in and, because sign-in matched the
stored address, took their chosen address with it. The rewrite line is older than
this epic, but S3 made it reachable here by matching adoption across spellings. It
now runs only when the mailbox GENUINELY changed, and lives in a named helper rather
than inline, so the rule has one home.

The three lookups that resolve a typed address -- password login, password reset,
verify resend -- go through find_user_by_email: exact spelling first, then the unique
canonical candidate, and None when several accounts share a mailbox rather than
guessing which one a password should open. Reset and resend previously answered a
different spelling with silence (they reply uniformly by design), which was a dead end
with no way out. Their mail now goes to the address on the account, never the one
typed into the form.
2026-09-04 01:07:54 +02:00
peter 876cece881 fix(auth): match identity on a canonical email, so one Gmail mailbox is one account (E9.5 S3)
Gmail delivers x.y@, xy@ and xy+tag@ to one inbox; the invite gate only lowercased.
On 2026-09-03 that turned an approved user's Google sign-in into a silent SECOND
access request -- OAuth returns the account's stored primary, which need not be the
spelling they typed -- and left prod with two user rows for one mailbox (ids 13, 14).
The +tag case fails every time: a Google account cannot contain +, so OAuth can only
ever answer with the base address.

Mail still goes to the address AS TYPED; only identity matching uses the canonical
form. utils.canonical_email is the single definition, and a model validator derives
users.email_canonical / invites.email_canonical on every write, so no call site can
forget it. Scope is narrow on purpose: dots are significant on most providers and the
Gmail rule does not hold for Workspace custom domains.

Swept every sibling of the same principle, not just the reported one: is_allowed,
upsert_pending_invite, the Google-callback adoption, _register_account (so no new
duplicate pair can be created), _activate_user_by_email, admin _accounts_for (an
invite backing a live user under another spelling used to look deletable) and
add_invite, plus purge_user, which now erases every canonically-equal invite instead
of leaving one behind that still grants access.

Migration 0079 backfills both columns and indexes them NON-uniquely: prod already
carries canonical duplicates (approved invite pairs since 2026-06-16), and merging
accounts is a judgement call about people's data, not something a migration does on
its own. A test pins the backfill SQL against the Python helper.
2026-09-04 00:14:12 +02:00
peter e5997ca567 fix(auth): consume a verification token that arrives without a page load (E9.5 S2)
Found in UAT. Navigating a tab that already runs the app to a URL differing only in
the fragment is a same-document navigation: nothing reloads, so the import-time entry
snapshot never sees the token. It sat unconsumed in the address bar and the previous
outcome stayed on screen -- the same silence this sprint set out to remove, wearing a
different hat. Pasting the emailed link into a running tab does exactly this, and our
verification mails were plain text until 0.78.8, which makes pasting the normal way to
use one.

A hashchange listener now picks up the new token, the guard tracks consumed tokens
rather than a single fired flag (still exactly one POST per token), and a fresh link
clears the standing outcome instead of leaving the old banner up.
2026-09-03 23:57:18 +02:00
peter c134b514c1 fix(auth): confirm an emailed verification link wherever it is opened (E9.5 S2)
The token was only ever consumed inside Welcome.tsx, which App renders ONLY when
there is no session. Opening the emailed link in a browser that already held a
Siftlode session posted nothing, the fragment was then stripped, and the user got
no signal at all -- the link simply did nothing. Incognito worked, which is why it
survived. Confirmation now happens in App, above that branch: signed out the outcome
still drives the auth card's banners, signed in it drives a top-of-app bar.

The entry URL is snapshotted once at import time (lib/entryUrl) instead of per
component, so no future URL writer can strip a token before whoever needs it has
read it -- that ordering race is what broke this in the first place. POST /auth/verify
now names the address it confirmed, which matters when the browser holds a session
for a different account; the same reasoning as password_reset_required already
covers the disclosure.

Also here, from the 2026-09-02 review tail: the dead GET /auth/verify goes (both
senders emit the fragment form, and the last query-token link expired 24h after the
0.78.7 deploy), a spent #verify=/#reset= fragment is cleared while signed in, and
the auth-card banners get role=status. The verifyOkReset copy no longer blames
admin approval, which E9.5 S1 made false.
2026-09-03 23:45:31 +02:00
peter 243869401a fix(auth): stop discarding a legitimate password on admin approval (E9.5 S1)
SEC-10 revoked any password set while an account's email was unverified, on both
the Google-adoption and the admin-approval path. On the approval path that was
redundant: password_login refuses an unverified account before it ever reads
is_active, so approval hands a pre-emptor's password nothing it could use. All the
revoke achieved there was destroying the password a legitimate registrant had just
chosen -- and since the admin normally decides before the user reads their mail,
'your password was cleared for security' was the DEFAULT experience, not an edge
case. The guard stays on the Google-adoption path, where the real owner does attest
ownership.

password_reset_required and the ok_reset banner stay: accounts stripped on prod
before this change have no other way back in.

Two existing tests pinned the old rule, so they are rewritten rather than deleted,
and the SEC-10 recovery pack now builds the revoked state through the surviving
producer.
2026-09-03 23:32:34 +02:00
peter 7339a0b84b release: 0.79.0 (improvement/e9-playlist)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m48s
2026-09-03 21:59:31 +02:00
peter c960c07a6a release: 0.79.0 notes + version bump
E9 Playlist module: total playtime in both playlist headers, the header
compacted to two rows, and the sticky column header sitting flush. Plus the
six defects the epic-end deep review confirmed.
2026-09-03 21:59:09 +02:00
peter 3cfdb36f06 test(format): pin the sub-minute truncation (E9 R2)
Round 2 mutation-checked round 1's own fix and found the one line it left
uncovered: every table row was an integer, so `Math.floor(sec)` in the new
seconds tier could be deleted or swapped for round/ceil with the suite still
green — exactly the "later cleanup removes it as redundant" failure the fix's
comment claims to prevent. Two fractional rows kill all three mutants.
2026-09-03 21:51:14 +02:00
peter ada95d04a4 fix(playlists): one rendering for a playlist total, honest below a minute (E9 R1)
Epic-end deep review, round 1. Six findings survived verification; five of them
were one defect wearing different hats.

formatRuntime floored 1-59s to "0m". Harmless while it only ever formatted a
Plex movie's length, but 12D repurposed it for an aggregate, and a total is the
only input that can land under a minute: a playlist of Shorts reported "0m"
while the rail beside it read "0:45". Fixed in the function, not behind a
`>= 60` guard at each of the seven call sites — the invariant belongs where it
cannot be forgotten. Its contract comment claimed "Plex movie/episode lengths"
and was two epics stale.

The rail rendered the same playlist total as formatDuration (1:23:45) while the
new header rendered formatRuntime (1h 23m) — two shapes of one number, side by
side on one screen. The rail now uses formatRuntime too; formatTotalHours had
already recorded that H:MM:SS is wrong for an aggregate.

The rename input kept min-w-0 without flex-1 in the compacted row, so it was
the only shrinkable child beside a shrink-0 action cluster. The Plex half of
this same epic already had the fix; this restores the symmetry.

plex.browse.items had no plural forms, so a one-item Plex playlist read
"1 items". Pre-existing, but the new header is the call site where count=1 is
ordinary. Both languages get both forms, no key drift.

formatRuntime had no unit tests at all across seven call sites. Added a tier
table plus a sweep asserting no non-zero second ever renders as "0m".

Dropped after 3/3 adversarial refutations: the meta-line segment ordering
(the filtered count is labelled, and count-then-runtime is the convention) and
the null-duration lower bound (the server's own SUM has skipped NULLs since
long before this branch; changing only the client would split them).
2026-09-03 21:43:04 +02:00
peter 1b004e6ccf feat(plex): total playtime in the playlist header (E9 S3)
Parity with the YouTube playlist header (E9 12D): the Plex playlist detail
now carries a meta line under the title — "N items · Xh Ym", summed from the
items' duration_seconds. No backend change: leaf_card already serializes
duration_seconds for both movies and episodes, and no new i18n key either —
the count reuses plex.browse.items, the same key the collection header uses.

The title/rename input moved into a min-w-0 flex-1 column so the meta line
sits under both; the line is hidden on an empty playlist.
2026-09-02 18:30:06 +02:00
peter 95c33e0b0f refactor(playlists): compact the playlist header to two rows (E9 12A)
The header portals into the shell's FIXED toolbar band, so its height is
list height the user never gets back — and it was spending four stacked
blocks on it: title, meta line, an action row, then a separate view-control
row under the cover. Option B collapses that into two tight rows beside a
smaller cover: title + rename/YouTube-link on the first, with Play-all,
push/revert/delete and the unsynced badge pushed right; the meta line
(count · playtime · filtered) and the view controls (group-by, reset view,
save order, view status, undo/redo) sharing the second.

Every control and its gating is unchanged — the view-control group still
appears only for `items.length > 1 || viewActive`, so a one-item playlist
shows just the meta line.

Measured on a 32-item playlist: the band goes 170px -> 88px at rest
(122px while a view is active, where the status sentence wraps), which is
~80px more visible list.
2026-09-02 03:45:07 +02:00
peter 4e6bcbc078 fix(playlists): stop the table header shifting up on first scroll (E9 12B)
The playlist content wrapper carried `pt-3`, so its sticky column header sat
12px below the scroller top at rest and jumped up to `top:0` on the first
scroll. Drop the top padding so the header is flush from rest and never
shifts — the fixed toolbar band above supplies the breathing room, matching
the channel table's already-correct pattern.
2026-09-02 03:39:17 +02:00
peter e17fb5bbf5 feat(playlists): total playtime in header + sticky-header fade fix (E9 S1)
Surface the playlist's total playtime beside the item count in the detail
header (12D) — sums every item's duration (formatRuntime "h m", matching the
approved mockup), hidden when nothing sums. Data is already client-side; no
backend change.

Pass fadeTop:false for the playlists page (12B) so the shell's top edge-fade
no longer sits over the already-sticky column header (tableHeader.tsx:257),
matching the channels/audit pages.
2026-09-02 03:39:17 +02:00
peter a9044ece50 release: 0.78.8 (improvement/email-html-alternative)
gate / frontend (push) Failing after 1m40s
gate / backend (push) Successful in 2m40s
2026-09-02 03:24:59 +02:00
peter 473aafd1c8 release: bump VERSION to 0.78.8 + release notes 2026-09-02 03:24:56 +02:00
peter 7019e3afdf fix(email): give every mail an HTML alternative so its link is clickable
The bodies are text/plain only, so whether the URL in a verification or
password-reset mail is a link at all was the client's decision. Gmail, Outlook and
Apple Mail linkify plain text; lighter web clients render the body verbatim and leave
the link as dead text — in a message whose entire purpose is that one link.

_send now attaches an HTML part generated from the SAME plain body, rather than
per-template HTML that would have to be kept in sync with it: one converter, all
eleven templates, and text/plain stays the primary part and the source of truth.
Escaping runs before linkification — bodies interpolate untrusted values (a
requester's address, the operator contact) — so no caller can inject markup, and a
query string's `&` is already `&amp;` when it reaches the href, which is what an HTML
attribute wants. Sentence-final punctuation is trimmed out of the href while
url-safe token characters (`-`, `_`) are left alone. Inline styles only, no images,
no external CSS, no tracking.
2026-09-02 03:13:23 +02:00
peter c81b110637 release: 0.78.7 (bug/verify-link-and-approval-lockout)
gate / frontend (push) Failing after 1m44s
gate / backend (push) Successful in 2m42s
2026-09-02 02:31:19 +02:00
peter 2ecc70ca6d release: bump VERSION to 0.78.7 + release notes 2026-09-02 02:31:14 +02:00
peter f8c63e80c2 fix(auth): widen the 0078 backfill so revoke victims who verified are rescued too
Review found the backfill's fourth predicate `AND email_verified = false` strands
exactly the subset that needs it most. The revoke does not invalidate the account's
outstanding verification token, so clicking the emailed link is the natural next
thing such a user does — and it sets email_verified=True while password_hash stays
NULL. That row then has no route back at all: reset ineligible, resend refuses an
already-verified address, set-password needs a session the account cannot obtain,
and nothing re-stamps it because the revoke itself is guarded on being unverified.

The remaining three predicates already pin the victim set exactly: registration and
the setup wizard always set a hash, every Google row carries a google_sub (adoption
assigns it before revoking), and nothing in the codebase ever clears google_sub — so
a local, non-demo, passwordless row can only have come from the revoke. Widening is
therefore safe, and it is the argument the docstring was already making while the
SQL contradicted it.

Also corrects that docstring: set-password is unreachable because the account cannot
authenticate at all, not because the epoch bump killed a session — an unverified
account is refused at login before a session is ever minted, so it never had one.
2026-09-02 02:21:46 +02:00
peter 3ac3d382e7 fix(auth): stop the address-bar mirror eating emailed auth tokens, and unlock revoked accounts
Two independent bugs made an email+password registration unrecoverable in prod.

The verification link did nothing. urlWithParams/clearParams rebuilt the URL as
pathname+query and dropped the fragment — but the one-time auth tokens ride the
fragment (#verify=/#reset=, SB3), and the providers that mirror nav params
(useUrlMirror in FeedFilters/Playlists/Plex/Channels) sit ABOVE the lazily-loaded
<App> in main.tsx. Their mount effect replaceState'd the token away before the auth
card's chunk had loaded, so the card read an empty hash: the link opened a bare
sign-in page, no request, no banner. Both helpers now carry the fragment over.

The account was then locked out for good. SEC-10 nulls password_hash when an
unverified account is attested (here: admin approval landing before the user could
verify — which the bug above made impossible). That is indistinguishable from a
Google-only account, and reset eligibility keyed off password_hash alone, so no
reset link was ever issued: no password to sign in with, and no way to get one.
users.password_revoked_at records the revocation, reset eligibility honours it, and
setting a password again (reset-confirm or set-password) clears it. An account that
never had a password stays ineligible on purpose — its owner adds one from Settings.

/auth/verify now also answers password_reset_required so the card can say why the
registered password stopped working, instead of leaving the user at the uniform
"Invalid email or password". Telling the holder of a valid verify token leaks
nothing: they have already proved they own the mailbox.

The 0078 migration backfills the stamp for rows carrying the exact lockout signature
(local, non-demo, no password, unverified), so existing victims recover unaided.
2026-09-02 01:28:45 +02:00
peter 94c62ca699 release: 0.78.6 (feat/admin-invite-delete-email-health)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m53s
2026-09-02 00:36:52 +02:00
peter 1fe24d374a docs(tests): correct the deny-vs-delete rationale in the invites test file
Round 4's only finding: the file header and one comment still said Deny stops a
refused person re-asking. It does not — upsert_pending_invite reopens a denied
row as pending — and the delete_invite docstring in the same branch already says
so. The real difference is that Deny records an auditable decision and mails the
requester; Delete forgets the address.
2026-09-02 00:15:21 +02:00
peter 2cba6b9fca docs: round-3 copy corrections, and pin the two behaviours nothing was guarding
Round 3's correctness/security pass came back clean; what was left said more than
the code does.

- emailOff promised the access-request list is where sign-ups show up. A sign-up
  on an address that is already approved or already pending adds nothing to it
  (roadmap UX-25), and "only" was falsified by this release's own inbox notice.
- deleteConfirm claimed the admin was dropping "an approval you already gave" —
  it is the confirmation for pending and denied rows too.
- sendFailed blamed the mail server for rejecting a message; the reason covers
  connect, DNS, TLS and login failures alike, so it now describes the symptom and
  leaves the diagnosis to the log.
- ADMIN_EMAILS re-promotion happens in the Google callback only, not on any
  sign-in.

Tests: the decided-list copy of the delete affordance was never rendered (both
new cases fail if that branch is dropped), and the throttle release — a
deliberate behaviour the notes advertise — was asserted by nothing; a drop with
no admin yet, or with a failing write, must not silence the following hour.
2026-09-02 00:06:01 +02:00
peter 8d7aaad9a8 fix: warn before the delete, not after — plus round-2 copy and a11y corrections
Round 2 on the feature diff (15 → 12):

- still_allowed_by_env now rides the invite LIST as well as the delete response,
  so the confirmation warns while the admin can still change their mind. It
  mattered most for a DENIED row on an env-allow-listed address: deleting it
  silently converted a refusal into a grant, and the only warning arrived after
  the irreversible click.
- Both inbox bodies claimed to name "the most recent" dropped message. The
  throttle records the FIRST of each hour and never updates it, so they now say
  "one of them", and the not-configured body finally names where to fix it.
- settings.invites.emailOff told the admin they "get no notice when someone
  signs up" — rendered directly above the queue that shows exactly that. Only
  the email is lost, and it now says so.
- The disabled delete buttons put their reason out of reach: `disabled` drops a
  control from the tab order and suppresses its title. Switched to
  aria-disabled + preventDefault, the pattern Playlists.tsx already established
  for the same reason.
- deleteInvite is quiet, so the 409 the UI handles with a translated toast no
  longer also raises the server's English detail as a modal.
- After a completed password reset, Back landed on the reset form again with a
  consumed token — the history change made that state reachable. The token is
  cleared on success so the reset view stops being the card's root.
- The hu deleteHasAccount named a tab that renders under a different label.

Tests: a dom test for the Access-requests panel covering both has_account
variants and all three delete outcomes, /api/me flag cases, an ADMIN_EMAILS
case, and the env-allowlist test now patches the raw setting so the parsing
stays under test.
2026-09-01 23:50:29 +02:00
peter 85043013f0 fix: address the feature review — accurate copy, reachable controls, real coverage
Round 1 on the feature diff found 15; the substantive ones:

- The delete control VANISHED on any invite whose address had registered, which
  is the ordinary sign-up shape (register writes the User and the pending invite
  together). The guard stays broad — that row is the account's approval gate, and
  orphaning it strands someone who can then neither sign in nor be approved — but
  the control is now a disabled button carrying the reason, on both lists, so an
  admin never sees a row silently missing a button its neighbours have.
- Deleting an entry is not always a revoke: is_allowed falls back to the env
  ALLOWED_EMAILS / ADMIN_EMAILS lists once no row exists, and an ADMIN_EMAILS
  address is force-promoted on its next Google sign-in. The route now reports
  still_allowed_by_env and the UI says so instead of claiming access was removed.
- "Deny keeps the row so they can't immediately re-ask" was simply false —
  upsert_pending_invite reopens a denied row. Corrected everywhere it was stated.
- The Configuration warning reads me.email_configured, which save() did not
  invalidate, so it kept warning about fields the admin had just filled in.
- The read-only variant was an aria-hidden span explained only by a Tooltip that
  the hints toggle can switch off; now a disabled button with title + aria-label.
- The delete error path threw away the 409's reason (the stale-list case) and
  left the stale row on screen.
- The throttle window was spent even when no notice was written, silencing the
  next hour on the path whose job is to stop silent drops.
- email_enabled() opened a second session and decrypted the SMTP password on
  every /api/me poll; it now takes the caller's session.
- welcomeMode.dom.test.ts never rendered the card, so reverting the history
  wiring to useState left it green. A render test now drives the real links and
  fails on that revert (verified).

Plus /api/me flag tests, the registration-shaped pending case, the env-allowlist
case, and a release note that matches what the code does.
2026-09-01 23:31:49 +02:00
peter 44f626f732 feat(admin): delete a whitelist entry, and stop hiding a broken mail setup
Three operator gaps, all found the same way — a person could not sign up and the
app showed nobody anything.

- DELETE /api/admin/invites/{id} with a delete control on both the pending
  requests and the previously-decided list. Deny keeps the row so the person
  cannot immediately re-ask; delete forgets the address. Refused with 409 while
  a User row exists for it, because is_allowed reads that row and removing it
  would revoke a live user's access with no other sign — the list ships
  has_account so the UI withholds the button instead of offering one that fails.
- Email is fail-soft by design, which had come to mean invisible. _send() now
  records an email_undelivered notification for every admin when a message is
  skipped or rejected, naming the message that was lost, throttled to one an
  hour. /api/me carries email_configured, driving a standing warning on
  Configuration → Email / SMTP and above the access-request list.
- The sign-in card's Create account / Forgot password / Didn't get the email
  views are history entries (useHistorySubview), so Back returns to the previous
  view instead of only moving the address bar.

Also: the per-test rate-limiter reset sweeps every loaded app.* module rather
than a hand-kept list of three — this change added a limiter outside it.
2026-09-01 23:09:25 +02:00
peter 24c93cbb83 docs: name the Configuration tab as it is rendered, drop an unused fixture
Round 6's two remaining nits. The operator note said "Configuration → Email"
but the group renders as "Email / SMTP" in both locales, and
test_an_admin_can_activate_a_pre_approved_registrant requested make_user without
using it — a leftover from the reverted rounds that made the test look like it
seeded an admin when it did not.
2026-09-01 22:15:56 +02:00
peter cb1f4f00b0 docs(auth): correct the last round of over-claiming comments and notes
Round 5's correctness/security pass came back clean; what was left was text that
said more than the code does.

- The register-path notify comment (and its test) claimed the admin "is still
  told". The call fires, but on a no-SMTP instance _send drops it — which is the
  open half of the problem, not something this branch fixed. Both now say so.
- The release notes tell admins what to actually do: an account whose address was
  whitelisted in advance has no pending request to approve, so the whitelist box
  is what activates it. That instruction only existed in the roadmap and in
  chores, which the dialog does not render.
- "That one line is the whole production change" now accounts for the test-only
  RateLimiter.reset() that also ships.
- The rate-limiter test claimed to pin that blocked attempts are not counted, on
  an assertion that held either way. Replaced with one that drives the clock
  across the window boundary and fails if hammering extends it.
2026-09-01 22:02:43 +02:00
peter f6e1d1e69f fix(auth): keep the admin notice unconditional on the register path
Round 4 showed the `inv is not None` gate added in b5a1002 removed a signal that
matters: for an address the admin had already approved, that email is their only
notice that someone has now created an account on it — the invite is not pending
so nothing lands in the Access-requests queue either. Suppressing it left the
registrant at "awaiting admin approval" with the admin unaware they existed. The
duplicate it avoided (Request access, then sign up) is mild noise by comparison.

The production change is now one db.commit(), which is all the reported bug ever
needed. Release notes corrected again: a previously-denied address was also
unaffected, the pending-request case never lost its queue entry (only the
account behind it), and the test inventory was missing a guard.
2026-09-01 21:48:40 +02:00
peter b64c3f514d fix(auth): drop the invite reopen — keep the fix to the lost account alone
Round 3 of the review killed the reopen from b5a1002: flipping an approved
invite back to pending let an unauthenticated POST /auth/register REVOKE a
standing approval for any whitelisted-but-unclaimed address, which then fails
is_allowed() and locks the real owner out of Google sign-in — the primary way
people get in here. It also re-armed the pre-emption window it was meant to
close, because with no SMTP email_verified is set True and the SEC-10 guard the
design leaned on never fires.

Two attempts at fixing the stranded-account UX in the same change both came back
worse than the wart. What remains is the narrow, real bug: the account was
silently rolled back and is now committed. An admin still binds the approval to
the account, exactly as before this branch. UX-25 in the roadmap records the
follow-up and both dead ends so they are not retried.

Also from round 3: the upsert docstring no longer claims the admin UI cannot
reach these accounts (re-entering the address in the same panel's whitelist box
does, and a test now pins it), the release notes are scoped to what the diff
actually changes with the operator note moved where the dialog renders it, and
RateLimiter has unit tests so reset() is not production code only a fixture
calls.
2026-09-01 21:34:38 +02:00
peter b5a1002412 fix(auth): reopen a pre-dated approval instead of auto-granting it
Replaces the activation added in ea931dc, which round 2 of the review showed was
a real weakening: on a no-SMTP instance it made a standing whitelist entry a
bearer grant — the first anonymous party to register an approved-but-unclaimed
address got an immediately usable account under that identity, with no
attestation, no admin action and no signal anywhere. The verify-time activation
had the same shape one step removed, and neither site ran the SEC-10 preset-
password guard or left an audit trail.

Keep a human in the loop instead. An approval granted before any account existed
was a decision about the ADDRESS, so registration reopens the invite
(upsert_pending_invite gains revive_approved, used only by the register worker).
The request lands in the Access-requests queue, approving it runs
_activate_user_by_email — and with it revoke_preset_password_if_unverified — and
the account activates. One mechanism, already audited, no new grant path, and
the "awaiting admin approval" copy is honest again.

The explicit commit from the previous commit still matters: with an invite
already pending, upsert returns without committing and _issue_token is gated on
SMTP, so a no-SMTP registration was still being rolled back.

Also from round 2: gate the admin notice on a fresh decision (matching the two
other upsert callers) so a retry no longer re-pings every admin, and reset the
module-level rate limiters between tests — the suite pooled every login into one
300s window and was two logins short of failing an unrelated test with a 429.
2026-09-01 21:18:14 +02:00
peter ea931dca30 fix(auth): honour an approval granted before sign-up instead of parking the account
Follow-up to the commit fix, from the deep-review. Persisting the account was
only half the story: when the admin had approved the address BEFORE the person
registered, the account landed is_active=False against an invite that is no
longer pending — absent from the Access-requests queue, and approve_invite 409s
on an already-decided invite. The sign-up worked and still could not sign in.

Derive activation from is_allowed() at the moment ownership is attested:
- verify_email_confirm activates a pre-approved account when the emailed link is
  confirmed (only the mailbox owner can follow it), so SEC-10 stays intact;
- _register_account activates only when there is no SMTP to verify with, where
  the admin whitelist is by design the sole gate — the same one add_invite
  already activates on.

Tests: the pre-approved no-SMTP case and the verify-activates case are real
regression tests (both fail without this change); guards cover the unknown-email
and denied->pending branches, verification of an unapproved account, and the
first end-to-end coverage of the /auth/register route and its
allow_registration gate. Release notes rewritten — the previous entry miscounted
its regression tests and advertised an email fix this release has no code for.
2026-09-01 20:56:43 +02:00
peter d954e71a77 fix(auth): commit the account on registration so a pre-approved sign-up isn't lost
_register_account() flushed the new User but relied on someone else to commit:
upsert_pending_invite() commits only when it actually writes an invite row, and
_issue_token() runs only when SMTP is enabled. For an email an admin had ALREADY
approved, on an instance where email is unavailable, neither fired — the session
closed and the account was rolled back. The would-be user got the uniform "ok"
and no account, could not sign in, and re-registering was a silent no-op.

Commit explicitly after the invite upsert (a no-op when upsert already committed
the same transaction). Three DB-lane regression tests cover the pre-approved
no-SMTP path (fails without the fix), the unknown-email path that always worked,
and the pre-approved path with SMTP on.
2026-09-01 16:25:05 +02:00
peter 602f62967a release: 0.78.4 (bug/plex-transcode-hls-startposition)
gate / frontend (push) Failing after 1m55s
gate / backend (push) Successful in 2m57s
2026-08-30 20:33:15 +02:00
peter ba00b50dea release: 0.78.4 — fix Plex transcode playback stall/rebuffer
hls.js startPosition:0 so the transcode/remux HLS session starts VOD-style
at the requested position instead of chasing the live edge of the growing
EVENT playlist.
2026-08-30 20:32:56 +02:00
peter 92adb593c8 fix(plex): start hls.js at position 0 for the transcode/remux HLS session
Our HLS is a growing EVENT playlist with no ENDLIST, which hls.js reads as
LIVE. Its default startPosition (-1) then starts at the live edge and hugs it
(no forward buffer → rebuffer spinners), and on a resume/seek session the
segments' absolute PTS (-copyts, offset by the server's -ss) don't line up
with the live-edge timeline → 0 fragments load → permanent stall. The
seek-restart model makes fragment 0 the requested content position, so start
VOD-style at 0 and buffer forward. Fixes the transcode playback stall/rebuffer
found in 0.78.2 UAT.
2026-08-30 20:09:00 +02:00
peter 66132b6dda release: 0.78.3 (feature/plex-playback-class-filter)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m35s
2026-08-30 19:09:22 +02:00
peter d09f021f0c release: 0.78.3 — filter the Plex feed by playback class 2026-08-30 19:09:08 +02:00
peter d7e9240e54 merge main (0.78.2) into playback-class before release
# Conflicts:
#	backend/app/plex/stream.py
2026-08-30 19:08:08 +02:00
peter f2fa49b778 release: 0.78.2 (feature/plex-hdr-hw-tonemap)
gate / frontend (push) Failing after 1m37s
gate / backend (push) Successful in 2m24s
2026-08-30 18:56:32 +02:00
peter d4ed45b2eb release: 0.78.2 — HW HDR→SDR tonemap for 10-bit/HDR Plex transcode 2026-08-30 18:56:18 +02:00
peter 375f6e58be merge main (0.78.1) into tonemap before release 2026-08-30 18:55:14 +02:00
peter 1a102ce81a release: 0.78.1 (bug/download-mediakind-probe)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m22s
2026-08-30 18:47:05 +02:00
peter bfd45a155e release: 0.78.1 — download media_kind from a file probe (BUG 2) 2026-08-30 18:46:33 +02:00
peter 1102ad6a28 fix(downloads): make the probe verdict authoritative over yt-dlp info by probe-ran, not per-field
Deep-review round 1 on bug/download-mediakind-probe found a real HIGH the prior
review missed:

- resolve_media_facts used a per-field `probe or info` fallback, so a probe that
  RAN and deliberately found no video track (cover-art-only audio: attached_pic
  skipped; a still image: format-guarded) had its None video overridden by
  yt-dlp's stale vcodec. An audio-mode download from a progressive source (info
  carries the source's video codec, embed-thumbnail on) was then classified VIDEO
  -> a Play button + a broken in-app <video> with no video track. Only the
  download path regressed (upload calls probe_media directly).

  Fix: gate precedence on whether ffprobe RAN, not per-field. If the probe read
  the file, its codec verdict wins outright (a None means 'no such track'); only a
  full ffprobe failure falls back to info. Dimensions still gap-fill from info.
  Refactored the parsing into _facts_from_probe so resolve_media_facts can see the
  raw probe-ran signal; probe_media keeps its 5-tuple (upload + TestProbeClassifier
  unchanged).

- tests: the cover-art-audio and still-image resurrection cases, a differing
  populated-info precedence case (the prior-flagged gap), and the ffprobe-failed
  info fallback — now stubbing run_ffprobe_json so the real parsing + precedence
  run together.

Backend gate green (ruff + full pytest).
2026-08-30 18:09:26 +02:00
peter c1a3b7b935 fix(plex): gate the part-key re-probe to single-version items; drop dead i18n key; cover cross-facet narrowing
Deep-review round 2 on the round-1 fixes (all low):

- part_key churn (state): round 1's part-key reset re-probed EVERY sync for a
  multi-version item (an original + a Plex 'Optimized' copy), because
  _media_facts reads media[0]/part[0] whose order Plex doesn't guarantee stable,
  oscillating transcode_hw between the two versions. Gate the part-key signal on
  a single-version item (len(Media)<=1); codec-change reset still applies to all.
  A multi-version same-codec re-encode not self-healing is an acceptable rare
  edge; a per-sync re-probe loop is not.

- dead i18n key: this diff's playableClass() collapses to transcode_hw/sw, so the
  bare plex.playable.transcode key it orphaned is now unused (knip can't see
  dynamic keys). Dropped from both locales, parity kept.

- test: cross-facet narrowing by an active playable class (a co-facet's year
  range collapses to the selected subset) — the flip side of self-exclusion,
  previously unpinned.

The aria-label+title double-announcement on the card dot is a deliberate accepted
trade-off (reliable SR name + mouse tooltip). Backend + FE gates green.
2026-08-30 17:58:38 +02:00
peter bf98b48517 fix(plex): re-probe on file replace, HU parity, public class const, playback-class test gaps
Deep-review round 1 on feature/plex-playback-class-filter:

- Sticky watermark (medium): _apply_item reset the bit-depth probe watermark
  only on a codec-string change, so a same-codec re-encode (8-bit->10-bit HEVC)
  kept a stale video_bit_depth/transcode_hw forever (the backfill only re-probes
  media_probed_at IS NULL rows). Also reset when the Plex Part key changes (the
  file was replaced), not just the codec.

- i18n parity (medium): the HU playback-class labels were mixed-language (only
  'direct' translated). Translated remux/transcode_hw/transcode_sw and the
  tooltips consistently to Hungarian (Kozvetlen streameles / Atkodolas).

- Layering (low): _PLAYBACK_CLASSES was module-private yet imported across into
  routes.plex.browse; renamed to public PLAYBACK_CLASSES (it is the shared
  source of truth for facet order + zero-fill).

- a11y (low): the browse-card playback-class dot conveyed meaning only via a
  mouse-hover title; added role=img + aria-label so SR/touch users get the class.

- tests: the NULL/unprobed transcode_hw -> SW fold (filter + facet), facet
  self-exclusion under an active playable filter, the migration codec-set ==
  HW_DECODABLE_CODECS drift guard, the probe-batch bound + oldest-first order,
  the transient probe-failure retry path, and the same-codec re-encode re-probe.

Backend gate green (ruff + full pytest); FE gate green (tsc/lint/knip/prettier/
vitest). NOTE: the facet-vs-runtime disagreement for 10-bit HEVC (C1) is the
known merge-order constraint — this branch's runtime is codec-only; the facet
becomes truthful once BUG1/tonemap's bit-depth-aware routing lands with it. Left
as a sequencing constraint, not code-changed here.
2026-08-30 17:51:33 +02:00
peter b563155a37 docs(plex): correct the cull-yield fallback comment to name the real route 404s
Deep-review round 3: the only surviving finding — a comment-drift nit. The
fallback's cull-yield comment named the route's 'Not available yet' path, but a
culled-key segment request actually returns 'No active playback session' (the
substantive invariant — never a None that 404s as file-not-found — is unchanged).
Reworded to match the route. Concurrency + correctness came back clean.
2026-08-30 15:18:26 +02:00
peter 915319fa35 fix(plex): make the HW->SW fallback supersede-check atomic; honest cull yield
Deep-review round 2 on the round-1 fix:

- Race (medium): the round-1 identity guard did the supersede check and the
  respawn in TWO separate _lock sections. A newer same-key session installed in
  the gap was still torn down by _spawn's unconditional pop+kill (the round-1
  comment even had the direction backwards). Fold the check into _spawn via a
  new replace_only= param: the identity test and the pop+replace now share ONE
  lock, so the respawn is a true no-op when superseded/culled.

- Honest states (low): the superseded branch returned current_session(key),
  which is None when the key was cap-culled during our out-of-lock wait; the
  route turns None into a dishonest 'Media file not found on disk' 404 for a
  file that exists. Yield 'current_session(key) or s' so a culled session
  degrades to the route's retryable 'not available yet' path instead.

- Hygiene: _want_hw docstring corrected (it is a test-only shim now; production
  decides via _select_encode) and the two stale _want_hw references refreshed
  (stream.py header, config.py plex_transcode_hw).

- tests: the cap-cull no-respawn/no-None sub-case; the _opencl_present except
  OSError branch (glob raising, not just empty); tightened a tautological -vf
  assertion. Backend gate green (ruff + full pytest).
2026-08-30 15:11:37 +02:00
peter a2a8f5681d fix(plex): don't tonemap 10-bit SDR; identity-guard the HW→SW fallback
Deep-review round 1 on feature/plex-hdr-hw-tonemap:

- F1 (correctness): _select_encode gated tonemap on bit-depth alone, so a
  10-bit *SDR* (bt709) source got the HDR hable tone curve applied, visibly
  altering an SDR picture where only a 10->8-bit downconvert was needed. Split
  the ffprobe gate into a 3-way classifier (_probe_video_kind: hwsafe / hdr /
  sdr10); only true-HDR (PQ/HLG) takes the OpenCL tonemap graph, 10-bit SDR
  drops to software libx264 (an exact downconvert). Applies to both auto and on.

- F2 (race): the runtime HW->SW fallback popped+killed _sessions[key] with no
  identity check. Because wait_for blocks OUTSIDE _lock, a concurrent same-key
  _spawn (rapid seek-restart) or _enforce_cap cull could kill this session; the
  fallback then respawned software over the NEWER session and breached the
  transcode cap. Guard the respawn: only fall back if _sessions[key] is still
  this session, else yield the currently-registered one.

- tests: 3-way classifier (incl. HLG + empty-probe), real _opencl_present glob,
  real wait_for proc fast-fail branch, tonemap-variant fallback, the F1
  regressions (auto + on), the supersede/no-respawn race, and the on/opencl
  matrix cell. Backend gate green (ruff + full pytest).
2026-08-30 15:02:41 +02:00
peter 4827aebb7f feat(plex): HW HDR→SDR tonemap for 10-bit/HDR transcode (OpenCL on the iGPU)
Upgrade mode-3's 10-bit/HDR branch from "route to software libx264" (BUG 1's
behaviour) to a full-GPU pipeline: VAAPI decode → OpenCL tonemap (hable,
HDR10/HLG → bt709 SDR nv12) → h264_vaapi. Gen 9.5 (UHD 630) can't tonemap in
the VAAPI VPP (tonemap_vaapi fails), so OpenCL is the only HW route — proven on
the prod iGPU at ~4.2x realtime, ~1 CPU core.

- stream.py: `_select_encode` returns (hw, tonemap); a 10-bit/HDR HW-decodable
  source picks the tonemap graph when the OpenCL runtime is present, else drops
  to software. `_opencl_present` gates on a registered ICD. `_venc_opts` /
  `_hwaccel_in_opts` gain the tonemap variant (named vaapi+opencl devices,
  `-filter_hw_device ocl`, hwmap→tonemap_opencl→hwmap reverse). `_want_hw` kept
  as the hw-half wrapper. The runtime HW→SW fallback still backstops a session
  that emits no segment.
- Dockerfile: add the Intel NEO OpenCL runtime (intel-opencl-icd). It was removed
  from Debian trixie, so it's pulled from bookworm (22.43 still supports Gen 9.5;
  the .deb runs on the trixie base — verified live). Harmless without /dev/dri.
- route + FE: thread `tonemap` through the session response; the stats overlay
  mode row reads "transcode · GPU · HDR→SDR" for the tonemap path.
- tests: tonemap command branch, the (hw, tonemap) selection matrix (auto/on ×
  8-bit/10-bit × opencl), start_session tonemap wiring, and the fmtMode tag.
2026-08-30 06:02:45 +02:00
peter 840526b1b8 feat(plex): filter the feed by playback class (Direct Play / Direct Stream / Transcode HW/SW)
Adds a movie-scoped "Playback" filter to the Plex feed so a user can isolate items by how
they play — the groundwork for collecting HW-transcodable (or no-transcode) versions.

Four classes derived from the stored `playable` + a new `transcode_hw` facet:
- Direct Play (direct), Direct Stream (remux)
- Transcode · HW  — playable=transcode AND the iGPU can hardware-transcode it
- Transcode · SW  — playable=transcode, software libx264 only

The HW/SW split is precise, not codec-only: a 10-bit/HDR HEVC is HW-decodable yet breaks the
VAAPI nv12 pipeline (the same case plex/stream special-cases), so it must count as SW.

Backend:
- New PlexItem columns (migration 0077): transcode_hw, video_bit_depth, media_probed_at.
  Migration backfills transcode_hw codec-only so the filter works immediately.
- `plex.sync.transcode_hw_capable` is the single classifier; the HW-decodable codec set moves
  to sync as `HW_DECODABLE_CODECS` and plex.stream imports it, so the catalog facet and the
  runtime encoder choice can't disagree by codec.
- Bit-depth backfill (`_probe_media_batch`, wired into sync): probes only HW-decodable transcode
  items' video Stream for bit-depth + HDR and refines a 10-bit/HDR HEVC to software. Bounded
  per run, watermarked by media_probed_at. A codec change resets the watermark (stale probe).
- Filter + self-excluded facet on a shared `playback_class_expr` CASE (movie-leg only — a show
  is an aggregate with no single class). transcode_hw threaded onto the feed/detail/card JSON.

Frontend:
- Multi-select "Playback" Combobox in the sidebar (movie scope, facet-empty drop-out), active
  chips, 4-way corner-dot badge (green/blue/amber/red) + tooltip, and an info-page playback chip
  with a codec · bit-depth subtitle. New `bg-info` (blue) token, i18n HU/EN parity.
- Fix a regression where the card's playback dot lost its hover tooltip: the full-card reveal
  overlay swallowed the cursor; the dot now sits above it (z-20).

Gate: backend full lane + frontend typecheck/lint/knip/prettier/vitest all green. Browser-UAT'd
on localdev (filter narrows to the SW set, facet counts, badge colour + tooltip, info chip).
2026-08-30 04:45:11 +02:00
peter 10d69cb316 fix(plex): route 10-bit/HDR HEVC to software + runtime VAAPI→libx264 fallback
A 10-bit/HDR HEVC source (e.g. an HDR10 x265 file) died at scale_vaapi=format=nv12
on the UHD 630 (Gen 9.5 VPP: "VAProfile not supported"); ffmpeg produced 0 segments
and the item never played, with no fallback — a retry just re-picked the failing HW
path. An 8-bit HEVC and the same file on libx264 both work.

Two defenses:
- `_want_hw` auto mode now also gates on an 8-bit-SDR source via a new
  `_probe_video_hwsafe` ffprobe (bit depth from pix_fmt/bits_per_raw_sample, HDR from
  color_transfer) — a 10-bit/HDR source takes the software libx264 path, which
  downconverts to yuv420p cleanly. The Plex mirror stores no bit-depth signal, so this
  is probed at session start. "on" still trusts the admin and skips the gate.
- `start_session` adds a runtime HW→SW fallback: a HW session that emits no first
  segment respawns on libx264 and reports hw=False (the route/stats overlay honestly
  show CPU). `wait_for` gained an optional `proc` so a dead ffmpeg fails fast instead
  of burning the 25s budget before the fallback fires.

Tests: TestProbeVideoHwsafe (parsing + probe-failure-assumes-safe), the 10-bit auto
gate + "on" skip in TestWantHw, and the no-segment HW→SW respawn in
TestTranscodeStartSession. Backend lane green.
2026-08-30 03:53:31 +02:00
peter a86131e062 fix(downloads): classify media_kind from a file probe, not yt-dlp info
A downloaded video was silently demoted to a download-only "file" (no
in-app play/edit) whenever yt-dlp's info dict omitted vcodec/acodec — which
it does for Facebook and other sources. Finalize derived media_kind from
those nulls (`derive(None, None)` -> file), so a genuine H.264/AAC MP4
became non-playable in the library.

Fix: probe the finished file on disk (the ground truth) at finalize and
derive from that, falling back to the info dict only for a field the probe
couldn't read. This is the same ffprobe the upload ingest already used, now
extracted to a shared downloads/probe.py so the download worker and the
upload path share one classifier (and can't drift). Also backfills the
stored vcodec/acodec/width/height from the real file.

Tests: TestResolveMediaFacts pins probe-wins-over-info (incl. the Facebook
no-codec case and yt-dlp's literal "none"); upload's probe tests repoint to
the shared module.
2026-08-30 03:32:14 +02:00
peter a84f2db310 release: 0.78.0 (feature/e6-plex-player-polish)
gate / frontend (push) Failing after 1m35s
gate / backend (push) Successful in 2m20s
2026-08-30 02:26:25 +02:00
peter 872d91f540 release: 0.78.0 — Plex transcode playback + player polish (E6) 2026-08-30 02:25:20 +02:00
peter 4eaa68ceeb feat(plex): S1a — add intel-media-va-driver for VAAPI transcode (E6)
The runtime image ships ffmpeg with h264_vaapi/libva2 but no VA driver, so
h264_vaapi can't initialise and a Plex transcode silently falls back to
libx264. Add the iHD driver (intel-media-va-driver) so the UHD 630 iGPU can
hardware-encode once /dev/dri is passed into LXC 118 (host + compose side,
tracked in siftlode-ops). Harmless where no render node is present.
2026-08-30 02:14:06 +02:00
peter 00539a4d97 fix(plex): deep-review follow-ups on E6 transcode + player polish
Eight verified findings from the sprint deep-review:

- Transcode cap: _enforce_cap now counts only transcode sessions, so a
  live remux (stream-copy) stream can no longer be LRU-evicted mid-play —
  matching the config comment + the HU/EN Config hint that promise remux
  is uncapped.
- Player Back: the browser/mouse Back interceptor (BackClose) now also
  mounts for the stats overlay and up-next card, so Back closes them first
  instead of ejecting the whole player (keyboard/Esc already did).
- Honest cold-start: loadSession clears sessionIsTranscode up front, so a
  transcode->direct/remux switch no longer flashes a stale "Preparing
  stream..." during the session request.
- Stats overlay a11y: the close (X) buttons in the stats, help and info
  overlays now read as "Close"/"Bezaras" (new plex.player.close key), not
  "Back"; dropped aria-live from the per-second stat grid so a screen
  reader is not flooded with telemetry.

Tests: route-level transcode routing (transcode item -> HLS re-encode
session, remux stays copy, direct opens none); start_session threading the
preset/crf/max_height knobs + hw/preexec into the ffmpeg argv; UpNextCard
countdown auto-advance, Play-now, Cancel and the S?-E? heading fallback.

Also brings UpNextCard.tsx / plexStats.ts / plexStats.test.ts to prettier.
2026-08-30 02:04:17 +02:00
peter dc24d353e0 feat(plex): honest transcode cold-start + encoder in stats (E6 S1d)
Surface the transcode session's runtime facts in the player. The session
response's transcode/hw flags now thread through: the stats overlay's mode row
reads "transcode · GPU" (VAAPI) or "transcode · CPU" (libx264) — the at-a-glance
confirmation of which S1 path engaged — and the initial-load overlay says
"Preparing stream…" instead of "Loading…" for a transcode session, since its
first segment is genuinely slower than a remux and shouldn't read as a hang.

fmtMode (pure, unit-tested) appends the encoder only for a transcode with a known
encoder. i18n HU/EN for the new preparing label. Browser-UAT'd on the XviD
fixture: mode row reads "transcode · CPU" live (software, no render node here).
2026-08-30 01:30:53 +02:00
peter ce96032edb feat(plex): transcode resource discipline + admin knobs (E6 S1c)
The software-fallback re-encode now runs deprioritized and admin-configurable.
The SW transcode ffmpeg child is niced (preexec_fn → os.nice, applied per-child
because the Popen lives in the long-lived API process, which must NOT itself be
niced — unlike the download worker's whole-process _deprioritize) so a CPU-bound
re-encode can't starve request handling or playback; the VAAPI path needs none of
it.

New sysconfig knobs (plex group, rendered on the admin Configuration page, i18n
HU/EN): plex_transcode_hw (auto/on/off, layered onto _want_hw), plex_transcode_
preset, plex_transcode_crf, plex_transcode_max_height (software-only downscale-to
cap; the GPU path encodes at source resolution). plex_max_transcodes stays the
concurrency cap (comment corrected — the fallback is shipped, not "later").

pytest extends the command-builder + _want_hw coverage (preset/crf/max_height
plumbing, the on/off/auto policy). Browser-UAT'd: max_height=240 drops the live
resolution to 330x240; the SW ffmpeg child runs at nice 10 (ppid=api process).
2026-08-30 01:22:30 +02:00
peter 0658f9550b feat(plex): transcode playback mode — VAAPI/libx264 re-encode (E6 S1b)
Non-h264 items (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) now play by
re-encoding the video to H.264 inside the existing HLS session, instead of the
old HTTP 501. VAAPI hardware encode (h264_vaapi, full GPU decode→scale→encode
pipeline) where the render node is present and the source is HW-decodable;
libx264 software fallback otherwise (yuv420p 8-bit, -threads reserve-one-core).

The re-encode mints a keyframe at every segment boundary (-force_key_frames) so
the HLS muxer still cuts independent segments and seeking works; everything else
— -copyts, the HLS tail, multi-audio renditions, the AAC audio path — is the
same machinery as remux, so hls.js/subtitles/seeking are unchanged. The session
route returns transcode/hw so the player can surface the active mode.

pytest pins both command branches (software + VAAPI shape) and the HW-selection
gate (render node present AND source HW-decodable). Browser-UAT'd on the XviD
fixture (Winning London): 501→plays, real 528×384 H.264 frames, 0 dropped, seek
lands clean; stats overlay Mode row reads "transcode".
2026-08-30 01:06:08 +02:00
peter f162788135 feat(plex): up-next end-card at episode end (E6 S4)
At episode end, when a next item exists and the pref is on, show an up-next
card (poster + title + ~10s countdown + Cancel) instead of the old instant jump,
so binge stays in the viewer's control (D6-A). Countdown or Play now advances;
Cancel/✕ dismisses and stays on the finished frame. Lives only on the
play-to-the-end path — credits-auto-skip advances earlier and never reaches the
card, so the two reconcile by construction (no double-advance). New pref
'upNext' (default on) with a Playback-tab toggle; the card closes on item change
and joins the Back/Escape cascade. i18n HU/EN.
2026-08-30 00:28:50 +02:00
peter 968674bc6a feat(plex): stats-for-nerds debug overlay (E6 S2)
Add a toggleable stats overlay to the Plex player: active mode
(direct/remux/transcode), transport, resolution, codec, bitrate, HLS level,
dropped frames, buffer-ahead, session offset and position — sampled once a
second off the live videoRef + hlsRef (no data layer). Non-modal corner panel,
aria-live polite, dismissable; toggled with 'd', a gauge control-bar button, or
its X. Number-crunching lives in a pure, unit-tested plexStats module. Doubles
as the S1 transcode-verification surface. i18n HU/EN.
2026-08-30 00:17:34 +02:00
peter 90ae9f6e88 feat(plex): on-screen ±10s skip buttons flanking play/pause (E6 S3)
Add discoverable rewind-10 / forward-10 controls to the Plex player control
bar, flanking play/pause (D5-A). New Skip10 glyph: an outline circular arrow
with a centred '10', deliberately distinct from the episode-level
SkipBack/SkipForward and from Restart's plain RotateCcw. Reuses seekTo; mirrors
the keyboard ←/→ (seekStep=10). i18n HU/EN aria-labels.
2026-08-30 00:09:36 +02:00
peter 3682e6a01f release: 0.76.0 (feature/e7-downloads-media-library)
gate / frontend (push) Failing after 1m36s
gate / backend (push) Successful in 2m26s
2026-08-29 21:54:19 +02:00
peter aca355f43b chore(release): 0.76.0 — E7 Downloads & Media Library
Bundles the whole E7 epic (library metadata, crop fix, share-by-default, in-app player,
browser-playable guarantee, local upload) plus the deep-review hardening. Bump VERSION and
add the 0.76.0 release-notes entry (RELEASE_NOTES[0] must match VERSION).
2026-08-29 21:52:21 +02:00
peter f41c81eb56 fix(downloads): media-library UI review fixes (E7)
Deep-review remediation on the frontend, all findings adversarially verified:

- LibraryPlayer picks audio vs video from the canonical media_kind (single source), not a
  hardcoded container whitelist, so an uploaded audio file with an uncommon/absent container
  renders in <audio> instead of a broken video box (legacy null media_kind keeps the heuristic).
- UploadDialog aborts the in-flight upload when the dialog is closed by X/ESC/backdrop/Back
  (not only the footer Cancel); drag highlight uses an enter/leave depth counter to stop the
  flicker; the progress bar gets a stable role=progressbar aria-label with the value on aria-valuenow.
- DownloadCenter's Shared tab renders honest loading/error states instead of a blank panel, and
  the library/shared row-action gating moves to a single tested helper (lib/downloadActions).

Tests: downloadActions gating matrix, the uploadReq XHR path (abort/401/422-localize/onerror),
and the media_kind-driven audio/video branch.

i18n: downloads.upload.uploadProgress (HU + EN).
2026-08-29 21:52:13 +02:00
peter 614c5528ad fix(downloads): harden upload serving + ingest against stored-XSS and leaks (E7 review)
Deep-review remediation for the E7 upload path, all findings adversarially verified:

- Serve every library file with a Content-Type chosen from the ffprobe-verified media_kind
  + a strict container allow-list (storage.served_media_type), never guessed from the
  user-controlled file extension. Closes a stored-XSS vector where a media/HTML polyglot
  uploaded as "evil.html" was served inline as text/html on the login-free /watch surface.
  Applied to both the authenticated /file and the public watch_file endpoints.
- Classify uploaded still images (jpg/png/gif/webp/…) as media_kind='file' so they are
  force-downloaded and don't get a broken in-app video player.
- Roll back the on-disk side of an upload when the ingest transaction fails after place_file
  (unlink the placed file + its deterministic poster path), so a failed commit can't orphan
  bytes in the GC-skipped .uploads tree.
- Floor download_upload_max_bytes at min=1 so a stored 0 can't silently disable the per-file cap.
- Dedupe the ffprobe-JSON invocation into edit.run_ffprobe_json (shared by probe_codecs and
  upload._probe) and the upload extension into storage._clean_upload_ext; drop the now-unused
  parse_probe_streams.

Tests: public + authenticated force-download and the polyglot-not-served-as-HTML guard, the
real _probe classifier (image/cover-art/dims/duration), the ingest rollback/cleanup path, the
max_jobs branch, the cap-floor invariant, and run_ffprobe_json error handling.
2026-08-29 21:52:02 +02:00
peter 104bd1e3a6 feat(downloads): public generic-file watch surface (E7 S5d)
The /watch/{token} SPA gets a third branch: media_kind==='file' renders an
info card (name / TYPE · size) + a Download button, no player. The <video>/
<audio> branches now guard on media_kind!=='file' so legacy links
(media_kind null) still play. Download is gated on the link's allow_download,
mirroring the media branch; the backend serves a file force-download
(attachment + octet-stream) regardless, so the gate only reflects owner ACL.

FE-only (public_meta already emits media_kind/size_bytes/container since S5a).
i18n HU/EN/DE 'file' fallback label. Two DOM tests: file -> info card +
Download, no player; view-only file -> no Download.
2026-08-29 19:51:58 +02:00
peter 584e9ced0c feat(downloads): local-upload UI — drag-drop dialog + progress (E7 S5c)
UploadDialog (drag-drop + native picker + optional display name) opened
from a new Upload button in the queue add-row. Uploads stream through a
new XHR helper (uploadReq in api/core) so the dialog shows real 0-100%
progress with a cancel — fetch/req can't report upload progress. The
upload lands as an already-done job; the S5b file-kind row gates it to
download+share. i18n HU/EN; upload cap/empty errors localize via the
shared structured-detail path.
2026-08-29 18:53:53 +02:00
peter b4ad59274b feat(downloads): gate library row actions by media_kind (E7 S5b)
Surface the S5a media_kind discriminator in the library UI. A generic
uploaded file (media_kind='file') is download+share only — no player and
no timeline to trim, so Play/Edit/Edit-details are media-only; it shows a
neutral file glyph instead of an (always-empty) thumbnail and no duration
chip (container/size still read). The "Shared with me" tab's file row is
gated the same way. Re-download is now hidden for edit clips and uploads
too (neither has a re-fetchable source — the backend already rejects both
with 409), removing a latent click-to-409 on edit rows.

FE-only: the serializer already emits media_kind (S5a). Adds media_kind to
the DownloadAsset type and "upload" to the job_kind union; DOM tests cover
the file-row presentation (glyph, no img, no duration).
2026-08-29 18:38:21 +02:00
peter fe0b34cd00 feat(downloads): local file upload ingest — media library backend (E7 S5a)
Add the non-yt-dlp ingest path so a user-uploaded file joins the same download
library and inherits its function set. Any file type is accepted: media
(video/audio) gets the full set (metadata, share, play/edit), a generic file
(pdf/zip/docx/…) is download-only.

- MediaAsset.media_kind ("video"|"audio"|"file") — the stored discriminator that
  gates library actions. Set at every finalize (download probe, editor carries the
  source's kind, upload probe); consolidates the ad-hoc audio-detect that lived only
  in links.py into one invariant (downloads.mediakind.derive). Migration 0076 adds
  the column + backfills existing rows.
- downloads/upload.py: ingest_upload probes the file (ffprobe → codecs/dims/duration),
  places it in a per-user .uploads tree, and creates an already-ready asset + done job
  (the same cache-hit shortcut enqueue uses). Uploads never expire (irreplaceable).
- POST /api/downloads/upload: streaming multipart with the per-file cap
  (download_upload_max_bytes, default 500 MiB) enforced on actual bytes read; quota
  charges the known size up front (quota.check_upload).
- Security: a generic file is always served force-download (attachment +
  application/octet-stream) on both the in-app and public endpoints, so uploaded
  HTML/SVG can't execute inline. Edit + redownload reject non-media / upload sources.
- python-multipart added (first multipart endpoint in the app).
2026-08-29 08:02:11 +02:00
peter 9eaa01617c feat(downloads): in-app player for library items (E7 S4b)
Play a completed library item in-app: a Play button on Library rows (and
Shared rows with download access) opens a modal that streams the local file
into a native <video controls> (or <audio> for audio-only assets).

Rather than the YouTube IFrame PlayerModal or the Plex HLS PlexPlayer — both
bound to their own domain APIs and unable to point at a local file — this
reuses the shared Modal a11y shell (focus-trap, ESC, layer-stack, portal) and
the range-aware /api/downloads/{id}/file endpoint. S4a's universal-codec
guarantee (mp4/H.264/AAC + m4a/mp3) makes native playback always work, so seek,
volume, fullscreen and speed come for free with zero transcode.
2026-08-29 07:18:58 +02:00
peter fdca12968b feat(downloads): one-click Share + downloadable-by-default links (E7 S3)
D2-A: the library row's Share button becomes a single action — quickShareLink()
ensures a public link (minting on demand), copies it to the clipboard, and toasts
with an "Open in new tab" action. A copy failure still toasts (warning) with that
action so the link is never a dead end. Advanced controls (password/expiry/revoke/
user-ACL) stay in the existing ShareDialog, reached via a new "Manage sharing" row
button.

D1 (forward-only): CreateLinkIn.allow_download defaults to true, so a newly minted
link is downloadable by default. The default lives in the API, not a migration —
the download_links column default stays false, so no existing link flips. The
ShareDialog create-default is flipped to match.

Tests: TestCreateLinkDefault (default true; explicit false honored); 4 vitest cases
for quickShareLink (mint-vs-reuse, copy ok/fail, create failure). HU/EN parity.
2026-08-29 06:36:07 +02:00
peter 94fcc40bde feat(downloads): guarantee every download is browser-playable (mp4/H.264/AAC)
The Download module's purpose is download -> share -> play in-browser, so the
output must play in EVERY browser (iOS/Safari/WebKit included). The only
universally-safe combo is mp4 + H.264 + AAC/MP3.

- formats.normalize() now CLAMPS every spec to that profile (container->mp4,
  vcodec->None, audio_format opus->m4a) at the single point all paths flow
  through (picker, stored profile, raw API) — not just by hiding UI options. A
  source exposing no H.264 is still caught by the worker's browser-safe
  re-encode, so playability is guaranteed end-to-end.
- build_ydl_opts drops the now-unreachable non-compat branches (+ _VCODEC_SORT).
- _SIG_VERSION 3->4 so assets cached under an old non-compat spec — or a stale
  non-H.264 mp4 that slipped the v3 net (the HEVC Facebook dls) — re-derive.
- ProfileEditor drops the container/codec pickers and Opus; audio is M4A/MP3.
- i18n: remove the now-unused container/vcodec/any keys (HU+EN parity).
2026-08-29 06:08:21 +02:00
peter fb2c843fa9 fix(downloads): anchor crop overlay to the letterboxed picture (E7 S2)
The video preview is object-fit: contain (the browser default for <video>), so a
non-16:9 source is letter/pillarboxed inside the element box. The crop marquee was
drawn in fractions of the whole box (bars included) and those fractions were then
multiplied by the source resolution, so the emitted ffmpeg crop diverged from what
the selector showed. Anchor the overlay to the computed content rect (and set
object-contain explicitly) so a marquee fraction is a fraction of the real picture;
extract the display-px<->source-px math into cropGeometry.ts with unit tests, and
clamp the even-rounded window in-bounds so a full-frame crop can't overflow.
2026-08-29 05:43:41 +02:00
peter bb6cb6e335 style(downloads): bump library-row meta text + action buttons a step (E7 S1)
Enlarge every non-title text on the card (channel, status/format/size/date meta,
source-ref line, progress readout) one step (xs→sm) plus their inline icons, and
grow the row action buttons (padding + icon) via the actions wrapper so the shared
IconBtn stays smaller in the modals. Buttons keep the title's line — the wrapper is
shrink-0/no-wrap, so the text column shrinks instead.
2026-08-29 05:29:19 +02:00
peter bdbff89507 feat(downloads): show container/format chip in library rows (E7 S1)
The completed download rows already surfaced size + duration; add the concrete
container (MP4/M4A/WebM…) as a compact chip so the library reads as a media list
with a visible format — the audio-vs-video distinction the running-only phase pill
can't show at rest. Pure display of the already-serialized asset.container; locked
by a small DOM test (present when ready, absent before the worker fills it).
2026-08-29 05:21:54 +02:00
peter 9b87838e3a release: 0.75.0 (improvement/quickfix-batch-2)
gate / frontend (push) Failing after 1m34s
gate / backend (push) Successful in 2m16s
2026-08-29 03:55:15 +02:00
peter f014416e31 chore(release): 0.75.0 — quickfix batch #2 (Plex/downloads/playlists) 2026-08-29 03:54:08 +02:00
peter 3369587412 fix(review): resolve deep-review findings across quickfix batch #2
- Button: `loading` always disables (|| not ??) — stops a double-submit
  when a call-site passes an explicit boolean `disabled` (VideoEditor).
- playlists: track un-ingestable-but-live YouTube ids in the sync baseline
  so a linked playlist containing a private/deleted video converges instead
  of re-classifying it as a pending pull forever; re-attempt ingestion each
  sync so a restored (private->public) video heals, and prune ids that
  leave YouTube.
- SettingsPanel: download-auth section renders an honest error/loading
  state (StateMessage + retry) instead of a bare empty section on a failed
  credentials fetch; HU+EN loadError string.
- tests: reorder-order, partial-failure baseline hold, un-ingestable
  converge/heal/prune, Button loading-disables, download-auth error/retry.
2026-08-29 03:52:13 +02:00
peter 9af4e8f4ff feat(playlists): sentence-style two-way sync preview (only changed sides)
The confirm dialog spelled out both sides even when one was "+0 / −0", which read
as engineer shorthand. Build the message from one sentence per side that actually
changes and add a short "stays unchanged" line for the untouched side; the zero
side is no longer shown as a bare +0/−0 to decode. Assembled in the component
from granular i18n clauses (add/remove per side, reorder, rename) with count
plurals; HU+EN. Replaces the single pushPlanMerge template.
2026-08-28 05:20:34 +02:00
peter a0b725d779 feat(playlists): two-way (3-way) merge YouTube sync + mandatory preview (S7)
"Sync to YouTube" was a one-directional destructive push (local wins, deletes
from YouTube every item not in the local list), so items added on YouTube were
lost and a dirty pull was skipped entirely. Replace it with a real 3-way merge:
each item is classified against the last-synced baseline as a local-add, a
YouTube-add, or a genuine delete, so an add or delete on either side survives.

- Persist the ordered baseline id set (new playlists.synced_video_ids, migration
  0075); the fingerprint hash alone couldn't drive a merge. NULL baseline (fresh
  export / pre-S7 link) unions losslessly on the first sync and records one.
- Pure merge_plan(baseline, local, youtube) -> add/remove per side + merged order
  (local order, YouTube-only additions appended). merge_playlist applies the
  local side directly and the YouTube side through the failure breaker; the
  baseline only advances on full success so a partial failure re-converges.
- plan_merge/push-plan now report what each side gains and loses; the confirm
  dialog shows the two-way plan before applying (pushPlanMerge, HU+EN).
- Tests: 10 new (pure classification incl. delete-vs-stale-copy; DB apply proving
  a YouTube-add and a Siftlode-add both survive on both sides, a genuine delete
  propagates without collateral loss, no-baseline never deletes, create path).
2026-08-28 04:20:29 +02:00
peter bc33b2ed16 feat(downloads): per-user, per-host yt-dlp cookie/auth support (S6)
Each user stores their own login cookies (Netscape cookies.txt) for an
auth-walled download host (Facebook, Instagram, X, TikTok, Reddit) in
Settings; a download of an auth-walled link on that host is retried with
the acting user's cookie.

Two load-bearing security controls, both in downloads.credentials as the
single source of truth so enqueue and the worker can't drift:
- cache privacy: a cookie-authed download salts its MediaAsset format_sig
  with the user id, so the authed asset is private and never cache-served
  to another user. The worker attaches a cookie ONLY when the asset it is
  downloading into IS that private, salted identity.
- host scope (SEC-12): a cookie is resolved only for a URL whose netloc
  the credential owns (netloc-scoped _host_is); a look-alike/unrelated
  host gets none.

cookies_enc is Fernet-encrypted at rest, only decrypted in-worker into a
0600 file under staging/.auth/ (survives the retry wipe + _produced_file
scan) and unlinked in finally. The blob is never returned by a route,
logged, audited, or written to a job's error/snapshot. cookiefile is set
after the adapter's extra merge so an adapter can't clobber it. Settings
UI is write-only (host + set/updated only, never echoed). New
download_credentials table + migration 0074; per-user routes; HU+EN i18n.

Tests: host-scope, cache-privacy (no cross-user serve), encrypt-at-rest,
write-only API, worker attach-gate; FE dom test for the write-only field.
2026-08-28 03:53:04 +02:00
peter 5280949667 feat(ui): migrate the canonical action-button surface onto <Button>
Sweep the rounded-lg solid/ghost/outline action-button family onto the shared primitive:
ConfirmProvider, ErrorDialog, DownloadDialog, DownloadCenter, downloadModals, downloadAdmin,
ProfileEditor, PersonalViewsModal, SettingsPanel, WatchPage, DraftSaveBar. Deliberately left
on their own styles per the agreed scope: login/onboarding rounded-xl CTAs (Welcome/Onboarding/
Setup/KeyGate), glass Plex surfaces, segmented pills/toggles/switches/chips, dropdown triggers,
data-driven Banner/tone maps, text-xs dense buttons, and tiny inline p-1/p-1.5 icon controls —
none map 1:1 to a variant+size without a visual change, and the repo has no tailwind-merge.
2026-08-28 03:12:29 +02:00
peter 4892c38466 feat(ui): shared <Button> primitive + migrate ShareDialog/VideoEditor/ChannelPage proofs
Introduce frontend/src/components/ui/Button.tsx — the one action-button primitive
(variants primary|danger|outline|ghost|glass|danger-outline; sizes xs|sm|md|lg|icon;
icon/iconOnly/label/loading/fullWidth/forwardRef). No focus-visible (already centralized
in index.css @layer base). Migrate the cleanest proof sites off btnCls, and flip the
channel Subscribe button to the outline variant so it matches Full history (the user's ask).
2026-08-28 02:59:21 +02:00
peter 9ae6c47d6c perf(downloads): de-prioritize worker transcodes + idle-back-off polls to stop LAN slowdown
The download/edit child (yt-dlp + ffmpeg) shares one LAN box with the API,
Postgres, and Plex playback. A CPU re-encode pegged every core for ~30s and
starved web requests + playback until it finished. Full-scope S4 mitigations:

- Child self-de-prioritizes at startup (inherited by every ffmpeg/yt-dlp
  descendant): CPU nice increment + best-effort idle ionice class + an
  OMP_NUM_THREADS budget, all config-gated and independently disable-able.
- Cap the browser-safe re-encode's encoder threads (`-threads`) so a transcode
  reserves a core for foreground work instead of claiming all of them.
- Throttle the retention GC's whole-tree orphan sweep (yield IO after each
  batch of files) so its rglob can't monopolize disk IO in the API process.
- Idle-back-off the Download Center (2s→15s when nothing queued/running) and
  UsageBar (2s→15s when running_jobs is 0) polls; both snap back to 2s on the
  next fetch after an enqueue (invalidateDownloads refetches immediately).

Tests: encode_thread_cap (pin/reserve-core/edge), -threads flag presence,
_deprioritize (nice/ionice/OMP branches), GC sweep throttle batching.
2026-08-28 02:37:56 +02:00
peter b7a6451dba fix(downloads): iPad audio playback + honest non-retryable failure for auth-walled hosts
5a — the public watch page rendered every asset in a <video>, but WebKit (all
iOS browsers) refuses to play an audio-only mp3 in <video>, so shared audio
links were broken on iPad while working on desktop Chromium. public_meta now
flags audio-only assets (codec present, no video stream) and WatchPage renders
a native <audio> in a padded surface card (no artwork exists for these).

5c — a login-walled Facebook post failed as a bare 'unknown', which offered a
misleading Retry. Classify closed-group / cookie-suggestion auth walls as
login_required and genuinely unsupported sites as a new unsupported_host class
(both non-retryable), and drop unknown from the modal's retryable set — an
unclassified failure is most often an unparsed auth wall. The queue row keeps
its own force-retry for real transients.
2026-08-28 02:22:28 +02:00
peter 11e32163b8 fix(ui): SortControl direction toggle no longer wraps on narrow popovers
The asc/desc buttons in the sort popover footer share one component across
Feed and Plex browse, but rendered differently: the Plex popover is narrower
(short option labels), so "Descending" wrapped icon-over-text and made the
active button taller. Add whitespace-nowrap to the direction buttons and bump
the popover min-width (min-w-52 -> min-w-56) so both call-sites render an
identical single-line toggle.
2026-08-28 02:06:06 +02:00
peter 257dc5741d fix(plex): resolve movies stuck "in progress" on the 15-min incremental sync
The incremental watch-sync (run_plex_watch_sync) learned a movie's watched
transition only from the history feed + onDeck. When a movie is finished on
Plex (commonly by scrubbing through the end credits) Plex sets viewCount>0,
clears viewOffset and DROPS it from Continue-Watching, so onDeck no longer
lists it (and onDeck can only ever ADD an in-progress row, never resolve one),
while a scrub-to-end frequently emits no history session row. The stale local
status='new', position>0 row then survived until the DAILY full reconcile —
which is why movies kept showing "in progress".

Add _reconcile_inprogress_movies: for each local in-progress MOVIE that Plex's
onDeck no longer lists, read its current per-account viewCount/viewOffset with
one metadata() call and apply it via the shared _pull_apply, flipping to
watched when Plex says so. Bounded by a probe cap (100), most-recently-touched
first; episodes are left to onDeck/history. No-signal rows are preserved (an
un-watch stays the daily reconcile's job).

Tests: watched-flip, onDeck+episode skip, and no-signal-preserved.
2026-08-28 01:34:02 +02:00
peter f52bc5898b release: 0.74.0 (improvement/quickfix-batch)
gate / frontend (push) Failing after 1m28s
gate / backend (push) Successful in 2m15s
2026-08-28 00:05:21 +02:00
peter d0033617e2 chore(release): 0.74.0 — quickfix batch (Plex sync/browse + Channel Manager) 2026-08-28 00:03:13 +02:00
peter f80ed115ad fix(plex,channels): deep-review fixes — durable watch-sync + no shared-link clobber
Deep-review of the quickfix batch surfaced two data/preference-loss bugs plus
smaller items; all fixed with tests.

- watch_sync _pull_apply: don't let the Plex-authoritative pull overwrite a DIRTY
  local row (a change whose immediate push failed) — that stranded the value
  forever since _repush_dirty only retries dirty rows. Leave it dirty for re-push.
- PlexProvider / ChannelsProvider: a partial shared link opened in a fresh tab
  persisted mount-time DEFAULTS over the account's stored scope/sort/filters
  (base read before the account was pinned). Persist/keep only URL-present dims;
  re-hydrate the omitted ones from storage.
- playback resume write-back: guard against out-of-order delivery (a superseded
  older push now drops) and evict the throttle map so it can't grow unbounded.
- PlexPlayer: announce mid-playback buffering to assistive tech (role=status +
  sr-only label); new plex.player.buffering (HU+EN).
- columnSort: hoist the shared key:dir sort codec used by both URL-state modules.
- Tests: dirty-guard + flag-settle + per-key throttle scoping + superseded-push
  (backend); hold-repeat/suppress-click/unmount + partial-link no-clobber +
  refetch-doesn't-clobber-undo (frontend). Backend pytest 475, FE vitest 519.
2026-08-27 23:54:44 +02:00
peter 7fbb996297 feat(channels): shareable manager view in the URL — tab+sort+column-filters (S8)
The Channels-manager content state now round-trips through the address bar,
so a manager view is bookmarkable/shareable — the S4 pattern applied to the
last content page the audit flagged.

- New lib/channelsUrlState.ts (channelsParamUpdates / channelsParamsToState /
  hasChannelsParams) under NAMESPACED keys ctab/csort/cf — channels-unique
  like ?v=/?pl/pscope, so no other provider reads or clears them and the
  feed's URL-stripping can't delete them. The mixed scalar/array FilterMap
  (sync/priority scalars + tags multi) rides in `cf` as a JSON blob so both
  shapes round-trip exactly. Only non-default values emit.
- ChannelsProvider mirrors {view, sort, filters} via useUrlMirror, page-gated
  on channels (off-page the keys clear; localStorage still holds the state and
  it re-mirrors on return). A shared link wins over stored state: readChannels
  UrlSeed captured once, applied before paint through the slice setters (so the
  account-pin re-hydrate sees them dirty and won't reload storage over the
  shared view), then persisted to the account once pinned so it survives a
  later plain reload — mirrors PlexProvider's seed/rehydrate. The name search
  stays ephemeral (out of the shared contract).
- NavigationProvider's page-switch nulls ctab/csort/cf alongside the plex keys
  so switching modules resets to the manager root.

Tests: channelsUrlState.dom.test.ts (9 round-trip) + ChannelsProvider.dom
seed/mirror (2); the existing tab-history test's beforeEach now resets the
query string too (the mirror writes it). FE vitest 503→514. FE-only.
2026-08-27 04:37:53 +02:00
peter 9cd4ae09d4 feat(views): editable saved-view detail — removable filter chips + undo + save-current (S7)
The Personal-views manager's detail pane becomes editable instead of
read-only:

- Captured filters render as REMOVABLE chips. Removing one prunes the set
  and writes it straight back to the stored view (PATCH filters), via the
  same live-row builders (useActiveFilters / buildPlexActiveFilters) that
  already compute the pruned snapshot — so the modal and the live chip row
  can't drift. The prune is wrapped in useUndoable + the shared UndoToolbar,
  so undo/redo re-persist the same way. The undoable draft is the single
  source of truth for the chip row, so a chip-prune refetch never clobbers
  the history.
- A "Save current" button beside Apply stores the module's currently-applied
  filters INTO the view (the live→stored inverse of Apply). Disabled when the
  view already equals the live filters (it's the active one). On save the
  draft re-seeds to the saved snapshot and clears the stale undo history.
- Rename gains a visible commit affordance: a check button beside the name
  input that enables once the name changes (Enter and blur still commit).

Provider gains two methods mirroring `rename`: updateFilters(id, filters)
and saveCurrent(id). One shared provider serves feed + plex, so both get it.
PATCH filters was already supported end-to-end — FE-only.

New i18n: views.saveCurrent / saveCurrentHint / saveCurrentUpToDate (HU+EN).
Tests: 5 new in PersonalViewsModal.dom.test.tsx (chip-remove persists, undo
re-persists, save-current fires + disables when up-to-date, rename commit).
2026-08-27 04:22:07 +02:00
peter 00527d0cf3 feat(channels): leading '+ Tag' pill + detail-page Unhide affordance (S6)
Item 6 — the tag add-trigger now LEADS the chip row (a stable left anchor)
instead of trailing it, so on the detail page (bounded=false) adding chips
grows the row to its right and the '+'/its editor popover no longer drift.
Compact icon in the width-bounded manager cell; a labelled '+ Tag' pill
where there's free width (the detail header).

Item 7 — surface the per-subscription 'hidden' flag on the detail GET
(channel_detail was omitting it) and add an Eye/EyeOff hide/unhide toggle
next to Block on the channel page, mirroring the manager. FE ChannelDetail
gains 'hidden'; new channel.* i18n keys (HU+EN).

Tests: +3 backend (channel_detail surfaces hidden, both states + unsubscribed
default). Gate green: backend pytest 471, FE vitest 498.
2026-08-27 04:02:25 +02:00
peter 8dc4976c2c feat(channels): horizontal −/+ priority stepper with pop-free inline edit (S5)
Redesign the shared PriorityStepper (channel manager cell + detail header)
from a value-plus-stacked-arrows control into a horizontal [−] value [+]
segmented stepper. The value now sits centred between the buttons and opens
an inline editor in a FIXED-width slot, so click-to-type no longer "pops"
the control wider (the old absolutely-positioned overlay grew w-6 → w-14).

One `size` prop drives the two surfaces: `sm` draws its own bordered pill
for the standalone table cell; `md` drops the outer border and keeps only
the segment dividers, blending into the detail header's lozenge instead of
double-bordering it. Hold-to-repeat, negatives, and the debounced-flush
persistence hook are unchanged.

The manager priority column widens 3.5rem → 5.5rem to fit the horizontal
control under fixed table layout.

Tests: 7 new DOM tests for the stepper interaction contract.
2026-08-27 03:29:42 +02:00
peter 56a2ecdec3 feat(plex): shareable Plex-browse state in the URL — scope+filters+sort+show (S4)
Mirror the Plex-browse content state into the address bar so a browse view is
bookmarkable/shareable, the Plex analogue of the feed's urlState. Keys are
NAMESPACED (pscope/psort/pshow/pf) rather than reusing the feed's bare
scope/sort/show: FeedFiltersProvider actively strips its own KEYS from the URL
whenever off the feed page, so a shared key would be deleted out from under the
Plex page. The namespaced keys are Plex-unique (like ?v=/?pl), so no other
provider ever reads or clears them.

- New lib/plexUrlState.ts: plexParamUpdates / plexParamsToState / hasPlexParams.
  The pf filter blob reuses the SAME appendPlexFilters wire serializer (new
  public plexFiltersToParams + inverse parsePlexFilters, kept next to each other
  so a new wire key can't be added on one side only).
- PlexProvider: URL-seed wins over localStorage at mount; the account-pin
  rehydrate persists the seed instead of clobbering it (mirrors feed's
  initialHadUrlFilters); useUrlMirror mirrors/clears the 4 keys, gated on
  page===plex. q/field stay out of the shareable contract.
- NavigationProvider: a page switch now also nulls the 4 plex keys.

Tests: 8 new round-trip cases in plexUrlState.dom.test.ts (FE vitest 483→491).
2026-08-27 03:14:54 +02:00
peter 8299ea72b7 feat(plex): 'last watched' browse sort + auto-apply in the in-progress view (S3)
Add a 'lastwatched' sort key to the unified Plex browse. Movies rank by
coalesce(progress_updated_at, watched_at); shows by max(coalesce(...)) across
their episodes (rollup.resumed_at, now broadened from progress-only so watched
episodes count too). Never-watched titles sink via nullslast.

Frontend auto-applies lastwatched:desc when entering the in-progress watch-state
view and restores the pre-entry sort on exit; a manual sort change while there
still wins. i18n HU+EN. Backend tests cover both union legs + mixed nullslast.
2026-08-27 02:50:41 +02:00
peter 6c918d1dff docs(plex): refresh watch-sync job docstrings for Plex-authoritative pull (S2)
The incremental/reconcile job docstrings still described last-write-wins and a
30-min cadence; update them to match the S2 behavior (Plex wins on divergence,
15-min incremental) so the "why" doesn't drift from the code.
2026-08-27 02:33:42 +02:00
peter 3f8257059d feat(plex): Plex-authoritative watch-sync — resume write-back + faster pull (S2)
Make Plex the source of truth for watch state:

- Write-back during playback: item_progress now pushes the resume position to
  Plex on the periodic checkpoint too (not only the final flush), throttled to
  one push per item every ~10s (in-memory, single-process rate limiter). The FE
  now also fires an immediate (debounced) push on seek.
- Faster incremental pull: 30 → 15 min (plex_watch_sync_interval_min).
- Plex-authoritative pull: _pull_apply resolves a genuine divergence to Plex's
  state instead of last-write-wins; the _same_state + synced_to_plex short-circuit
  still prevents a freshly-pushed row from bouncing. Removed the now-unused
  _LWW_TOLERANCE_S / _sift_ts.

Tests: 5 new in test_plex_watch_sync.py (Plex-wins resume/watched, no-bounce,
throttle final vs periodic).
2026-08-27 02:07:45 +02:00
peter 12d9561ba1 fix(plex): warm up HLS buffer on session start + player loading/buffering spinner
The Plex player stalled ~2s a few seconds into every auto-advanced next
episode. Root cause: each auto-advance spins up a cold ffmpeg HLS remux
session, and for a fresh start (start_s==0) start_session returned
immediately with zero buffer — hls.js drained the first segment, then
rebuffered waiting for the still-warming ffmpeg to write the next one.
Once ffmpeg gets ahead (stream-copy is faster than realtime) it never
stalls again.

- stream.py: before reporting a session ready, wait for the first
  _WARMUP_SEGMENTS (3) video segments to exist — the seg0 wait is now
  unconditional (was start_s>0 only) and warm-up runs for all starts.
  Best-effort and bounded: stop at the first segment not ready in time.
- PlexPlayer.tsx: a loading/buffering spinner. The existing overlay only
  covered the INITIAL load (!ready); add a `buffering` state driven by the
  video's waiting/stalled/seeking (set) and playing/canplay/seeked/pause
  (clear) events so a mid-playback stall or in-flight seek reads as
  "loading" instead of a frozen frame, and surface a fatal loadError even
  once playback has started. Spinner uses the shared Loader2.

Merges roadmap E6·item1 (spinner while loading/seeking) into this fix,
since the server-side warm-up deliberately holds ready a beat longer.

Tests: 3 new in test_plex_stream.py (warm-up waits for the first few
segments; stops at the first missing one; no warm-up when seg0 never
arrives). Gate: backend pytest 460 (Docker lane), FE tsc/lint/knip/
prettier + vitest 483.
2026-08-27 00:50:58 +02:00
535 changed files with 51509 additions and 3905 deletions
+1897
View File
File diff suppressed because it is too large Load Diff
+35 -24
View File
@@ -26,8 +26,8 @@ The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc project
keep commits logical and Conventional-Commit-styled.
- **No AI attribution in commits.** No `Co-Authored-By` trailer, no "generated by" lines. (This file
being AI-written is the intended, visible exception; commits are not.)
- **Release: user-owned in timing, agent-run in execution.** The user decides *when* to ship; on their
go-ahead *you* run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
- **Release: user-owned in timing, agent-run in execution.** The user decides _when_ to ship; on their
go-ahead _you_ run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
verify `/api/version`). Never self-ship — bring work to a green, committed state and stop for the
go-ahead first, then run it yourself.
- **Public repo.** Keep ops/topology/secrets out — those live in a separate private repo. Nothing here
@@ -41,7 +41,7 @@ how it gets built. Concretely:
- **Plan before you build.** For anything non-trivial, present a short plan (sprint-level for larger
work) and get a go-ahead before writing code.
- **Offer choices, mark the optimal — and explain the trade-off.** When several solutions are viable,
list them and mark the recommended one. This is the *one* place to spend extra words: give the *why*
list them and mark the recommended one. This is the _one_ place to spend extra words: give the _why_
behind each option and the weight of the decision. These decision points are where the product owner
engages with the build and learns from it, so make them understandable — a reasoned menu, not a bare one.
- **Mockups for design/UX decisions.** For any visual or interaction choice, show mockup options to
@@ -59,31 +59,42 @@ how it gets built. Concretely:
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → components; data
via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through `lib/api/*`
slices over `core.req<T>`.
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
feature component or a domain API slice, and one layer down `lib/` may not import from `components/` at
all, not even a type (`no-restricted-imports` in `eslint.config.js`, with its own test). A new generic
component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
`lib/api/*` slices over `core.req<T>`.
**Not wanted** (reviewed and rejected as over-engineering for this size): microservices, event bus,
CQRS, a repository/DI framework, a router library, a generic CRUD-router factory or base serializer.
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
|---|---|---|
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| Area | Use | Not |
| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
## House rules that bite (learned from real bugs)
@@ -100,6 +111,6 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
---
*Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
_Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
code conventions in [`backend/app/README.md`](backend/app/README.md). This file stays the source of
truth for the canonical-patterns table and the gate/workflow/collaboration rules above.*
truth for the canonical-patterns table and the gate/workflow/collaboration rules above._
+20 -4
View File
@@ -45,18 +45,34 @@ RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
# ffmpeg: merge separate video+audio streams + postprocessing. Deno: the JavaScript runtime
# yt-dlp uses to solve YouTube's "n" signature challenge for the web player clients (required
# alongside the PO token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode (E6 S1).
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
# no driver, so without this `vainfo`/`h264_vaapi` fail and transcode silently falls back to libx264.
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg unzip curl ca-certificates \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/deno \
&& rm /tmp/deno.zip \
&& rm -rf /var/lib/apt/lists/*
# Intel NEO OpenCL runtime (intel-opencl-icd) — needed by the Plex transcode's HDR/10-bit HW path, which
# tone-maps HDR→SDR on the GPU via `tonemap_opencl` (Gen 9.5 / UHD 630 can't tonemap in the VAAPI VPP, so
# OpenCL is the only HW route; without this the 10-bit branch drops to CPU libx264). NEO was REMOVED from
# Debian trixie (this base = python:3.12-slim = trixie), so it's pulled from **bookworm** — the 22.43 there
# still supports Gen 9.5 and its .deb runs on trixie (verified live on the prod UHD 630). The bookworm apt
# source is added only for this one install and removed after, so it can't drift the rest of the image.
# Harmless on hosts with no /dev/dri or no OpenCL use (the runtime just sits unused).
RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nComponents: main\nSigned-By: /usr/share/keyrings/debian-archive-keyring.pgp\n' > /etc/apt/sources.list.d/bookworm.sources \
&& apt-get update \
&& apt-get install -y --no-install-recommends -t bookworm intel-opencl-icd \
&& rm /etc/apt/sources.list.d/bookworm.sources \
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
+661
View File
@@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
+18 -3
View File
@@ -94,9 +94,18 @@ docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
docker compose -f docker-compose.selfhost.yml up -d
```
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
Windows) and restore with `scripts/restore.sh`.
That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a
version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md)
and in the app's Release Notes. The registry keeps only the newest few of those, so a pin decides
*when* you take the next hop rather than letting you stay put forever — see
[docs/self-hosting.md](docs/self-hosting.md#upgrading).
**Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling
the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is
what decrypts the Google, SMTP and Plex secrets stored in that database, so a dump restored without it
is only half an instance. Use `scripts/backup.sh` (or `backup.ps1` on Windows) and `scripts/restore.sh`;
the equivalent one-liners, the pinning recipe and the rollback steps are in
[docs/self-hosting.md](docs/self-hosting.md#backups).
## How it works
@@ -111,6 +120,12 @@ Windows) and restore with `scripts/restore.sh`.
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
frontend; packaged as a single Docker image with Docker Compose.
## License
[GNU AGPL-3.0](LICENSE). You can run, study, modify and share Siftlode; if you modify it and let other
people use it over a network, they are entitled to your modified source. Copyright © 2026 Siftlode
contributors.
## Note
This project is developed with AI assistance.
+1 -1
View File
@@ -1 +1 @@
0.73.0
0.85.0
@@ -0,0 +1,46 @@
"""download_credentials (S6: per-user, per-host yt-dlp cookie/auth store)
Each user stores their OWN login cookies (Netscape cookies.txt) for an auth-walled download host
(Facebook, Instagram, …), Fernet-encrypted at rest. A download of a yt-dlp-supported, auth-walled
link on that host is retried with the acting user's cookie. Unique per (user_id, host_key) so a
user holds at most one credential per host. Empty for every existing install; forward-only in
practice (a downgrade just drops the table + its secrets).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0074_download_credentials"
down_revision: Union[str, None] = "0073_saved_view_source"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"download_credentials",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("host_key", sa.String(length=64), nullable=False),
sa.Column("cookies_enc", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "host_key", name="uq_download_credential"),
)
op.create_index("ix_download_credentials_user_id", "download_credentials", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_download_credentials_user_id", table_name="download_credentials")
op.drop_table("download_credentials")
@@ -0,0 +1,26 @@
"""playlist synced_video_ids (S7: 3-way merge baseline for two-way YouTube sync)
The one-directional "local wins" push is replaced by a 3-way merge that classifies each item
against the last-synced state. That needs the raw ordered id set as a baseline, not just the
`synced_fingerprint` hash. Add a nullable `synced_video_ids` (JSON list) column; NULL on every
existing row means "no baseline yet" — the first merge then unions losslessly and records one.
Forward-only in practice (a downgrade just drops the column).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0075_playlist_sync_baseline"
down_revision: Union[str, None] = "0074_download_credentials"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("playlists", sa.Column("synced_video_ids", sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column("playlists", "synced_video_ids")
@@ -0,0 +1,41 @@
"""media_assets.media_kind (E7 S5a: the video|audio|file discriminator for the media library)
Adds a stored media-kind to each asset so the library can gate its actions on it (play/edit for
media, download-only for a generic uploaded file) instead of re-deriving from codecs in prose. The
column is the single home of the invariant `downloads.mediakind.derive` computes at finalize.
Backfill for existing rows: a stored video codec -> 'video', an audio-only codec -> 'audio', an
editor derivative (source_kind='edit', which carries no codecs but is always a video clip) ->
'video'. Rows still pending/errored (no codecs, not an edit) stay NULL — kind is set when they
finalize. Empty `media_kind` never reaches the UI: only `ready`/`done` rows render actions, and
those are all backfilled here.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0076_media_asset_kind"
down_revision: Union[str, None] = "0075_playlist_sync_baseline"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("media_kind", sa.String(length=16), nullable=True))
op.execute(
"""
UPDATE media_assets
SET media_kind = CASE
WHEN vcodec IS NOT NULL THEN 'video'
WHEN acodec IS NOT NULL THEN 'audio'
WHEN source_kind = 'edit' THEN 'video'
ELSE NULL
END
"""
)
def downgrade() -> None:
op.drop_column("media_assets", "media_kind")
@@ -0,0 +1,54 @@
"""plex_items playback-class facet (E7: filter the feed by Direct Play / Direct Stream / Transcode)
Adds the columns the playback-class filter needs:
* transcode_hw — for a `playable="transcode"` item, whether the prod iGPU can hardware-transcode it
(codec HW-decodable AND 8-bit SDR) vs software-only. NULL for non-transcode items.
* video_bit_depth — the source video bit depth (from the Plex media Stream), NULL until probed.
* media_probed_at — watermark for the bit-depth backfill (NULL = not yet probed).
Backfill for existing rows: `transcode_hw` is derived from the ALREADY-stored `codec_video` alone (the
same HW-decodable allow-list `plex.sync.HW_DECODABLE_CODECS` / `plex.stream` use), so the filter works
immediately after migration. Bit depth is unknown for existing rows (the cheap Plex listing doesn't
carry it), so a 10-bit/HDR HEVC is provisionally 'hw' here and later refined to 'sw' by the bit-depth
backfill (`plex.sync._probe_media_batch`) once its Stream is probed. `video_bit_depth`/`media_probed_at`
start NULL.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0077_plex_playback_class"
down_revision: Union[str, None] = "0076_media_asset_kind"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("plex_items", sa.Column("transcode_hw", sa.Boolean(), nullable=True))
op.add_column("plex_items", sa.Column("video_bit_depth", sa.Integer(), nullable=True))
op.add_column(
"plex_items", sa.Column("media_probed_at", sa.DateTime(timezone=True), nullable=True)
)
op.create_index("ix_plex_items_transcode_hw", "plex_items", ["transcode_hw"])
# Codec-only derivation for existing rows — keep this codec set identical to
# plex.sync.HW_DECODABLE_CODECS. Bit depth is unknown here, so provisionally 8-bit (→ hw for a
# decodable codec); the bit-depth backfill downgrades 10-bit/HDR HEVC to sw later.
op.execute(
"""
UPDATE plex_items
SET transcode_hw = CASE
WHEN playable = 'transcode'
THEN lower(codec_video) IN ('h264','hevc','h265','vp9','vc1','mpeg2video')
ELSE NULL
END
"""
)
def downgrade() -> None:
op.drop_index("ix_plex_items_transcode_hw", table_name="plex_items")
op.drop_column("plex_items", "media_probed_at")
op.drop_column("plex_items", "video_bit_depth")
op.drop_column("plex_items", "transcode_hw")
@@ -0,0 +1,58 @@
"""users.password_revoked_at — remember that SEC-10 dropped a preset password
Revision ID: 0078_password_revoked_at
Revises: 0077_plex_playback_class
Create Date: 2026-09-02
SEC-10 (`auth.revoke_preset_password_if_unverified`) nulls `password_hash` when an account whose email
was never verified is attested (Google adoption, or admin approval). That left NO trace, and
password-reset eligibility keyed off `password_hash IS NOT NULL` — so such an account could neither
sign in (no password) nor request a reset link (not "eligible"): a permanent lockout with a silent,
uniform response at every step.
This column is that trace. A row with `password_revoked_at` set is a local account that HAD a password
and had it revoked, so the reset flow treats it as eligible again. Setting a password (reset-confirm or
/auth/set-password) clears the stamp. A Google-only account that never had a password keeps it NULL and
stays reset-ineligible by design — its owner adds a password from Settings while signed in.
Backfill: the signature is exact and narrow — a local (google_sub IS NULL), non-demo account with NO
password could ONLY have got there through the revoke. Registration always sets a hash, the setup
wizard sets one too, every Google row carries a google_sub (the adoption path assigns it BEFORE
revoking), and nothing in the codebase ever clears google_sub. Stamp those with the current time so
they become recoverable without operator intervention.
Deliberately NOT also requiring `email_verified = false`, even though the revoke only fires on an
unverified account: verifying is exactly what such a victim does NEXT. The revoke does not invalidate
the outstanding verification token, so clicking the emailed link sets email_verified=True while
password_hash stays NULL — and that row has no route back at all (reset ineligible, resend refuses an
already-verified address, and set-password needs a signed-in session the account can no longer obtain:
no password to sign in with, and no Google identity). Those are the victims who most need the stamp,
so the predicate must not exclude them.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0078_password_revoked_at"
down_revision: Union[str, None] = "0077_plex_playback_class"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"users",
sa.Column("password_revoked_at", sa.DateTime(timezone=True), nullable=True),
)
op.execute(
"UPDATE users SET password_revoked_at = now() "
"WHERE password_hash IS NULL "
"AND google_sub IS NULL "
"AND is_demo = false"
)
def downgrade() -> None:
op.drop_column("users", "password_revoked_at")
@@ -0,0 +1,65 @@
"""email_canonical on users + invites, for matching one mailbox across Gmail spellings
Revision ID: 0079_email_canonical
Revises: 0078_password_revoked_at
Create Date: 2026-09-04
Gmail delivers `x.y@gmail.com`, `xy@gmail.com` and `xy+tag@gmail.com` to ONE mailbox; the invite gate
only lowercased. On 2026-09-03 that turned an approved user's Google sign-in into a silent second
access request (OAuth returns the account's stored primary, which need not be the spelling they typed)
and left prod with two user rows for one mailbox.
The backfill below reproduces `app.utils.canonical_email` in SQL. That duplication is deliberate — a
migration must not import application code that will keep changing — but it means the two have to
agree; `test_canonical_email_backfill_matches_the_helper` pins them together.
Deliberately NOT unique. Prod already carries canonical duplicates: two approved invites for one
mailbox dating back to 2026-06-16, and two active user rows from the 2026-09-03 walkthrough. Merging
accounts is a judgement call about people's data, not something a migration gets to do on its own, so
this migration only records the canonical form and leaves the duplicates in place for a human.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0079_email_canonical"
down_revision: str | None = "0078_password_revoked_at"
branch_labels: str | None = None
depends_on: str | None = None
# Mirrors app.utils.canonical_email: Gmail (and its googlemail alias) loses dots and any `+tag` from
# the local part and lands on gmail.com; every other domain is only lowercased, because the dot rule
# is not guaranteed elsewhere — and does not hold for Workspace custom domains unless their admin
# enabled it.
_CANON_SQL = """
CASE
WHEN split_part(lower({col}), '@', 2) IN ('gmail.com', 'googlemail.com')
-- an empty local part after stripping falls through to plain lowercase, as the helper does
AND replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') <> ''
THEN replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') || '@gmail.com'
ELSE lower({col})
END
"""
def _backfill(table: str) -> None:
op.execute(
f"UPDATE {table} SET email_canonical = {_CANON_SQL.format(col='email')} "
f"WHERE email IS NOT NULL"
)
def upgrade() -> None:
for table in ("users", "invites"):
op.add_column(
table,
sa.Column("email_canonical", sa.String(length=320), nullable=False, server_default=""),
)
_backfill(table)
op.create_index(f"ix_{table}_email_canonical", table, ["email_canonical"])
def downgrade() -> None:
for table in ("users", "invites"):
op.drop_index(f"ix_{table}_email_canonical", table_name=table)
op.drop_column(table, "email_canonical")
@@ -0,0 +1,32 @@
"""users.google_email — the address Google asserts, kept beside the one the user registered with
Revision ID: 0080_google_email
Revises: 0079_email_canonical
Create Date: 2026-09-04
Since 0079's follow-up a Google sign-in no longer overwrites `users.email`, so a deliberate
`me+siftlode@gmail.com` survives being linked to a Google account whose primary is `me@gmail.com`.
That leaves two true addresses for one account and only one of them stored. This column holds the
other, so the account menu and the Settings sign-in row can show which address Google actually
answers with.
No backfill is possible: Google's answer is only known at sign-in. Accounts linked before this
column existed carry NULL until their owner next signs in with Google — and for the majority the two
addresses are identical anyway, where the UI shows nothing either way.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0080_google_email"
down_revision: str | None = "0079_email_canonical"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("google_email", sa.String(length=320), nullable=True))
def downgrade() -> None:
op.drop_column("users", "google_email")
@@ -0,0 +1,34 @@
"""invites.invited_at — when the admin actually told this person the door was open
Revision ID: 0081_invite_invited_at
Revises: 0080_google_email
Create Date: 2026-09-05
Whitelisting an address stays SILENT by design: the operator is opening a door as a possibility,
usually after agreeing it with the person, and a row can sit approved for weeks without anyone
registering. Telling them is therefore a separate, deliberate act — a send button on the row.
Which raises the one question the admin cannot otherwise answer: "did I already write to them?"
Without a record the only failure mode is writing to a real person twice, which is exactly the
nudging the silent-by-default design exists to avoid. The audit log holds the event, but not where
the decision is made.
NULL means "never invited from here", which is true of every existing row: no backfill is possible
or wanted, since the historic approval mails answered a REQUEST rather than inviting anyone.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0081_invite_invited_at"
down_revision: str | None = "0080_google_email"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("invites", sa.Column("invited_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.drop_column("invites", "invited_at")
@@ -0,0 +1,34 @@
"""invites.lang — the language this person asked us in
Revision ID: 0082_invite_lang
Revises: 0081_invite_invited_at
Create Date: 2026-09-05
The mails an access request produces — approved, denied, invited — go to an address that often has
no account yet, so there is no `preferences.language` to read. The signal does exist, though, and
only at the moment of the request: the language of the page the form was submitted from, or the
locale Google reported for a sign-in we refused. Recording it there is what lets the decision mail
answer in the requester's own language instead of an admin's guess about a stranger.
Captured rather than derived, for the third time in this epic: `preferences.language` (S6) and
`google_email` (S3) both taught that a signal which exists only in one moment cannot be backfilled
later. So no backfill here either — NULL means "we were never told", which is the honest state for
every historic row, and for every address an admin whitelists by hand. Those fall back to English
unless the admin picks otherwise on the invitation send.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0082_invite_lang"
down_revision: str | None = "0081_invite_invited_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("invites", sa.Column("lang", sa.String(length=8), nullable=True))
def downgrade() -> None:
op.drop_column("invites", "lang")
@@ -0,0 +1,78 @@
"""comfort_pins — the shelf you fall back to when you cannot decide
Revision ID: 0083_comfort_pins
Revises: 0082_invite_lang
Create Date: 2026-09-06
E24's one new table. The shelf holds five different things — a video, a YouTube channel, a Plex show,
a Plex movie, a playlist — because comfort media is usually "put on X", a series or a channel, and
almost never one particular video.
Three shapes were possible and two are wrong here:
- a boolean on `video_states` / `plex_states` cannot ORDER a shelf, and has nowhere to put a channel
or a show (neither has a per-user state row at all);
- a polymorphic `(kind, ref)` string pair orders and holds anything, but drops the foreign key — and a
Plex item removed from the library, or a channel the user unsubscribes and prunes, would then leave
a dead tile on the one surface whose entire job is to be reliable when nothing else is.
So: one nullable CASCADE-ing FK per kind, plus `num_nonnulls(...) = 1`. The union invariant lives in
the schema rather than in a comment at every insert site, which is the same rule that the
`cancelled`/`canceled` bug taught this codebase.
NULLs are DISTINCT in a Postgres unique index, so the five per-kind `(user_id, <ref>)` uniques each
stop a double-pin without colliding with the other four kinds' NULL rows.
No backfill: an empty shelf is the honest state for every existing account. The surface renders its
own empty state and invites the first pin.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0083_comfort_pins"
down_revision: str | None = "0082_invite_lang"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"comfort_pins",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("video_id", sa.String(length=16), sa.ForeignKey("videos.id", ondelete="CASCADE")),
sa.Column(
"channel_id", sa.String(length=32), sa.ForeignKey("channels.id", ondelete="CASCADE")
),
sa.Column("plex_item_id", sa.Integer(), sa.ForeignKey("plex_items.id", ondelete="CASCADE")),
sa.Column("plex_show_id", sa.Integer(), sa.ForeignKey("plex_shows.id", ondelete="CASCADE")),
sa.Column("playlist_id", sa.Integer(), sa.ForeignKey("playlists.id", ondelete="CASCADE")),
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.CheckConstraint(
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
name="ck_comfort_pin_one_ref",
),
sa.UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
sa.UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
sa.UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
sa.UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
sa.UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
)
op.create_index("ix_comfort_pins_user_id", "comfort_pins", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_comfort_pins_user_id", table_name="comfort_pins")
op.drop_table("comfort_pins")
+12
View File
@@ -27,9 +27,18 @@ class AuditAction:
USER_SUSPEND = "user_suspend"
USER_UNSUSPEND = "user_unsuspend"
USER_DELETE = "user_delete"
# E9.5 S5: an admin let a stuck registrant in, or re-sent their confirmation link. Both are
# recorded because both are the admin acting ON someone's access, not merely reading a list.
USER_ACTIVATE = "user_activate"
USER_VERIFY_RESEND = "user_verify_resend"
USER_SETPW_LINK = "user_setpw_link"
INVITE_APPROVE = "invite_approve"
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
INVITE_DELETE = "invite_delete"
# E9.5 S6: the admin chose to tell someone their address is whitelisted. Adding the address is
# silent, so this is the act that actually reaches a person — and the one worth a record.
INVITE_SEND = "invite_send"
DEMO_ADD = "demo_add"
DEMO_REMOVE = "demo_remove"
DEMO_RESET = "demo_reset"
@@ -44,6 +53,9 @@ class AuditAction:
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
+140 -9
View File
@@ -7,11 +7,13 @@ from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app import email as email_mod
from app import mailtext
from app import sysconfig
from app.config import settings
from app.db import get_db
from app.models import Invite, User
from app.security import decrypt_optional
from app.utils import canonical_email
PASSWORD_MIN_LEN = 10
@@ -63,16 +65,126 @@ def has_write_scope(user: User) -> bool:
return WRITE_SCOPE in tok.scopes.split()
def env_allowed_canonicals() -> set[str]:
"""The env bootstrap allow-list, in CANONICAL form — the only shape it may be compared in.
S3 canonicalised the Invite branch of `is_allowed` and left this one matching the literal string,
which is the 2026-09-03 lockout with a different source of truth: an operator who writes
`first.last@gmail.com` into ALLOWED_EMAILS is refused when Google answers with the undotted
primary, and a `+tag`ged entry can never match at all, because a Google account cannot contain a
`+`. The addresses themselves stay literal wherever we MAIL them (`admin_notify_groups`); only
the comparison is canonical.
"""
return env_admin_canonicals() | {canonical_email(e) for e in settings.allowed_email_set}
def env_admin_canonicals() -> set[str]:
"""The env ADMIN_EMAILS bootstrap list, canonical — for the role check only. Kept separate from
`env_allowed_canonicals` because being listed as an admin also grants access, but being allowed
does not make anyone an admin."""
return {canonical_email(e) for e in settings.admin_email_set}
def is_allowed(db: Session, email: str) -> bool:
"""May this email sign in? An approved Invite is the source of truth; the env
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB)."""
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB).
Matched on the CANONICAL form (utils.canonical_email), not the literal string. Gmail delivers
every dotted / `+tagged` spelling to one mailbox, so approving one spelling always meant approving
that mailbox — matching literally only produced the 2026-09-03 lockout, where an approved user's
Google sign-in became a silent second access request because OAuth returned the undotted primary.
One canonical form can match SEVERAL invite rows (prod carries approved pairs for one mailbox,
the oldest from 2026-06-16). An approved row among them grants access: the API cannot flip an
approved invite to denied (`_decide` only accepts a pending one), so approved-beside-denied means
the admin approved that mailbox under a different spelling, not that they revoked it. Revoking is
deleting the invite row or suspending the account — both still work, since a deleted row is no
longer here to match and suspension is enforced at sign-in.
"""
if not email:
return False
email = email.lower()
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
if inv is not None:
return inv.status == "approved"
return email in settings.allowed_email_set or email in settings.admin_email_set
canon = canonical_email(email)
statuses = (
db.execute(select(Invite.status).where(Invite.email_canonical == canon)).scalars().all()
)
if statuses:
return "approved" in statuses
return canon in env_allowed_canonicals()
def can_activate(db: Session, user: User) -> bool:
"""Is this account merely waiting for an admin to finish letting it in?
True when it is inactive but its address is ALREADY allowed — the shape roadmap UX-25 describes:
the admin approved the address, the person signed up afterwards, and nothing connects the two
(the Access-requests queue only offers Approve on a still-pending invite).
Gated on `is_allowed`, the same function the sign-in paths consult, because activating an address
that is NOT allowed manufactures an inconsistent account: password sign-in only checks
`is_active` and would let them in, while their Google sign-in would be turned away and silently
reopened as a fresh access request. Letting in someone with no approved invite is the whitelist
box's job — that action says so out loud.
Lives in core so the admin surface and the /api/me badge count answer from ONE definition; a
button that appears and a number that counts must never disagree about who is stuck.
"""
if user.is_demo or user.is_suspended or user.is_active:
return False
return is_allowed(db, user.email)
def count_pending_activations(db: Session) -> int:
"""How many sign-ups are waiting on an admin click (see can_activate). Drives the admin badge.
Deliberately NOT "everyone who cannot sign in": an account that has simply not confirmed its
address yet is waiting on its owner, not on the admin, and badging that would nag about something
the admin did not cause. It is still listed on the admin screen — with a re-send — because the
mail may never have arrived.
"""
# Narrowed in SQL, then decided by can_activate itself — so the number the badge shows and the
# button the admin sees cannot drift apart by one of them re-deriving the rule.
rows = (
db.execute(
select(User).where(
User.is_demo.is_(False),
User.is_suspended.is_(False),
User.is_active.is_(False),
)
)
.scalars()
.all()
)
return sum(1 for u in rows if can_activate(db, u))
def find_user_by_email(db: Session, email: str) -> User | None:
"""The account someone means when they type an address at sign-in, password-reset or
verify-resend. None when there isn't exactly one answer.
Exact spelling wins. Failing that, the address is resolved to its mailbox
(utils.canonical_email) — so a password chosen at registration still works when the person types
another spelling of their own Gmail, and so the three flows that look an account up by typed
address stop being dead ends. Before this, requesting a reset for `xy@gmail.com` on an account
registered as `x.y@gmail.com` issued nothing and said nothing: the endpoint answers uniformly by
design, so the user got silence and no way in.
SEVERAL canonical candidates return None rather than a guess. That case is real — prod carries two
accounts for one mailbox — and picking one of them for a password sign-in would hand someone the
wrong account. The callers all treat None as "no such account", which is the uniform,
non-enumerating answer they already gave.
"""
if not email:
return None
email = email.strip().lower()
exact = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
if exact is not None:
return exact
candidates = (
db.execute(select(User).where(User.email_canonical == canonical_email(email)))
.scalars()
.all()
)
return candidates[0] if len(candidates) == 1 else None
def get_or_create_demo_user(db: Session) -> User:
@@ -119,6 +231,9 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
touch any browser session — each caller handles its own."""
email = user.email.lower()
user_id = user.id
# Read the language off the row while it still exists — the confirmation below is queued as a
# background task that runs after this session (and this user) are gone.
lang = mailtext.lang_for_user(user)
tok = user.token
# Both are stored encrypted — decrypt for the revoke call (the access-token fallback was passing
# ciphertext, so a token row with no refresh token silently failed to revoke). decrypt_optional so a
@@ -130,13 +245,25 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
)
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
db.execute(delete(User).where(User.id == user_id))
db.execute(delete(Invite).where(Invite.email == email))
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
# account was just erased — and prod really does carry approved pairs for one mailbox.
#
# Unless ANOTHER account still lives on that mailbox — prod carries that too (two user rows for
# one Gmail mailbox). Then NO invite goes, not even the literal one: is_allowed matches
# canonically, so every row here is what keeps the survivor allowed, and removing any of them
# would refuse their next Google sign-in and reopen it as a fresh access request — this epic's
# founding bug, caused by its own cleanup. `delete_invite` refuses for exactly this reason; the
# user row is already gone by now, so this query sees only the others.
canon = canonical_email(email)
if db.execute(select(User.id).where(User.email_canonical == canon)).first() is None:
db.execute(delete(Invite).where(Invite.email_canonical == canon))
db.commit()
if google_token:
background.add_task(revoke_google_token, google_token)
# Confirm the erasure to the (now former) user — GDPR good practice, and covers both the
# self-service and admin deletion paths since both funnel through here.
background.add_task(email_mod.send_account_deleted, email, operator_contact())
background.add_task(email_mod.send_account_deleted, email, lang, operator_contact())
ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
@@ -214,7 +341,11 @@ def require_human(user: User = Depends(current_user)) -> User:
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
where there's no scope check to lean on."""
if user.is_demo:
raise HTTPException(status_code=403, detail="Not available in the demo account.")
# Structured, because `req` leaves 403 to the caller and `notifyActionError` only shows a
# STRUCTURED detail — a plain string here reached the demo user as their caller's generic
# "couldn't do that", with no hint that the demo account is the reason (round 6). Affordances
# this gate sits behind are not all UI-gated on `is_demo` (the comfort shelf is not).
raise HTTPException(status_code=403, detail={"code": "demo"})
return user
+42 -3
View File
@@ -186,6 +186,21 @@ class Settings(BaseSettings):
download_supervisor_poll_seconds: int = 3
# Grace between SIGTERM and SIGKILL when stopping a child (matches run_ffmpeg's escalation).
download_kill_grace_seconds: int = 5
# --- worker-child resource de-prioritization ---
# The download/edit child (yt-dlp + ffmpeg) shares ONE LAN box with the API, Postgres, and Plex
# playback. Without a priority handicap a CPU re-encode pegs every core for ~30s and starves web
# requests + playback until it finishes. The child lowers its OWN priority at startup — inherited
# by every ffmpeg/yt-dlp descendant — so any foreground work always wins. Each knob is best-effort
# and independently disable-able with 0.
download_child_nice: int = 10 # CPU nice increment applied to the child; 0 = don't renice
download_child_ionice_class: int = 3 # best-effort IO class (3 = idle); 0 = leave IO untouched
# Cap the browser-safe re-encode's encoder threads so a transcode can't claim every core. 0 =
# reserve one core for foreground work (max(1, cpu_count - 1)); >0 pins exactly that many.
download_ffmpeg_threads: int = 0
# Throttle the retention GC's whole-tree orphan file sweep so its rglob can't monopolize disk IO
# in the API process: sleep `pause_ms` after every `batch` files touched. pause_ms=0 disables.
download_gc_sweep_pause_ms: int = 20
download_gc_sweep_batch: int = 400
# Total cache size cap across ALL users (bytes). When >0 and exceeded, the GC evicts the
# least-recently-accessed ready assets until under it. 0 = unlimited (disk is the limit).
download_total_max_bytes: int = 0
@@ -193,6 +208,10 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -232,14 +251,34 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
plex_watch_sync_interval_min: int = 30
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
# that catches what the incremental history feed can't — notably an un-watch done on the Plex
# side — so it runs far less often (daily) than the incremental pass.
plex_watch_reconcile_interval_min: int = 1440
# Max concurrent transcodes for the P3 fallback. Low by default — CPU-only transcode is
# expensive; direct-serve (browser-compatible files) has no such limit.
# Max concurrent transcode sessions (playable="transcode"). Low by default — the software-fallback
# re-encode is CPU-heavy (the VAAPI hardware path is far cheaper and could sustain more); direct-serve
# and remux (stream-copy) are not capped by this.
plex_max_transcodes: int = 4
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
plex_transcode_hw: str = "auto"
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
# encodes at source resolution.
# Loudness normalisation of the downmixed stereo audio on the HLS path (EBU R128 `loudnorm`).
# OFF by default: it makes a film ~11 dB louder and narrows its dynamic range, which is a taste
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
plex_audio_loudnorm: bool = False
plex_transcode_preset: str = "veryfast"
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
+10
View File
@@ -0,0 +1,10 @@
"""E24 — the read-model behind the Home surface ("Mit nézzek most?").
The anti-paralysis answer layer: one hero suggestion, a cross-source resume rail, and the user's own
comfort shelf, all served from our own Postgres with no upstream call at serve time (the same rule
the Plex Discover surface follows).
Layering, mirroring `app/plex/`: `budget` (the time bands) and `cards` (the unified row shape) are
pure; `queries` holds the SQL; `rails`, `pins` and `hero` compose. Nothing here imports from
`app.routes` — `routes/discover.py` is a thin HTTP shell over these.
"""
+62
View File
@@ -0,0 +1,62 @@
"""The time budget — E24's replacement for "what mood are you in?".
`random-gondolatok` 15/D asked for a mood questionnaire before the recommendations. The same file's
later BA-interview note (#23) says the mood half is not what makes this work: the friction is
deciding at all. A mood→genre mapping would also be invented data — we store no mood signal and never
will — whereas "how much time do you have" is a question the library can actually answer, from a
column we already have.
So the surface asks about TIME, and the answer is deterministic: four bands, no model, no latency.
The band also picks the source for free — ten minutes is a YouTube video, an evening is a movie —
which is exactly the cross-source behaviour the Home surface exists to provide.
The bands overlap on purpose. "I have 30 minutes" should not be answered with a two-minute Short just
because it fits; a lower bound is as much a part of the promise as the upper one.
"""
from typing import Literal
Budget = Literal["short", "medium", "long", "any"]
DEFAULT_BUDGET: Budget = "any"
# budget -> (min_seconds, max_seconds); None = unbounded on that side.
BANDS: dict[str, tuple[int | None, int | None]] = {
"short": (None, 12 * 60), # "10 perc" — a bit of slack over the round number the chip shows
"medium": (8 * 60, 40 * 60), # "30 perc"
"long": (40 * 60, None), # "Egy este"
"any": (None, None), # "Mindegy"
}
def band(budget: str) -> tuple[int | None, int | None]:
"""The (min, max) seconds for a budget. Unknown budgets fall back to the unbounded band rather
than raising: the route validates the vocabulary with a Literal, so anything reaching here with a
bad value is our own bug, and an unfiltered shelf is a better failure than a 500 on the landing
page."""
return BANDS.get(budget, BANDS["any"])
def remaining_seconds(duration_seconds: int | None, position_seconds: int) -> int | None:
"""What is actually LEFT to watch. The question is "how much time do you have", so a 90-minute
film with 8 minutes to go belongs in the ten-minute band, not the evening one. Unknown duration
stays unknown — see `fits`."""
if duration_seconds is None:
return None
return max(0, duration_seconds - max(0, position_seconds))
def fits(seconds: int | None, budget: str) -> bool:
"""Does a runtime fit the budget?
An unknown duration fits only the unbounded band. We cannot promise an item finishes inside ten
minutes when we do not know how long it is, and quietly including it would make the one control
on this page dishonest — the failure mode the house rules call an un-honest state."""
lo, hi = band(budget)
if seconds is None:
return lo is None and hi is None
if lo is not None and seconds < lo:
return False
if hi is not None and seconds > hi:
return False
return True
+141
View File
@@ -0,0 +1,141 @@
"""The unified discover card: one row shape the Home surface can render without knowing which
module it came from.
This is deliberately a THIN mapping, not a new serializer layer. The per-source projections already
exist and are already the shapes the frontend's `VideoCard` and `PlexPosterCard` consume —
`feedquery.serialize` for YouTube, `plex.cards.movie_card`/`show_card`/`leaf_card` for Plex — so each
unified card carries the native payload under `video` / `plex` alongside the handful of fields the
rail chrome itself needs (title, thumbnail, progress, runtime). The Home page therefore renders with
the existing card components rather than a third one, and a field added to a feed row or a Plex card
still gets added in exactly one place. CLAUDE.md's "not wanted" list names a base serializer for
precisely this reason.
`aspect` is the one genuinely new field. A rail has to know whether a tile is a 16:9 still or a 2:3
poster, and that is a property of the ARTWORK, not of the module: a Plex episode ships a landscape
still exactly like a YouTube video does, while a movie and a show ship posters. Emitting it lets the
rail layer stay domain-free instead of branching on `source`.
`subtitle` only ever carries DATA — a channel name, a show title, a year. Nothing here composes an
English phrase: this repo is HU+EN with zero key drift, and a sentence assembled in the backend is a
string no translation file can reach. Anything that needs wording (season counts, "N videó") is left
to the client, which has both the native payload and the translations.
"""
from app.discover.budget import remaining_seconds
from app.models import Channel, Playlist
# Plex artwork is a poster for the things you browse and a landscape still for the things you play.
_POSTER_KINDS = ("movie", "show")
def _base(
*,
source: str, # "youtube" | "plex"
kind: str, # "video" | "movie" | "show" | "episode" | "channel" | "playlist"
id: str,
title: str | None,
aspect: str, # "wide" (16:9) | "poster" (2:3)
subtitle: str | None = None,
thumb: str | None = None,
duration_seconds: int | None = None,
position_seconds: int = 0,
status: str = "new",
) -> dict:
return {
"source": source,
"kind": kind,
"id": id,
"title": title or "",
"subtitle": subtitle,
"thumb": thumb,
"aspect": aspect,
"duration_seconds": duration_seconds,
"position_seconds": position_seconds,
"remaining_seconds": remaining_seconds(duration_seconds, position_seconds),
"status": status,
# Native payloads — at most one is ever set. The client opens the player from these.
"video": None,
"plex": None,
}
def key(card: dict) -> str:
"""The identity of a card ACROSS sources — `<source>:<id>`. A Plex rating_key and a YouTube video
id could in principle collide as bare strings, and the reroll's exclusion list has to be able to
say "not that one" about either."""
return f"{card['source']}:{card['id']}"
def from_video(row: dict) -> dict:
"""`row` is a `feedquery.serialize` dict — the same object the feed grid renders."""
card = _base(
source="youtube",
kind="video",
id=row["id"],
title=row["title"],
aspect="wide",
subtitle=row.get("channel_title"),
thumb=row.get("thumbnail_url"),
duration_seconds=row.get("duration_seconds"),
position_seconds=row.get("position_seconds") or 0,
status=row.get("status") or "new",
)
card["video"] = row
return card
def from_plex(row: dict) -> dict:
"""`row` is a `plex.cards` dict (movie / show / leaf). Its own `type` is the kind."""
kind = row.get("type") or "movie"
if kind == "episode":
subtitle = row.get("show_title")
elif kind == "show":
subtitle = None # season count lives in the native payload; the client words it
else:
subtitle = str(row["year"]) if row.get("year") else None
card = _base(
source="plex",
kind=kind,
id=row["id"],
title=row.get("title"),
aspect="poster" if kind in _POSTER_KINDS else "wide",
subtitle=subtitle,
thumb=row.get("thumb"),
duration_seconds=row.get("duration_seconds"),
position_seconds=row.get("position_seconds") or 0,
status=row.get("status") or "new",
)
card["plex"] = row
return card
def from_channel(ch: Channel, unwatched: int) -> dict:
"""A pinned channel. It is not itself playable — the shelf resolves it to its newest unwatched
video on click (see `pins._shelf_card`) — so the tile advertises what is waiting there instead of a
runtime it does not have."""
card = _base(
source="youtube",
kind="channel",
id=ch.id,
title=ch.title,
aspect="wide",
thumb=ch.thumbnail_url,
)
card["unwatched_count"] = unwatched
return card
def from_playlist(pl: Playlist, item_count: int, thumb: str | None) -> dict:
"""A pinned playlist. `thumb` is its first item's thumbnail — a shelf of blank tiles would defeat
the point of a surface you are supposed to be able to scan at a glance."""
card = _base(
source="youtube",
kind="playlist",
id=str(pl.id),
title=pl.name,
aspect="wide",
thumb=thumb,
)
card["item_count"] = item_count
return card
+104
View File
@@ -0,0 +1,104 @@
"""The hero: the single suggestion the Home surface leads with.
One rule, in one function, because the page's entire promise is that it answers rather than offers.
Three tiers, tried in order:
1. **resume** — something already started whose REMAINING time fits the budget. Almost always the
right answer: it is the thing the user themselves chose most recently, and finishing it costs no
decision at all.
2. **comfort** — a pinned shelf entry that fits. The user's own declared fallback, which beats
anything we could infer.
3. **surprise** — a seeded draw from what is unseen and fits.
That order is the PAGE's answer. "Mást kérek" rotates it (`start`), so each press offers a different
KIND of answer rather than the next row of the same list — the user's call, 2026-09-06, after the
button was measured cycling nine resume cards and never once reaching the shelf.
The tier travels with the card so the client can label it honestly ("Ott hagytad abba" vs "A
polcodról" vs "Meglepetés") instead of guessing from the shape of the payload — a kicker that says
"where you left off" over a film the user has never opened is exactly the kind of quiet lie the house
rules call a dishonest state.
Budgets are matched against REMAINING time throughout (`budget.remaining_seconds`, which equals the
runtime for anything not yet started): with eight minutes left, a 90-minute film is a ten-minute
answer.
"""
from sqlalchemy.orm import Session
from app.discover import cards, pins, rails
from app.discover.budget import fits
from app.models import User
HERO_TIERS = ("resume", "comfort", "surprise")
# How deep the surprise tier draws before giving up. Deliberately deeper than one, so an excluded
# pick moves the tier along instead of emptying it (the shuffle is seeded — the same seed always
# hands back the same order, so without the headroom "already shown" would mean "nothing left").
SURPRISE_DRAW = 12
def pick(
db: Session,
user: User,
budget: str,
seed: int,
*,
resume: list[dict] | None = None,
shelf: list[dict] | None = None,
exclude: set[str] | None = None,
start: int = 0,
) -> dict | None:
"""`{"tier": ..., "card": ...}`, or None when nothing at all fits.
`resume` and `shelf` may be passed pre-computed so the consolidated `/api/discover` response does
not build the same two rails twice; the reroll endpoint passes neither and pays only for what it
reaches.
`exclude` is what makes "Mást kérek" mean something. A new seed alone only re-draws the SURPRISE
tier: the resume tier deterministically returns the most recently progressed thing that fits, so
pressing reroll on a user who has something half-finished handed back the identical card and the
button looked dead (measured in the browser, 2026-09-06). Skipping the cards already shown makes
every press produce a different answer, at every tier.
`start` ROTATES which tier is tried first — 0 keeps the priority order (the page's own first
answer), 1 leads with comfort, 2 with surprise. Exclusion alone was not enough: the resume rail
holds up to 20 rows, so the button walked down it and never reached the shelf or a surprise at
all (measured, 2026-09-06: twelve presses returned nine cards, every one of them `resume`). The
fallthrough is kept, so a press does not come back empty just because the tier it led with has
nothing to offer."""
skip = exclude or set()
def _resume() -> dict | None:
for card in rails.continue_watching(db, user) if resume is None else resume:
playable = rails.playable(db, user.id, card)
# Excluded on the card the USER saw (the show), not only on what it resolves to, so a
# rerolled-past show does not come straight back as its own next episode.
if cards.key(card) in skip or (playable is not None and cards.key(playable) in skip):
continue
if playable is not None and fits(playable["remaining_seconds"], budget):
return playable
return None
def _comfort() -> dict | None:
for card in pins.playables(pins.shelf(db, user) if shelf is None else shelf):
if cards.key(card) not in skip and fits(card["remaining_seconds"], budget):
return card
return None
def _surprise() -> dict | None:
# A wider draw than the one card we need: the shuffle is seeded, so without the headroom an
# excluded pick would leave the tier empty rather than move past it.
for card in rails.surprise(db, user, budget, seed, limit=SURPRISE_DRAW):
if cards.key(card) not in skip:
return card
return None
tiers = {"resume": _resume, "comfort": _comfort, "surprise": _surprise}
offset = start % len(HERO_TIERS)
for tier in HERO_TIERS[offset:] + HERO_TIERS[:offset]:
card = tiers[tier]()
if card is not None:
return {"tier": tier, "card": card}
return None
+173
View File
@@ -0,0 +1,173 @@
"""The comfort shelf: pin a thing you can always fall back to, and have it be one click from playing.
Five kinds are pinnable — a video, a YouTube channel, a Plex show, a Plex movie, a playlist — because
comfort media is almost always "put on X", a series or a channel, and only sometimes one particular
video. `models.ComfortPin` carries that union as five nullable FKs under a `num_nonnulls(...) = 1`
check; this module is where the union is spoken about in one vocabulary, so no call site invents a
second one.
Every shelf card carries a resolved `next`: the thing that actually plays when you click the tile —
a show's in-progress-or-next episode, a channel's newest unwatched video, a playlist's first
unwatched item. It costs a query per non-leaf pin, which is affordable precisely because this shelf is
short and hand-curated, and it buys the property the whole surface exists for: one click, no
intermediate page, no second decision.
Domain module, so nothing here raises HTTP. `resolve_ref` returns None for "no such thing, or not
yours" and the route turns that into its 404 — core never imports routes.
"""
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.discover import cards, queries
from app.models import Channel, ComfortPin, PlexItem, PlexShow, Playlist, User, Video
from app.plex import rollup
from app.plex.cards import leaf_card, show_card
from app.plex.lookups import by_rating_key
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
PIN_COLUMNS: dict[str, str] = {
"video": "video_id",
"channel": "channel_id",
"plex_item": "plex_item_id",
"plex_show": "plex_show_id",
"playlist": "playlist_id",
}
PIN_KINDS = tuple(PIN_COLUMNS)
def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, object] | None:
"""Translate the client's (kind, ref) into the (column, value) the row stores, or None.
The client speaks the ids the rest of the API speaks: a video id, a channel id, a Plex
`rating_key`, a playlist id. Plex's internal integer keys never leave the backend, so the
translation belongs here rather than in five call sites.
The COLUMN half of every answer comes from `PIN_COLUMNS`; only the LOOKUP differs per kind
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before the
E24 review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
all — a constant that called itself the one vocabulary while nothing spoke it."""
col = PIN_COLUMNS.get(kind)
if col is None:
return None
if kind == "video":
return (col, ref) if db.get(Video, ref) is not None else None
if kind == "channel":
return (col, ref) if db.get(Channel, ref) is not None else None
if kind == "plex_item":
it = by_rating_key(db, PlexItem, ref)
return (col, it.id) if it is not None else None
if kind == "plex_show":
sh = by_rating_key(db, PlexShow, ref)
return (col, sh.id) if sh is not None else None
try:
pid = int(ref)
except ValueError:
return None
pl = db.get(Playlist, pid)
# A playlist is per-user, unlike the other four (which are shared catalog rows whose PRIVACY
# lives in the state tables). Someone else's list must read as "no such thing", not as a 403 —
# the same reason `owned_or_404` 404s rather than 403s.
return (col, pl.id) if pl is not None and pl.user_id == user.id else None
def list_pins(db: Session, user: User) -> list[ComfortPin]:
return list(
db.execute(
select(ComfortPin)
.where(ComfortPin.user_id == user.id)
.order_by(ComfortPin.position.asc(), ComfortPin.id.asc())
)
.scalars()
.all()
)
def next_position(db: Session, user: User) -> int:
"""New pins land at the end of the shelf.
`max(position) + 1`, not the row count. Positions are never renumbered on removal — `remove_pin`
deletes the row, and every FK cascade silently removes a pin when the thing it points at goes
away — so past the first hole a count under-counts and would drop the new pin into the MIDDLE of
the shelf, or onto a position another pin already holds (nothing in the schema forbids that).
Same idiom as `routes/playlists._next_playlist_position`."""
return (
db.scalar(
select(func.coalesce(func.max(ComfortPin.position), -1)).where(
ComfortPin.user_id == user.id
)
)
+ 1
)
def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
"""One pin as a unified card with its resolved `next`. None when the pinned row has gone (the FK
cascade removes the pin itself, so this is only the window between a delete and a refetch)."""
card: dict | None = None
nxt: dict | None = None
if pin.video_id is not None:
row = queries.yt_video_row(db, user, pin.video_id)
if row is None:
return None
card = cards.from_video(row)
nxt = dict(card) # a video plays itself
elif pin.channel_id is not None:
ch = db.get(Channel, pin.channel_id)
if ch is None:
return None
card = cards.from_channel(ch, queries.channel_unwatched_count(db, user, ch.id))
row = queries.channel_next_video(db, user, ch.id)
nxt = cards.from_video(row) if row else None
elif pin.plex_item_id is not None:
it = db.get(PlexItem, pin.plex_item_id)
if it is None:
return None
st = rollup.item_state_map(db, user.id, [it.id]).get(it.id)
card = cards.from_plex(leaf_card(it, st))
nxt = dict(card) # a movie or a single episode plays itself
elif pin.plex_show_id is not None:
sh = db.get(PlexShow, pin.plex_show_id)
if sh is None:
return None
status = rollup.state_map(db, user.id, [sh.id]).get(sh.id, "new")
card = cards.from_plex(show_card(sh, status))
ep = queries.next_episode(db, user.id, sh.id)
nxt = cards.from_plex(ep) if ep else None
elif pin.playlist_id is not None:
pl = db.get(Playlist, pin.playlist_id)
if pl is None:
return None
count, row = queries.playlist_summary(db, user, pl.id)
card = cards.from_playlist(pl, count, row.get("thumbnail_url") if row else None)
nxt = cards.from_video(row) if row else None
if card is None:
return None
# A COPY, never the card itself: a self-playing pin (a video, a movie) would otherwise make
# `card["next"] is card`, which serializes as a circular reference and 500s the whole shelf. The
# copy is taken before `pin_id`/`next` are attached, so `next` stays a plain playable card.
card["pin_id"] = pin.id
card["next"] = nxt
return card
def shelf(db: Session, user: User) -> list[dict]:
"""The whole shelf, in the user's own order."""
out = []
for pin in list_pins(db, user):
card = _shelf_card(db, user, pin)
if card is not None:
out.append(card)
return out
def playables(shelf_cards: list[dict]) -> list[dict]:
"""The `next` of every shelf entry that has one — what the hero may draw from. A pinned channel
with nothing unwatched left, or an empty playlist, simply is not an answer right now."""
return [c["next"] for c in shelf_cards if c.get("next")]
+348
View File
@@ -0,0 +1,348 @@
"""The row fetchers behind the Home surface — one place where "what is startable right now" is
expressed as SQL, for both sources.
Two rules shape this module:
**Reuse the query layer, not the route.** The YouTube half calls `feedquery.filtered_query` /
`sort_keys`, so the Home surface inherits the feed's own definitions of subscribed, blocked,
explore-visible and in-progress for free, and cannot drift from them. The Plex half calls
`plex.rollup` and the `plex.cards` serializers rather than `routes/plex/discover.py`: that route
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
sidebar and wrong for a landing page that does not.
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
here takes the feed's seeded-md5 idiom instead of `random()`.
"""
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.discover.budget import band
from app.feedquery import (
feed_columns,
filtered_query,
not_blocked_cond,
order_by,
serialize,
sort_keys,
)
from app.models import (
Channel,
PlaylistItem,
PlexItem,
PlexShow,
PlexState,
User,
Video,
VideoState,
)
from app.plex import rollup
from app.plex.cards import leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
# How many rows a single rail may carry. Matches `routes/plex/discover.py`'s RAIL_LIMIT on purpose —
# the two surfaces should feel like the same app, and a rail is a rail.
RAIL_LIMIT = 20
# The feed parameters a Home rail runs with. Spelled out rather than borrowed from the route's
# `_feed_params` defaults, because those are an HTTP contract that may change for the feed's own
# reasons; the landing page's scope should not move with them silently.
_FEED_BASE: dict = {
"tags": [],
"tag_modes": {},
"untagged": False,
"channel_ids": [],
"q": None,
"min_duration": None,
"max_duration": None,
"max_age_days": None,
"published_after": None,
"published_before": None,
"show_normal": True,
"include_shorts": False,
"include_live": False,
"show": "unwatched",
"scope": "my",
"library_source": "organic",
}
def _seeded(col, seed: int):
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
return func.md5(func.concat(col, str(seed)))
# ── YouTube ───────────────────────────────────────────────────────────────────────────────────────
def yt_in_progress(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
"""Started-but-not-finished videos, most recently progressed first, as (progressed_at, row).
The predicate is `feedquery`'s `show="in_progress"` verbatim (a stored resume position, status not
watched/hidden) — but the query is built here rather than through `filtered_query` because this
rail wants an ordering by `progress_updated_at`, and that column lives on the state alias the feed
query keeps private. Driving from `video_states` is also the faster plan: it is a per-user table of
a few hundred rows, against a catalog of many thousands.
Building it here means the feed's OTHER predicates have to be chosen deliberately rather than
inherited. `not_blocked_cond` is shared, because "never show me this channel" outranks "you
started it" — blocking leaves the state row untouched, so without this the video the user just
banned heads the resume rail. The feed's SCOPE clauses (subscribed / via_explore accessibility)
are deliberately NOT applied, for the reason `yt_video_row` gives: something you have already
started must stay resumable even after you unsubscribe or an exploration expires."""
st = aliased(VideoState)
q = (
select(*feed_columns(user, st), st.progress_updated_at.label("progressed_at"))
.join(Channel, Channel.id == Video.channel_id)
.join(st, and_(st.video_id == Video.id, st.user_id == user.id))
.where(
st.position_seconds > 0,
st.status.notin_(("watched", "hidden")),
Video.unavailable_since.is_(None),
not_blocked_cond(user),
)
.order_by(st.progress_updated_at.desc().nullslast())
.limit(limit)
)
return [(r.progressed_at, serialize(r)) for r in db.execute(q).all()]
def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) -> list[dict]:
"""Unwatched videos from the user's own feed scope, in seeded-shuffle order, inside the budget.
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
actually a good response to, and the feed's own default keeps them out otherwise."""
lo, hi = band(budget)
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("shuffle", seed, rank_expr)
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
# go: the resume tier has already had its chance at them, and it may have passed precisely because
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
out = [serialize(r) for r in rows]
return [r for r in out if not r["position_seconds"]][:limit]
def channel_next_video(db: Session, user: User, channel_id: str) -> dict | None:
"""The newest unwatched video of a pinned channel — what "put on that channel" resolves to."""
# `library_source="all"`: a pinned channel means the whole channel, including videos that only
# ever arrived through a search. The feed's "organic" default is a feed-scoping choice, not a
# statement about what belongs to a channel.
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("newest", 0, rank_expr)
row = db.execute(query.order_by(*order_by(keys)).limit(1)).first()
return serialize(row) if row is not None else None
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
"""One video as a feed row, with this user's watch state.
Built directly rather than through `filtered_query` on purpose: a PINNED video must keep working
even when it falls outside the feed's scope — the user unsubscribed from the channel, or blocked
it, or the video only ever arrived through a search. The shelf is the surface you fall back to;
it silently losing an entry because a filter moved is exactly the failure it must not have."""
st = aliased(VideoState)
row = db.execute(
select(*feed_columns(user, st))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.where(Video.id == video_id)
).first()
return serialize(row) if row is not None else None
def channel_unwatched_count(db: Session, user: User, channel_id: str) -> int:
"""How many unwatched videos a pinned channel is holding — the tile's only honest promise, since a
channel has no runtime of its own."""
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, _ = filtered_query(db, user, **params)
return int(db.execute(select(func.count()).select_from(query.subquery())).scalar() or 0)
def playlist_summary(db: Session, user: User, playlist_id: int) -> tuple[int, dict | None]:
"""(item count, the row to play next) for a pinned playlist — its first not-yet-watched item, or
its first item once the whole list is watched. The same row doubles as the tile's artwork, so a
shelf of playlists is scannable instead of a row of blank boxes."""
st = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, st), PlaylistItem.position)
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.where(PlaylistItem.playlist_id == playlist_id)
.order_by(PlaylistItem.position.asc())
).all()
if not rows:
return 0, None
nxt = next((r for r in rows if (r.status or "new") not in ("watched", "hidden")), rows[0])
return len(rows), serialize(nxt)
# ── Plex ──────────────────────────────────────────────────────────────────────────────────────────
def _unseen_movie_conds(uid: int):
"""A movie the user has neither finished nor hidden. A LEFT-JOIN-IS-NULL would also drop rows that
merely carry a resume position, which are exactly the ones Continue watching wants back."""
seen = (
select(1)
.select_from(PlexState)
.where(
PlexState.item_id == PlexItem.id,
PlexState.user_id == uid,
or_(
PlexState.status.in_(("watched", "hidden")),
# Started counts as "not a surprise" too — see the note in `yt_surprise`.
PlexState.position_seconds > 0,
),
)
.exists()
)
return ~seen
def plex_in_progress(db: Session, uid: int, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
"""In-progress movies + partially-watched shows as (progressed_at, card).
Mirrors `routes/plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
NULL status still counts as started, so `notin_` alone would drop it), and the same shared
`rollup.agg_subquery` + `apply_show_state_filter` for shows, so the two rails cannot disagree on
what "in progress" means. What it does NOT inherit is that route's `LibraryFilters` narrowing:
this surface has no Plex sidebar to read a filter from."""
movie_libs, show_libs = content_libs(db)
out: list[tuple[object, dict]] = []
if movie_libs:
mq = (
select(PlexItem, PlexState)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
PlexState.position_seconds > 0,
or_(
PlexState.status.is_(None),
PlexState.status.notin_(("watched", "hidden")),
),
)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(limit)
)
for it, st in db.execute(mq).all():
out.append((st.progress_updated_at, movie_card(it, st)))
if show_libs:
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at).join(prog, prog.c.show_id == PlexShow.id),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_libs))
for sh, resumed_at in db.execute(
sq.order_by(prog.c.resumed_at.desc().nullslast()).limit(limit)
).all():
out.append((resumed_at, show_card(sh, "in_progress")))
return out
def plex_surprise(db: Session, uid: int, budget: str, seed: int, limit: int) -> list[dict]:
"""Unseen playable movies inside the budget, in seeded-shuffle order.
Movies only. A show is not a thing you can start playing, and resolving every candidate show to
its next episode just to draw three cards would be a query per row for a rail whose whole value is
that it answers instantly — shows reach this surface through Continue watching and the comfort
shelf, which is where "put the next episode on" actually belongs.
No `playable` filter, matching the Plex surprise rail: `PlexItem.playable` is a playback CLASS
("direct" / "remux" / "transcode"), not a yes/no, and NULL there means "not probed yet" rather than
"will not play". Excluding un-probed items would quietly hide a fresh library from the one surface
that is supposed to have an answer. The class travels on the card, so the client still shows it."""
movie_libs, _ = content_libs(db)
if not movie_libs:
return []
lo, hi = band(budget)
q = (
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
_unseen_movie_conds(uid),
)
.order_by(_seeded(PlexItem.rating_key, seed))
.limit(limit)
)
if lo is not None:
q = q.where(PlexItem.duration_s >= lo)
if hi is not None:
q = q.where(PlexItem.duration_s <= hi)
items = db.execute(q).scalars().all()
states = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, states.get(it.id)) for it in items]
# Watch states that take an episode out of the running for "what would play next".
_DONE = ("watched", "hidden")
def next_episode(db: Session, uid: int, show_id: int) -> dict | None:
"""What "put that show on" means: the episode in progress, else the first unwatched one, else the
first — over the same `(season, episode)` broadcast ordering `detail.py::_episode_nav` uses.
Three ordered `LIMIT 1` probes rather than one scan picked apart in Python. The Python version
loaded EVERY episode of the show to choose one of them — 236 rows for Friends, 227 for JAG — which
the comfort shelf could just about afford (a handful of hand-picked pins) and the resume rail
could not, since it resolves a show per rail row on the landing page.
Returned as a `leaf_card` so it carries the show title the episode row itself does not."""
asc = (
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
base = (
select(PlexItem, PlexState)
.outerjoin(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
.limit(1)
)
# In progress — the LAST one in broadcast order, which is `plex.rollup.rollup`'s `inprog[-1]`,
# i.e. what the show page's own Resume button plays. Two episodes of one show are in progress at
# once more often than it sounds (`watch_sync._reconcile_inprogress_movies` documents that a
# scrub to the end leaves a stale `status='new', position>0` row until the daily reconcile), and
# taking opposite ends of that set is how one show came to have two different "continue" answers:
# the Home tile ranked it by its LATEST touch and then offered the EARLIEST started episode.
# `position_seconds > 0` already requires the state row to exist, so no null guard.
pick = db.execute(
base.where(PlexState.position_seconds > 0, PlexState.status.notin_(_DONE)).order_by(
PlexItem.season_number.desc().nullslast(),
PlexItem.episode_number.desc().nullslast(),
)
).first()
# Unwatched — INCLUDING an episode with no state row at all, which is the common case and the one
# a bare `status NOT IN (...)` silently drops (NULL is not false, it is unknown, so the row fails
# the filter).
pick = (
pick
or db.execute(
base.where(or_(PlexState.id.is_(None), PlexState.status.notin_(_DONE))).order_by(*asc)
).first()
)
# Fully watched show: start it again from the top.
pick = pick or db.execute(base.order_by(*asc)).first()
if pick is None:
return None
ep, st = pick
show = db.get(PlexShow, show_id)
return leaf_card(
ep, st, show_title=show.title if show else None, show_rk=show.rating_key if show else None
)
+97
View File
@@ -0,0 +1,97 @@
"""The Home surface's rails: cross-source lists, assembled from `queries.py` and emitted as unified
cards.
The whole point of this module is that a rail is ONE list, not a YouTube list beside a Plex list. Two
side-by-side rails would just move the choice the user could not make ("which half do I look at?") one
level up, which is the opposite of what this epic is for.
"""
from sqlalchemy.orm import Session
from app import sysconfig
from app.discover import cards, queries
from app.discover.queries import RAIL_LIMIT
from app.models import PlexShow, User
from app.plex.lookups import by_rating_key
from app.plex.rollup import NO_TS
def plex_available(db: Session) -> bool:
"""Plex is a deployment-level toggle, not a per-user one (see `routes/me.py`). With it off, the
Home rails are YouTube-only and say nothing about Plex."""
return sysconfig.get_bool(db, "plex_enabled")
def playable(db: Session, uid: int, card: dict) -> dict | None:
"""The thing that would actually START from this card — itself for anything with a runtime, and
for a Plex SHOW the episode that is next (in progress, else the first unwatched, else the first).
ONE function, because three surfaces need the same answer and a second implementation is how the
shelf and the rail came to disagree in the first place: the hero calls it to decide what it is
offering, `continue_watching` calls it to fill each show tile's `next`, and the comfort shelf
resolves the same way in `pins`. None when the show has no episodes mirrored yet.
A card that already carries a resolved `next` (a rail row) is taken at its word rather than
re-queried — the hero reads the rail it was handed."""
if not (card.get("source") == "plex" and card.get("kind") == "show"):
return card
if "next" in card:
return card["next"]
sh = by_rating_key(db, PlexShow, card["id"])
if sh is None:
return None
ep = queries.next_episode(db, uid, sh.id)
return cards.from_plex(ep) if ep else None
def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[dict]:
"""One resume rail across both sources, most recently progressed first.
A Plex SHOW enters as a show CARD — that is what the Plex rail shows, and it is the artwork the
user recognises — but it carries `next`: the episode that would actually start. Without it a
click on a half-watched series landed on the show page with six seasons to choose between, which
is the decision this whole surface exists to remove (the comfort shelf and the hero already
resolved; only this rail did not). The client plays `next ?? card`, so the tile stays a poster and
the click stays one click."""
rows: list[tuple[object, dict]] = [
(ts, cards.from_video(r)) for ts, r in queries.yt_in_progress(db, user, limit)
]
if plex_available(db):
rows += [(ts, cards.from_plex(c)) for ts, c in queries.plex_in_progress(db, user.id, limit)]
rows.sort(key=lambda r: r[0] or NO_TS, reverse=True)
out = [card for _, card in rows[:limit]]
# Resolve AFTER the slice: only the rows that survive it are worth a query.
for card in out:
if card["kind"] == "show":
card["next"] = playable(db, user.id, card)
return out
def surprise(db: Session, user: User, budget: str, seed: int, limit: int = 3) -> list[dict]:
"""The hero's draw pool: unseen things that fit the budget, from both sources under ONE seed.
Interleaved rather than concatenated — the same alternation `routes/plex/discover.py` uses for its
movie/show split — so a short pool never comes back all-YouTube or all-Plex just because one side
happened to be queried first.
WHICH source leads is decided by the seed, not by a hardcoded order. The hero draws a pool of ONE,
so a fixed order would mean the Plex library could never answer "egy este" while a single YouTube
video also fit — the reroll would look broken and the cross-source promise would quietly be a
YouTube promise. Parity on the same seed that orders the rows keeps it deterministic: the same
seed always gives the same answer, and pressing "Mást kérek" moves both halves together."""
halves = [
(cards.from_video, queries.yt_surprise(db, user, budget, seed, limit)),
(
cards.from_plex,
queries.plex_surprise(db, user.id, budget, seed, limit) if plex_available(db) else [],
),
]
if seed % 2:
halves.reverse()
out: list[dict] = []
for i in range(max(len(rows) for _, rows in halves)):
for to_card, rows in halves:
if i < len(rows):
out.append(to_card(rows[i]))
return out[:limit]
+258
View File
@@ -0,0 +1,258 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
registry via the netloc-scoped `_host_is`, so a cookie is only ever offered to the exact site it
belongs to; an arbitrary `?url=` (or a look-alike host) resolves to no host_key and gets no
cookie.
* **secret handling** — cookies are Fernet-encrypted at rest; the blob is only ever decrypted
in-worker into a 0600 file that is unlinked in a `finally`, and is never returned by a route,
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.security import decrypt_optional, encrypt
log = logging.getLogger("siftlode.downloads.credentials")
class HostSpec:
"""A supported cookie host: a short canonical `key` (stored on the credential + folded into the
cache salt), a human `label` for the UI, and the netloc `domains` it matches."""
__slots__ = ("key", "label", "domains")
def __init__(self, key: str, label: str, domains: tuple[str, ...]):
self.key = key
self.label = label
self.domains = domains
# Curated registry (user's call: seed the common auth-walled hosts now). One `key` can cover several
# domains of the same site. Add a host by adding a row — the FE dropdown, host-scope guard, and salt
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
HostSpec("reddit", "Reddit", ("reddit.com",)),
)
_BY_KEY: dict[str, HostSpec] = {h.key: h for h in SUPPORTED_HOSTS}
def is_supported_host(host_key: str) -> bool:
return host_key in _BY_KEY
def host_label(host_key: str) -> str:
spec = _BY_KEY.get(host_key)
return spec.label if spec else host_key
def host_key_for_url(url: str | None) -> str | None:
"""The supported host_key whose domains own `url`'s netloc, or None. Netloc-scoped (via
`_host_is`), so a look-alike host ('myfacebook.com', 'facebook.com.evil') never matches. When
several specs could match, the most-specific (longest matched domain) wins so the result is
deterministic across the enqueue and worker call sites."""
if not url:
return None
from urllib.parse import urlparse
host = urlparse(url).netloc.lower()
if not host:
return None
best_key: str | None = None
best_len = -1
for spec in SUPPORTED_HOSTS:
for dom in spec.domains:
if _host_is(host, dom) and len(dom) > best_len:
best_key, best_len = spec.key, len(dom)
return best_key
def cookie_salt(user_id: int, host_key: str) -> str:
"""The cache-signature salt for a cookie-authed download. Keyed by user id + host so the authed
asset is private to that user (never dedup-shared) and stable across that user's own re-adds."""
return f"cookie:{user_id}:{host_key}"
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
`format_sig(spec, salt=...)`."""
host_key = host_key_for_url(url)
if host_key is None:
return None
if get_credential(db, user_id, host_key) is None:
return None
return cookie_salt(user_id, host_key)
# --- storage (CRUD) ---------------------------------------------------------------------------
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
class InvalidCookies(ValueError):
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
def validate_cookies(raw: str) -> str:
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
the Netscape format: comment/`#` lines plus TAB-separated cookie lines (domain, flag, path,
secure, expiry, name, value). We don't fully parse — just reject obvious non-cookie paste (empty,
or no tab-separated data line) so a mistaken paste fails HERE with a clear message rather than
silently producing a cookie file yt-dlp ignores."""
text = (raw or "").strip()
if not text:
raise InvalidCookies("empty")
has_data_line = any(
line.strip() and not line.lstrip().startswith("#") and "\t" in line
for line in text.splitlines()
)
if not has_data_line:
raise InvalidCookies("format")
# Netscape cookie files must end with a newline for some parsers; normalise.
return text if text.endswith("\n") else text + "\n"
def upsert_credential(db: Session, user_id: int, host_key: str, raw: str) -> DownloadCredential:
"""Create or replace this user's cookie blob for `host_key`. Caller has validated host_key is
supported and `raw` via `validate_cookies`. Does NOT commit (the route commits with the audit
row so the trail can't drift). The blob is encrypted here; the plaintext never leaves this call."""
cred = get_credential(db, user_id, host_key)
enc = encrypt(raw)
if cred is None:
cred = DownloadCredential(user_id=user_id, host_key=host_key, cookies_enc=enc)
db.add(cred)
else:
cred.cookies_enc = enc
return cred
def delete_credential(db: Session, user_id: int, host_key: str) -> bool:
"""Delete this user's cookie for `host_key`. Returns whether a row existed. Does NOT commit."""
cred = get_credential(db, user_id, host_key)
if cred is None:
return False
db.delete(cred)
return True
def status_list(db: Session, user_id: int) -> list[dict]:
"""The supported hosts with this user's configured/updated state — the write-only Settings view.
Never includes the blob."""
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
).scalars()
}
out = []
for spec in SUPPORTED_HOSTS:
cred = have.get(spec.key)
out.append(
{
"host_key": spec.key,
"label": spec.label,
"configured": cred is not None,
"updated_at": cred.updated_at.isoformat() if cred is not None else None,
}
)
return out
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
return None # shared asset — never attach a cookie to a non-private identity
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
def _write_cookiefile(raw: str, staging: Path) -> Path:
"""Write the cookie blob to a private 0600 file under `staging/.auth/` and return its path.
Deliberately NOT in the staging root: the download retry loop unlinks every file it finds there,
and `_produced_file` picks the largest file there as the output — either would corrupt a
cookies.txt in the root. A `.auth` SUBDIR survives both (they skip directories) and is still
removed by the parent worker's staging cleanup on a hard kill. The caller unlinks it in a
`finally` on the normal/exception paths."""
auth_dir = staging / ".auth"
auth_dir.mkdir(mode=0o700, exist_ok=True)
path = auth_dir / "cookies.txt"
# Open with 0600 from the start (don't create world-readable then chmod).
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
os.write(fd, raw.encode("utf-8"))
finally:
os.close(fd)
return path
+66 -32
View File
@@ -25,6 +25,7 @@ import hashlib
import json
import logging
import math
import os
import queue
import shutil
import subprocess
@@ -32,15 +33,31 @@ import threading
import time
from pathlib import Path
from app.config import settings
log = logging.getLogger("siftlode.edit")
def encode_thread_cap() -> int:
"""How many encoder threads a browser-safe re-encode may use. `download_ffmpeg_threads` when the
admin pinned one (>0), else reserve a core for foreground work (max(1, cpu_count - 1)). Shared by
the ffmpeg `-threads` flag and the child's `OMP_NUM_THREADS` so both honour the same budget."""
n = settings.download_ffmpeg_threads
if n > 0:
return n
return max(1, (os.cpu_count() or 2) - 1)
class EditAborted(Exception):
"""Raised by run_ffmpeg when the job was paused/canceled mid-encode."""
# --- edit spec ------------------------------------------------------------------------------
# SEC-13: ceiling on a join's cut-list. An accurate join emits two filters per segment into a single
# ffmpeg `filter_complex`, so "however many the caller sent" is an unbounded command line.
_MAX_SEGMENTS = 100
def normalize_edit_spec(spec: dict | None) -> dict:
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
@@ -75,6 +92,12 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if en > st:
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
norm.sort(key=lambda x: x["start_s"])
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
# The cap is far above any real cut-list (the UI's own splitter produces a handful) and the
# excess is DROPPED rather than raising: the surplus segments are the attack, the first
# _MAX_SEGMENTS are a legitimate edit.
del norm[_MAX_SEGMENTS:]
if norm:
out["segments"] = norm
# A join always chooses a codec path: accurate=filter-concat re-encode,
@@ -425,18 +448,42 @@ _BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) out of ffprobe's JSON stream list, lowercased.
JSON, not the flat `default=nw=1` output: there the fields arrive in ffprobe's own struct
order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line "remember the last
codec_type" parser attributes every name to the PREVIOUS stream. Measured on real downloads:
an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download."""
def run_ffprobe_json(
path: Path,
*,
show_streams: bool = False,
show_format: bool = False,
entries: str | None = None,
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the shared media probe (`probe.probe_media`,
used by both the download worker's finalize and the upload ingest) both go through it, so the
subprocess flags/timeout/error handling live in one place instead of drifting copies."""
args = ["ffprobe", "-v", "error"]
if show_streams:
args.append("-show_streams")
if show_format:
args.append("-show_format")
if entries:
args += ["-show_entries", entries]
args += ["-of", "json", str(path)]
try:
streams = json.loads(payload or "{}").get("streams") or []
except ValueError:
return None, None
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
@@ -451,26 +498,10 @@ def parse_probe_streams(payload: str) -> tuple[str | None, str | None]:
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
try:
out = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-show_entries",
"stream=codec_type,codec_name",
"-of",
"json",
str(path),
],
capture_output=True,
text=True,
timeout=30,
check=True,
).stdout
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError):
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return parse_probe_streams(out)
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
@@ -510,11 +541,12 @@ def browser_transcode_flags(
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg."""
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
@@ -534,6 +566,8 @@ def build_browser_safe_cmd(
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
+50
View File
@@ -8,9 +8,28 @@ explanation (and decides whether a retry makes sense). The raw text stays availa
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
"""
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
# does, it must carry the original code rather than re-deriving one from this text.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
_RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -24,6 +43,16 @@ _RULES: tuple[tuple[str, str], ...] = (
# Some extractors (e.g. Reddit) gate content behind a logged-in account: "Account
# authentication is required." — an auth wall we deliberately don't cross, not a transient error.
("authentication is required", "login_required"),
# Closed-group / private-post auth walls (Facebook groups, Instagram private, …). yt-dlp phrases
# these many ways; the common thread is "you must be signed in" or its own suggestion to supply
# cookies. Treating the cookie suggestion as an auth signal is what makes a login-walled link read
# honestly (and stay non-retryable) when the acting user has no usable cookie for that host.
("log in to", "login_required"),
("login required", "login_required"),
("you must be logged in", "login_required"),
("only available for registered users", "login_required"),
("use --cookies", "login_required"),
("--cookies-from-browser", "login_required"),
# Geo-restricted.
("in your country", "geo_blocked"),
("available in your location", "geo_blocked"),
@@ -42,6 +71,10 @@ _RULES: tuple[tuple[str, str], ...] = (
("account associated with this video has been terminated", "unavailable"),
("this video is not available", "unavailable"),
("this video has been removed", "unavailable"),
# yt-dlp doesn't support this site at all ("Unsupported URL: …") — distinct from an auth wall on a
# supported site. Non-retryable: no cookie or retry can make an unsupported host downloadable.
("unsupported url", "unsupported_host"),
("no video could be found in this url", "unsupported_host"),
# DRM.
("drm protected", "drm"),
("drm-protected", "drm"),
@@ -77,3 +110,20 @@ def classify(msg: str | None) -> str:
if marker in low:
return category
return "unknown"
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
stops the fetch aiming at our own network; this stops the remaining readout.
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
function rather than an inline branch in the worker so it can be pinned by a test — the raw
text leaking back is a security regression that would otherwise pass a green suite."""
if source_kind == "url" and code == "network":
return NETWORK_REDACTED_MESSAGE
return msg
+76 -48
View File
@@ -5,13 +5,16 @@ of truth for what file gets produced. `format_sig` hashes exactly the fields tha
OUTPUT BYTES, so two users picking equivalent settings share one cached MediaAsset.
`build_ydl_opts` turns a spec into a yt-dlp options dict (format selector + postprocessors).
spec shape (all keys optional; defaults applied by `normalize`):
spec shape (all keys optional; defaults + browser-playability clamp applied by `normalize`):
mode "av" (audio+video) | "v" (video-only) | "a" (audio-only)
max_height int | null (null = best)
container "mp4" | "mkv" | "webm" | null (merge/remux target for av/v)
audio_format "m4a" | "mp3" | "opus" | null (extract target for mode "a")
vcodec "h264" | "vp9" | "av1" | null (preference, best-effort)
container always coerced to "mp4" (the only universally browser-playable container)
audio_format "m4a" | "mp3" (mode "a" extract target; anything else — e.g. opus — → "m4a")
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import hashlib
@@ -50,10 +53,12 @@ _DEFAULTS = {
# prefer H.264+AAC (browser/iOS-compatible) rather than yt-dlp's default VP9/Opus best.
# v3: mp4 compat now *selects* H.264 (avc1) with a progressive fallback + re-encodes the rare
# AV1/VP9-only result to H.264, so cached AV1 assets must re-derive (fixes iOS broken-player).
_SIG_VERSION = 3
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
_VCODEC_SORT = {"h264": "vcodec:h264", "vp9": "vcodec:vp9", "av1": "vcodec:av01"}
def normalize(spec: dict) -> dict:
@@ -76,6 +81,18 @@ def normalize(spec: dict) -> dict:
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
s["audio_format"] = "m4a"
return s
@@ -98,17 +115,31 @@ def build_ydl_opts(
outtmpl: str,
progress_hook,
extra: dict | None = None,
cookiefile: str | None = None,
max_bytes: int | None = None,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
`extra` is source-specific options contributed by the SourceAdapter (e.g. YouTube's
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic."""
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it.
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
than it rather than filling the disk and only then being caught by the quota accounting."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
"outtmpl": outtmpl,
"noplaylist": True,
# SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to the single video. A
# URL that is PURELY a playlist or channel has no such video to fall back to, so it still
# expands into every entry — one enqueue, an unbounded number of downloads. One job means
# one item; a user who wants the whole list enqueues it item by item, under their quota.
"playlist_items": "1",
"quiet": True,
"no_warnings": True,
"noprogress": True,
@@ -132,6 +163,21 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
#
# A non-positive budget is left unset because 0 is not an expressible yt-dlp filter, NOT because
# nothing needs to happen: the worker refuses that job outright before extraction (`_OwnerRefused`
# → `quota_exceeded`). Enqueue cannot make that call — it ran while the user was still under the
# cap. Delete the worker's refusal and this branch silently uncaps the one user already over it.
if max_bytes and max_bytes > 0:
opts["max_filesize"] = max_bytes
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
if s["mode"] == "a":
opts["format"] = "bestaudio/best"
opts["postprocessors"].append(
@@ -143,51 +189,33 @@ def build_ydl_opts(
)
else:
hcap = f"[height<=?{h}]" if h else ""
# Compatibility-first for mp4 (the universal container): produce H.264 video + AAC audio
# so shared/downloaded files play in EVERY browser, including iOS/Safari. WebKit (which
# every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there, even though Chromium
# plays it fine. A custom profile that explicitly sets `vcodec` opts out.
compat = s["container"] == "mp4" and s["vcodec"] is None
# Every video download is the mp4/H.264 compat profile (normalize() clamps container→mp4,
# vcodec→None), so shared/downloaded files play in EVERY browser, iOS/Safari included. WebKit
# (which every iOS browser, Brave included, is forced to use) decodes only H.264/H.265 + AAC
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
if s["mode"] == "v":
if compat:
# Video-only: keep it audio-free. Prefer an H.264 *video-only* stream, then any
# video-only, and only fall to a progressive (audio-bearing) `best` as a last
# resort — same audio-free-when-possible guarantee as the old `bestvideo/best`.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}"
f"/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}/best{hcap}"
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}/bestvideo{hcap}/best[vcodec^=avc1]{hcap}/best{hcap}"
)
else: # av
if compat:
# Prefer H.264 at the SELECTION level, not just via format_sort: a sort key is only
# a *preference* and can't conjure an H.264 stream a source doesn't expose as
# video-only. A non-YouTube source (e.g. Facebook) may ship AV1 as the best
# video-only stream and H.264 only as a *progressive* (a+v) format — which
# `bestvideo` skips — so `bestvideo+bestaudio` picked AV1 and iOS broke. This chain
# tries avc1 video-only, then the avc1 progressive, then anything.
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
else:
opts["format"] = f"bestvideo{hcap}+bestaudio/best{hcap}"
if s["container"]:
opts["merge_output_format"] = s["container"]
sort = []
# Keep the codec preference as a tiebreaker (ranks H.264/AAC above resolution among the
# selected candidates); the selector above is the hard guarantee, this is belt-and-braces.
if compat:
sort += ["vcodec:h264", "acodec:aac"]
# avc1 video-only + m4a audio, then the avc1 progressive, then anything (re-encoded).
opts["format"] = (
f"bestvideo[vcodec^=avc1]{hcap}+bestaudio[acodec^=mp4a]"
f"/best[vcodec^=avc1]{hcap}"
f"/bestvideo{hcap}+bestaudio/best{hcap}"
)
opts["merge_output_format"] = "mp4"
# Codec preference tiebreaker (ranks H.264/AAC among the selected candidates); the selector
# above is the hard guarantee, this is belt-and-braces.
sort = ["vcodec:h264", "acodec:aac"]
if h:
sort.append(f"res:{h}")
if s["vcodec"] in _VCODEC_SORT:
sort.append(_VCODEC_SORT[s["vcodec"]])
if sort:
opts["format_sort"] = sort
opts["format_sort"] = sort
# SponsorBlock first (marks segments), then strip them from the media + chapters.
if s["sponsorblock"]:
+14
View File
@@ -86,11 +86,25 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
cutoff = time.time() - _ORPHAN_MIN_AGE_S
removed = 0
# Throttle the whole-tree walk: this runs in the API process, and an unpaced rglob+stat over a
# large download root monopolizes disk IO long enough to make web requests + playback stutter.
# Yield the IO channel after every `batch` files touched so foreground work interleaves.
pause_s = max(0, settings.download_gc_sweep_pause_ms) / 1000.0
batch = max(1, settings.download_gc_sweep_batch)
touched = 0
def _throttle() -> None:
nonlocal touched
touched += 1
if pause_s and touched % batch == 0:
time.sleep(pause_s)
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
_throttle()
try:
if f.resolve() in keep or f.stat().st_mtime > cutoff:
continue
+9 -1
View File
@@ -80,9 +80,13 @@ def public_meta(
Resolves the owner's per-download display overrides (custom title/channel) over the asset's
auto-extracted values, and surfaces the source + any extra reference links as clickable URLs."""
from app.downloads import service
from app.downloads import mediakind, service
source_url = service.reference_url(job, asset)
# Audio-only assets (mp3 rips, extracted audio) can't play inside a <video> element on WebKit
# (all iOS browsers), so the watch page renders a native <audio> instead. A generic uploaded
# file (media_kind='file') has no player at all — the watch page shows file info + a download.
# Both branches read the stored media_kind (the single source), not a re-derived codec check.
return {
"title": (job and job.display_name) or asset.title,
"uploader": (job and job.display_uploader) or asset.uploader,
@@ -92,6 +96,10 @@ def public_meta(
"duration_s": asset.duration_s,
"width": asset.width,
"height": asset.height,
"audio": asset.media_kind == mediakind.AUDIO,
"media_kind": asset.media_kind,
"size_bytes": asset.size_bytes,
"container": asset.container,
"allow_download": link.allow_download,
"file_url": file_url,
"poster_url": poster_url,
+23
View File
@@ -0,0 +1,23 @@
"""The media-kind discriminator for a library asset — the single home of one invariant.
`MediaAsset.media_kind` is `"video" | "audio" | "file"`. It decides which library actions apply:
media (video/audio) gets play/edit/share; a generic uploaded file gets download + share only, and
is never rendered inline (see the force-download serving in the file/watch routes). Every finalize
path computes it HERE — the download worker + the upload ingest via `derive`, the editor by
carrying the source's kind over — so the rule lives in code, not in a comment that can drift (an
earlier ad-hoc `vcodec is None and acodec is not None` audio check lived only in links.py).
"""
VIDEO = "video"
AUDIO = "audio"
FILE = "file"
def derive(vcodec: str | None, acodec: str | None) -> str:
"""Media kind from the probed codecs: a video stream -> video, audio-only -> audio, neither
(a non-media upload: pdf/zip/docx/…) -> file."""
if vcodec:
return VIDEO
if acodec:
return AUDIO
return FILE
+113 -15
View File
@@ -8,13 +8,15 @@ from datetime import date, datetime, timedelta, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import select
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, formats, service, storage
from app.downloads import adapters, credentials, formats, mediakind, probe, quota, service, storage
from app.downloads import urlguard
from app.downloads.jobdb import (
_edit_staging_dir,
_job_status,
@@ -35,6 +37,18 @@ class _Aborted(Exception):
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
class _OwnerRefused(Exception):
"""A refusal that belongs to THIS job's owner, not to the source.
MediaAssets are shared: two users asking for the same video and format get one row (see
`service.get_or_create_asset`), and the generic failure handler writes the message onto it —
correctly, because an upstream failure really is everyone's. A per-user refusal is not. Written
to the shared row it would fail a second user's queued job for the same video with the first
user's private reason (`worker._process_job` reads `asset.error` verbatim), telling someone with
an empty library to free up space. So this is caught separately: the job fails, the asset goes
back to `pending` for whoever else is holding it."""
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
@@ -304,10 +318,17 @@ def _media_info(info: dict) -> dict:
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
def _extract_with_fallback(
job_id, asset_id, url, spec, staging, adapter, cookiefile=None, max_bytes=None
):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt."""
error); other sources make a single attempt. Returns the info dict of the successful attempt.
`cookiefile` (S6) is the acting user's per-host cookies.txt, already gated by
`credentials.resolve_cookiefile`; passed to every attempt so the fallback stays authenticated.
`max_bytes` (SEC-13) is the owner's remaining storage budget, applied to every attempt."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
@@ -317,6 +338,8 @@ def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
outtmpl,
_make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
cookiefile=cookiefile,
max_bytes=max_bytes,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
@@ -381,7 +404,9 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(produced, dest, reenc_v, reenc_a),
editmod.build_browser_safe_cmd(
produced, dest, reenc_v, reenc_a, threads=editmod.encode_thread_cap()
),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
should_cancel=_cancel_poll(job_id),
@@ -395,8 +420,11 @@ def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict
return dest
def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict) -> None:
def _download(
job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict, user_id: int
) -> None:
staging = _staging_dir(asset_id)
cookiefile: Path | None = None
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
@@ -405,7 +433,50 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
info = _media_info(_extract_with_fallback(job_id, asset_id, url, spec, staging, adapter))
# SEC-12, second half: the enqueue guard ran when the job was created, which may have been
# hours ago and against a different DNS answer. This is where the socket actually gets
# opened, so the check is repeated here and the job fails rather than fetching.
if source_kind == "url":
try:
urlguard.check_url(url)
except urlguard.BlockedUrl as e:
raise RuntimeError(f"Refusing to download from this address ({e.reason}).") from e
# S6: attach the acting user's per-host cookie ONLY when it's host-scoped to this URL AND
# this asset is that user's private, salted identity (the double gate lives in credentials).
remaining_bytes: int | None = None
with SessionLocal() as cdb:
asset_sig = cdb.execute(
select(MediaAsset.format_sig).where(MediaAsset.id == asset_id)
).scalar_one_or_none()
if asset_sig is not None:
cookiefile = credentials.resolve_cookiefile(
cdb, user_id, url, asset_sig, spec, staging
)
# SEC-13: what this user has left. `check_enqueue` only compared the footprint as it
# stood when the job was QUEUED — a download's size is unknown until it finishes, so a
# single job could previously blow through the cap by any amount. Handing the remainder
# to yt-dlp as `max_filesize` makes the format selector refuse an oversized file up
# front instead.
remaining_bytes = quota.remaining_bytes(cdb, user_id)
# …and zero is a REFUSAL, not "no limit". Earlier jobs of this user may have filled the cap
# while this one sat in the queue, and 0 is not an expressible yt-dlp `max_filesize`, so
# passing it down would leave the ceiling unset — the one user already at their cap would be
# the only one downloading unbounded. Fail the job here instead. Not retryable until they
# free space, which is what the `quota_exceeded` category tells them.
if remaining_bytes == 0:
raise _OwnerRefused(dlerrors.QUOTA_FULL_MESSAGE)
info = _media_info(
_extract_with_fallback(
job_id,
asset_id,
url,
spec,
staging,
adapter,
cookiefile=str(cookiefile) if cookiefile else None,
max_bytes=remaining_bytes,
)
)
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
@@ -445,15 +516,15 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
asset.rel_path = rel
asset.size_bytes = size
asset.duration_s = meta.duration_s
asset.width = int(info["width"]) if info.get("width") else None
asset.height = int(info["height"]) if info.get("height") else None
# Codecs/dims come from a PROBE of the finished file — the ground truth — NOT yt-dlp's
# `info` dict, whose codec fields are absent for some sources (Facebook among them):
# deriving media_kind from those nulls misclassified a real H.264 video as a
# download-only 'file' with no in-app play/edit. Same probe the upload ingest uses.
asset.vcodec, asset.acodec, asset.width, asset.height = probe.resolve_media_facts(
final, info
)
asset.container = ext
asset.vcodec = (
info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
)
asset.acodec = (
info.get("acodec") if info.get("acodec") not in (None, "none") else None
)
asset.media_kind = mediakind.derive(asset.vcodec, asset.acodec)
asset.title = meta.title
asset.uploader = meta.uploader
asset.uploader_url = adapter.uploader_url(info)
@@ -495,8 +566,31 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
_reset_asset_pending(asset_id)
log.info("job %s aborted (pause/cancel)", job_id)
except _OwnerRefused as exc:
# Fail this job only. The asset is never marked `error`; it goes back to `pending` (below),
# so another user's job for the same video is claimed and downloaded normally instead of
# inheriting a reason that is about someone else's account.
msg = str(exc)[:500]
with SessionLocal() as db:
service.finalize_if_running(
db,
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
phase=None,
)
db.commit()
# Released AFTER the job is terminal: the moment the asset is `pending` another job can claim
# it, and this one must already be out of the way. (The staging tree is still torn down in
# `finally` below — the same narrow window the pause/cancel path above has always had.)
_reset_asset_pending(asset_id)
log.info("job %s refused for its owner: %s", job_id, msg)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
code = dlerrors.classify(msg)
# SEC-12: redact a transport-level readout for user-supplied URLs (see user_error_text).
msg = dlerrors.user_error_text(source_kind, msg, code)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
@@ -509,12 +603,16 @@ def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spe
job_id,
status=JobStatus.ERROR,
error=msg,
error_code=dlerrors.classify(msg),
error_code=code,
phase=None,
)
db.commit()
log.warning("job %s failed: %s", job_id, msg)
finally:
# Unlink the decrypted cookie file explicitly (the rmtree below removes the whole staging
# tree too, but never leave a plaintext session cookie on disk a moment longer than needed).
if cookiefile is not None:
cookiefile.unlink(missing_ok=True)
shutil.rmtree(staging, ignore_errors=True)
+107
View File
@@ -0,0 +1,107 @@
"""Ground-truth media probe for a produced library file — the SINGLE place both ingest paths ask
"what is actually on disk?".
`media_kind` (and the stored `vcodec`/`acodec`/`width`/`height`) must reflect the real file, not an
extractor's metadata: yt-dlp's `info` dict omits codec fields for some sources (e.g. Facebook), and
deriving the kind from those nulls misclassified a genuine H.264 video as a download-only `file`
(no in-app play). So the download worker (`pipeline`) and the local upload ingest (`upload`) both
probe the finished file here; `mediakind.derive` then turns the probed codecs into the kind.
"""
from pathlib import Path
from app.downloads import edit as editmod
# ffprobe container formats that mean "this is a still image / image sequence", NOT playable media.
# A standalone jpg/png/gif/webp/bmp reports a real video stream (codec mjpeg/png/…) with no
# `attached_pic` flag, so without this guard it would classify as media_kind='video' and get a Play
# button + a broken in-app <video> player (and inline serving on the public share). Keyed on the
# container format so an animated gif/webp also lands as a generic 'file' (no browser plays those in
# a <video> either). SVG and anything else ffprobe can't decode already fails the probe → 'file'.
_IMAGE_FORMATS = {
"image2",
"image2pipe",
"png_pipe",
"mjpeg",
"mjpeg_pipe",
"jpeg_pipe",
"gif",
"webp_pipe",
"bmp_pipe",
"tiff_pipe",
"ico",
"svg_pipe",
}
def _facts_from_probe(
data: dict,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) parsed from a NON-None ffprobe JSON dict. A still
image / image-sequence container yields no vcodec (→ generic 'file'); a video stream flagged
`attached_pic` (embedded cover art in an mp3/m4a) is skipped — it's a thumbnail, not a real video
track, so an audio file with album art still classifies as audio."""
fmt = data.get("format") or {}
fmt_names = {n.strip() for n in (fmt.get("format_name") or "").split(",")}
if fmt_names & _IMAGE_FORMATS:
return None, None, None, None, None
vcodec = acodec = None
width = height = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and vcodec is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
vcodec = stream.get("codec_name")
width = stream.get("width")
height = stream.get("height")
elif kind == "audio" and acodec is None:
acodec = stream.get("codec_name")
duration = None
try:
raw = fmt.get("duration")
if raw is not None:
duration = int(float(raw))
except (TypeError, ValueError):
duration = None
return vcodec, acodec, width, height, duration
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (an unprobeable file — or a still image — is treated as a generic
non-media file, so it becomes a download-only 'file' rather than a broken 'video')."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
if data is None:
return None, None, None, None, None
return _facts_from_probe(data)
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None]:
"""(vcodec, acodec, width, height) for the stored download asset: PROBE the finished file on disk
(the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources (Facebook among
them), so deriving media_kind from it there labelled a real H.264 video a download-only 'file'.
Precedence is by whether the probe RAN, not per-field: if ffprobe read the file, its codec verdict
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
`probe or info` fallback would let `info`'s stale vcodec resurrect a video track the file doesn't
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h
pv, pa, pw, ph, _dur = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the cosmetic width/height the probe couldn't read.
return pv, pa, pw if pw is not None else info_w, ph if ph is not None else info_h
+38
View File
@@ -7,6 +7,13 @@ Limits come from a per-user DownloadQuota row if the admin set one, else the sys
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
DOWNLOAD jobs additionally consult `remaining_bytes` in the worker, because enqueue can only compare
the footprint as it stood when the job was created and earlier jobs may have filled the cap since.
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
"""
from dataclasses import dataclass
@@ -122,6 +129,37 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def remaining_bytes(db: Session, user_id: int) -> int | None:
"""Storage this user may still spend on ONE download: ``None`` for an unlimited user, else the
cap minus the current footprint, floored at 0.
Zero is a real answer, not "no limit" — it means the cap is already met and the caller must
REFUSE rather than proceed unbounded. `check_enqueue` cannot make that refusal on the caller's
behalf: it ran when the job was queued, while the user was still under the cap, and the jobs
that filled it have landed since. Keeping the arithmetic here (rather than inline in the worker)
is what keeps every max_bytes decision in this module."""
lim = resolve(db, user_id)
if lim.unlimited:
return None
return max(0, lim.max_bytes - footprint(db, user_id))
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
until the worker finishes) so check_enqueue only compares the current footprint; an upload can,
so it's charged against the cap before it lands. Raises QuotaExceeded(max_jobs|max_bytes)."""
lim = resolve(db, user_id)
if lim.unlimited:
return
n = active_jobs(db, user_id)
if n >= lim.max_jobs:
raise QuotaExceeded("max_jobs", lim.max_jobs, n)
projected = footprint(db, user_id) + add_bytes
if projected > lim.max_bytes:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
+34 -1
View File
@@ -12,16 +12,48 @@ progress hook), then exits 0. An unexpected crash exits non-zero and the parent
"""
import logging
import os
import shutil
import subprocess
import sys
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
def _deprioritize() -> None:
"""Lower THIS child's CPU + IO priority so its yt-dlp/ffmpeg work — every descendant inherits the
handicap — can never starve the co-located API, Postgres, or Plex playback. Run once at startup,
before any heavy work. Every step is best-effort: a failed renice must never fail a download."""
nice_inc = settings.download_child_nice
if nice_inc > 0:
try:
os.nice(nice_inc) # relative increment; the child starts at the worker's nice (0)
except OSError as exc:
log.info("could not renice download child: %s", str(exc)[:120])
cls = settings.download_child_ionice_class
if cls > 0 and shutil.which("ionice"): # util-linux; absent on some minimal images
try:
subprocess.run(
["ionice", "-c", str(cls), "-p", str(os.getpid())],
capture_output=True,
timeout=5,
check=False,
)
except (OSError, subprocess.SubprocessError) as exc:
log.info("could not ionice download child: %s", str(exc)[:120])
# Cap OpenMP-parallel ffmpeg filters to the same budget as the encoder threads, so a filter
# graph can't fan out across every core behind ffmpeg's own `-threads` cap.
os.environ.setdefault("OMP_NUM_THREADS", str(editmod.encode_thread_cap()))
def main() -> None:
logging.basicConfig(level=logging.INFO)
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
raise SystemExit(2)
@@ -35,6 +67,7 @@ def main() -> None:
job_kind = job.job_kind
spec = dict(job.profile_snapshot or {})
source_kind, source_ref = job.source_kind, job.source_ref
user_id = job.user_id
if asset_id is None:
return
@@ -45,7 +78,7 @@ def main() -> None:
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec)
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
if __name__ == "__main__":
+15 -2
View File
@@ -15,8 +15,9 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, quota, storage
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
@@ -155,7 +156,12 @@ def enqueue(
Raises quota.QuotaExceeded if the user is at their job-count or footprint cap."""
quota.check_enqueue(db, user_id)
spec = formats.normalize(spec)
sig = formats.format_sig(spec)
# If this user has a stored cookie for the download's host, salt the cache signature with their
# id so the cookie-authed result is a PRIVATE asset — never dedup-shared with, or cache-served
# to, another user (the load-bearing privacy fix; the worker attaches the cookie to exactly this
# salted identity). A normal download (no matching cookie) gets salt=None and shares as before.
salt = credentials.salt_for_enqueue(db, user_id, source_url(source_kind, source_ref))
sig = formats.format_sig(spec, salt=salt)
asset = get_or_create_asset(db, source_kind, source_ref, sig)
# Show a real title + thumbnail the moment it's queued (feed downloads are catalog videos).
populate_from_catalog(db, asset)
@@ -210,6 +216,10 @@ def enqueue_edit(
src_asset = db.get(MediaAsset, source_job.asset_id) if source_job.asset_id else None
if src_asset is None or src_asset.status != "ready" or not src_asset.rel_path:
raise EditError("source_not_ready")
# A generic uploaded file (media_kind='file') has no timeline to trim/crop — the editor path is
# media-only. Reject here so an upload's file never reaches the ffmpeg edit worker.
if src_asset.media_kind == mediakind.FILE:
raise EditError("not_media")
spec = editmod.normalize_edit_spec(edit_spec)
if not spec:
@@ -226,6 +236,9 @@ def enqueue_edit(
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
asset.duration_s = editmod.clip_duration(spec, src_asset.duration_s)
# A clip inherits its source's kind (a video edit stays video, an audio trim stays audio);
# the ffmpeg edit worker doesn't re-probe codecs, so set it here where the source is known.
asset.media_kind = src_asset.media_kind or mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
+77
View File
@@ -19,6 +19,8 @@ from datetime import date
from pathlib import Path
from xml.sax.saxutils import escape
from app.downloads import mediakind
# Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
_MAX_TITLE = 120
# User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -147,6 +149,81 @@ def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
return f".edits/{user_id}/clip_{asset_id}.{ext}"
def _clean_upload_ext(filename: str) -> str:
"""The upload's extension reduced to a shell-safe, lowercased, bounded token (``""`` when the
upload has no extension). The SINGLE definition feeding both the on-disk suffix and the stored
`container` — they must agree (the served Content-Disposition suffix has to match the container),
so the sanitize/truncation rule lives in one place, not two drifting copies."""
return _UNSAFE.sub("", Path(filename).suffix.lstrip(".")).lower()[:16]
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
episode sidecars). The name is system-decided and bound to the asset id (`_a{id}` keeps two
same-named uploads apart); the original filename is display metadata only. The extension is
preserved (shell-safe, bounded) so the served file + its Content-Disposition carry a correct
suffix; an extensionless upload stays extensionless."""
stem = sanitize(Path(filename).stem, 80)
ext = _clean_upload_ext(filename)
suffix = f".{ext}" if ext else ""
return f".uploads/{user_id}/{stem}_a{asset_id}{suffix}"
def upload_ext(filename: str) -> str | None:
"""The stored `container` for an uploaded file: its shell-safe lowercased extension, or None when
the upload has no extension (a bare name). Shares `_clean_upload_ext` with `upload_rel_path` so the
stored container and the on-disk suffix can never diverge."""
return _clean_upload_ext(filename) or None
# The Content-Type used to serve an inline-playable media asset, keyed by its stored `container`.
# DELIBERATELY a strict allow-list of known audio/video containers: the served type is chosen from
# the asset's ffprobe-verified media_kind + this table, NEVER guessed from the raw (user-controlled)
# upload extension. Without this, an uploaded media/HTML polyglot named "evil.html" — a real MP3/MP4
# that ffprobe classifies as media, so it dodges the generic-file force-download — would be served
# with a Content-Type Starlette guesses from ".html" (text/html) and execute as stored XSS on the
# serving origin. A media asset whose container isn't in this table falls back to a neutral
# force-download attachment (see `served_media_type`), so nothing script-executable is ever inline.
_MEDIA_MIME: dict[str, str] = {
# video containers
"mp4": "video/mp4",
"m4v": "video/mp4",
"mov": "video/quicktime",
"webm": "video/webm",
"mkv": "video/x-matroska",
"ogv": "video/ogg",
"avi": "video/x-msvideo",
# audio containers
"m4a": "audio/mp4",
"mp3": "audio/mpeg",
"aac": "audio/aac",
"ogg": "audio/ogg",
"oga": "audio/ogg",
"opus": "audio/ogg",
"weba": "audio/webm",
"wav": "audio/wav",
"flac": "audio/flac",
}
def served_media_type(media_kind: str | None, container: str | None) -> str | None:
"""The Content-Type a library file MUST be served with — a real audio/video MIME for a recognised
media container, else ``None`` meaning "serve as a neutral force-download attachment".
Returns None for a generic ``file`` kind AND for any media whose container we don't positively
recognise as safe. Callers serve None as ``application/octet-stream`` + ``attachment`` so the
served Content-Type is NEVER derived from the (attacker-controlled) file extension — the guard
that stops uploaded HTML/SVG or a media/HTML polyglot from executing inline on the serving
origin. A ``None`` media_kind (a legacy asset before the discriminator was backfilled) is treated
as media so an old download keeps its inline player."""
if media_kind == mediakind.FILE:
return None
return _MEDIA_MIME.get((container or "").lower())
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
+122
View File
@@ -0,0 +1,122 @@
"""Local file ingest for the download center (E7 S5) — the non-yt-dlp path into the media library.
A user-uploaded file joins the SAME library as a downloaded one, so it inherits the whole function
set already built (metadata chips, share links, and — for media — in-app play/edit). Unlike a
download it bypasses yt-dlp AND the worker entirely: the file is already on the host, so ingest
probes it, places it on disk, and creates an already-`ready` MediaAsset + `done` DownloadJob in one
step — the same "the file exists, mark it done now" shortcut `service.enqueue` takes on a cache hit.
Kinds (the invariant lives in `mediakind`): a probe with a real video stream -> video, audio-only
-> audio, neither -> a generic 'file' (pdf/zip/docx/…) that is download-only and never rendered
inline (see the force-download serving in the file/watch routes). This module is the ingest seam a
future sub-function (e.g. convert) plugs a new adapter into, mirroring how it sidesteps the worker.
An upload does NOT get an `expires_at`: it's user-provided and may be irreplaceable (there's no
source to re-download), so it doesn't age out on the retention clock the way a re-fetchable download
does — it lives until the user deletes it.
"""
import secrets
from datetime import datetime, timezone
from pathlib import Path
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, probe, quota, service, storage
from app.models import DownloadJob, MediaAsset
def _title_from_filename(filename: str) -> str:
"""A human display title from the original upload name: the stem with separators tidied to
spaces (kept as-is otherwise — this is display metadata, not the shell-safe on-disk name)."""
stem = Path(filename).stem.replace("_", " ").replace(".", " ").strip()
return (stem or "Upload")[:255]
def ingest_upload(
db: Session,
user_id: int,
*,
filename: str,
src_path: Path,
size: int,
display_name: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return the `done`
job. `src_path` is MOVED into place on success (the caller stages it to a temp file first).
Raises quota.QuotaExceeded (holdings/footprint cap) — checked BEFORE any disk write, so a
rejected upload leaves nothing behind and the caller just cleans up its temp file."""
quota.check_upload(db, user_id, size) # the file's size is known → charge it up front
vcodec, acodec, width, height, duration = probe.probe_media(src_path)
kind = mediakind.derive(vcodec, acodec)
# An upload never dedups (each file is its own asset): a random ref makes the cache key unique,
# so `format_sig` is a constant marker rather than a format hash.
ref = f"{user_id}:{secrets.token_hex(8)}"[:512]
asset = MediaAsset(
source_kind="upload", source_ref=ref, format_sig="upload", status="pending", ref_count=0
)
db.add(asset)
db.flush() # need asset.id for the on-disk path
rel = storage.upload_rel_path(user_id, asset.id, filename)
storage.place_file(src_path, settings.download_root, rel)
# From here the file is on disk under `.uploads/`. If the transaction fails before it commits, the
# asset/job rows roll back but the placed file would be ORPHANED — nothing references it, the
# per-user delete path keys off asset.rel_path (which is gone), and the GC orphan sweep skips the
# dot-prefixed `.uploads/` tree. So roll back the on-disk side too on any failure.
try:
final = storage.abs_path(settings.download_root, rel)
actual_size = final.stat().st_size if final.exists() else size
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = actual_size
asset.container = storage.upload_ext(filename)
asset.vcodec = vcodec
asset.acodec = acodec
asset.width = width
asset.height = height
asset.duration_s = duration
asset.media_kind = kind
asset.title = (display_name or "").strip() or _title_from_filename(filename)
asset.last_access_at = datetime.now(timezone.utc)
# No expires_at — see the module docstring (an upload is irreplaceable, so it doesn't age out).
if kind == mediakind.VIDEO:
# A frame poster gives the library row a thumbnail + the share link an og:image. Audio has
# no frame and a generic file no player, so neither gets one. Best-effort (ffmpeg decodes
# server side even for a codec the browser can't play).
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="upload",
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=(display_name or "").strip() or None,
status="done",
progress=100,
queue_pos=service.next_queue_pos(db),
)
db.add(job)
service.incr_ref_count(db, asset.id)
db.commit()
except Exception:
db.rollback()
# The rows are gone; remove the now-unreferenced file so no bytes leak. Clean the poster by its
# DETERMINISTIC path (rel with a .jpg suffix, the same path ensure_poster writes to) rather than
# ensure_poster's return value: it writes the frame directly to that path and returns None on an
# ffmpeg failure, so a partial poster can be on disk while the return is None — gating on the
# return would skip that orphan. unlink is missing_ok, so the poster path is a harmless no-op
# when none was generated (audio/file, or a video whose poster never started).
for leaked in (rel, str(Path(rel).with_suffix(".jpg"))):
storage.abs_path(settings.download_root, leaked).unlink(missing_ok=True)
raise
db.refresh(job)
return job
+108
View File
@@ -0,0 +1,108 @@
"""Refuse a user-supplied download URL that points back into our own network (SEC-12).
The Download Center deliberately accepts an arbitrary ``http(s)`` URL — that is the feature — and
hands it to yt-dlp running on the WORKER. Without a guard, any approved user can make the server
fetch anything the server can reach: other containers on the compose network, the LAN, a cloud
metadata endpoint. The failure text then comes back to them in ``job.error``, so "connection
refused" vs "HTTP Error 401" vs "timed out" turns the download box into an internal port scanner.
What this closes: a URL whose host resolves to a loopback / private / link-local / reserved address
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
differently between the two moments, and the worker is where the socket is actually opened.
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
with an internal address.
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
kept). Tracked as SEC-17.
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
hostname into the box), which is the reachable one.
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
user-controlled destination to guard.
"""
import ipaddress
import socket
from urllib.parse import urlparse
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
# vocabulary (frontend `localizeDetail`) — keep them stable.
REASON_SCHEME = "scheme" # not http(s)
REASON_NO_HOST = "no_host" # URL carries no hostname
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
class BlockedUrl(Exception):
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
REASON_* codes above."""
def __init__(self, reason: str):
super().__init__(reason)
self.reason = reason
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
"""Every address family we must not open a connection to. Checked explicitly rather than
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
mapped = getattr(ip, "ipv4_mapped", None)
if mapped is not None:
ip = mapped
return bool(
ip.is_private
or ip.is_loopback
or ip.is_link_local
or ip.is_multicast
or ip.is_reserved
or ip.is_unspecified
)
def _resolve(host: str, port: int | None) -> list[str]:
"""Every address ``host`` resolves to. Raises BlockedUrl(REASON_UNRESOLVABLE) when DNS has no
answer — reported as its own outcome, because "we could not look that up" and "we refuse to
fetch that" are different things to tell a user."""
try:
infos = socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
except socket.gaierror as exc:
raise BlockedUrl(REASON_UNRESOLVABLE) from exc
return [info[4][0] for info in infos]
def check_url(raw: str) -> None:
"""Raise BlockedUrl unless ``raw`` is an http(s) URL whose host resolves only to addresses the
server is allowed to fetch from. A host with even ONE forbidden address is refused: a
round-robin answer must not become a coin flip."""
# `urlparse` itself raises ValueError on a malformed authority ("http://[::1/x" — an unclosed
# bracket), and `.port` raises on ":99999" / ":abc" when READ rather than at parse time. Both
# escape the caller's `except BlockedUrl` as a 500 unless caught here, which is the opposite of
# what this module exists to do: turn a bad URL into a structured, localizable refusal.
try:
u = urlparse((raw or "").strip())
port = u.port
except ValueError as exc:
raise BlockedUrl(REASON_MALFORMED) from exc
if u.scheme not in ("http", "https"):
raise BlockedUrl(REASON_SCHEME)
host = u.hostname
if not host:
raise BlockedUrl(REASON_NO_HOST)
for addr in _resolve(host, port):
try:
ip = ipaddress.ip_address(addr)
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
raise BlockedUrl(REASON_PRIVATE) from None
if _is_forbidden(ip):
raise BlockedUrl(REASON_PRIVATE)
+252 -114
View File
@@ -3,36 +3,103 @@
Gmail SMTP + App Password by default. Deliberately fail-soft: if SMTP isn't configured
or a send errors, we log and return False so the caller can fall back to in-app
notifications — email is never load-bearing for the app's core flow.
Fail-soft must not mean invisible, though: every drop also leaves an in-app notice in the admins'
inbox (see `_alert_admins`). An instance whose SMTP was never filled in silently swallows
verification links, password resets and access-request notices, and the operator's first clue used
to be a person telling them nothing arrived.
"""
import html
import logging
import re
import smtplib
import ssl
from email.message import EmailMessage
from email.utils import formatdate
from app import sysconfig
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import mailtext, sysconfig
from app.config import settings
from app.db import SessionLocal
from app.models import User
from app.notifications import create_notification
from app.ratelimit import RateLimiter
log = logging.getLogger("siftlode.email")
def _smtp() -> dict:
"""Effective SMTP config (admin DB override over env defaults). Opens its own short
session because email is often sent from a BackgroundTask, outside a request's db."""
with SessionLocal() as db:
return {
"host": sysconfig.get_str(db, "smtp_host"),
"port": sysconfig.get_int(db, "smtp_port"),
"user": sysconfig.get_str(db, "smtp_user"),
"password": sysconfig.get_str(db, "smtp_password"),
"from": sysconfig.get_str(db, "smtp_from"),
}
# One admin notice an hour per reason. A relay that is down fails every send, and the inbox must
# stay readable; the log keeps the full record either way.
_alert_throttle = RateLimiter(max_events=1, window_seconds=3600)
def email_enabled() -> bool:
c = _smtp()
def _alert_admins(reason: str, subject: str, recipients: list[str]) -> None:
"""Record an in-app notice that a message did NOT go out. Best effort in every direction: it
opens its own session (callers are often background tasks), swallows its own errors, and never
reports failure to the caller — this is the fallback path, it must not become a new way to
break a send."""
if not _alert_throttle.allow(reason):
return
try:
with SessionLocal() as db:
admins = (
db.execute(
select(User.id).where(
User.role == "admin",
User.is_active.is_(True),
User.is_suspended.is_(False),
)
)
.scalars()
.all()
)
if not admins:
# Nothing was recorded, so don't let this cost the next hour of alerts on the very
# path whose job is to stop drops going unseen. Same on the failure branch below.
_alert_throttle.reset()
return
for uid in admins:
create_notification(
db,
user_id=uid,
type="email_undelivered",
# English fallback; the inbox renders a translated version from `data`.
title="Email could not be sent",
body=subject,
data={"reason": reason, "subject": subject, "recipients": recipients},
commit=False,
)
db.commit()
except Exception:
_alert_throttle.reset()
log.exception("Could not record the undelivered-email notice (%r)", subject)
def _read_smtp(db: Session) -> dict:
return {
"host": sysconfig.get_str(db, "smtp_host"),
"port": sysconfig.get_int(db, "smtp_port"),
"user": sysconfig.get_str(db, "smtp_user"),
"password": sysconfig.get_str(db, "smtp_password"),
"from": sysconfig.get_str(db, "smtp_from"),
}
def _smtp(db: Session | None = None) -> dict:
"""Effective SMTP config (admin DB override over env defaults). Opens its own short session when
the caller has none, because email is usually sent from a BackgroundTask outside a request's db.
A caller that IS inside a request should pass its session: reading the password decrypts a Fernet
value, and /api/me polls this for every admin tab."""
if db is not None:
return _read_smtp(db)
with SessionLocal() as own:
return _read_smtp(own)
def email_enabled(db: Session | None = None) -> bool:
c = _smtp(db)
return bool(c["host"] and c["user"] and c["password"])
@@ -40,6 +107,48 @@ def _admin_contact() -> str | None:
return next(iter(sorted(settings.admin_email_set)), None)
# A bare URL in a text/plain mail is only clickable if the CLIENT decides to linkify it. Gmail,
# Outlook and Apple Mail do; plenty of lighter web clients render the body verbatim and leave the
# link as dead text — which, for a verification or password-reset mail, is the whole message. So
# every mail also carries an HTML alternative built from the SAME plain body (below), rather than
# per-template HTML that would have to be kept in sync with it.
#
# `[^\s<]` stops at whitespace and at the `<` of any following markup; the trailing-punctuation trim
# keeps a sentence-final period out of the href. `;` and `:` are deliberately NOT trimmed — escaping
# turns `&` into `&amp;`, so a query string can legitimately end in a semicolon.
_URL_RE = re.compile(r"https?://[^\s<]+")
def _html_body(text: str) -> str:
"""Render a plain-text mail body as minimal HTML: escaped, blank-line-separated paragraphs, with
real links. Escaping happens FIRST and the linkifier then runs over the escaped text — bodies
interpolate untrusted values (a requester's email address, the operator contact), so no caller
can inject markup, and `&` inside a URL is already `&amp;` by the time it reaches the href, which
is what an HTML attribute wants anyway."""
def _anchor(m: re.Match[str]) -> str:
url = m.group(0)
trail = ""
while url and url[-1] in ".,!?)":
url, trail = url[:-1], url[-1] + trail
if not url:
return trail
return f'<a href="{url}">{url}</a>{trail}'
paragraphs = [p for p in html.escape(text).split("\n\n") if p.strip()]
rendered = "".join(
f"<p>{_URL_RE.sub(_anchor, p).replace(chr(10), '<br>')}</p>" for p in paragraphs
)
# Inline styles only — many clients drop <style> blocks. No images, no external CSS, no tracking:
# the HTML part exists to make the link clickable, not to become a template to maintain.
return (
'<html><body style="margin:0;padding:16px;font-family:system-ui,-apple-system,'
"'Segoe UI',sans-serif;font-size:15px;line-height:1.6\">"
f'<div style="max-width:560px">{rendered}</div>'
"</body></html>"
)
def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -> bool:
recipients = [e for e in to if e]
if not recipients:
@@ -47,6 +156,7 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
c = _smtp()
if not (c["host"] and c["user"] and c["password"]):
log.info("SMTP not configured; skipping email %r to %s", subject, recipients)
_alert_admins("not_configured", subject, recipients)
return False
msg = EmailMessage()
msg["Subject"] = subject
@@ -57,7 +167,10 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
msg["Date"] = formatdate(localtime=True)
if reply_to:
msg["Reply-To"] = reply_to
# text/plain stays the primary part (it is the source of truth, and a text-only client still
# gets the full message); the HTML alternative below is what makes the link clickable everywhere.
msg.set_content(body)
msg.add_alternative(_html_body(body), subtype="html")
try:
with smtplib.SMTP(c["host"], c["port"], timeout=20) as s:
s.starttls(context=ssl.create_default_context())
@@ -67,128 +180,153 @@ def _send(to: list[str], subject: str, body: str, reply_to: str | None = None) -
return True
except Exception:
log.exception("SMTP send failed (%r to %s)", subject, recipients)
_alert_admins("send_failed", subject, recipients)
return False
def send_access_approved(email: str) -> bool:
def _next_step(verify_link: str | None, has_account: bool, hours: int, lang: str) -> str:
"""What this person actually has to do next, in the one paragraph that carries the whole flow.
Three states, because a mail that guesses wrong sends someone looking for a link that isn't
coming — or worse, tells them to sign in when their address is still unconfirmed and sign-in will
refuse them (E9.5 S4 named that state on screen; the mail must not contradict it).
The link branch is the point of the C option: in the common ordering — approve first, the person
signs up later, or signs up and waits — they would otherwise have to find and open TWO separate
messages. One mail, one click.
"""
if verify_link:
return mailtext.text(
"next_step.verify", lang, link=verify_link, expiry=mailtext.link_expiry(hours, lang)
)
if has_account:
return mailtext.text("next_step.sign_in", lang, app=settings.app_base)
return mailtext.text("next_step.register", lang, app=settings.app_base)
def _reply_contact(lang: str) -> tuple[str | None, str]:
"""The admin address to answer to, and the closing line that names it — empty when no admin
contact is configured, so the mail never promises a reply nobody will read."""
admin = _admin_contact()
body = (
"Hi,\n\n"
"Your request to use Siftlode has been approved — welcome aboard.\n\n"
"Open Siftlode and sign in, and your YouTube subscriptions feed will be ready:\n\n"
f"{settings.app_base}\n\n"
"If you weren't expecting this, you can ignore the message.\n\n"
"— Siftlode" + (f"\n\nQuestions? Just reply to this email ({admin})." if admin else "")
return admin, (mailtext.text("contact.reply", lang, admin=admin) if admin else "")
def send_access_approved(
email: str,
lang: str,
verify_link: str | None = None,
has_account: bool = False,
hours: int = 24,
) -> bool:
"""Answer someone who ASKED for access. Sent automatically on approval, because a person who
made a request is owed a reply — unlike the admin-initiated invitation below."""
admin, contact = _reply_contact(lang)
body = mailtext.text(
"access_approved.body",
lang,
next_step=_next_step(verify_link, has_account, hours, lang),
contact=contact,
)
return _send([email], "You're in — your Siftlode access is approved", body, reply_to=admin)
return _send([email], mailtext.text("access_approved.subject", lang), body, reply_to=admin)
def send_access_denied(email: str) -> bool:
admin = _admin_contact()
body = (
"Hi,\n\n"
"Thanks for your interest in Siftlode. After review, your access request wasn't approved "
"this time.\n\n"
"If you think this was a mistake, you're welcome to reply and let us know.\n\n"
"— Siftlode" + (f"\n\nQuestions? Just reply to this email ({admin})." if admin else "")
def send_access_invitation(
email: str,
lang: str,
verify_link: str | None = None,
has_account: bool = False,
hours: int = 24,
) -> bool:
"""Invite someone the admin let in WITHOUT them asking — sent only when the admin presses send.
Whitelisting an address is opening a door, not knocking on one: the operator has usually already
agreed it with the person, and an address can sit approved for weeks without anyone registering.
So adding to the whitelist stays silent and this is a separate, deliberate act. That is also why
the closing line offers no deadline — nudging is the one thing this mail must not do.
Its language is the one place the app has no signal to read: the admin typed this address in, so
nobody has ever told us how to address its owner. That is why the send button asks (E9.5 S7).
"""
admin, contact = _reply_contact(lang)
body = mailtext.text(
"access_invitation.body",
lang,
next_step=_next_step(verify_link, has_account, hours, lang),
contact=contact,
)
return _send([email], "About your Siftlode access request", body, reply_to=admin)
return _send([email], mailtext.text("access_invitation.subject", lang), body, reply_to=admin)
def send_role_changed(to: str, role: str, operator: str | None) -> bool:
line = (
"You've been granted admin access on Siftlode — you can now manage settings, the "
"scheduler and users."
if role == "admin"
else "Your admin access on Siftlode has been removed — you're now a regular user."
def send_access_denied(email: str, lang: str) -> bool:
admin, contact = _reply_contact(lang)
body = mailtext.text("access_denied.body", lang, contact=contact)
return _send([email], mailtext.text("access_denied.subject", lang), body, reply_to=admin)
def send_role_changed(to: str, lang: str, role: str, operator: str | None) -> bool:
line = mailtext.text(
"role_changed.granted" if role == "admin" else "role_changed.revoked", lang
)
contact = f"\n\nQuestions? Contact the operator at {operator}." if operator else ""
body = "Hi,\n\n" + line + contact + "\n\n— Siftlode"
return _send([to], "Your Siftlode role has changed", body, reply_to=operator)
contact = mailtext.text("contact.operator", lang, operator=operator) if operator else ""
body = mailtext.text("role_changed.body", lang, line=line, contact=contact)
return _send([to], mailtext.text("role_changed.subject", lang), body, reply_to=operator)
def send_account_unsuspended(to: str, operator: str | None) -> bool:
contact = f"\n\nQuestions? Contact the operator at {operator}." if operator else ""
body = (
"Hi,\n\n"
"Your Siftlode account has been reinstated — the suspension was lifted and you can sign "
"in again." + contact + "\n\n— Siftlode"
)
return _send([to], "Your Siftlode account has been reinstated", body, reply_to=operator)
def send_account_unsuspended(to: str, lang: str, operator: str | None) -> bool:
contact = mailtext.text("contact.operator", lang, operator=operator) if operator else ""
body = mailtext.text("account_unsuspended.body", lang, contact=contact)
return _send([to], mailtext.text("account_unsuspended.subject", lang), body, reply_to=operator)
def send_account_deleted(to: str, operator: str | None) -> bool:
def send_account_deleted(to: str, lang: str, operator: str | None) -> bool:
contact = mailtext.text("contact.deleted", lang, operator=operator) if operator else ""
body = mailtext.text("account_deleted.body", lang, contact=contact)
return _send([to], mailtext.text("account_deleted.subject", lang), body, reply_to=operator)
def send_account_suspended(to: str, lang: str, operator: str | None) -> bool:
contact = (
f"\n\nIf you didn't request this, contact the Siftlode operator at {operator}."
mailtext.text("contact.suspended", lang, operator=operator)
if operator
else ""
else mailtext.text("contact.suspended_unknown", lang)
)
body = (
"Hi,\n\n"
"Your Siftlode account and all associated data — subscriptions, tags, watch history, "
"playlists and settings — have been permanently deleted. Nothing is retained.\n\n"
"Thanks for having used Siftlode." + contact + "\n\n— Siftlode"
body = mailtext.text("account_suspended.body", lang, contact=contact)
return _send([to], mailtext.text("account_suspended.subject", lang), body, reply_to=operator)
def send_admin_new_request(admins: list[str], lang: str, requester: str) -> bool:
"""Tell the admins someone is waiting. One call per language group (see
`routes.auth.notify_admins_of_request`) — admins are people with their own UI language, and a
single blast to all of them would have to pick one and be wrong for the rest.
The link deep-links straight to the admin Access-requests page (the SPA reads ?admin=…). The
requester address is spelled out (mail clients auto-link both the URL and the address), and the
reply note is accurate: _send sets Reply-To to the requester, so a reply reaches THEM — not the
sending mailbox. (The old copy said "Settings → Account", which was the wrong menu.)
"""
body = mailtext.text(
"admin_new_request.body",
lang,
requester=requester,
link=f"{settings.app_base}/?admin=access-requests",
)
return _send([to], "Your Siftlode account has been deleted", body, reply_to=operator)
subject = mailtext.text("admin_new_request.subject", lang, requester=requester)
return _send(admins, subject, body, reply_to=requester)
def send_account_suspended(to: str, operator: str | None) -> bool:
contact = (
f"If you think this is a mistake or have questions, contact the Siftlode operator "
f"at {operator}.\n\n"
if operator
else "If you think this is a mistake, please contact the Siftlode operator.\n\n"
def send_verify_email(to: str, lang: str, link: str, hours: int = 24) -> bool:
# The body says what the link's rules are. The reset mail always has; this one left the reader to
# discover both by meeting a dead link, which is precisely the silence this epic removes.
body = mailtext.text(
"verify_email.body", lang, link=link, expiry=mailtext.link_expiry(hours, lang)
)
body = (
"Hi,\n\n"
"A sign-in to your Siftlode account was just attempted, but the account is currently "
"suspended, so access was denied.\n\n" + contact + "— Siftlode"
)
return _send([to], "Your Siftlode account is suspended", body, reply_to=operator)
return _send([to], mailtext.text("verify_email.subject", lang), body)
def send_admin_new_request(admins: list[str], requester: str) -> bool:
# The link deep-links straight to the admin Access-requests page (the SPA reads ?admin=…). The
# requester address is spelled out (mail clients auto-link both the URL and the address), and the
# reply note is accurate: _send sets Reply-To to the requester, so a reply reaches THEM — not the
# sending mailbox. (The old copy said "Settings → Account", which was the wrong menu.)
approve_url = f"{settings.app_base}/?admin=access-requests"
body = (
f"{requester} just requested access to Siftlode.\n\n"
f"Approve or deny it here:\n{approve_url}\n"
f"(or in Siftlode: open Users → Access requests)\n\n"
f"Requester's email: {requester}\n"
f"Replying to this message goes straight to the requester (it's the Reply-To address).\n"
)
return _send(admins, f"Siftlode access request from {requester}", body, reply_to=requester)
def send_password_reset(to: str, lang: str, link: str) -> bool:
body = mailtext.text("password_reset.body", lang, link=link)
return _send([to], mailtext.text("password_reset.subject", lang), body)
def send_verify_email(to: str, link: str) -> bool:
body = (
"Hi,\n\n"
"Confirm your email to finish creating your Siftlode account:\n\n"
f"{link}\n\n"
"If you didn't sign up, you can ignore this message.\n\n"
"— Siftlode"
)
return _send([to], "Confirm your Siftlode email", body)
def send_password_reset(to: str, link: str) -> bool:
body = (
"Hi,\n\n"
"Use this link to set a new Siftlode password (it expires in an hour):\n\n"
f"{link}\n\n"
"If you didn't request this, you can ignore this message — your password is unchanged.\n\n"
"— Siftlode"
)
return _send([to], "Reset your Siftlode password", body)
def send_test(to: str) -> bool:
body = (
"This is a test email from Siftlode.\n\n"
"If you're reading this, your SMTP settings work.\n\n"
"— Siftlode"
)
return _send([to], "Siftlode SMTP test", body)
def send_test(to: str, lang: str) -> bool:
return _send([to], mailtext.text("test.subject", lang), mailtext.text("test.body", lang))
+14 -7
View File
@@ -117,6 +117,19 @@ def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
return out
def not_blocked_cond(user: User):
"""Per-user channel blocklist: never show this user videos from a channel they blocked
(applies to feed, Library, explore, the channel page and the Home resume rail alike).
A shared expression rather than a clause each surface writes for itself. Blocking does not touch
`video_states`, so a half-watched video from a newly blocked channel survives with its resume
position — and the one query that copied only PART of this predicate set is exactly how that
video came back as the landing page's suggestion after the user had said "never again"."""
return Video.channel_id.notin_(
select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)
)
def filtered_query(
db: Session,
user: User,
@@ -207,13 +220,7 @@ def filtered_query(
)
)
# Per-user channel blocklist: never show this user videos from a channel they blocked
# (applies to feed, Library, explore, and the channel page alike).
query = query.where(
Video.channel_id.notin_(
select(BlockedChannel.channel_id).where(BlockedChannel.user_id == user.id)
)
)
query = query.where(not_blocked_cond(user))
# Source filter. In the shared Library ("all" scope) it uses the GLOBAL via_search flag
# (search-discovered by anyone). In "my" scope it's the user's OWN provenance:
+373
View File
@@ -0,0 +1,373 @@
"""Bilingual text for every outbound mail, and the one resolver that decides which language a
recipient gets (E9.5 S7).
Why a catalog instead of literals in `email.py`: HU/EN parity is a house rule for every user-facing
string, and a mail is the one user-facing surface the reader CANNOT re-render by switching the UI
language — it arrives once, in whatever language we picked, and stays that way. `test_mail_language.py`
enforces that every key carries both languages *and* the same `{placeholders}`, so a half-translated
mail fails the gate instead of reaching a person.
The language is resolved by the REQUEST handler and handed to `send_*` as a plain code. It is never
resolved inside the mail: every send runs as a `BackgroundTasks` job, after the request's session is
gone — the same reason `_welcome_mail_parts` computes its link in the handler and not in the task.
"""
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models import Invite, User
from app.utils import canonical_email
SUPPORTED: tuple[str, ...] = ("en", "hu")
DEFAULT = "en"
def normalize(lang: str | None) -> str:
"""Coerce anything to a language we can actually render. Storage is not a guarantee: a
`preferences.language` written by an older build (or a locale we no longer ship) must degrade to
English rather than raise inside a background task nobody is watching."""
return lang if lang in SUPPORTED else DEFAULT
def text(key: str, lang: str | None, **values: object) -> str:
"""Render one catalog entry. Unknown languages fall back to English (see `normalize`); an
unknown KEY raises, because that is a programming error the tests must surface, not something to
paper over with an empty line in someone's inbox."""
return MAIL_TEXT[key][normalize(lang)].format(**values)
def link_expiry(hours: int, lang: str) -> str:
"""How long a link lasts, in a unit a reader converts on sight. A whole number of days is
said in days — the approval mail's token lives seven of them, and "168 hours" is not an answer
to "how long have I got". A single day stays "24 hours", which is how everyone writes it."""
if hours >= 48 and hours % 24 == 0:
return text("duration.days", lang, n=hours // 24)
return text("duration.hours", lang, n=hours)
def lang_for_user(user: User | None) -> str:
"""The language a signed-up recipient gets: the one they picked in the app."""
prefs = (user.preferences or {}) if user is not None else {}
return normalize(prefs.get("language"))
def lang_for_email(db: Session, email: str) -> str:
"""The language for an address that may not have an account yet: their own account preference,
else the language they filled our access-request form in, else English.
Matched CANONICALLY like every other identity lookup in this epic, and it walks past rows that
carry no usable code instead of stopping at the first one — prod really does hold two accounts
for a single mailbox, and only one of them may have ever picked a language.
"""
canon = canonical_email(email)
for prefs in (
db.execute(select(User.preferences).where(User.email_canonical == canon).order_by(User.id))
.scalars()
.all()
):
code = (prefs or {}).get("language")
if code in SUPPORTED:
return code
for code in (
db.execute(select(Invite.lang).where(Invite.email_canonical == canon).order_by(Invite.id))
.scalars()
.all()
):
if code in SUPPORTED:
return code
return DEFAULT
# The brand line ("— Siftlode") and the product name are deliberately NOT catalog entries: they are
# a name, not a translated string, and giving them keys would invite someone to "translate" them.
MAIL_TEXT: dict[str, dict[str, str]] = {
# ── How long a link lasts ───────────────────────────────────────────────────────────────
# Rendered, not interpolated as a raw hour count: the approval mail's token lives seven days
# (E9.5 S6), and "expires in 168 hours" is a unit no reader converts.
"duration.hours": {"en": "{n} hours", "hu": "{n} óra"},
"duration.days": {"en": "{n} days", "hu": "{n} nap"},
# ── Shared closing fragments ────────────────────────────────────────────────────────────────
"contact.reply": {
"en": "\n\nQuestions? Just reply to this email ({admin}).",
"hu": "\n\nKérdésed van? Csak válaszolj erre a levélre ({admin}).",
},
"contact.operator": {
"en": "\n\nQuestions? Contact the operator at {operator}.",
"hu": "\n\nKérdésed van? Írj az üzemeltetőnek: {operator}.",
},
"contact.deleted": {
"en": "\n\nIf you didn't request this, contact the Siftlode operator at {operator}.",
"hu": "\n\nHa nem te kérted, írj a Siftlode üzemeltetőjének: {operator}.",
},
"contact.suspended": {
"en": (
"If you think this is a mistake or have questions, contact the Siftlode operator "
"at {operator}.\n\n"
),
"hu": (
"Ha szerinted ez tévedés, vagy kérdésed van, írj a Siftlode üzemeltetőjének: "
"{operator}.\n\n"
),
},
"contact.suspended_unknown": {
"en": "If you think this is a mistake, please contact the Siftlode operator.\n\n",
"hu": "Ha szerinted ez tévedés, keresd a Siftlode üzemeltetőjét.\n\n",
},
# ── The one paragraph that carries the whole onboarding flow (see email._next_step) ──────────
"next_step.verify": {
"en": (
"You already started an account, so there is just one step left. Confirm your email "
"address and you're in:\n\n"
"{link}\n\n"
"The link works once and expires in {expiry}. If it does, you can ask for a fresh "
"one from the sign-in page."
),
"hu": (
"A fiókodat már elkezdted, úgyhogy egyetlen lépés maradt. Erősítsd meg az "
"e-mail-címed, és bent is vagy:\n\n"
"{link}\n\n"
"A link egyszer használható, és {expiry} múlva lejár. Ha lejárt, a belépési oldalon "
"kérhetsz újat."
),
},
"next_step.sign_in": {
"en": "Your account is ready — just sign in:\n\n{app}",
"hu": "A fiókod készen áll — csak lépj be:\n\n{app}",
},
"next_step.register": {
"en": (
"Create your account whenever you feel like it:\n\n"
"{app}\n\n"
"We'll send you a confirmation link to finish, and then you're in."
),
"hu": (
"Hozd létre a fiókod, amikor kedved tartja:\n\n"
"{app}\n\n"
"Küldünk egy megerősítő linket a befejezéshez, és már bent is vagy."
),
},
# ── Access decision ─────────────────────────────────────────────────────────────────────────
"access_approved.subject": {
"en": "You're in — your Siftlode access is approved",
"hu": "Bent vagy — jóváhagytuk a Siftlode-hozzáférésed",
},
"access_approved.body": {
"en": (
"Hi,\n\n"
"Your request to use Siftlode has been approved — welcome aboard.\n\n"
"{next_step}\n\n"
"If you weren't expecting this, you can ignore the message.\n\n"
"— Siftlode{contact}"
),
"hu": (
"Szia,\n\n"
"Jóváhagytuk a Siftlode-hozzáférési kérelmed — üdv a fedélzeten.\n\n"
"{next_step}\n\n"
"Ha nem számítottál erre, nyugodtan hagyd figyelmen kívül ezt a levelet.\n\n"
"— Siftlode{contact}"
),
},
"access_invitation.subject": {
"en": "You're invited to Siftlode",
"hu": "Meghívtak a Siftlode-ra",
},
"access_invitation.body": {
"en": (
"Hi,\n\n"
"Someone has opened up Siftlode for you — a quiet, self-hosted reader for your YouTube "
"subscriptions. Every upload from the channels you follow lands in one Feed you can "
"actually filter and sort, with no algorithm deciding what you see, and no Shorts or "
"livestreams unless you want them.\n\n"
"{next_step}\n\n"
"No rush — the door stays open.\n\n"
"— Siftlode{contact}"
),
"hu": (
"Szia,\n\n"
"Valaki megnyitotta előtted a Siftlode-ot — ez egy csendes, saját üzemeltetésű olvasó "
"a YouTube-feliratkozásaidhoz. A követett csatornák minden feltöltése egyetlen "
"Feedbe érkezik, amit tényleg tudsz szűrni és rendezni; nincs algoritmus, ami "
"eldönti, mit látsz, és nincs Shorts vagy élő adás, hacsak nem kéred.\n\n"
"{next_step}\n\n"
"Nem sürget semmi — az ajtó nyitva marad.\n\n"
"— Siftlode{contact}"
),
},
"access_denied.subject": {
"en": "About your Siftlode access request",
"hu": "A Siftlode-hozzáférési kérelmedről",
},
"access_denied.body": {
"en": (
"Hi,\n\n"
"Thanks for your interest in Siftlode. After review, your access request wasn't "
"approved this time.\n\n"
"If you think this was a mistake, you're welcome to reply and let us know.\n\n"
"— Siftlode{contact}"
),
"hu": (
"Szia,\n\n"
"Köszönjük az érdeklődésed a Siftlode iránt. Átnéztük a kérelmed, de most nem hagytuk "
"jóvá.\n\n"
"Ha szerinted ez tévedés, válaszolj nyugodtan erre a levélre.\n\n"
"— Siftlode{contact}"
),
},
# ── Account lifecycle ───────────────────────────────────────────────────────────────────────
"role_changed.subject": {
"en": "Your Siftlode role has changed",
"hu": "Megváltozott a szereped a Siftlode-on",
},
"role_changed.granted": {
"en": (
"You've been granted admin access on Siftlode — you can now manage settings, the "
"scheduler and users."
),
"hu": (
"Admin jogot kaptál a Siftlode-on — mostantól kezelheted a beállításokat, az ütemezőt "
"és a felhasználókat."
),
},
"role_changed.revoked": {
"en": "Your admin access on Siftlode has been removed — you're now a regular user.",
"hu": "Elvettük az admin jogod a Siftlode-on — mostantól sima felhasználó vagy.",
},
"role_changed.body": {
"en": "Hi,\n\n{line}{contact}\n\n— Siftlode",
"hu": "Szia,\n\n{line}{contact}\n\n— Siftlode",
},
"account_unsuspended.subject": {
"en": "Your Siftlode account has been reinstated",
"hu": "Visszaállítottuk a Siftlode-fiókod",
},
"account_unsuspended.body": {
"en": (
"Hi,\n\n"
"Your Siftlode account has been reinstated — the suspension was lifted and you can "
"sign in again.{contact}\n\n— Siftlode"
),
"hu": (
"Szia,\n\n"
"Visszaállítottuk a Siftlode-fiókod — a felfüggesztést feloldottuk, újra be tudsz "
"lépni.{contact}\n\n— Siftlode"
),
},
"account_deleted.subject": {
"en": "Your Siftlode account has been deleted",
"hu": "Töröltük a Siftlode-fiókod",
},
"account_deleted.body": {
"en": (
"Hi,\n\n"
"Your Siftlode account and all associated data — subscriptions, tags, watch history, "
"playlists and settings — have been permanently deleted. Nothing is retained.\n\n"
"Thanks for having used Siftlode.{contact}\n\n— Siftlode"
),
"hu": (
"Szia,\n\n"
"A Siftlode-fiókod és minden hozzá tartozó adat — feliratkozások, címkék, megtekintési "
"előzmények, lejátszási listák és beállítások — véglegesen törölve. Semmit nem őrzünk "
"meg.\n\n"
"Köszönjük, hogy használtad a Siftlode-ot.{contact}\n\n— Siftlode"
),
},
"account_suspended.subject": {
"en": "Your Siftlode account is suspended",
"hu": "A Siftlode-fiókod fel van függesztve",
},
"account_suspended.body": {
"en": (
"Hi,\n\n"
"A sign-in to your Siftlode account was just attempted, but the account is currently "
"suspended, so access was denied.\n\n{contact}— Siftlode"
),
"hu": (
"Szia,\n\n"
"Valaki most próbált belépni a Siftlode-fiókodba, de a fiók jelenleg fel van "
"függesztve, ezért a belépést elutasítottuk.\n\n{contact}— Siftlode"
),
},
# ── To the admins ───────────────────────────────────────────────────────────────────────────
"admin_new_request.subject": {
"en": "Siftlode access request from {requester}",
"hu": "Siftlode hozzáférési kérelem: {requester}",
},
"admin_new_request.body": {
"en": (
"{requester} just requested access to Siftlode.\n\n"
"Approve or deny it here:\n{link}\n"
"(or in Siftlode: open Users → Access requests)\n\n"
"Requester's email: {requester}\n"
"Replying to this message goes straight to the requester (it's the Reply-To address).\n"
),
"hu": (
"{requester} most kért hozzáférést a Siftlode-hoz.\n\n"
"Itt hagyhatod jóvá vagy utasíthatod el:\n{link}\n"
"(vagy a Siftlode-ban: Felhasználók → Hozzáférési kérések)\n\n"
"A kérelmező e-mail-címe: {requester}\n"
"Ha erre a levélre válaszolsz, az egyenesen a kérelmezőhöz megy (ő a Reply-To cím).\n"
),
},
# ── Link mails ──────────────────────────────────────────────────────────────────────────────
"verify_email.subject": {
"en": "Confirm your Siftlode email",
"hu": "Erősítsd meg a Siftlode e-mail-címed",
},
"verify_email.body": {
"en": (
"Hi,\n\n"
"Confirm your email to finish creating your Siftlode account:\n\n"
"{link}\n\n"
"The link works once and expires in {expiry}. If it does, you can ask for a fresh "
"one from the sign-in page.\n\n"
"If you didn't sign up, you can ignore this message.\n\n"
"— Siftlode"
),
"hu": (
"Szia,\n\n"
"Erősítsd meg az e-mail-címed, hogy elkészüljön a Siftlode-fiókod:\n\n"
"{link}\n\n"
"A link egyszer használható, és {expiry} múlva lejár. Ha lejárt, a belépési oldalon "
"kérhetsz újat.\n\n"
"Ha nem te regisztráltál, nyugodtan hagyd figyelmen kívül ezt a levelet.\n\n"
"— Siftlode"
),
},
"password_reset.subject": {
"en": "Reset your Siftlode password",
"hu": "Állíts be új Siftlode-jelszót",
},
"password_reset.body": {
"en": (
"Hi,\n\n"
"Use this link to set a new Siftlode password (it expires in an hour):\n\n"
"{link}\n\n"
"If you didn't request this, you can ignore this message — your password is "
"unchanged.\n\n"
"— Siftlode"
),
"hu": (
"Szia,\n\n"
"Ezzel a linkkel állíthatsz be új Siftlode-jelszót (egy óra múlva lejár):\n\n"
"{link}\n\n"
"Ha nem te kérted, hagyd figyelmen kívül ezt a levelet — a jelszavad változatlan.\n\n"
"— Siftlode"
),
},
"test.subject": {
"en": "Siftlode SMTP test",
"hu": "Siftlode SMTP-teszt",
},
"test.body": {
"en": (
"This is a test email from Siftlode.\n\n"
"If you're reading this, your SMTP settings work.\n\n"
"— Siftlode"
),
"hu": (
"Ez egy teszt e-mail a Siftlode-tól.\n\n"
"Ha ezt olvasod, az SMTP-beállításaid működnek.\n\n"
"— Siftlode"
),
},
}
+66 -4
View File
@@ -1,10 +1,13 @@
import asyncio
import logging
import math
import sys
from contextlib import asynccontextmanager, suppress
from pathlib import Path
from fastapi import FastAPI, HTTPException
from fastapi import FastAPI, HTTPException, Request
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
# When started via uvicorn --log-config the "siftlode" logger is already configured
# (see log_config.json). This block is a timestamped fallback for other entrypoints
@@ -21,18 +24,21 @@ log = logging.getLogger("siftlode.app")
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import state
from app.config import settings
from app.db import SessionLocal
from app.plex import stream as plex_stream
from app.plex import subtitles as plex_subtitles
from app.routes import (
admin,
audit as audit_routes,
auth as auth_routes,
channels,
config as config_routes,
discover as discover_routes,
downloads,
feed,
health,
@@ -76,6 +82,7 @@ async def lifespan(app: FastAPI):
# alone doesn't buy that; not waiting does. The sweep only touches directories no live session
# owns, so racing it against early playback is safe.
sweep = asyncio.create_task(_sweep_hls_orphans())
reaper = asyncio.create_task(_reap_hls_sessions())
start_scheduler()
try:
yield
@@ -85,10 +92,36 @@ async def lifespan(app: FastAPI):
# pending ("Task was destroyed but it is pending!" — noise that reads like a bug during
# triage). Cancelling does NOT interrupt the rmtree thread itself; this only makes our own
# bookkeeping honest.
sweep.cancel()
with suppress(asyncio.CancelledError):
await sweep
for task in (sweep, reaper):
task.cancel()
with suppress(asyncio.CancelledError):
await task
shutdown_scheduler()
# Kills a running subtitle extraction: its worker would otherwise hold interpreter exit for
# the rest of the read (see plex_subtitles.shutdown).
plex_subtitles.shutdown()
# How often idle HLS sessions are reaped (see plex/stream.py `reap_idle`).
_HLS_REAP_EVERY_S = 60.0
async def _reap_hls_sessions() -> None:
"""The periodic reaper plex/stream.py's docstring promises. `reap_idle` used to run only inside
the NEXT session request, so after the last playback of the day an abandoned session — its
ffmpeg still working through the film, its directory a whole segmented copy of it — lived until
somebody pressed play again or the process restarted.
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
held one whole segmented copy of a film per item on the download volume (round 5)."""
while True:
await asyncio.sleep(_HLS_REAP_EVERY_S)
try:
await asyncio.to_thread(plex_stream.reap_idle)
except Exception:
log.exception("HLS session reap failed")
async def _sweep_hls_orphans() -> None:
@@ -107,6 +140,34 @@ async def _sweep_hls_orphans() -> None:
app = FastAPI(title=settings.app_name, lifespan=lifespan)
@app.exception_handler(RequestValidationError)
async def _validation_error(request: Request, exc: RequestValidationError) -> JSONResponse:
"""FastAPI's own 422, in its own shape — except that it can be ENCODED. The JSON parser accepts
`NaN`/`Infinity`, a float field refuses them, and the refusal echoes the value back: the default
handler then failed to serialise its own error (JSON has no NaN) and the request became a 500
with a traceback. A non-finite value is echoed as its name instead."""
finite = {float: lambda x: x if math.isfinite(x) else str(x)}
return JSONResponse(
status_code=422, content={"detail": jsonable_encoder(exc.errors(), custom_encoder=finite)}
)
@app.exception_handler(DataError)
async def _nul_in_text(request: Request, exc: DataError) -> JSONResponse:
"""A NUL byte in a string the request sent is a 422, on every route at once. Postgres cannot hold
NUL in text, so the driver refuses to even send the query — and a `%00` in a query parameter or
an escaped NUL in a JSON body (a playlist's `contains=`, a reorder's key list, a collection
search) was a 500 with a traceback wherever it reached a comparison. Guarding each site did not
hold: round 3 guarded the by-rating_key lookup and missed the rest. Any other DataError is left
alone — re-raised, it stays the 500 it always was."""
if "NUL (0x00)" not in str(exc.orig):
raise exc
return JSONResponse(
status_code=422, content={"detail": {"code": "invalid_input", "reason": "nul"}}
)
app.add_middleware(
SessionMiddleware,
secret_key=settings.secret_key,
@@ -187,6 +248,7 @@ app.include_router(auth_routes.router)
app.include_router(sync.router)
app.include_router(tags.router)
app.include_router(feed.router)
app.include_router(discover_routes.router)
app.include_router(search_routes.router)
app.include_router(me.router)
app.include_router(notifications.router)
+162 -3
View File
@@ -3,6 +3,7 @@ from datetime import date, datetime
from sqlalchemy import (
BigInteger,
Boolean,
CheckConstraint,
Computed,
Date,
DateTime,
@@ -17,9 +18,15 @@ from sqlalchemy import (
func,
)
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.utils import canonical_email
# The resume-position columns (VideoState / PlexState.position_seconds) are int4. The checkpoint
# routes bound their bodies by this, so an out-of-range position is a 422 rather than a 500.
POSITION_MAX_S = 2**31 - 1
class TimestampMixin:
@@ -29,6 +36,21 @@ class TimestampMixin:
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
class CanonicalEmailMixin:
"""Keeps `email_canonical` derived from `email` on every write.
The invariant — "identity is matched on the canonical form, mail is sent to the typed one" —
lives HERE, in code, instead of in a comment every write site has to remember. That distinction
is what the 2026-09-03 lockout was made of: one gate lowercased, nothing else did, and an
approved user's Google sign-in silently became a second access request.
"""
@validates("email")
def _derive_email_canonical(self, _key: str, value: str | None) -> str | None:
self.email_canonical = canonical_email(value)
return value
class UpdatedAtMixin:
"""Adds an `updated_at` that server-defaults on insert and bumps on every UPDATE."""
@@ -37,7 +59,7 @@ class UpdatedAtMixin:
)
class User(Base, TimestampMixin):
class User(Base, TimestampMixin, CanonicalEmailMixin):
__tablename__ = "users"
id: Mapped[int] = mapped_column(primary_key=True)
@@ -45,8 +67,26 @@ class User(Base, TimestampMixin):
# multiple NULLs under a unique index). Set on Google sign-in / in-app linking.
google_sub: Mapped[str | None] = mapped_column(String(64), unique=True, index=True)
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
# The same mailbox, in the form used to MATCH identities (utils.canonical_email) — Gmail dots and
# `+tags` folded away. Kept by the validator below so no write site can forget it. Deliberately
# NOT unique: prod already carries two rows for one mailbox (a password registration and the
# Google sign-in that failed to find it), and a migration must not merge accounts on its own.
email_canonical: Mapped[str] = mapped_column(String(320), index=True, server_default="")
# The address Google asserted for this account, recorded at sign-in / link time. NOT the same
# thing as `email`: since E9.5 S3 a Google sign-in no longer overwrites the address the user
# registered with, so a deliberate `me+siftlode@gmail.com` survives — and this is where the other
# half of the truth lives, for the UI to show when the two differ. NULL until the next Google
# sign-in for accounts linked before this column existed.
google_email: Mapped[str | None] = mapped_column(String(320))
# argon2id hash for email+password login; NULL for Google-only / demo accounts.
password_hash: Mapped[str | None] = mapped_column(String(255))
# When SEC-10 dropped a preset password on Google adoption (see the single caller of
# auth.revoke_preset_password_if_unverified; admin approval no longer revokes — E9.5 S1). Without
# this trace a revoked account was indistinguishable from a Google-only one, so it failed the
# password-reset eligibility check and could never be recovered — it could neither sign in nor
# reset. Set = "had a password, lost it to the guard" -> reset-eligible. Cleared whenever a
# password is set again (reset-confirm / set-password).
password_revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Email ownership proven (Google sign-in is implicitly verified; password registration
# verifies via a link). Login requires this AND is_active.
email_verified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
@@ -114,7 +154,7 @@ class AuthToken(Base, TimestampMixin):
used_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class Invite(Base):
class Invite(Base, CanonicalEmailMixin):
"""Access-request / whitelist row. Source of truth for who may sign in; the env
ALLOWED_EMAILS/ADMIN_EMAILS act only as a bootstrap fallback (see auth.is_allowed)."""
@@ -122,6 +162,9 @@ class Invite(Base):
id: Mapped[int] = mapped_column(primary_key=True)
email: Mapped[str] = mapped_column(String(320), unique=True, index=True)
# See User.email_canonical. Not unique here either: prod has approved invites for both spellings
# of the same mailbox, dating back to 2026-06-16.
email_canonical: Mapped[str] = mapped_column(String(320), index=True, server_default="")
# pending | approved | denied
status: Mapped[str] = mapped_column(String(16), default="pending", server_default="pending")
note: Mapped[str | None] = mapped_column(Text)
@@ -130,6 +173,17 @@ class Invite(Base):
)
decided_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
decided_by: Mapped[str | None] = mapped_column(String(320)) # admin email
# When the admin last SENT this person an invitation from the whitelist row. Adding an address
# stays silent — it opens a door, it does not knock — so telling them is a separate, deliberate
# act, and this is what stops it happening twice to a real person.
invited_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The language this person asked us in ("en"/"hu"), captured when the request was made: the page
# their form was on, or the locale Google reported for a refused sign-in. NULL for a row the
# admin whitelisted by hand — nobody has told us anything about that mailbox, which is exactly
# why the invitation send asks (E9.5 S7). Captured rather than derived for the same reason
# `preferences.language` and `google_email` are: the signal exists only in that one moment and
# cannot be backfilled afterwards.
lang: Mapped[str | None] = mapped_column(String(8))
class DemoWhitelist(Base, TimestampMixin):
@@ -374,6 +428,7 @@ class VideoState(Base, UpdatedAtMixin):
# status: a partially-watched video keeps status "new" but a non-zero position, so
# it can render a progress bar and match the "in progress" feed filter. 0 = not
# started / finished (trivially-early and near-finished positions are not stored).
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(
Integer, default=0, server_default="0", nullable=False
)
@@ -551,6 +606,10 @@ class Playlist(Base, TimestampMixin, UpdatedAtMixin):
# SHA-256 of name + ordered video ids as last synced from / pushed to YouTube; the
# baseline `dirty` is derived against. NULL = no baseline yet (treated as dirty).
synced_fingerprint: Mapped[str | None] = mapped_column(String)
# JSON list of the ordered video ids as of the last sync — the 3-way-merge baseline the
# two-way YouTube sync classifies each item against (local-add vs YT-add vs real delete).
# NULL = no baseline yet; the first merge then does a lossless union (never deletes).
synced_video_ids: Mapped[str | None] = mapped_column(Text)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
items: Mapped[list["PlaylistItem"]] = relationship(
@@ -704,6 +763,12 @@ class MediaAsset(Base, TimestampMixin):
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Set once the GC has warned holders of the upcoming expiry, so it doesn't warn every run.
gc_notified: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
@@ -867,6 +932,35 @@ class DownloadLink(Base, TimestampMixin):
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
class MessageKey(Base, TimestampMixin):
"""A user's end-to-end-encryption key material for direct messaging (phase 2).
@@ -1088,6 +1182,22 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
# Browser-playability class from ffprobe: "direct" | "remux" | "transcode" (None = unprobed).
playable: Mapped[str | None] = mapped_column(String(12), index=True)
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Transcode HW-capability facet (E7 playback-class filter). NULL for non-transcode items; for a
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
# probed (the rest are SW-only by codec and need no bit-depth); see `_probe_media_batch`.
media_probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Intro/credit markers: [{"type": "intro"|"credits", "start_s": int, "end_s": int}, ...]
markers: Mapped[list | None] = mapped_column(JSON)
added_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
@@ -1303,3 +1413,52 @@ class PlexPlaylistItem(Base):
ForeignKey("plex_items.id", ondelete="CASCADE"), index=True
)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
class ComfortPin(Base, TimestampMixin):
"""One entry on a user's comfort shelf — the anti-paralysis fallback: "just put on X" (E24).
Deliberately NOT a boolean on `VideoState`/`PlexState`, and NOT a polymorphic `(kind, ref)` string
pair. A flag cannot ORDER a shelf and cannot hold a channel or a show — and comfort media is
usually "put on X", a series or a channel, not one particular video. A string ref would order and
hold anything, but it would also lose the foreign key: a Plex item dropped from the library, or an
unsubscribed channel, would leave a dead tile on the one surface whose whole job is to be
reliable. So instead: one nullable CASCADE-ing FK per pinnable kind, and a check constraint that
exactly one of them is set.
That constraint is the house rule applied to a five-way union — the invariant lives in the schema,
not in a comment every insert site has to remember. `kind`/`ref` below derive FROM the columns, so
there is no second vocabulary to keep in sync.
NULLs are DISTINCT in a Postgres unique index, so the five per-kind unique constraints each stop a
double-pin of the same thing without colliding across kinds."""
__tablename__ = "comfort_pins"
__table_args__ = (
CheckConstraint(
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
name="ck_comfort_pin_one_ref",
),
UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
video_id: Mapped[str | None] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"))
channel_id: Mapped[str | None] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
plex_item_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_items.id", ondelete="CASCADE")
)
plex_show_id: Mapped[int | None] = mapped_column(
ForeignKey("plex_shows.id", ondelete="CASCADE")
)
playlist_id: Mapped[int | None] = mapped_column(ForeignKey("playlists.id", ondelete="CASCADE"))
# 0-based shelf order. The shelf is short and hand-curated, so a dense integer the reorder
# endpoint rewrites wholesale beats a fractional index nobody would ever exercise here.
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0", nullable=False)
+2
View File
@@ -33,6 +33,7 @@ def movie_card(it: PlexItem, st: PlexState | None) -> dict:
"thumb": f"/api/plex/image/{it.rating_key}",
"art": _art_url(it.rating_key, it.art_key),
"playable": it.playable,
"transcode_hw": it.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
"status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0,
}
@@ -65,6 +66,7 @@ def episode_card(e: PlexItem, st: PlexState | None) -> dict:
"duration_seconds": e.duration_s,
"thumb": f"/api/plex/image/{e.rating_key}",
"playable": e.playable,
"transcode_hw": e.transcode_hw, # HW/SW transcode split → the 4-way corner-dot badge
"status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0,
}
+6 -1
View File
@@ -197,7 +197,12 @@ class PlexClient:
)
def unscrobble(self, rating_key: str) -> None:
"""Mark the item *unwatched* on the token's Plex account (clears viewCount)."""
"""Mark the item *unwatched* on the token's Plex account.
Clears the view count AND the resume point (`viewOffset`) — measured against the live server
on 2026-09-09, from both a watched+offset state and an offset-only one. This one call is the
whole un-mark; `/:/timeline?time=0` and `/:/progress?time=0` were both tried there and do
NOT clear an offset, so do not reach for them."""
self._write(
"GET",
f"/:/unscrobble?identifier={self._LIBRARY_IDENTIFIER}"
+7 -1
View File
@@ -11,5 +11,11 @@ _M = TypeVar("_M")
def by_rating_key(db: Session, model: type[_M], rating_key: str) -> _M | None:
"""The mirrored row for a Plex rating_key, or None. `model` is any of the rating_key-keyed Plex
tables (PlexItem / PlexShow / PlexSeason / PlexCollection); the return type is that model, so call
sites keep their `Model | None` static type."""
sites keep their `Model | None` static type.
A key with a NUL in it (`%00` in a path segment) names no row — a 404, like any unknown key.
Postgres refuses NUL in text, so asking would be a DataError; the app-wide handler turns the
places that DO ask (lists and filters compared to rating_key) into a 422 (see main.py)."""
if "\x00" in str(rating_key):
return None
return db.execute(select(model).where(model.rating_key == str(rating_key))).scalars().first()
+71
View File
@@ -8,9 +8,28 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import and_, case, func, or_, select, type_coerce
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Session
from app.models import PlexItem, PlexLibrary
TS_CONFIG = "public.unaccent_simple"
# The ONE Plex sort vocabulary, shared by every surface that offers the Sort control (the browse grid
# and the collection info page) and mirrored by the frontend's `plexSort.ts`. It lives here rather than
# inside a router because the two surfaces order DIFFERENT things — browse orders a union subquery in
# SQL, a collection orders its already-loaded member list in Python — and a key that one of them cannot
# honour would put an option in the UI that silently does nothing. Tests pin both implementations to
# this tuple, so adding a key here fails the gate until every surface can order by it.
PLEX_SORT_KEYS: tuple[str, ...] = (
"added",
"lastwatched",
"release",
"year",
"rating",
"duration",
"title",
)
# --- Language-aware column pickers (E5 S6) ----------------------------------
#
# The HU-overlay twin (E5 S5) stores Hungarian metadata in `*_hu` columns on the SAME EN row. When
@@ -206,6 +225,33 @@ def apply_meta_filters(q, model, p: dict, lang: str = "en"):
return q.where(*conds) if conds else q
# --- Playback-class filter (E7) — MOVIE-ONLY (a show is an aggregate with no single class) ---------
# The 4 classes the feed filter + facet expose, derived from the stored `playable` + `transcode_hw`.
# ONE definition (this SQL CASE) so the filter WHERE and the facet counts can't drift: `playable_conds`
# filters on it, `facets` groups by it. transcode_hw IS NOT True groups NULL (unprobed) with software.
# Public — it is the shared source of truth imported by routes.plex.browse (facet order + zero-fill).
PLAYBACK_CLASSES = ("direct", "remux", "transcode_hw", "transcode_sw")
def playback_class_expr():
"""SQL CASE mapping a PlexItem (movie) to its playback-class string, or NULL. The HW branch must
precede the generic transcode branch (ordered evaluation)."""
return case(
(PlexItem.playable == "direct", "direct"),
(PlexItem.playable == "remux", "remux"),
(and_(PlexItem.playable == "transcode", PlexItem.transcode_hw.is_(True)), "transcode_hw"),
(PlexItem.playable == "transcode", "transcode_sw"),
else_=None,
)
def playable_conds(sel: list[str]) -> list:
"""A one-element condition list (or empty) selecting the given playback classes — an OR via
`class IN (...)` over the shared CASE. Unknown keys are dropped."""
keep = [c for c in sel if c in PLAYBACK_CLASSES]
return [playback_class_expr().in_(keep)] if keep else []
@dataclass
class LibraryFilters:
"""The shared sidebar filter query-params for /library + /facets, declared once and injected via
@@ -228,6 +274,7 @@ class LibraryFilters:
keywords: str | None = None # TMDB keyword chips (E5 S5), CSV; language-neutral
collections: str | None = None # CSV of collection rating_keys
collection_mode: str = "any" # how multiple collections combine: any (OR, default) | all (AND)
playable: str | None = None # CSV of playback classes: direct|remux|transcode_hw|transcode_sw
def as_dict(self) -> dict:
return {
@@ -247,6 +294,7 @@ class LibraryFilters:
"keywords": self.keywords,
"collections": self.collections,
"collection_mode": self.collection_mode,
"playable": self.playable,
}
@@ -311,3 +359,26 @@ def apply_show_state_filter(query, tot, wat, inp, show: str):
if show == "unwatched":
return query.where(wat == 0, inp == 0)
return query
def content_libs(db: Session) -> tuple[list[int], list[int]]:
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5).
Lives here, in the query layer, rather than in `routes/plex/_shared.py` where it started: E24's
Home surface needs the same scope, and a domain package importing from `routes/` would invert the
one-way dependency CLAUDE.md draws. It never touched request state, so nothing was lost by the
move."""
libs = (
db.execute(
select(PlexLibrary).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
)
.scalars()
.all()
)
movie_lib_ids = [lb.id for lb in libs if lb.kind == "movie"]
show_lib_ids = [lb.id for lb in libs if lb.kind == "show"]
return movie_lib_ids, show_lib_ids
+21 -3
View File
@@ -3,11 +3,25 @@
Drives the show/season grid badges and the Resume / Play-from-start / mark-all buttons.
"""
from datetime import datetime, timezone
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.models import PlexItem, PlexState
# Sort sentinel for a rail row whose timestamp is missing, so `key=lambda r: r[0] or NO_TS` reads the
# same at every rail that merges two type-homogeneous lists in Python.
#
# It is NOT a crash fix, whatever the E24 commit that introduced it says. The `(ts is not None, ts)`
# key it replaced cannot raise: tuple comparison finds the first UNEQUAL element via `==`, so two
# `(False, None)` keys compare equal and `None < None` is never reached — verified by execution
# during the E24 epic-end review, over every permutation of null/non-null rows. What the sentinel
# buys is one idiom instead of two, and a key that says "missing sorts last" outright rather than
# encoding it in a boolean flag. A BARE `r[0]` would genuinely raise, which is the mutation the
# rails' tests do guard against.
NO_TS = datetime.min.replace(tzinfo=timezone.utc)
def show_status(total: int, watched: int, inprog: int) -> str:
"""Roll a show's episodes up into a single watch state: fully watched, partially started, or new."""
@@ -43,8 +57,10 @@ def rollup(cards: list[dict]) -> dict:
def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
"""Per-show episode counts for a user: total, watched, in-progress. Reused to (a) filter the show
grid by aggregate watch-state, (b) compute each card's badge, and (c) drive the Discover
continue-watching rail. `include_resumed` adds `resumed_at` = max(progress_updated_at) across the
show's episodes, so the rail can order shows by how recently they were last progressed."""
continue-watching rail, and (d) the browse "last watched" sort. `include_resumed` adds `resumed_at`
= max(coalesce(progress_updated_at, watched_at)) across the show's episodes — i.e. how recently ANY
episode was last touched (resumed OR marked watched) — so callers can order shows by recency. A
fully-watched show (no resume progress) still gets a timestamp from its episodes' watched_at."""
ps = aliased(PlexState)
cols = [
PlexItem.show_id.label("show_id"),
@@ -63,7 +79,9 @@ def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
).label("inprog"),
]
if include_resumed:
cols.append(func.max(ps.progress_updated_at).label("resumed_at"))
cols.append(
func.max(func.coalesce(ps.progress_updated_at, ps.watched_at)).label("resumed_at")
)
return (
select(*cols)
.outerjoin(ps, and_(ps.item_id == PlexItem.id, ps.user_id == user_id))
+642 -53
View File
@@ -3,7 +3,10 @@
Playback is from the LOCAL physical file. Browser-compatible files (playable="direct") are served
raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
when needed. HEVC/VP9 (playable="transcode") needs a full re-encode and is deferred to P3.
when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
software libx264 as the automatic fallback where the render node is absent or the source codec isn't
hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
(fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
@@ -16,19 +19,24 @@ CPU-only prod host; only P3's full transcode is CPU-heavy.
"""
import logging
import os
import re
import secrets
import shutil
import subprocess
import threading
import time
from collections.abc import Sequence
from pathlib import Path
from sqlalchemy.orm import Session as DbSession
from app import sysconfig
from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.models import PlexItem
from app.plex import paths
from app.plex.sync import HW_DECODABLE_CODECS
log = logging.getLogger("siftlode.plex")
@@ -38,6 +46,21 @@ _HLS_ROOT = (
else Path(settings.download_root) / ".plex-hls"
)
_SEG_SECONDS = 6
# Cold-start warm-up. A freshly-spawned ffmpeg has produced only the first segment (or two) by the
# time we could report the session ready; hls.js parses the EVENT playlist, drains that tiny buffer,
# then STALLS a few seconds into playback waiting for the next segment the still-warming ffmpeg
# hasn't written yet. Very visible on auto-advance to the next episode (start_s==0, which previously
# returned with ZERO buffer). So before reporting ready we wait for the first few segments to exist:
# the client then starts with ~_WARMUP_SEGMENTS×_SEG_SECONDS of buffer and, since stream-copy runs
# faster than realtime, ffmpeg stays ahead from there and never stalls again. Best-effort and bounded
# — we stop at the first segment that isn't ready within the per-segment budget (a slow miss is no
# worse than the old zero-buffer behaviour).
_WARMUP_SEGMENTS = 3
_WARMUP_SEG_TIMEOUT_S = 8.0
# The HLS variant ffmpeg gives the VIDEO rendition of a multi-audio session. `-var_stream_map` names
# each audio rendition (`name:a0`, `a1`, …) but leaves the video one unnamed, so ffmpeg falls back to
# its index for that one. See `segment_name` for why this is a constant and not a comment.
_VIDEO_VARIANT = "0"
_SESSION_IDLE_S = 600 # reap a session with no access for this long
# A session directory is named `{rating_key}_{int(start_s)}_{tag}_{aoff:+.2f}` (see start_session);
# before multi-audio it was just `{rating_key}_{int(start_s)}`. `sweep_orphans` deletes ONLY names
@@ -72,11 +95,30 @@ _sessions: dict[str, "HlsSession"] = {}
class HlsSession:
def __init__(
self, key: str, directory: Path, proc: subprocess.Popen, start_s: float, entry: str
self,
key: str,
directory: Path,
proc: subprocess.Popen,
start_s: float,
variants: "Sequence[str]",
transcode: bool = False,
hw: bool = False,
tonemap: bool = False,
):
# Unguessable per-session id, carried in the URLs the client is handed. Sessions are keyed
# by rating_key alone, so a seek-restart or a SECOND viewer of the same item replaces the
# registered one — and the old viewer's next request for `seg_12.ts`, resolved by rating_key,
# was answered 200 with the NEW session's twelfth segment, produced from a different offset
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it
# (deep-review round 5). A request naming a session that is no longer registered is a 404,
# which the player recovers from by restarting at the playhead.
self.id = secrets.token_urlsafe(9)
self.key = key
self.dir = directory
self.proc = proc
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
self.start_s = start_s # the REQUESTED seek offset
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
@@ -85,7 +127,13 @@ class HlsSession:
# subtitle cue-shift both key off the true content position — no drift. Provisional = X until
# the first segment is probed.
self.media_start_s = start_s
self.entry = entry # the playlist filename hls.js should load (master.m3u8 with subs, else index.m3u8)
# The rendition names this session's ffmpeg was told to write — () for a single-audio
# session. Held on the session because it is what `artifact_may_arrive` needs to tell a name
# this session CAN produce from one belonging to the other argv branch (see it).
self.variants = tuple(variants)
# The playlist filename hls.js should load: the master of a multi-audio session, else the
# single index. Derived, not passed, so it cannot disagree with the argv.
self.entry = _MASTER_PL if self.variants else _INDEX_PL
self.last_access = time.time()
@@ -116,6 +164,293 @@ _HLS_TAIL = [
]
# --- Transcode (playable="transcode") video re-encode -----------------------------------------
# The remux path stream-copies the video (`-c:v copy`); a non-h264 source can't be copied and must be
# re-encoded to H.264. The encoder + tonemap are chosen per-session by `_select_encode`:
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
# 1080p transcodes are fine and the API stays responsive.
# * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
# iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
# it reserves a core via `-threads` (see encode_thread_cap).
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
_VAAPI_DEVICE = "/dev/dri/renderD128"
# Where an OpenCL ICD registers its driver. The Intel NEO runtime (intel-opencl-icd) drops an .icd file
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
# codec; aliased to the old name to keep the call sites below unchanged.
_HW_DECODABLE = HW_DECODABLE_CODECS
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
# (_venc_opts tonemap branch) when the OpenCL runtime is present, else software libx264. A 10-bit source
# WITHOUT one of these transfers is SDR, not HDR, and must not be tone-mapped. See _probe_video_kind.
_HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
# The three source classes that drive the transcode-pipeline choice (see _probe_video_kind).
_VK_HWSAFE = "hwsafe" # 8-bit SDR — the plain scale_vaapi=format=nv12 GPU path
_VK_HDR = "hdr" # PQ/HLG HDR — needs the OpenCL tonemap graph (or software if no runtime)
_VK_SDR10 = "sdr10" # 10-bit *SDR* — a plain 10→8-bit downconvert, NEVER the HDR tone curve
def _probe_video_kind(src: Path) -> str:
"""Classify the source for the transcode pipeline — one of `_VK_HWSAFE` / `_VK_HDR` / `_VK_SDR10`:
* `_VK_HWSAFE` — 8-bit SDR: safe for the plain VAAPI path (`scale_vaapi=format=nv12` + h264_vaapi).
* `_VK_HDR` — PQ/HLG HDR (any depth): the plain scale path breaks on Gen 9.5 VPP, so it needs the
OpenCL tonemap graph (`tonemap_opencl=hable`) when the runtime is present, else software libx264.
* `_VK_SDR10` — 10-bit but *SDR* (bt709 etc.): NOT HDR. Applying the HDR hable tone curve here would
visibly alter an SDR picture (crushed highlights/midtones) where only a 10→8-bit downconvert was
needed, so it must NOT take the tonemap graph — it goes to software libx264, which CPU-decodes and
downconverts to 8-bit yuv420p exactly. (Kept off the plain scale_vaapi 10-bit path too: that path
is unverified on Gen 9.5 VPP, and software is the known-correct downconvert.)
Ground truth from ffprobe (bit depth via pix_fmt / bits_per_raw_sample, HDR via color_transfer); the
Plex mirror doesn't store these. Returns `_VK_HWSAFE` on any probe failure: the runtime HW→SW fallback
in start_session backstops a HW session that still produces nothing, so a probe miss only costs the
rare wasted first attempt, never a stranded item."""
try:
out = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=pix_fmt,bits_per_raw_sample,color_transfer",
"-of",
"default=noprint_wrappers=1",
str(src),
],
capture_output=True,
text=True,
timeout=10,
)
except (subprocess.SubprocessError, OSError):
return _VK_HWSAFE
pix_fmt = bits = transfer = ""
for line in out.stdout.splitlines():
k, _, v = line.strip().partition("=")
v = v.strip().lower()
if k == "pix_fmt":
pix_fmt = v
elif k == "bits_per_raw_sample":
bits = v
elif k == "color_transfer":
transfer = v
if transfer in _HDR_TRANSFERS:
return _VK_HDR # PQ/HLG — needs a real HDR→SDR tonemap, not a plain downconvert
# >8-bit but not HDR = 10-bit SDR. bits_per_raw_sample OR the pix_fmt bit-depth suffix
# (yuv420p10le, p010le, yuv444p12le…) signals >8-bit even when the other is absent.
if (bits.isdigit() and int(bits) > 8) or re.search(r"(?:10|12|16)(?:le|be)$", pix_fmt):
return _VK_SDR10
return _VK_HWSAFE
def _opencl_present() -> bool:
"""Whether an OpenCL runtime (Intel NEO / intel-opencl-icd) is installed — the precondition for the
HW HDR→SDR tonemap graph. Gen 9.5 VPP can't tonemap natively (tonemap_vaapi fails), so the working
HW path for a 10-bit/HDR source is VAAPI-decode → OpenCL tonemap → VAAPI-encode, which needs an ICD
registered here. Absent on the dev box and until the image bakes NEO in → those sources take
software libx264. Its own function so it's monkeypatchable in tests, mirroring _render_node_present."""
try:
return any(_OPENCL_ICD_DIR.glob("*.icd"))
except OSError:
return False
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
round-trip to system memory. Empty for the software path.
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
if not hw:
return []
if tonemap:
return [
"-init_hw_device",
f"vaapi=va:{_VAAPI_DEVICE}",
"-init_hw_device",
"opencl=ocl@va",
"-hwaccel",
"vaapi",
"-hwaccel_output_format",
"vaapi",
"-hwaccel_device",
"va",
]
return ["-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", _VAAPI_DEVICE]
# Audio is ALWAYS re-encoded on the HLS path: the source is typically AC3/DTS 5.1, which no browser
# plays, so it is downmixed to stereo AAC. `-ac 2` uses ffmpeg's standard downmix matrix; nothing is
# normalised, which is why a film mastered for a cinema arrives with whispered dialogue and deafening
# action (bug hunt 2026-09, item 14).
_AUDIO_ENC = ["-c:a", "aac", "-ac", "2", "-b:a", "192k"]
# Optional loudness normalisation (admin knob `plex_audio_loudnorm`, default OFF). EBU R128 single-pass
# `loudnorm` — two-pass is impossible here, since a seek-restart session must start emitting segments
# immediately and cannot analyse the file first.
#
# The downmix is FORCED FIRST (`aformat=channel_layouts=stereo`) rather than left to `-ac 2`: measuring
# 5.1 and then folding it down would normalise something other than what the viewer hears. `loudnorm`
# runs its own resampler at 192 kHz internally, so the chain resamples back to 48 kHz for the AAC
# encoder. Measured on a real 5.1 AC3 film (Westworld S1E6, 60 s from 15:00): I -27.6 → -16.6 LUFS,
# LRA 13.3 → 9.8 LU — the quiet/loud gap narrows, and everything gets ~11 dB louder, so expect to turn
# the volume down after switching this on. The filter's 3 s lookahead does NOT desync A/V: measured
# under this pipeline's own `-copyts`, the audio start PTS is identical with and without it.
_LOUDNORM_AF = "aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000"
def _aenc_opts(loudnorm: bool) -> list[str]:
"""Audio encoder args for one output, with the normalisation filter when the admin enabled it."""
return [*_AUDIO_ENC, *(["-af", _LOUDNORM_AF] if loudnorm else [])]
def _venc_opts(
transcode: bool,
hw: bool,
tonemap: bool = False,
preset: str = "veryfast",
crf: int = 20,
max_height: int = 0,
) -> list[str]:
"""The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
path leaned on the source's keyframes, but a re-encoded stream has none the HLS muxer can cut on
unless we force them (`-force_key_frames` every `_SEG_SECONDS`), else segments aren't independent and
seeking breaks. `preset`/`crf`/`max_height` (software only) come from admin sysconfig."""
if not transcode:
return ["-c:v", "copy"]
force_kf = ["-force_key_frames", f"expr:gte(t,n_forced*{_SEG_SECONDS})"]
if hw and tonemap:
# HDR/10-bit HW path: Gen 9.5 VPP can't tonemap (scale_vaapi=format=nv12 fails), so the decoded
# VAAPI surfaces are mapped to OpenCL (hwmap=derive_device=opencl), tone-mapped HDR→SDR to 8-bit
# nv12 there (tonemap_opencl, hable curve, bt709 out), then mapped back to VAAPI (reverse=1) for
# h264_vaapi. `-filter_hw_device ocl` tells the filter graph which device to run the OpenCL
# filters on. Full GPU pipeline — ~1 CPU core, several×realtime on the UHD 630 (measured).
return [
"-filter_hw_device",
"ocl",
"-vf",
"hwmap=derive_device=opencl,"
"tonemap_opencl=tonemap=hable:transfer=bt709:matrix=bt709:primaries=bt709:format=nv12,"
"hwmap=derive_device=vaapi:reverse=1",
"-c:v",
"h264_vaapi",
*force_kf,
]
if hw:
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
# The GPU encode is cheap, so it runs at source resolution (max_height is a software-only knob).
return ["-vf", "scale_vaapi=format=nv12", "-c:v", "h264_vaapi", *force_kf]
# Software fallback: same browser-safe recipe as downloads/edit.py (libx264). yuv420p forces 8-bit
# High (not High10) so any browser decodes it; -threads reserves a core so the CPU-bound encode can't
# starve request handling / playback on the shared box.
args: list[str] = []
if max_height and max_height > 0:
# Downscale-ONLY to the height cap (min(ih,H)), keeping aspect with an even width (-2), so a tall
# source doesn't make the software re-encode ruinously expensive. The comma in min() is escaped
# (\,) so ffmpeg's filter parser reads one filter, not two. Never upscales a smaller source.
args += ["-vf", f"scale=-2:min(ih\\,{max_height})"]
args += [
"-c:v",
"libx264",
"-preset",
preset,
"-crf",
str(crf),
"-pix_fmt",
"yuv420p",
"-threads",
str(encode_thread_cap()),
*force_kf,
]
return args
def _render_node_present() -> bool:
"""Whether the VAAPI render node is available — i.e. `/dev/dri` was passed into the container.
Present on prod after the iGPU enablement; absent on the CPU-only dev box and in the localdev
Docker. Its own function so the selection is monkeypatchable in tests and one place for S1c's knob."""
return Path(_VAAPI_DEVICE).exists()
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
Honours the admin `plex_transcode_hw` policy: "off" → always software; "on" → HW wherever the render
node exists (trusting the admin, skipping the decodability gate — the runtime HW→SW fallback still
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
if not transcode:
return (False, False)
mode = (sysconfig.get_str(db, "plex_transcode_hw") or "auto").lower()
if mode == "off" or not _render_node_present():
return (False, False)
if mode == "on":
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
return (True, tonemap)
# auto
if (item.codec_video or "").lower() not in _HW_DECODABLE:
return (False, False)
kind = _probe_video_kind(src)
if kind == _VK_HWSAFE:
return (True, False) # 8-bit SDR → full-GPU scale path
if kind == _VK_HDR:
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
return (True, True) if _opencl_present() else (False, False)
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
return (False, False)
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
"""The HW gate on its own (the `hw` half of `_select_encode`). A readability shim for the tests that
assert only whether a session runs on the GPU (production decides via `_select_encode` directly)."""
return _select_encode(db, item, transcode, src)[0]
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
_SW_TRANSCODE_NICE = settings.download_child_nice
def _sw_child_preexec() -> None:
"""Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
playback."""
try:
os.nice(_SW_TRANSCODE_NICE)
except OSError:
pass
def _probe_audio_langs(src: Path) -> list[str]:
"""One entry per audio stream (its language tag, or "" if untagged). len() = audio-track count.
Used to build the multi-rendition var_stream_map. Empty on probe failure → single-audio path."""
@@ -149,8 +484,20 @@ def _ffmpeg_cmd(
audio_ord: int | None,
aoff: float = 0.0,
audio_langs: list[str] | None = None,
transcode: bool = False,
hw: bool = False,
tonemap: bool = False,
preset: str = "veryfast",
crf: int = 20,
max_height: int = 0,
loudnorm: bool = False,
) -> list[str]:
args = ["ffmpeg", "-nostdin", "-loglevel", "error"]
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
@@ -167,11 +514,15 @@ def _ffmpeg_cmd(
if audio_langs:
# MULTI-RENDITION: map EVERY audio track as an alternate HLS rendition in one group, so hls.js
# switches audio CLIENT-SIDE on the same timeline (no session restart, no drift) — the premium
# audio-switch path. Produces master.m3u8 + stream_%v.m3u8 (v0=video, v1..=audio) + seg_%v_%d.ts.
# audio-switch path. Produces master.m3u8 + one stream_%v.m3u8 and seg_%v_*.ts per variant,
# where %v is the video rendition's index ("0") and each audio rendition's `name:` (a0, a1,
# …). Those names are spelled by `segment_name` / `variant_playlist_name` below, which the
# guard and the warm-up read back — see them for the measurement behind the ordering.
args += ["-map", "0:v:0"]
for i in range(len(audio_langs)):
args += ["-map", f"{ai}:a:{i}?"]
args += ["-sn", "-c:v", "copy", "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
# One `-af` covers EVERY audio output of this file, so each rendition is normalised.
args += ["-sn", *venc, *_aenc_opts(loudnorm)]
var = ["v:0,agroup:aud"]
for i, lang in enumerate(audio_langs):
entry = f"a:{i},agroup:aud,name:a{i}"
@@ -180,23 +531,25 @@ def _ffmpeg_cmd(
if i == 0:
entry += ",default:yes"
var.append(entry)
# The name patterns come from the vocabulary functions, with ffmpeg's own placeholders passed
# through them — so the producer and the parser cannot spell the layout differently (round 7).
args += _HLS_TAIL + [
"-master_pl_name",
"master.m3u8",
_MASTER_PL,
"-var_stream_map",
" ".join(var),
"-hls_segment_filename",
str(out_dir / "seg_%v_%d.ts"),
str(out_dir / "stream_%v.m3u8"),
str(out_dir / segment_name("%d", "%v")),
str(out_dir / variant_playlist_name("%v")),
]
else:
ao = audio_ord if audio_ord is not None else 0
args += ["-map", "0:v:0", "-map", f"{ai}:a:{ao}?", "-sn"]
args += ["-c:v", "copy", "-c:a", "aac", "-ac", "2", "-b:a", "192k"]
args += [*venc, *_aenc_opts(loudnorm)]
args += _HLS_TAIL + [
"-hls_segment_filename",
str(out_dir / "seg_%d.ts"),
str(out_dir / "index.m3u8"),
str(out_dir / segment_name("%d")),
str(out_dir / _INDEX_PL),
]
return args
@@ -247,12 +600,14 @@ def _kill(s: HlsSession) -> None:
def _enforce_cap(cap: int) -> None:
# Caller holds _lock. Drop the least-recently-accessed sessions over the cap.
if len(_sessions) <= cap:
# Caller holds _lock. Only TRANSCODE sessions count toward plex_max_transcodes — remux
# (stream-copy) and direct-serve are cheap and deliberately uncapped (see config.py:
# plex_max_transcodes, and the user-facing config hint). Drop the least-recently-accessed
# transcode sessions over the cap, leaving any concurrent remux streams untouched.
tx = [(key, s) for key, s in _sessions.items() if s.transcode]
if len(tx) <= cap:
return
for key, s in sorted(_sessions.items(), key=lambda kv: kv[1].last_access)[
: len(_sessions) - cap
]:
for key, s in sorted(tx, key=lambda kv: kv[1].last_access)[: len(tx) - cap]:
_kill(s)
_sessions.pop(key, None)
@@ -264,55 +619,151 @@ def start_session(
audio_ord: int | None = None,
aoff: float = 0.0,
multi: bool = False,
transcode: bool = False,
) -> HlsSession | None:
"""(Re)start the HLS remux for an item at the given offset. `multi` (item has >1 audio track) maps
every audio track as an HLS rendition so the client switches audio without a restart; otherwise a
single audio track (`audio_ord`) is muxed. Subtitles are separate WebVTT tracks (not muxed here).
Returns None if the local file can't be read."""
"""(Re)start the HLS session for an item at the given offset. `transcode` (item is non-h264,
playable="transcode") re-encodes the video to H.264 (VAAPI or libx264) instead of stream-copying it;
otherwise the video is copied (remux). `multi` (item has >1 audio track) maps every audio track as an
HLS rendition so the client switches audio without a restart; otherwise a single audio track
(`audio_ord`) is muxed. Subtitles are separate WebVTT tracks (not muxed here). Returns None if the
local file can't be read."""
src = paths.local_media_path(db, item.file_path)
if src is None:
return None
# Read once, here: the knob has to be identical for the directory name and the ffmpeg argv, and a
# session respawn (the HW->SW fallback) must not pick up a mid-playback config change.
loudnorm = sysconfig.get_bool(db, "plex_audio_loudnorm")
key = item.rating_key
start_s = max(0.0, float(start_s))
audio_langs = _probe_audio_langs(src) if multi else []
is_multi = len(audio_langs) > 1 # only worth a master playlist when there really are ≥2 tracks
with _lock:
old = _sessions.pop(key, None)
if old is not None:
_kill(old)
tag = "multi" if is_multi else str(audio_ord)
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src, start_s, directory, audio_ord, aoff, audio_langs if is_multi else None
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
# The rendition names the argv below will produce — one source for the argv, the warm-up's
# segment names, the entry playlist and the artifact guard.
variants = session_variants(audio_langs if is_multi else None)
# Loudness normalisation changes the AUDIO THE SEGMENTS CONTAIN, so it belongs in the directory
# name beside the other parameters that do (offset, audio track, A/V offset): a session left over
# from before the admin flipped the knob must not be mistaken for one that honours it. Appended to
# the tag with a DASH — `_SESSION_DIR_RE` treats the tag as one underscore-free field, and that
# regex is the only thing keeping the orphan sweep off a shared scratch path (see its comment).
tag = ("multi" if is_multi else str(audio_ord)) + ("-ln" if loudnorm else "")
directory = _HLS_ROOT / f"{key}_{int(start_s)}_{tag}_{aoff:+.2f}"
def _seg(i: int) -> Path:
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
def _spawn(
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
) -> "HlsSession | None":
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
# whoever is registered now). The INITIAL spawn passes replace_only=None and always supersedes —
# that is the seek-restart's intended last-writer-wins.
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
if transcode and not hw:
preset = sysconfig.get_str(db, "plex_transcode_preset") or "veryfast"
crf = sysconfig.get_int(db, "plex_transcode_crf")
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
with _lock:
if replace_only is not None and _sessions.get(key) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(key, None)
if old is not None:
_kill(old)
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src,
start_s,
directory,
audio_ord,
aoff,
audio_langs if is_multi else None,
transcode=transcode,
hw=hw,
tonemap=tonemap,
preset=preset,
crf=crf,
max_height=max_height,
loudnorm=loudnorm,
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
preexec_fn=preexec,
)
s = HlsSession(
key,
directory,
proc,
start_s,
variants,
transcode=transcode,
hw=hw,
tonemap=tonemap,
)
_sessions[key] = s
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
# at least 1 so playback can't be capped to zero.
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
return s
s = _spawn(*_select_encode(db, item, transcode, src))
# Wait for the first video segment outside the lock (this blocks on ffmpeg, and we must not hold up
# other sessions). Pass the proc so a HW session that dies at VAAPI init fails fast (not after 25s).
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if not ready and s.hw:
# We waited on _seg(0) OUTSIDE _lock, so `s.proc` dying is ambiguous: it may have failed on its
# own (VAProfile/init/tonemap refusal), OR a concurrent same-key _spawn (a rapid seek-restart) or
# _enforce_cap may have killed it to install/keep a newer session. Respawn on libx264 (no HW, no
# tonemap) so a genuinely-failed GPU session never strands playback — but pass `replace_only=s` so
# the respawn is a no-op if we've been superseded: the identity check and the pop+replace share
# ONE lock in _spawn, so we can never tear down the newer session. s.hw is now False, so the route
# honestly reports "CPU" and the player's Preparing-stream state stays correct.
log.warning(
"plex transcode: HW session produced no segments — falling back to software key=%s", key
)
s = HlsSession(key, directory, proc, start_s, "master.m3u8" if is_multi else "index.m3u8")
_sessions[key] = s
# Admin-configurable concurrency cap (Configuration → Plex → Max concurrent transcodes);
# at least 1 so playback can't be capped to zero.
_enforce_cap(max(1, sysconfig.get_int(db, "plex_max_transcodes")))
# Learn the REAL start offset K from the first VIDEO segment (outside the lock — this blocks on
# ffmpeg producing it, and we must not hold up other sessions). The client reads `media_start_s`, so
# its absolute clock + subtitle shift key off the true keyframe, not the requested offset. On
# timeout/probe failure we keep the provisional requested offset (old behaviour, small drift).
if start_s > 0:
seg0 = directory / ("seg_0_0.ts" if is_multi else "seg_0.ts") # multi: video variant = v0
if wait_for(seg0, timeout=25.0):
k = _probe_first_pts(seg0)
respawned = _spawn(False, False, replace_only=s)
if respawned is None:
# Superseded/culled mid-fallback — yield whoever is registered now (a newer seek-restart), or
# our own now-dead session if the key was cap-culled. Never None here: the POST /session route
# turns a None return into a "Media file not found on disk" 404 — dishonest for a file that
# exists. A (dead) session instead degrades to a retryable 404 ("No active playback session" /
# "Not available yet") the client re-requests.
return current_session(key) or s
s = respawned
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
if ready:
# For a seek (start_s>0) learn the REAL start offset K from the first segment's absolute PTS,
# so the client's absolute clock + subtitle shift key off the true keyframe, not the requested
# offset. On probe failure we keep the provisional requested offset (old behaviour, small drift).
if start_s > 0:
k = _probe_first_pts(_seg(0))
if k is not None and k >= 0:
s.media_start_s = k
# Warm up the buffer (see _WARMUP_SEGMENTS) so the client doesn't outrun the cold ffmpeg. Stop
# at the first segment that isn't ready in time — ffmpeg produces them faster than realtime once
# warm, so a miss means something slow and we proceed rather than block the request further.
for i in range(1, _WARMUP_SEGMENTS):
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
break
log.info(
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s",
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
key,
start_s,
s.media_start_s,
is_multi,
audio_ord,
transcode,
s.hw,
s.tonemap,
)
return s
@@ -325,10 +776,109 @@ def current_session(key: str) -> HlsSession | None:
return s
def wait_for(path: Path, timeout: float = 20.0) -> bool:
# The names ffmpeg writes into a session directory, in ONE place — and "one place" means the argv
# BUILDS its patterns from these same functions (`segment_name("%d", "%v")`), not that a comment
# says so. Getting this wrong is invisible in both directions: a name we failed to anticipate used
# to fall back to "wait and see" (round 6's HIGH), and now that the rule is positive it falls back
# to a 404 instead — so the producer and the parser have to be literally the same spelling.
#
# MEASURED (ffmpeg 7.1.5 in the API container — the binary that actually writes these files — and
# re-checked on 8.1.1): `-hls_segment_filename seg_%v_%d.ts` substitutes `%v` with the VARIANT and
# `%d` with the SEQUENCE — variant FIRST. A dual-audio session writes seg_0_0.ts… (video, the
# unnamed variant) plus seg_a0_0.ts… / seg_a1_0.ts… (the `name:aN` audio renditions), and the
# playlists beside them are master.m3u8 + stream_0/a0/a1.m3u8. Single-audio uses `seg_%d.ts` +
# index.m3u8, with no variant field at all.
#
# The two argv branches are DISJOINT vocabularies, which is why the guard below is keyed to the
# SESSION and not to HLS in general: a single-audio session can never write `master.m3u8`,
# `stream_*.m3u8` or any `seg_<variant>_<n>.ts`, and a multi-audio one can never write `index.m3u8`
# or `seg_<n>.ts`. Accepting the union meant a hopeless name of the other shape was waited out for
# the full playlist/segment timeout — the threadpool parking this guard exists to end (round 7).
_MASTER_PL = "master.m3u8"
_INDEX_PL = "index.m3u8"
_SEG_NAME_RE = re.compile(r"^seg_(?:([0-9a-z]{1,4})_)?(\d{1,9})\.ts$")
def session_variants(audio_langs: "Sequence[str | None] | None") -> tuple[str, ...]:
"""The `-var_stream_map` rendition names a session will write, or () for a single-audio one.
The video rendition is left unnamed in the map, so ffmpeg falls back to its index
(`_VIDEO_VARIANT`); each audio rendition is named `aN` in track order."""
if not audio_langs:
return ()
return (_VIDEO_VARIANT, *(f"a{i}" for i in range(len(audio_langs))))
def segment_name(seq: "int | str", variant: "str | None" = None) -> str:
"""The file ffmpeg writes for the `seq`-th segment of `variant`.
`variant=None` is a single-audio session, whose argv carries no `%v` field at all. `seq` and
`variant` are typed loosely because `_ffmpeg_cmd` builds its own `-hls_segment_filename` pattern
through this function, passing ffmpeg's `"%d"` / `"%v"` placeholders — that is what keeps the
producer and the parser from drifting, rather than a comment asking the next editor not to."""
return f"seg_{variant}_{seq}.ts" if variant else f"seg_{seq}.ts"
def variant_playlist_name(variant: str) -> str:
"""The sub-playlist ffmpeg writes for one rendition of a multi-audio session."""
return f"stream_{variant}.m3u8"
def playlist_names(variants: "Sequence[str]") -> frozenset[str]:
"""Every playlist name this session's argv can write — the master plus one per rendition, or
just `index.m3u8` for a single-audio session."""
if not variants:
return frozenset({_INDEX_PL})
return frozenset({_MASTER_PL, *(variant_playlist_name(v) for v in variants)})
def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
"""Whether waiting for `filename` can possibly end in a file OF THIS SESSION.
Positive rule, not a blocklist: a session produces segments and playlists and NOTHING else, so
any other name is refused at once. ffmpeg also produces segments strictly in order, so a segment
whose PREDECESSOR does not exist yet is not the next one — it is beyond the generated region,
and no amount of waiting will conjure it. Playlists are the one thing that must still be waited
for: they are rewritten continuously and have no ordering to read.
Keyed to the SESSION's own vocabulary, not to HLS's: the two argv branches are disjoint, so a
name belonging to the other shape is as hopeless here as a random string. Reading those as "an
HLS name, so wait and see" left the whole class reachable — measured through the real route
against a live single-audio session, `master.m3u8` and `stream_q7z.m3u8` were 404s after 30.0 s
and `seg_a0_0.ts` / `seg_zzzz_0.ts` after 20.1 s each (deep-review round 7).
Saying no at once matters because the route that waits is a SYNC one: it holds a thread off the
~40-slot pool, and a pooled DB connection with it, for the whole timeout. An ordinary seek past
the produced region was enough to park one for 20 s, and forty concurrent requests for a name
that will never exist parked the whole API (deep-review round 5). Round 5's guard read only
single-audio segment names, so both halves of that were still fully reachable: every multi-audio
name fell through, and so did any `.m3u8`/`.vtt` name at all."""
if filename in playlist_names(s.variants):
return True
m = _SEG_NAME_RE.match(filename)
if m is None:
return False
variant, seq = m.group(1), int(m.group(2))
# The variant field must be one THIS session writes: present and known for a multi-audio
# session, absent for a single-audio one.
if variant not in (s.variants or (None,)):
return False
if seq == 0:
return True
try:
return (s.dir / segment_name(seq - 1, variant)).exists()
except OSError:
return False
def wait_for(path: Path, timeout: float = 20.0, proc: subprocess.Popen | None = None) -> bool:
"""Wait until a session file (playlist / segment) exists and is non-empty. Segments are
produced ~faster than realtime, so a segment just ahead of playback appears quickly; a segment
far beyond the generated region won't (the frontend restarts the session at a seek instead)."""
far beyond the generated region won't (the frontend restarts the session at a seek instead).
With `proc` given, also stop as soon as that ffmpeg exits: a VAAPI session that fails to init
(10-bit/HDR VAProfile refusal) dies within ~1s producing nothing, and without this we'd block the
whole `timeout` before the HW→SW fallback could kick in — turning the fix into a stall."""
end = time.time() + timeout
while time.time() < end:
try:
@@ -336,6 +886,12 @@ def wait_for(path: Path, timeout: float = 20.0) -> bool:
return True
except OSError:
pass
if proc is not None and proc.poll() is not None:
# the producer exited — no more segments are coming; one last check then give up.
try:
return path.exists() and path.stat().st_size > 0
except OSError:
return False
time.sleep(0.15)
return path.exists()
@@ -386,14 +942,47 @@ def sweep_orphans(older_than: float | None = None) -> int:
return dropped
def reap_idle() -> int:
# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
# it (see reap_idle).
_FINISHED_IDLE_S = 30
def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
"""Kill sessions nobody has said they still want for `_SESSION_IDLE_S` — or, once their ffmpeg
has finished, for `finished_idle_s`.
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
killed a one-minute pause, and playback died ~30 s after resuming (deep-review round 4).
What makes the short rule safe again is that fetching is no longer the only signal: a player
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
viewer who is merely paused says so far more often than this rule asks (round 5). The heartbeat
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing": round 6
found that claim in two comments and not in the code. Round 4's blanket
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
one a whole segmented copy of the film on a volume shared with the download center."""
now = time.time()
dropped = 0
with _lock:
for key, s in list(_sessions.items()):
done = s.proc.poll() is not None
if now - s.last_access > _SESSION_IDLE_S or (done and now - s.last_access > 30):
idle = now - s.last_access
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
_kill(s)
_sessions.pop(key, None)
dropped += 1
return dropped
def touch_session(key: str) -> None:
"""Say a viewer is still there for `key`, without fetching anything. `last_access` used to move
only on an HLS file fetch, which a paused player does not do — so the only way to keep a paused
viewer's session alive was to keep EVERY abandoned one just as long."""
with _lock:
s = _sessions.get(key)
if s is not None:
s.last_access = time.time()
+730 -14
View File
@@ -1,15 +1,57 @@
"""WebVTT conversion, HLS-clock time-shift, and on-disk cache for Plex subtitle streams.
"""WebVTT conversion, HLS-clock time-shift, and the on-disk cache + background extractor for Plex
subtitle streams.
Text subtitles (embedded or external) are turned into WebVTT so the browser <track> can render them;
bitmap subtitles (Blu-ray/DVD) can't be (would need burn-in). Converted VTTs are cached on the shared
Plex on-disk cache root (``cache.CACHE_DIR``).
WHY THERE IS A BACKGROUND EXTRACTOR HERE — measured on prod, 2026-09-10, against a 17.06 GB 1080p
MKV with two embedded SubRip tracks on the 81 TB array:
=========================================================== ========
full pass, ONE track (what this module used to do) 154 s
full pass, BOTH text tracks in a SINGLE ffmpeg run 111 s
`-c:s copy` instead of converting to WebVTT 143 s
from 1:00:00 to the end of the file 66 s
a 630 s WINDOW at 1:00:00, both tracks 8 s
=========================================================== ========
The conversion is not the cost (dropping it saved 7%): ffmpeg has to stream every byte of the
container between its seek point and its stop point, and the array delivers ~140 MB/s — so the time
is (bytes read) / 140 MB/s, and the ONLY lever is reading less. Hence:
* extract a WINDOW around the playhead first — the viewer sees subtitles in seconds — then complete
the whole track in the background so every later seek is instant and the next session is free;
* extract EVERY embedded text track in ONE pass, because the read is shared: picking Hungarian used
to cost a full read and then switching to English cost another one (the "hektikus" report);
* never do any of it inside a request (it froze the <track> with no feedback AND held an API worker
thread for the whole read).
`-copyts` CANNOT be used for a window: it makes ffmpeg ignore `-t`/`-to` (measured — a [0, 600 s]
window still took the full 145 s and emitted the whole track), so windows are extracted on a
zero-based clock and shifted back to absolute time here with `shift_vtt`.
"""
import json
import logging
import re
import subprocess
import tempfile
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import PlexItem
from app.plex import paths as plex_paths
from app.plex.cache import CACHE_DIR
from app.plex.client import PlexError
from app.plex.client import PlexClient, PlexError
from app.plex.lookups import by_rating_key
log = logging.getLogger("siftlode.plex")
# Image-based subtitle codecs (Blu-ray/DVD) — can't be turned into text WebVTT (would need burn-in).
BITMAP_SUBS = {
@@ -31,6 +73,9 @@ def srt_to_vtt(text: str) -> str:
def _vtt_ts_to_s(ts: str) -> float:
# Also reads ffmpeg's spelling of a NEGATIVE time (a windowed pass, see `_ffmpeg_extract`): its
# webvtt muxer splits the milliseconds with C's truncating division, so every field carries the
# sign — -5.005 s is `00:-5.-05`, -65.3 s is `-1:-5.-300` — and summing the signed fields is exact.
parts = ts.split(":")
h = "0" if len(parts) == 2 else parts[0]
m, rest = parts[-2], parts[-1]
@@ -39,16 +84,12 @@ def _vtt_ts_to_s(ts: str) -> float:
def _vtt_s_to_ts(x: float) -> str:
x = max(0.0, x)
h = int(x // 3600)
x -= h * 3600
m = int(x // 60)
x -= m * 60
s = int(x)
ms = int(round((x - s) * 1000))
if ms >= 1000:
s += 1
ms -= 1000
# Rounded to whole milliseconds FIRST, then split: rounding the fraction last carried a rounded-up
# millisecond into the seconds and no further, so 59.9996 came out as `00:00:60.000` — a timestamp
# the WebVTT parser rejects, silently dropping that cue.
h, rest = divmod(int(round(max(0.0, x) * 1000)), 3_600_000)
m, rest = divmod(rest, 60_000)
s, ms = divmod(rest, 1000)
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"
@@ -109,7 +150,7 @@ def ffmpeg_to_vtt(src: str, sub_ord: int | None) -> str:
"webvtt",
"pipe:1",
]
r = subprocess.run(cmd, capture_output=True)
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S)
if r.returncode != 0 or not r.stdout:
raise PlexError(
(r.stderr or b"").decode("utf-8", "replace")[:200] or "ffmpeg subtitle convert failed"
@@ -125,11 +166,686 @@ def cache_read(key: str) -> str | None:
return None
def cache_write(key: str, vtt: str) -> None:
def cache_write(key: str, vtt: str) -> bool:
"""Atomically replace the cached VTT; False when it could not be written (`publish` must then
not claim the region it was carrying)."""
try:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
tmp = CACHE_DIR / f"{key}.vtt.tmp"
tmp.write_text(vtt, encoding="utf-8")
tmp.replace(CACHE_DIR / f"{key}.vtt") # atomic publish
return True
except OSError:
return False
# --- cache identity -----------------------------------------------------------------------------
# The cached VTT holds ABSOLUTE cue times (per rating_key + subtitle ordinal); the per-session shift
# is applied on the way out, so a resume/seek to any offset reuses the same file. Beside it, a small
# JSON sidecar records WHICH absolute-time ranges the file actually covers — a windowed extraction
# publishes a usable but incomplete track, and "incomplete" must never be mistaken for "all there is".
_FULL_END_S = 1_000_000_000.0 # stand-in for "to the end of the file" (JSON can't hold infinity)
def cache_key(rating_key: str, sub_ord: int) -> str:
return f"sub_{rating_key}_{sub_ord}"
def _vtt_path(key: str) -> Path:
return CACHE_DIR / f"{key}.vtt"
def _meta_path(key: str) -> Path:
return CACHE_DIR / f"{key}.json"
def read_meta(key: str) -> tuple[list[list[float]], bool]:
"""Covered absolute-time ranges + whether the track is fully extracted. A VTT with no sidecar
(written by an older build, which only ever cached complete tracks) counts as complete."""
try:
raw = json.loads(_meta_path(key).read_text(encoding="utf-8"))
return [[float(a), float(b)] for a, b in raw.get("ranges", [])], bool(raw.get("complete"))
except (OSError, ValueError, TypeError):
return ([[0.0, _FULL_END_S]], True) if _exists(_vtt_path(key)) else ([], False)
def _exists(p: Path) -> bool:
"""`Path.exists` that answers False for a path the OS refuses outright: pathlib swallows only
ENOENT-like errors, so an over-long name (ENAMETOOLONG, from an absurd rating_key or ordinal)
escaped as a 500 from the status route."""
try:
return p.exists()
except (OSError, ValueError):
return False
def _write_meta(key: str, ranges: list[list[float]], complete: bool) -> None:
try:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
tmp = CACHE_DIR / f"{key}.json.tmp"
tmp.write_text(json.dumps({"ranges": ranges, "complete": complete}), encoding="utf-8")
tmp.replace(_meta_path(key))
except OSError:
pass
def generation(key: str) -> int:
"""Monotonic id of the cached file's current content. It goes into the <track> URL so that when a
window is superseded by a wider one the browser refetches instead of keeping the short VTT."""
try:
return _vtt_path(key).stat().st_mtime_ns
except (OSError, ValueError): # ValueError: a NUL in the key — `stat` refuses the path outright
return 0
def merge_ranges(ranges: list[list[float]]) -> list[list[float]]:
"""Sort + coalesce touching/overlapping covered ranges."""
out: list[list[float]] = []
for a, b in sorted((r for r in ranges if r[1] > r[0]), key=lambda r: r[0]):
if out and a <= out[-1][1]:
out[-1][1] = max(out[-1][1], b)
else:
out.append([a, b])
return out
def covers(ranges: list[list[float]], at: float) -> bool:
return any(a <= at <= b for a, b in ranges)
def merge_vtt(*sources: str) -> str:
"""Union of the cue blocks of several absolute-time VTTs, ordered by start time.
Two extraction passes over the same file emit identical cues for the region they share, so they
are de-duplicated — but NOT on the raw timing text. The two passes do not spell a timestamp the
same way: a window comes back through `shift_vtt`, which always writes `HH:MM:SS.mmm`, while a
full pass is ffmpeg's own output, and its webvtt muxer drops the hour below one hour
(`10:00.123`). Keyed on the text, every cue of a window in the first hour survived twice and the
viewer saw each line stacked on itself. So each timing line is parsed and re-emitted in one
canonical form, and the key is that canonical timing PLUS the cue text: two different cues that
happen to share a time (two speakers, a sign over dialogue) are both kept."""
seen: set[str] = set()
cues: list[tuple[float, str]] = []
for src in sources:
for block in re.split(r"\n[ \t]*\n", src or ""):
canon = _canonical_cue(block.strip())
if canon is None:
continue # header / NOTE / STYLE — the merged file gets its own single header
start, key, text = canon
if key in seen:
continue
seen.add(key)
cues.append((start, text))
cues.sort(key=lambda c: c[0])
return "WEBVTT\n\n" + "\n\n".join(b for _, b in cues) + "\n"
def _canonical_cue(block: str) -> tuple[float, str, str] | None:
"""(start seconds, identity, the block with its timing line canonicalised) — or None for a block
that is not a cue. The identity leaves out anything before the timing line (an SRT-style cue
number), which is a label, not content."""
lines = block.split("\n")
for i, ln in enumerate(lines):
if "-->" not in ln:
continue
left, _, right = ln.partition("-->")
rparts = right.strip().split(None, 1)
settings = (" " + rparts[1]) if len(rparts) > 1 else ""
try:
st = _vtt_ts_to_s(left.strip())
en = _vtt_ts_to_s(rparts[0])
except (ValueError, IndexError):
return None
lines[i] = f"{_vtt_s_to_ts(st)} --> {_vtt_s_to_ts(en)}{settings}"
return st, "\n".join(lines[i:]), "\n".join(lines)
return None
def _last_cue_end(vtt: str) -> float:
"""Where the last cue of a VTT ends — how far a pass that died early provably got."""
end = 0.0
for ln in vtt.split("\n"):
if "-->" not in ln:
continue
try:
end = max(end, _vtt_ts_to_s(ln.partition("-->")[2].strip().split(None, 1)[0]))
except (ValueError, IndexError):
continue
return end
def publish(key: str, vtt: str, covered: tuple[float, float], complete: bool) -> bool:
"""Merge a freshly extracted region into the cached track and widen its covered ranges. False
when the region did not land (the cache is unwritable, or the VTT there cannot be read) — the
caller must then record a FAILED pass: returning quietly let the next poll schedule the same read
again, a whole-container pass every time, while the player said "preparing" for good.
A read-merge-write, so it runs under `_PUBLISH_LOCK`: two jobs publish into the same keys (a
window and the full pass, and every embedded track of the file rides each of them), and without
the lock the later writer drops whatever the earlier one merged — permanently, once `complete` is
set. It is also the only writer of the cache files, which is what keeps their fixed `.tmp`
staging names safe."""
with _PUBLISH_LOCK:
old = cache_read(key)
if old is None and _vtt_path(key).exists():
# There, but unreadable (`cache_read` swallows the OSError). Merging into "" would replace
# a whole track with this one region while its sidecar still claimed all of it.
log.warning("subtitle cache %s unreadable; not publishing into it", key)
return False
old_ranges, was_complete = read_meta(key)
if old is None:
# A sidecar with no VTT behind it claims cues that are not there, so it is not merged
# into. And a VTT with no sidecar reads as a COMPLETE track (see read_meta — how older
# builds cached), while a publish is two renames: so before a key's first VTT lands, its
# sidecar is made to claim nothing. Dying between the two then leaves a window that is
# extracted again, never one taken for the whole track for good.
old_ranges, was_complete = [], False
_write_meta(key, [], False)
if not cache_write(key, merge_vtt(old or "", vtt)):
log.warning("subtitle cache %s could not be written", key)
return False # the region never landed, so the sidecar must not claim it
_write_meta(key, merge_ranges(old_ranges + [list(covered)]), complete or was_complete)
return True
# --- background extraction ----------------------------------------------------------------------
# How much of the track to extract before the viewer sees anything. WINDOW_PRE_S starts a little
# before the playhead so a cue that is already on screen is not missed; WINDOW_LEN_S is the span
# published as the first, provisional VTT (~4 s of work on the measured file). It only has to hold
# until the full pass lands behind it — 111 s for that file — so it is deliberately much longer than
# any realistic full pass.
WINDOW_PRE_S = 30.0
WINDOW_LEN_S = 300.0
# …but it does not always land: a full pass can be backed off (FULL_RETRY_S) or simply slower than
# the window lasts (a file past ~40 GB, or several passes sharing the array). A window was only asked
# for once the playhead had LEFT coverage, so every edge unmounted the track for a poll plus an
# extraction. Within this distance of a covered stretch's end, the next window is read ahead.
LOOKAHEAD_S = 60.0
# Ceiling on one ffmpeg run so a wedged extraction can't hold a pool thread forever (the old
# in-request extraction had no timeout at all).
EXTRACT_TIMEOUT_S = 900.0
# How long a timed-out ffmpeg gets, after a TERM, to close its outputs before it is killed.
_TERM_GRACE_S = 10.0
# A failed track (or window) is remembered briefly so a 1 s status poll doesn't respawn ffmpeg every
# second, and forgotten again soon after so fixing the file (or Plex coming back) doesn't need a
# restart.
FAIL_TTL_S = 60.0
# …except a FULL pass that failed. That is a read of the whole container (minutes), a failure that
# reproduces does so every time, and the viewer is typically inside what it did publish — so the
# short memo re-armed a whole-file read every minute for as long as they watched. Windows around the
# playhead still run meanwhile; only the full pass waits this out.
FULL_RETRY_S = 3600.0
# The furthest position the status poll accepts. No film is this long, and an absurd one used to
# reach ffmpeg's `-ss` (which rejects anything past its int64 microseconds).
MAX_POSITION_S = 100_000.0
# Windows and full passes run on SEPARATE pools, so a window never waits behind a full pass. They
# shared one FIFO pool of three whose comment promised "room for a window" that nothing enforced:
# stepping through three episodes with subtitles on left three full passes (minutes each) holding every
# worker, and a seek window on the third queued behind them. Two of each: bounded, so a burst can't
# thrash the array.
_WIN_POOL = ThreadPoolExecutor(max_workers=2, thread_name_prefix="plex-subs-win")
_FULL_POOL = ThreadPoolExecutor(max_workers=2, thread_name_prefix="plex-subs-full")
# Bounds on what may wait for those pools — separate too, so held full-pass slots (minutes each)
# cannot refuse a window (seconds). The queues are unbounded and every job is a read of the library:
# a viewer scrubbing through a film queued a window per stop (the one where they stopped ran LAST),
# and a script could queue thousands. Past a bound nothing is queued — the next poll asks again, by
# which time something has finished.
_MAX_WINDOWS_INFLIGHT = 8
_MAX_FULL_INFLIGHT = 6
_MAX_TRACK_WINDOWS = 2
_LOCK = threading.Lock()
# Queued or running jobs → the absolute span a window job reads (None for a full pass).
_INFLIGHT: dict[tuple, tuple[float, float] | None] = {}
# Track-level failures: a track that cannot be served at all (no item, a bitmap track, Plex down…).
_FAILURES: dict[str, tuple[str, float]] = {}
# rating_key → when its full pass last failed (see FULL_RETRY_S).
_FULL_FAILED: dict[str, float] = {}
# cache key → the spans windows failed to read, with when. A failed window says nothing about the rest
# of the track — the full pass may be about to deliver it — so it is remembered as a SPAN: it must be
# neither respawned every poll nor taken as a failure of positions it never tried. (It used to set the
# track-level memo, which told every viewer at any uncovered position "failed" — a final answer that
# ends the player's poll — including one planted by a request for an absurd position.)
_WIN_FAILED: dict[str, list[tuple[float, float, float]]] = {}
_PUBLISH_LOCK = threading.Lock()
# The ffmpeg children currently running, so a shutdown can kill them (see `shutdown`), and whether it
# has: a worker that reaches its Popen after the kill sweep must not start a read.
_PROCS: set[subprocess.Popen] = set()
_STOPPING = False
class SubtitleUnavailable(Exception):
"""A track that cannot be served as text, carrying the stable code the route maps to a status."""
def __init__(self, code: str):
super().__init__(code)
self.code = code
# code -> HTTP status. An extraction/upstream failure is 422 with a structured detail (502 is
# reserved for the raw byte proxies); the rest are genuine "no such thing" / "not text".
UNAVAILABLE_STATUS = {
"no_item": 404,
"no_track": 404,
"no_file": 404,
"not_fetchable": 404,
"bitmap": 415,
"extract_failed": 422,
"plex_error": 422,
}
# The codes that are a FINAL answer about the track: it can never be served as text, so the player
# stops asking. The others (`plex_error`, `extract_failed`) are about now or about one stretch — a
# memo that lapses, a window that failed where the rest is fine — and answered as `unavailable`: said,
# but asked again. Given as `failed`, one Plex hiccup ended the subtitles for the rest of the session,
# at cached positions too, and after the full pass had completed behind it.
FINAL_CODES = frozenset({"no_item", "no_track", "no_file", "not_fetchable", "bitmap"})
def reset_state() -> None:
"""Drop the in-flight/failure registries (tests; nothing in the app needs it)."""
global _STOPPING
with _LOCK:
_INFLIGHT.clear()
_FAILURES.clear()
_FULL_FAILED.clear()
_WIN_FAILED.clear()
_STOPPING = False
def shutdown() -> None:
"""Stop extracting, for the app's shutdown. The pool's workers are non-daemon threads and Python
joins them at interpreter exit — so a worker still inside a full pass (minutes on a big file, up
to EXTRACT_TIMEOUT_S) held the process open after uvicorn had stopped, until the container runtime
SIGKILLed it. Queued jobs are dropped and running ffmpeg children are killed, which returns their
workers at once — and a worker still BEFORE its ffmpeg (the Plex lookup, the DB) finds the flag
and never starts one. Nothing that matters is lost: an interrupted publish cannot pass for a whole
track (see `publish`), and an interrupted track is simply extracted again on the next poll."""
global _STOPPING
with _LOCK:
_STOPPING = True
procs = list(_PROCS)
for pool in (_WIN_POOL, _FULL_POOL):
pool.shutdown(wait=False, cancel_futures=True)
for p in procs:
p.kill()
def _sweep_memos() -> None:
"""(Caller holds _LOCK.) Drop every lapsed failure memo. They were only ever pruned when the SAME
key was read again, and the status route takes any rating_key/ordinal — so a script polling fresh
keys grew the API process without bound. Swept on every write, they hold at most a TTL's worth."""
now = time.monotonic()
for k in [k for k, (_, t) in _FAILURES.items() if now - t >= FAIL_TTL_S]:
del _FAILURES[k]
for k in [k for k, t in _FULL_FAILED.items() if now - t >= FULL_RETRY_S]:
del _FULL_FAILED[k]
for k in list(_WIN_FAILED):
_live_window_failures(k)
def _fail(key: str, code: str) -> None:
with _LOCK:
_sweep_memos()
_FAILURES[key] = (code, time.monotonic())
def _recent_failure(key: str) -> str | None:
with _LOCK:
hit = _FAILURES.get(key)
if hit and time.monotonic() - hit[1] < FAIL_TTL_S:
return hit[0]
_FAILURES.pop(key, None)
return None
def _full_failed(rating_key: str) -> bool:
with _LOCK:
when = _FULL_FAILED.get(rating_key)
if when is not None and time.monotonic() - when < FULL_RETRY_S:
return True
_FULL_FAILED.pop(rating_key, None)
return False
def _live_window_failures(key: str) -> list[tuple[float, float, float]]:
"""(Caller holds _LOCK.) The key's failed-window spans that are still remembered."""
now = time.monotonic()
live = [f for f in _WIN_FAILED.get(key, []) if now - f[2] < FAIL_TTL_S]
if live:
_WIN_FAILED[key] = live
else:
_WIN_FAILED.pop(key, None)
return live
def _window_fail(key: str, span: tuple[float, float]) -> None:
with _LOCK:
_sweep_memos()
_WIN_FAILED[key] = [*_live_window_failures(key), (span[0], span[1], time.monotonic())]
def _window_failed(key: str, at: float) -> bool:
with _LOCK:
return any(a <= at <= b for a, b, _ in _live_window_failures(key))
def ensure(rating_key: str, sub_ord: int, at: float = 0.0) -> dict:
"""State of one subtitle track, scheduling whatever extraction is still missing. NEVER blocks.
``ready`` = fully extracted · ``partial`` = usable, covers ``at``, still growing · ``preparing``
= nothing to show for ``at`` yet · ``unavailable`` = nothing for ``at`` and nothing being read
for it right now (``code`` says why), ask again · ``failed`` = the track can never be shown
(``code`` says why) — the only final answer. ``gen`` changes whenever the cached file grows, so
the client can refetch a window that has been superseded; 0 = nothing is cached at all."""
key = cache_key(rating_key, sub_ord)
ranges, complete = read_meta(key)
have = cache_read(key) is not None
covered = have and covers(ranges, at)
gen = generation(key)
if complete and have:
return {"state": "ready", "code": None, "gen": gen}
if not have and _exists(_vtt_path(key)):
# On disk but unreadable (permissions, a bad volume): nothing can serve it, and no extraction
# can help — `publish` refuses to merge into it, and a job whose sidecar covers the position
# skips as already done — so scheduling only kept the player on "preparing" for good. `gen` 0:
# there is nothing a <track> could fetch.
return {"state": "unavailable", "code": "cache_unreadable", "gen": 0}
code = _recent_failure(key)
if code and not covered:
return {
"state": "failed" if code in FINAL_CODES else "unavailable",
"code": code,
"gen": gen,
}
# Nothing left that could cover the position for now: the full pass failed, AND so did a window
# here. Still not final — the memos lapse, and the rest of the track may be right there.
full_failed = _full_failed(rating_key)
window_failed = not covered and _window_failed(key, at)
if full_failed and window_failed:
return {"state": "unavailable", "code": "extract_failed", "gen": gen}
if not code:
probe, want_window = at, not covered and not window_failed
if covered:
# Read ahead of an edge instead of waiting to fall off it (see LOOKAHEAD_S). The probe is
# just PAST the edge: `covers` is inclusive, so the edge itself reads as already done.
edge = max(b for a, b in ranges if a <= at <= b)
if edge - at <= LOOKAHEAD_S and edge < _FULL_END_S:
probe = edge + 1.0
want_window = not _window_failed(key, probe)
_schedule(rating_key, sub_ord, probe, want_window=want_window, want_full=not full_failed)
return {"state": "partial" if covered else "preparing", "code": None, "gen": gen}
def cached_track(rating_key: str, sub_ord: int) -> tuple[str | None, str | None]:
"""What the <track> fetch can serve RIGHT NOW, scheduling nothing: the cached VTT (whatever it
covers), else the recent failure that explains why there is none.
Scheduling belongs to `ensure` — the status poll, which knows where the viewer is. The fetch does
not: when it scheduled too, it judged readiness at position 0, so a viewer who resumed an hour in
got an unrequested window over the head of the file (plus a remount when it landed), and with a
failure memo alive it was refused a window that WAS on disk while the poll called it showable."""
key = cache_key(rating_key, sub_ord)
hit = cache_read(key)
return (hit, None) if hit is not None else (None, _recent_failure(key))
def _schedule(
rating_key: str, sub_ord: int, at: float, want_window: bool, want_full: bool = True
) -> None:
"""Kick off the jobs this track still needs: the window that makes it watchable now, and the one
full pass that makes it complete. A 1 s status poll (or a second viewer) joins a job already
queued or running instead of starting another ffmpeg.
A window is joined when one of this track's in-flight windows will COVER the position asked about
— decided by what it reads, not by an identity derived from the position. The poll sends the live
playhead, so a position-keyed window was a new job every second; bucketing the position instead
still queued a second read when the playhead crossed a bucket edge, and joined a backward seek to
a window that could not cover it. Nor may it be one window per track: a window cut around the
wrong spot (the first poll after a resume can still report 0) made the right one wait for it. The
full pass is per ITEM, because one read fans out to every embedded text track of the file."""
jobs: list[tuple[tuple, float | None]] = []
with _LOCK:
if _STOPPING:
return
n_windows = sum(1 for span in _INFLIGHT.values() if span is not None)
n_full = len(_INFLIGHT) - n_windows
if want_window and n_windows < _MAX_WINDOWS_INFLIGHT:
spans = [
span
for ident, span in _INFLIGHT.items()
if span is not None and ident[1:3] == (rating_key, sub_ord)
]
if not any(a <= at <= b for a, b in spans) and len(spans) < _MAX_TRACK_WINDOWS:
# Rounded to what ffmpeg is told (`-ss %.3f`): the cues come back on ffmpeg's zero,
# and shifting them by an unrounded start put them up to half a millisecond off —
# enough to spell a timestamp differently from the full pass and double the cue.
w_start = float(f"{max(0.0, at - WINDOW_PRE_S):.3f}")
ident = ("win", rating_key, sub_ord, w_start)
_INFLIGHT[ident] = (w_start, w_start + WINDOW_PRE_S + WINDOW_LEN_S)
jobs.append((ident, w_start))
if want_full and n_full < _MAX_FULL_INFLIGHT and ("full", rating_key) not in _INFLIGHT:
_INFLIGHT[("full", rating_key)] = None
jobs.append((("full", rating_key), None))
for ident, window_start in jobs:
pool = _FULL_POOL if window_start is None else _WIN_POOL
pool.submit(_run, ident, rating_key, sub_ord, window_start, at)
def _already_done(key: str, window_start: float | None, at: float) -> bool:
"""A queued job looks again before it reads anything. While it waited, another job may have
covered it: every embedded track rides each pass over its file, so a window for one language also
publishes the others."""
if not _exists(_vtt_path(key)):
return False
ranges, complete = read_meta(key)
return complete or (window_start is not None and covers(ranges, at))
def _run(
ident: tuple, rating_key: str, sub_ord: int, window_start: float | None, at: float
) -> None:
key = cache_key(rating_key, sub_ord)
try:
if _STOPPING or _already_done(key, window_start, at):
return
_extract_and_publish(rating_key, sub_ord, window_start)
except SubtitleUnavailable as e:
if e.code == "extract_failed":
_pass_failed(rating_key, [key], window_start)
else:
_fail(key, e.code)
except PlexError as e:
log.warning("subtitle extraction failed for %s/%s: %s", rating_key, sub_ord, e)
_fail(key, "plex_error")
except (
Exception
): # a worker thread has nobody to raise to — record it, let the next poll see it
log.exception("subtitle extraction crashed for %s/%s", rating_key, sub_ord)
_pass_failed(rating_key, [key], window_start)
finally:
with _LOCK:
_INFLIGHT.pop(ident, None)
def _pass_failed(
rating_key: str,
keys: list[str],
window_start: float | None,
span: tuple[float, float] | None = None,
) -> None:
"""Record an ffmpeg pass that failed: a full pass backs off the whole item (FULL_RETRY_S); a
window marks only the stretch it failed to read, for the tracks it was reading."""
if window_start is None:
with _LOCK:
_sweep_memos()
_FULL_FAILED[rating_key] = time.monotonic()
return
for key in keys:
_window_fail(key, span or (window_start, window_start + WINDOW_PRE_S + WINDOW_LEN_S))
def _text_streams(db: Session, rating_key: str) -> tuple[PlexItem, list[dict]]:
"""The item plus its subtitle streams, in the ordinal order the API and the UI both use."""
it = by_rating_key(db, PlexItem, rating_key)
if it is None:
raise SubtitleUnavailable("no_item")
with PlexClient(db) as plex:
part = ((plex.metadata(it.rating_key) or {}).get("Media") or [{}])[0].get("Part") or [{}]
return it, [s for s in (part[0].get("Stream") or []) if s.get("streamType") == 3]
def _is_bitmap(stream: dict) -> bool:
return (stream.get("codec") or stream.get("format") or "").lower() in BITMAP_SUBS
def _extract_and_publish(rating_key: str, sub_ord: int, window_start: float | None) -> None:
"""One extraction job. A sidecar track is fetched whole from Plex (cheap, no ffmpeg); an embedded
one is extracted together with EVERY other embedded text track in the file, because they all ride
on the same single read — the whole point of the measurement at the top of this module."""
with SessionLocal() as db:
it, subs = _text_streams(db, rating_key)
if sub_ord < 0 or sub_ord >= len(subs):
raise SubtitleUnavailable("no_track")
want = subs[sub_ord]
if _is_bitmap(want):
raise SubtitleUnavailable("bitmap")
if want.get("key"):
# An external sidecar is one small fetch from Plex, never a read of the media file, so a
# window job fetches the whole thing too. It must not be left to the full job: that
# identity is per ITEM (the embedded fan-out shares it), so a sidecar picked while an
# embedded track's full pass was running sat behind that pass for minutes.
vtt = _sidecar_vtt(db, want)
if not publish(cache_key(rating_key, sub_ord), vtt, (0.0, _FULL_END_S), True):
raise SubtitleUnavailable("extract_failed")
return
if want.get("index") is None:
raise SubtitleUnavailable("not_fetchable")
src = plex_paths.local_media_path(db, it.file_path)
if src is None:
raise SubtitleUnavailable("no_file")
# Ordinal among the file's EMBEDDED subtitle streams — what `-map 0:s:N` counts. Sidecars sit
# in the same list but not in the file, so they must not shift the mapping.
targets = {
sum(1 for x in subs[:i] if x.get("index") is not None): i
for i, s in enumerate(subs)
if s.get("index") is not None and not _is_bitmap(s)
}
start = window_start or 0.0
length = (WINDOW_LEN_S + WINDOW_PRE_S) if window_start is not None else None
end = _FULL_END_S if length is None else start + length
texts, clean = _ffmpeg_extract(str(src), sorted(targets), window_start, length)
for emb_ord, vtt in texts.items():
key = cache_key(rating_key, targets[emb_ord])
# A pass that exited non-zero holds only what ffmpeg reached before it died. Claim no more
# than its last cue, never call it complete, and remember the failure: published as complete,
# the track read "ready" for good while its subtitles stopped wherever ffmpeg did (the cache
# has no eviction, so nothing would ever have extracted it again).
covered_end = end if clean else start + _last_cue_end(vtt)
landed = publish(
key,
shift_vtt(vtt, -start), # zero-based window clock → absolute cue times
(start, covered_end),
clean and length is None,
)
if not landed:
# A read that could not be kept is a failed read (see publish) — backed off like one.
_pass_failed(rating_key, [key], window_start, (start, end))
elif not clean:
_pass_failed(rating_key, [key], window_start, (covered_end, end))
def _sidecar_vtt(db: Session, stream: dict) -> str:
"""An external subtitle file, fetched through Plex and converted to WebVTT."""
with PlexClient(db) as plex:
raw = plex.raw_get(stream["key"])
codec = (stream.get("codec") or stream.get("format") or "").lower()
if codec in ("srt", "subrip", ""):
return srt_to_vtt(raw.decode("utf-8", "replace"))
with tempfile.NamedTemporaryFile(suffix="." + (codec or "sub"), delete=False) as fh:
fh.write(raw)
tmp = fh.name
try:
return ffmpeg_to_vtt(tmp, None)
finally:
Path(tmp).unlink(missing_ok=True)
def _ffmpeg_extract(
src: str, emb_ords: list[int], start: float | None, length: float | None
) -> tuple[dict[int, str], bool]:
"""Extract several embedded subtitle streams to WebVTT in ONE ffmpeg run (= one read of the file).
Returns the texts and whether ffmpeg exited cleanly — a non-zero exit can still leave partial
output, which is worth publishing but must never be taken for the whole track.
`start`/`length` bound the read to a window. Without `-copyts` the cues come out on a zero-based
clock (the caller shifts them back) — which is exactly why `-copyts` is absent: with it ffmpeg
ignores `-t` and reads the whole container anyway (measured)."""
if not emb_ords:
return {}, True
with tempfile.TemporaryDirectory(prefix="plexsub-") as tmpdir:
cmd = ["ffmpeg", "-nostdin", "-loglevel", "error"]
if start:
cmd += ["-ss", f"{start:.3f}"]
cmd += ["-i", src]
outs = {o: Path(tmpdir) / f"{o}.vtt" for o in emb_ords}
for emb_ord, out in outs.items():
cmd += ["-map", f"0:s:{emb_ord}"]
if length is not None:
cmd += ["-t", f"{length:.3f}"]
# The input seek lands on the keyframe at or BEFORE `start`, so a cue already running
# at `start` comes out at a negative time — and by default the muxer then shifts the
# WHOLE output so that cue sits at 0: every cue of the window late by the distance to
# that keyframe (1–5 s measured on a real library file, a whole GOP on a synthetic
# one), and doubled for good once the full pass merged in beside it. Kept negative, the
# times are exact on the zero-based clock (`_vtt_ts_to_s` reads the sign).
cmd += ["-avoid_negative_ts", "disabled"]
cmd += ["-f", "webvtt", "-y", str(out)]
# Popen rather than run(): the child is registered so `shutdown` can kill it — and one that
# would start after the shutdown's kill sweep never does, or is killed right here.
with _LOCK:
if _STOPPING:
raise SubtitleUnavailable("extract_failed")
proc = subprocess.Popen(
cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE
)
with _LOCK:
_PROCS.add(proc)
if _STOPPING:
proc.kill()
timed_out = False
try:
try:
_, err = proc.communicate(timeout=EXTRACT_TIMEOUT_S)
except subprocess.TimeoutExpired:
# A pass that ran out of time is a pass that died partway: what it reached is kept and
# published as partial (the caller's unclean path), like any other non-zero exit. It
# used to be killed and its outputs deleted unread — and with the full-pass backoff, the
# same doomed read came back every hour. TERM first: ffmpeg closes its outputs on it,
# whereas a KILL leaves only what was already flushed in whole blocks (measured: none).
timed_out = True
proc.terminate()
try:
_, err = proc.communicate(timeout=_TERM_GRACE_S)
except subprocess.TimeoutExpired:
proc.kill()
_, err = proc.communicate()
finally:
with _LOCK:
_PROCS.discard(proc)
got = {
o: p.read_text(encoding="utf-8", errors="replace")
for o, p in outs.items()
if p.exists()
}
if not got:
raise SubtitleUnavailable("extract_failed")
clean = proc.returncode == 0 and not timed_out
if not clean:
log.warning(
"ffmpeg subtitle pass %s: %s",
"timed out" if timed_out else f"exited {proc.returncode}",
(err or b"")[:200],
)
return got, clean
+130 -3
View File
@@ -18,7 +18,7 @@ item-detail/player path (P2), not here.
import logging
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
@@ -58,6 +58,57 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
return "direct"
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
# own encoder).
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
# Color-transfer values marking an HDR source (PQ / HLG). A 10-bit/HDR source breaks the VAAPI
# scale=nv12 path on Gen 9.5 (see plex.stream) → software transcode, so it counts as NOT HW-capable.
_HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
def transcode_hw_capable(
playable: str | None,
vcodec: str | None,
bit_depth: int | None = None,
hdr: bool = False,
) -> bool | None:
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
if playable != "transcode":
return None
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
return False
if bit_depth is not None and bit_depth > 8:
return False
return not hdr
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
"""(bit_depth, is_hdr) from a FULL Plex metadata dict's video Stream (Media→Part→Stream, streamType
1). Only the rich metadata call carries Stream; the cheap listing does not — so this is read by the
bit-depth backfill, not the listing sync. (None, False) when absent/unparseable."""
for m in meta.get("Media") or []:
for part in m.get("Part") or []:
for s in part.get("Stream") or []:
if s.get("streamType") != 1: # 1 = video
continue
bd = s.get("bitDepth")
try:
depth = int(bd) if bd else None
except (TypeError, ValueError):
depth = None
hdr = str(s.get("colorTrc") or "").lower() in _HDR_TRANSFERS
return depth, hdr
return None, False
def _epoch(v) -> datetime | None:
try:
return datetime.fromtimestamp(int(v), tz=timezone.utc) if v else None
@@ -217,6 +268,13 @@ def sync(db: Session) -> dict:
stats["people_enriched"] = _enrich_people_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex people enrich batch failed: %s", e)
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception as e: # best-effort — never fail the sync over the backfill
log.warning("Plex media probe batch failed: %s", e)
except PlexNotConfigured as e:
return {"skipped": str(e)}
except PlexError as e:
@@ -236,6 +294,49 @@ def sync(db: Session) -> dict:
# How many titles to rich-fetch per sync run for the cast backfill — bounds the extra Plex load; the
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
_ENRICH_BATCH = 60
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
_PROBE_BATCH = 60
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
rows = (
db.execute(
select(PlexItem)
.where(
PlexItem.playable == "transcode",
func.lower(PlexItem.codec_video).in_(HW_DECODABLE_CODECS),
PlexItem.media_probed_at.is_(None),
)
.order_by(PlexItem.id)
.limit(limit)
)
.scalars()
.all()
)
now = datetime.now(timezone.utc)
n = 0
for row in rows:
try:
meta = plex.metadata(row.rating_key) or {}
except Exception as e: # transient — leave media_probed_at NULL so it retries next run
log.debug("Plex media probe fetch failed for %s: %s", row.rating_key, e)
continue
bit_depth, hdr = _video_stream_props(meta)
row.video_bit_depth = bit_depth
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, bit_depth, hdr)
row.media_probed_at = (
now # probed (even if Plex carried no Stream — don't re-fetch forever)
)
n += 1
db.commit()
return n
def _enrich_people_batch(db: Session, plex: PlexClient, limit: int = _ENRICH_BATCH) -> int:
@@ -369,15 +470,41 @@ def _apply_item(row: PlexItem, lib_id: int, kind: str, meta: dict) -> None:
row.added_at = _epoch(meta.get("addedAt"))
_apply_facet_fields(row, meta) # filterable/orderable extras (shared with shows)
mf = _media_facts(meta)
new_vcodec = mf.get("codec_video")
new_part_key = mf.get("part_key")
# The physical file was replaced when its video codec changes OR — for a SINGLE-version item — when its
# Plex Part key changes (a re-encode / quality upgrade gets a new part id). Either way any probed
# bit-depth is stale, so reset the probe watermark and re-derive transcode_hw from codec alone until the
# backfill re-probes. Codec alone is NOT enough: a same-codec bit-depth re-encode (8-bit→10-bit HEVC)
# keeps the codec `hevc` but flips the HW class, and only the part-key move catches it.
#
# The part-key check is gated on single-version because `_media_facts` reads media[0]/part[0], and for a
# MULTI-version item (an original + a Plex "Optimized" copy) that order isn't guaranteed stable across
# listings — an unguarded part-key compare would then reset+re-probe every sync and oscillate the
# transcode_hw bucket between the two versions. A multi-version same-codec re-encode not self-healing is
# an acceptable rare edge; a per-sync re-probe loop is not.
single_version = len(meta.get("Media") or []) <= 1
codec_changed = (row.codec_video or "").lower() != (new_vcodec or "").lower()
if codec_changed or (single_version and row.part_key != new_part_key):
row.video_bit_depth = None
row.media_probed_at = None
row.file_path = mf.get("file_path")
row.part_key = mf.get("part_key")
row.part_key = new_part_key
row.container = mf.get("container")
row.codec_video = mf.get("codec_video")
row.codec_video = new_vcodec
row.codec_audio = mf.get("codec_audio")
row.size_bytes = mf.get("size_bytes")
row.playable = classify_playable(
mf.get("codec_video"), mf.get("codec_audio"), mf.get("container")
)
# transcode_hw: keep the bit-depth backfill's authoritative value (it folds in bit-depth + HDR) once
# probed; otherwise (re)derive the provisional codec-only class. A non-transcode item is always None,
# even if it had been probed as a transcode before (an audio/container flip to remux without a codec
# change wouldn't reset the watermark, so pin it here).
if row.playable != "transcode":
row.transcode_hw = None
elif row.media_probed_at is None:
row.transcode_hw = transcode_hw_capable("transcode", row.codec_video, row.video_bit_depth)
def _prune_stale(
+109 -22
View File
@@ -11,11 +11,14 @@ already pages) and upsert them into the owner's `plex_states`. Plex wins on over
exist only in Siftlode are left untouched (union, Plex-authoritative on the intersection).
Phases B (Siftlode→Plex push via scrobble/timeline) and C (incremental Plex→Siftlode via the watch
history + onDeck, last-write-wins) build on this and land in later ships.
history + onDeck) build on this and land in later ships. As of S2 Plex is the source of truth: the
player pushes the resume position back to Plex on the periodic checkpoint (not just the final flush),
and the incremental pull resolves a genuine divergence to Plex's state (see `_pull_apply`) rather than
running a last-write-wins comparison.
"""
import logging
from datetime import datetime, timedelta, timezone
from datetime import datetime, timezone
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
@@ -293,12 +296,6 @@ def push_bulk_state_to_plex(user_id: int, changes: list[tuple[int, str, str]]) -
# --- Phase C: incremental + full Plex ↔ Siftlode reconcile (scheduler jobs) ------------------------
# Clock-skew / round-trip slack when comparing a Plex timestamp to a Siftlode one. A state we just
# pushed (Plex stamps viewedAt a beat after our watched_at) must not read as "newer on Plex" and get
# pulled back with a different shape. Genuine Plex-side edits are minutes from our writes — far
# outside this — so the window only absorbs skew, never real changes.
_LWW_TOLERANCE_S = 90
def _active_sync_links(db: Session) -> list[PlexLink]:
"""Every link with two-way sync actually running (owner MVP = one row). Iterated uniformly so
@@ -338,11 +335,6 @@ def _resolve_account_id(db: Session, plex: PlexClient, link: PlexLink) -> int:
return acct_id
def _sift_ts(st: PlexState) -> datetime | None:
"""The freshest Siftlode-side timestamp on a row, for last-write-wins."""
return max([t for t in (st.watched_at, st.progress_updated_at) if t], default=None)
def _same_state(st: PlexState | None, status: str, pos: int) -> bool:
"""Whether a row already matches a target (status, pos) — the core ping-pong guard: when nothing
would change we skip the write entirely (no row churn, no timestamp bump, no re-pull)."""
@@ -370,9 +362,20 @@ def _pull_apply(
if st is not None and not st.synced_to_plex:
st.synced_to_plex = True # already matches Plex → just settle the flag, no churn
return None
# Conflict: keep Siftlode when it's newer than Plex (plus skew slack).
sts = _sift_ts(st) if st else None
if sts and plex_ts and plex_ts <= sts + timedelta(seconds=_LWW_TOLERANCE_S):
# Plex-authoritative (S2): on a genuine divergence for an item Plex knows about, Plex wins — no
# last-write-wins tug-of-war. This is safe because prompt write-back (playback.py pushes the resume
# position to Plex on the ~10s checkpoint, not just on the final flush) keeps Plex current, so a
# local edit that's "newer" than Plex has almost always already been pushed and matched above by
# `_same_state`; a row that reaches here really is out of step with the source of truth. The
# `_same_state` + `synced_to_plex` short-circuit above still absorbs the freshly-pushed case (Plex
# echoes the same shape a beat later) so a just-pushed row never bounces.
if st is not None and not st.synced_to_plex:
# Durability guard: this row holds a local change that never reached Plex — its immediate push
# failed, so it's still dirty. Do NOT let the Plex-authoritative pull overwrite it and clear the
# flag: that would strand the local value forever, since `_repush_dirty` only retries dirty rows.
# Leave it dirty; `_repush_dirty` (later in this same run) pushes the local value up to Plex. This
# closes the "Plex flapped mid-session" lost-update — the one case the Plex-wins comment's
# "almost always already pushed" assumption above excludes.
return None
if st is None:
st = PlexState(user_id=user_id, item_id=item_id)
@@ -436,14 +439,89 @@ def _repush_dirty(db: Session, plex: PlexClient, link: PlexLink) -> int:
return n
# How many stale in-progress movies we probe per incremental run (one metadata() call each). The set
# is normally tiny (movies started but no longer on Plex's Continue-Watching); the daily full reconcile
# is the backstop for anything beyond the cap. Most-recently-touched are probed first.
_MOVIE_PROBE_CAP = 100
def _reconcile_inprogress_movies(
db: Session, plex: PlexClient, link: PlexLink, on_deck_ids: set[int]
) -> dict[str, int]:
"""Close the movie "stuck in progress" gap that the history + onDeck feeds structurally miss.
When a movie is finished on Plex (often by scrubbing through the end credits) Plex sets
``viewCount>0``, clears ``viewOffset`` and DROPS the movie from Continue-Watching. So onDeck no
longer lists it (and onDeck can only ever ADD an in-progress row, never resolve one to watched —
see `_pull_apply`), and a scrub-to-end frequently emits no `/status/sessions/history` session row,
so the history feed misses it too. The stale local ``status='new', position>0`` row then survives
until the *daily* full reconcile. Movies are hit hardest: a movie is a single leaf with no
episode-rollup to mask the stuck row, and once watched it leaves onDeck entirely.
Fix: for each local in-progress MOVIE that Plex's onDeck no longer lists, read the item's current
per-(token-)account ``viewCount``/``viewOffset`` with one cheap `metadata()` call and apply it via
the shared `_pull_apply` — flipping to watched when Plex says so. Returns ``{"watched", "in_progress",
"probed"}`` counts. Episodes are intentionally left to onDeck/history (they promote the next episode
and aren't affected). Note: like the rest of this job, `metadata()` reads the TOKEN account (the
owner) — a HTPC playing under a managed Plex user (accountID>1) is out of scope here as elsewhere."""
conds = [
PlexState.user_id == link.user_id,
PlexState.status.notin_(("watched", "hidden")),
PlexState.position_seconds > _PROGRESS_MIN_S,
PlexItem.kind == "movie",
]
if on_deck_ids:
conds.append(PlexState.item_id.notin_(on_deck_ids))
rows = db.execute(
select(PlexState, PlexItem.rating_key)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.where(*conds)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(_MOVIE_PROBE_CAP + 1)
).all()
if len(rows) > _MOVIE_PROBE_CAP:
rows = rows[:_MOVIE_PROBE_CAP]
log.warning(
"in-progress movie probe hit the %d cap (user %s); reconcile catches the tail",
_MOVIE_PROBE_CAP,
link.user_id,
)
out = {"watched": 0, "in_progress": 0, "probed": 0}
for st, rk in rows:
try:
meta = plex.metadata(rk)
except PlexError as e:
log.warning(
"movie watch-probe failed (user %s, item %s): %s", link.user_id, st.item_id, e
)
continue
out["probed"] += 1
if meta is None:
continue
res = _plex_watch_to_state(
meta.get("viewCount"), meta.get("viewOffset"), meta.get("lastViewedAt")
)
if res is None:
# Plex holds no watch signal now (viewCount 0, no offset) — don't clear it here; an un-watch
# is the daily reconcile's job, and a Siftlode-only in-progress row must be preserved.
continue
status, pos, watched_at, prog_at = res
ch = _pull_apply(db, link.user_id, st.item_id, status, pos, watched_at or prog_at)
if ch:
out[ch] += 1
return out
def run_plex_watch_sync(db: Session) -> dict:
"""Scheduler job (default 30 min) — the cheap incremental pass. Pulls Plex-side changes since the
high-water mark (watch history + on-deck, filtered to the owner account) into Siftlode under
last-write-wins, then re-pushes any still-unsynced local states. Does NOT propagate un-watches
(the history feed only carries positive views) — that's `run_plex_watch_reconcile`'s job."""
"""Scheduler job (default 15 min) — the cheap incremental pass. Pulls Plex-side changes since the
high-water mark (watch history + on-deck, filtered to the owner account) into Siftlode, resolving a
divergence to Plex (the source of truth — see `_pull_apply`), reconciles stale in-progress movies
Plex has since finished (see `_reconcile_inprogress_movies`), then re-pushes any still-unsynced
local states. Does NOT propagate un-watches (the history feed only carries positive views) — that's
`run_plex_watch_reconcile`'s job."""
if not sysconfig.get_bool(db, "plex_enabled"):
return {"skipped": "disabled"}
total = {"links": 0, "watched": 0, "in_progress": 0, "repushed": 0}
total = {"links": 0, "watched": 0, "in_progress": 0, "repushed": 0, "movie_probes": 0}
for link in _active_sync_links(db):
total["links"] += 1
try:
@@ -463,10 +541,14 @@ def run_plex_watch_sync(db: Session) -> dict:
if ch:
total[ch] += 1
# onDeck resolves resume-position changes AND tells us which items Plex still considers
# in-progress (needed by the movie reconcile below to know which local rows are stale).
on_deck_ids: set[int] = set()
for row in plex.on_deck():
iid = rk_to_id.get(str(row.get("ratingKey") or ""))
if iid is None:
continue
on_deck_ids.add(iid)
off = int(row.get("viewOffset") or 0) // 1000
if off < _PROGRESS_MIN_S:
continue
@@ -476,6 +558,11 @@ def run_plex_watch_sync(db: Session) -> dict:
if ch:
total[ch] += 1
mv = _reconcile_inprogress_movies(db, plex, link, on_deck_ids)
total["watched"] += mv["watched"]
total["in_progress"] += mv["in_progress"]
total["movie_probes"] += mv["probed"]
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = now
db.commit()
@@ -491,7 +578,7 @@ def run_plex_watch_sync(db: Session) -> dict:
def run_plex_watch_reconcile(db: Session) -> dict:
"""Scheduler job (default daily) — the authoritative full reconcile. Rescans every section and,
per item, uses `synced_to_plex` to settle what the incremental feed can't:
- Plex has a state, Siftlode differs → last-write-wins (Plex usually the newer here).
- Plex has a state, Siftlode differs → Plex wins (the source of truth — see `_pull_apply`).
- Siftlode has a state, Plex has NO record:
· synced_to_plex=True → Plex un-watched it since our last sync → clear it (un-watch).
· synced_to_plex=False → a local change that never reached Plex → re-push it (via
+7
View File
@@ -18,6 +18,13 @@ class RateLimiter:
self._hits: dict[str, list[float]] = {}
self._lock = threading.Lock()
def reset(self) -> None:
"""Forget every recorded attempt. Each limiter is a module-level singleton, so a test suite
sharing one process would otherwise pool all of its attempts into a single window and
eventually trip an unrelated test with a 429."""
with self._lock:
self._hits.clear()
def allow(self, key: str) -> bool:
"""Record an attempt for ``key`` and return whether it is within the limit.
+13 -5
View File
@@ -17,12 +17,20 @@ from app.models import User
M = TypeVar("M", bound=Base)
def get_or_404(db: Session, model: type[M], id_: Any, *, detail: str = "Not found") -> M:
def not_found(what: str = "generic") -> HTTPException:
"""The 404 for a row that is gone — structured, so the bilingual UI says it in the reader's
language (`errors.notFound.<what>`, generic fallback). This is the 404 a user actually meets: the
thing was deleted in another tab or on another device. It used to be a bare English string
("Unknown view", "Not found"), which the Hungarian UI toasted verbatim."""
return HTTPException(status_code=404, detail={"code": "not_found", "reason": what})
def get_or_404(db: Session, model: type[M], id_: Any, *, what: str = "generic") -> M:
"""Fetch a row by primary key or raise 404. Replaces the `db.get(Model, id); if None: 404`
idiom repeated across routes."""
obj = db.get(model, id_)
if obj is None:
raise HTTPException(status_code=404, detail=detail)
raise not_found(what)
return obj
@@ -33,13 +41,13 @@ def owned_or_404(
user: User,
*,
owner_attr: str = "user_id",
detail: str = "Not found",
what: str = "generic",
) -> M:
"""Fetch a row by primary key that `user` owns, or raise 404. A row owned by someone
else 404s (NOT 403) on purpose — a 403 would confirm the id exists to a stranger.
Generalizes the near-identical per-route ownership helpers (e.g. plex's
`_own_playlist_or_404`, the saved-views owner check)."""
obj = get_or_404(db, model, id_, detail=detail)
obj = get_or_404(db, model, id_, what=what)
if getattr(obj, owner_attr) != user.id:
raise HTTPException(status_code=404, detail=detail)
raise not_found(what)
return obj
+466 -31
View File
@@ -1,20 +1,26 @@
"""Admin-only onboarding: review access requests (Invites), approve/deny, manual add.
Also the demo-account admin surface: the demo email whitelist + a manual state reset."""
from collections.abc import Iterable
from datetime import datetime, timezone
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app import audit
from app import email as email_mod
from app import mailtext
from app.audit import AuditAction
from app.auth import (
admin_user,
can_activate,
count_admins,
env_allowed_canonicals,
find_user_by_email,
get_or_create_demo_user,
is_allowed,
operator_contact,
purge_user,
)
@@ -25,13 +31,47 @@ from app.models import (
Invite,
User,
)
from app.routes.auth import revoke_preset_password_if_unverified
from app.utils import iso, valid_email
from app.routes._common import get_or_404
from app.routes.auth import (
APPROVAL_VERIFY_TTL,
issue_set_password_link,
issue_verification_link,
send_verification_link,
)
from app.utils import canonical_email, iso, valid_email
router = APIRouter(prefix="/api/admin", tags=["admin"])
def _serialize(inv: Invite) -> dict:
def _accounts_for(db: Session, emails: Iterable[str]) -> set[str]:
"""Which of these addresses already have an account, as a set of CANONICAL forms. Batched, so
listing the invites costs one extra query. An invite whose address has an account must not be
deletable: `is_allowed` reads this row, so removing it would silently revoke a live user's access.
Canonical on both sides, because `is_allowed` matches that way: an invite for `x.y@gmail.com`
whose account registered as `xy@gmail.com` would otherwise look account-less, offer the admin a
delete button, and take away a live user's access — precisely the accident this guard exists to
prevent."""
wanted = {canonical_email(e) for e in emails if e}
if not wanted:
return set()
rows = db.execute(
select(User.email_canonical).where(User.email_canonical.in_(wanted))
).scalars()
return {e for e in rows if e}
def _env_allowed(email: str) -> bool:
"""Whether the env bootstrap lists still let this address in. `is_allowed` consults them only
when no Invite row exists, so this is exactly what deleting the row would fall back to — and the
admin has to know that BEFORE they confirm, not from a toast afterwards.
Canonical, because `is_allowed` is: a warning that disagrees with the gate it is warning about is
worse than no warning."""
return canonical_email(email or "") in env_allowed_canonicals()
def _serialize(inv: Invite, *, has_account: bool = False) -> dict:
return {
"id": inv.id,
"email": inv.email,
@@ -40,6 +80,19 @@ def _serialize(inv: Invite) -> dict:
"requested_at": iso(inv.requested_at),
"decided_at": iso(inv.decided_at),
"decided_by": inv.decided_by,
# When (if ever) the admin sent this person an invitation. Adding an address is silent, so
# without this the admin has no way to answer "did I already write to them?" — and the only
# failure mode is writing to a real person twice.
"invited_at": iso(inv.invited_at),
# Drives the delete affordance: an entry backed by an account is not removable here.
"has_account": has_account,
# …and whether removing it would actually take access away, or leave the env allow-list
# granting it anyway (for an ADMIN_EMAILS address, as an admin).
"still_allowed_by_env": _env_allowed(inv.email),
# The language this person asked us in, or null when nobody ever told us (a hand-added
# whitelist row). The invitation dialog preselects it, and falls back to the admin's own UI
# language — a guess, but a visible one the admin can correct before sending.
"lang": inv.lang,
}
@@ -56,26 +109,71 @@ def list_invites(
rows = db.execute(stmt.order_by(Invite.requested_at.desc())).scalars().all()
order = {"pending": 0, "approved": 1, "denied": 2}
rows.sort(key=lambda i: order.get(i.status, 9))
return [_serialize(i) for i in rows]
accounts = _accounts_for(db, (i.email for i in rows))
return [_serialize(i, has_account=canonical_email(i.email) in accounts) for i in rows]
def _welcome_mail_parts(db: Session, email: str) -> tuple[str | None, bool, int]:
"""What a welcome or invitation mail should tell this address to do next: a confirmation link (if
there is an unconfirmed account behind it), whether an account exists at all, and how long the
link lasts.
This is the C option's whole mechanic. In the common ordering — the address is approved and the
person signs up either side of that — they would otherwise have to find and open TWO messages:
one saying they are in, another asking them to confirm. Carrying the link inside the mail that is
already going out makes it one mail and one click.
Resolved through `find_user_by_email`, so an account registered under another spelling of the
same Gmail mailbox is still recognised — and, deliberately, so several candidates resolve to
"no single answer" rather than a guess: an invitation that carried the wrong account's link would
be worse than one that simply pointed at the sign-up page.
"""
hours = int(APPROVAL_VERIFY_TTL.total_seconds() // 3600)
user = find_user_by_email(db, email)
if user is None or user.is_demo:
return None, False, hours
if user.email_verified or not email_mod.email_enabled(db):
return None, True, hours
return issue_verification_link(db, user, APPROVAL_VERIFY_TTL), True, hours
def _activate_user_by_email(db: Session, email: str) -> None:
"""Approving access activates a matching (e.g. pending password-registration) account so
it can sign in. No-op if no such user exists yet (a Google user is created at first login)."""
user = db.execute(select(User).where(User.email == email.lower())).scalar_one_or_none()
it can sign in. No-op if no such user exists yet (a Google user is created at first login).
E9.5 S1: this path deliberately does NOT revoke a preset password. Revoking here destroyed the
password a legitimate registrant had just chosen and made "your password was cleared for
security" the DEFAULT experience, because approve-before-verify is the normal ordering.
S1 justified that by saying `password_login` refuses an unverified account BEFORE it looks at
`is_active`, so a pre-emptor could not sign in "approved or not". A review of the finished epic
showed that argument only holds while the address stays unverified — and S6's approval mail
hands the real owner a confirmation link precisely so they will end that state. Nothing replaced
the revoke, and that is deliberate: round 2 of the same review tried "the newest registration
owns an unverified account" and reverted it, because the rule is symmetric — an attacker can
always register last — and it handed anonymous callers a write onto a real account's credential.
The owner's recourse is password-reset, which proves the mailbox, rewrites the hash and bumps
the session epoch. The residual is logged as SEC-16. The revoke also stays on the Google-adoption path
(auth.revoke_preset_password_if_unverified), where ownership is actually attested."""
# Canonical, with the literal spelling preferred when both exist: approving
# `npeter83.temu@gmail.com` must activate the account that registered as `npeter83temu@gmail.com`,
# since Gmail delivers both to one mailbox and the admin decided about the mailbox.
email = email.lower()
candidates = (
db.execute(
select(User).where(User.email_canonical == canonical_email(email)).order_by(User.id)
)
.scalars()
.all()
)
user = next((u for u in candidates if u.email == email), None) or next(iter(candidates), None)
if user is not None and not user.is_active:
# SEC-10: same guard as the Google-adoption path — approving an account whose email was never
# verified must not hand a working login to a password set on it by whoever registered the
# address (a pre-emption). Shared single-source helper so the two paths can't drift.
revoke_preset_password_if_unverified(user)
user.is_active = True
db.commit()
def _decide(db: Session, invite_id: int, admin: User, approved: bool) -> Invite:
inv = db.get(Invite, invite_id)
if inv is None:
raise HTTPException(status_code=404, detail="No such invite")
inv = get_or_404(db, Invite, invite_id, what="invite")
# Idempotency guard: only a still-pending invite can be decided. Without this a repeat/stale
# approve or deny re-runs its side effects — re-activating a user, or (now that deny emails)
# sending a duplicate decision mail, even flipping an already-approved invite to "denied" and
@@ -106,8 +204,13 @@ def approve_invite(
) -> dict:
inv = _decide(db, invite_id, admin, approved=True)
_activate_user_by_email(db, inv.email)
background.add_task(email_mod.send_access_approved, inv.email)
return _serialize(inv)
# Computed here, not in the background task: issuing the token needs this request's session, and
# the task must not outlive it. Approval answers a request the person MADE, so unlike the
# invitation below it is sent automatically — a reply is owed, not a nudge.
link, has_account, hours = _welcome_mail_parts(db, inv.email)
lang = mailtext.lang_for_email(db, inv.email)
background.add_task(email_mod.send_access_approved, inv.email, lang, link, has_account, hours)
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
@router.post("/invites/{invite_id}/deny")
@@ -120,8 +223,10 @@ def deny_invite(
# Symmetry with approve: the requester hears back either way, instead of a silent limbo. No
# account is activated, so this is an email only (the requester has no in-app inbox yet).
inv = _decide(db, invite_id, admin, approved=False)
background.add_task(email_mod.send_access_denied, inv.email)
return _serialize(inv)
background.add_task(
email_mod.send_access_denied, inv.email, mailtext.lang_for_email(db, inv.email)
)
return _serialize(inv, has_account=bool(_accounts_for(db, [inv.email])))
class AddInviteIn(BaseModel):
@@ -138,7 +243,16 @@ def add_invite(
email = body.email.strip().lower()
if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.")
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
# Reuse a canonically-equal row instead of adding a second one for the same mailbox. Prod carries
# exactly that pair for one person from 2026-06-16 (hartmann.peter1986@ / hartmannpeter1986@),
# which is how an admin ends up approving the same human twice and still seeing them bounce.
canon = canonical_email(email)
rows = (
db.execute(select(Invite).where(Invite.email_canonical == canon).order_by(Invite.id))
.scalars()
.all()
)
inv = next((r for r in rows if r.email == email), None) or next(iter(rows), None)
if inv is None:
inv = Invite(email=email)
db.add(inv)
@@ -150,7 +264,124 @@ def add_invite(
)
db.commit()
_activate_user_by_email(db, email)
return _serialize(inv)
return _serialize(inv, has_account=bool(_accounts_for(db, [email])))
class InviteSendIn(BaseModel):
# Which language to write this person in. Optional so an existing caller (and a test) can post
# an empty body, but the admin UI always sends it: this is the one recipient the app has no
# signal about, because the admin typed the address in themselves.
lang: str | None = None
@router.post("/invites/{invite_id}/invite")
def send_invitation(
invite_id: int,
body: InviteSendIn | None = None,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Tell someone their address is whitelisted — only when the admin asks for it.
Whitelisting stays SILENT on purpose. The operator opens a door as a possibility, usually having
already agreed it with the person, and an address can sit approved for weeks with nobody
registering; a mail fired automatically would be nudging someone who never asked to be nudged.
So the send is its own act, on its own button, and `invited_at` records it so the admin can see
they have already written rather than writing twice.
Only an APPROVED row can be invited: there is nothing to invite someone to while their request is
undecided, and inviting a denied address would contradict the decision."""
inv = get_or_404(db, Invite, invite_id, what="invite")
if inv.status != "approved":
raise HTTPException(
status_code=409,
detail={"code": "invite_send", "reason": "not_approved", "email": inv.email},
)
if not email_mod.email_enabled(db):
# Same reasoning as the admin-side verification resend: mail is fail-soft everywhere else,
# but a button the admin pressed must not answer with a success toast for a mail that was
# never sent.
raise HTTPException(status_code=409, detail={"code": "invite_send", "reason": "email_off"})
link, has_account, hours = _welcome_mail_parts(db, inv.email)
# An explicit pick from the dialog is recorded, so a later mail to this address matches the one
# the admin already sent. Only the explicit pick — writing the fallback back would turn a column
# that means "what this person told us" into "what we last guessed".
picked = body.lang if body else None
if picked in mailtext.SUPPORTED:
lang = inv.lang = picked
else:
lang = mailtext.lang_for_email(db, inv.email)
# Sent INLINE, so `invited_at` records the send rather than the intent. Every other mail here is
# fail-soft in the background, which is right when nobody is waiting on it — but this column
# exists to answer "did I already write to them?", and the UI dims the button and warns about a
# second mail on the strength of it. Stamping it before a send that then failed makes the one
# question it answers a lie, and talks the admin out of the retry that would fix it.
#
# One thing a failed send does leave behind: the confirmation token `_welcome_mail_parts` minted
# to render the link, which commits inside `_issue_token`. Accepted rather than chased — the
# token is single-use, expires on its own, and was mailed nowhere; deferring it would mean
# rewriting a helper three callers share so the handler owns its commit.
if not email_mod.send_access_invitation(inv.email, lang, link, has_account, hours):
raise HTTPException(
status_code=422, detail={"code": "invite_send", "reason": "send_failed"}
)
# Only after the mail is actually out. Same as the other two paths that decide for an address
# (`approve_invite`, `add_invite`): an approved row means this mailbox may come in, so finish it,
# or the mail can promise "your account is ready — just sign in" to someone `password_login` will
# refuse for `is_active`. Doing it BEFORE the send would mean a request the UI reports as failed
# had quietly activated an account behind the admin's back.
_activate_user_by_email(db, inv.email)
inv.invited_at = datetime.now(timezone.utc)
audit.record(
db,
admin.id,
AuditAction.INVITE_SEND,
target_type="invite",
target_id=inv.id,
summary=inv.email,
)
db.commit()
return _serialize(inv, has_account=has_account)
@router.delete("/invites/{invite_id}")
def delete_invite(
invite_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Remove a whitelist/request entry outright — the counterpart to add_invite, for an address that
never became an account (an approval added by mistake, or one that is no longer wanted). Deny
records a decision and keeps the row for the audit trail; delete forgets the address. Neither
stops the person asking again: `upsert_pending_invite` reopens a denied row, and a deleted one
simply comes back as a fresh pending request.
Refused once an account exists for the address, whatever its state: this row is the account's
approval gate, and orphaning the account from it would leave a person who can neither sign in nor
be approved. Delete the USER instead — purge_user removes the invite along with it.
Reports `still_allowed_by_env`, because deleting the row is NOT always a revoke: `is_allowed`
falls back to the env ALLOWED_EMAILS / ADMIN_EMAILS bootstrap lists once no invite exists, so for
an address on those the deletion leaves access intact (and an ADMIN_EMAILS address is still
promoted to admin on its next Google sign-in). The admin has to hear that, not guess it."""
inv = get_or_404(db, Invite, invite_id, what="invite")
if _accounts_for(db, [inv.email]):
raise HTTPException(
status_code=409,
detail="That address already has an account — delete the account instead.",
)
still_allowed_by_env = _env_allowed(inv.email)
audit.record(
db,
admin.id,
AuditAction.INVITE_DELETE,
target_type="invite",
target_id=inv.id,
summary=inv.email,
)
db.delete(inv)
db.commit()
return {"deleted": invite_id, "still_allowed_by_env": still_allowed_by_env}
# --- Users & roles --------------------------------------------------------------------
@@ -167,7 +398,15 @@ def _serialize_user(u: User) -> dict:
"email_verified": u.email_verified,
"is_demo": u.is_demo,
"has_password": u.password_hash is not None,
# Distinguishes "this account never had a password" (Google-only) from "SEC-10 cleared it" —
# the same distinction _send_reset_if_eligible needs, and the one an admin has to see before
# deciding what kind of help someone needs.
"password_revoked_at": iso(u.password_revoked_at),
"has_google": u.google_sub is not None,
# Only when it DIFFERS from the account address (E9.5 S3): "connected" is not the same
# statement as "connected as someone else", and the second one is the one that explains a
# confusing sign-in.
"google_email": u.google_email if u.google_email != u.email else None,
"created_at": iso(u.created_at),
}
@@ -195,9 +434,7 @@ def set_user_role(
role = body.role
if role not in ("user", "admin"):
raise HTTPException(status_code=400, detail="role must be 'user' or 'admin'")
target = db.get(User, user_id)
if target is None:
raise HTTPException(status_code=404, detail="No such user")
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account's role can't be changed.")
if target.id == admin.id:
@@ -224,7 +461,13 @@ def set_user_role(
target.role = role
db.commit()
if changed:
background.add_task(email_mod.send_role_changed, target.email, role, operator_contact())
background.add_task(
email_mod.send_role_changed,
target.email,
mailtext.lang_for_user(target),
role,
operator_contact(),
)
return _serialize_user(target)
@@ -246,9 +489,7 @@ def set_user_suspended(
un-suspend the user is emailed they can return (suspend is announced reactively, on a blocked
sign-in)."""
suspended = body.suspended
target = db.get(User, user_id)
if target is None:
raise HTTPException(status_code=404, detail="No such user")
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be suspended.")
if target.id == admin.id:
@@ -273,7 +514,12 @@ def set_user_suspended(
target.is_suspended = suspended
db.commit()
if was_suspended and not suspended:
background.add_task(email_mod.send_account_unsuspended, target.email, operator_contact())
background.add_task(
email_mod.send_account_unsuspended,
target.email,
mailtext.lang_for_user(target),
operator_contact(),
)
return _serialize_user(target)
@@ -287,9 +533,7 @@ def admin_delete_user(
"""Admin-triggered account deletion — the same full erasure a user can perform on themselves
(cascade delete of all personal data + access-request cleanup + Google-grant revoke). Guards:
not the demo account, not yourself (use Settings → Account), and not the last admin."""
target = db.get(User, user_id)
if target is None:
raise HTTPException(status_code=404, detail="No such user")
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be deleted.")
if target.id == admin.id:
@@ -318,6 +562,197 @@ def admin_delete_user(
# --- Demo account: email whitelist + state reset -------------------------------------
# --- Stuck sign-ups: the admin lifeline (E9.5 S5, roadmap UX-25) ---------------------------------
def _pending_access_rows(db: Session) -> list[User]:
"""Accounts that have signed up but still cannot sign in, newest first.
Two different dead ends share this list, because from the admin's chair they look the same — a
person who registered and is still outside:
· `is_active=False` — approved BEFORE they signed up, so nothing ever landed in the
Access-requests queue and its Approve button doesn't apply (`_decide` only takes a pending
row). This is roadmap UX-25, and it is why the list exists.
· `email_verified=False` — the confirmation mail hasn't been acted on. Not the admin's to
decide, but theirs to help with: on 2026-09-03 the first mail to a fresh Gmail landed in
Spam, so "send it again" is a real move.
A suspended account is deliberately excluded: it cannot sign in either, but that is a decision
already made, not a person stuck waiting on one.
"""
return list(
db.execute(
select(User)
.where(
User.is_demo.is_(False),
User.is_suspended.is_(False),
or_(User.is_active.is_(False), User.email_verified.is_(False)),
)
.order_by(User.created_at.desc())
)
.scalars()
.all()
)
@router.get("/pending-access")
def list_pending_access(_: User = Depends(admin_user), db: Session = Depends(get_db)) -> list[dict]:
"""The sign-ups that are still outside, with which gate is shut for each. `can_activate` cannot
be derived on the client — it needs the invite/allow-list lookup — so it is answered here.
One `is_allowed` query per row rather than a batched re-implementation of it: the row count is
the admin's own backlog of stuck sign-ups (single digits on any real instance), and a second,
subtly different copy of the "may this address sign in" rule is exactly the drift this codebase
keeps getting bitten by."""
out = []
for u in _pending_access_rows(db):
# The whole rule in one line: this address MAY sign in, and this account still can't. An
# undecided request fails it (nothing is allowed yet) and so does a denied one — both belong
# to the queue above, which owns that decision. Without this, every fresh registration
# appeared in both lists at once, which UAT caught on 2026-09-05.
if not is_allowed(db, u.email):
continue
out.append(_serialize_user(u) | {"can_activate": can_activate(db, u)})
return out
@router.get("/users/{user_id}/access")
def user_access(
user_id: int,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Everything the account panel shows about why this person can or cannot get in (E12 S5).
Its own endpoint rather than extra columns on /users: `is_allowed` and `can_activate` each run a
query, and the panel opens for ONE user at a time — folding them into the list would pay that cost
per row on every load to answer a question nobody asked yet."""
target = get_or_404(db, User, user_id, what="user")
return _serialize_user(target) | {
"is_allowed": is_allowed(db, target.email),
"can_activate": can_activate(db, target),
"email_configured": email_mod.email_enabled(db),
}
@router.post("/users/{user_id}/set-password-link")
def create_set_password_link(
user_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Mint a one-time link that lets this account set a password, with no email involved (E12 S5).
The answer to SEC-15, SEC-16 and "no SMTP and someone forgot their password" at once. The admin
hands the link over by whatever channel they trust; opening it sets a password and signs out
whoever held the old one.
Deliberately NOT gated on the account's state. It is useful for a pre-empted address, for a
revoked password, for a Google-only account that wants one, and for a plain forgotten password on
an instance with no mail — narrowing it to one of those would only make the admin's real case the
one it refuses. The demo account is the single exception: it is shared, so handing anyone a way to
change its password is handing them everyone else's session."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(
status_code=400, detail={"code": "setpw", "reason": "demo", "email": target.email}
)
url, expires_at = issue_set_password_link(db, target)
# The URL itself is NOT recorded: the audit log is read by humans and kept long, and a live
# credential-setting link is the last thing that belongs in it. That the link was issued, by whom,
# for whom, is the auditable fact.
audit.record(
db,
admin.id,
AuditAction.USER_SETPW_LINK,
target_type="user",
target_id=target.id,
summary=target.email,
)
db.commit()
return {"ok": True, "url": url, "expires_at": iso(expires_at), "email": target.email}
@router.post("/users/{user_id}/activate")
def activate_user(
user_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Let in a registrant whose address was approved before they signed up (roadmap UX-25).
Admin-side ONLY, and that is the whole design. Two earlier attempts at this were reverted after
review, and both moved authority rather than visibility: auto-activating on registration turns a
standing whitelist entry into a bearer grant for whoever registers the address first, and
reopening the approved invite to `pending` lets an anonymous sign-up REVOKE a standing approval.
This endpoint changes nothing about what an unauthenticated caller can cause — the admin could
already do exactly this by re-entering the address in the whitelist box (`add_invite` →
`_activate_user_by_email`). What was missing was any way to know they had to."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo:
raise HTTPException(status_code=400, detail="The demo account can't be activated.")
if target.is_active:
return _serialize_user(target) # idempotent: a double click is not an error
if not is_allowed(db, target.email):
raise HTTPException(
status_code=409,
detail={"code": "activate", "reason": "not_allowed", "email": target.email},
)
audit.record(
db,
admin.id,
AuditAction.USER_ACTIVATE,
target_type="user",
target_id=target.id,
summary=target.email,
before={"is_active": False},
after={"is_active": True},
)
target.is_active = True
db.commit()
return _serialize_user(target)
@router.post("/users/{user_id}/resend-verification")
def resend_user_verification(
user_id: int,
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Re-send an account's confirmation link from the admin side.
The public /auth/verify/resend answers uniformly and reveals nothing, which is right for an
anonymous caller but leaves the admin unable to tell a delivered mail from a dropped one. Here
the caller is already authenticated as an admin, so this one says plainly what happened.
Not rate-limited: an admin is not a threat model to enumerate against, and the deliberate silence
of the public endpoint is exactly what this exists to escape."""
target = get_or_404(db, User, user_id, what="user")
if target.is_demo or target.email_verified:
raise HTTPException(
status_code=409,
detail={"code": "resend", "reason": "not_applicable", "email": target.email},
)
if not email_mod.email_enabled(db):
# Mail is fail-soft everywhere else, which is what left the 0.78.5 sign-ups with nothing to
# fall back on. An admin pressing a button deserves the failure, not a success toast.
raise HTTPException(status_code=409, detail={"code": "resend", "reason": "email_off"})
if not send_verification_link(db, target):
# Same rule as the invitation above: a button an admin pressed must not answer with a
# success for a mail that was dropped. Mail is fail-soft everywhere else because nobody is
# waiting on the answer; here somebody is.
raise HTTPException(status_code=422, detail={"code": "resend", "reason": "send_failed"})
audit.record(
db,
admin.id,
AuditAction.USER_VERIFY_RESEND,
target_type="user",
target_id=target.id,
summary=target.email,
)
db.commit()
return {"ok": True, "email": target.email}
def _serialize_demo(row: DemoWhitelist) -> dict:
return {
"id": row.id,
+513 -111
View File
@@ -6,6 +6,8 @@ helpers) stays in app/auth.py, imported below."""
import ipaddress
import secrets
from datetime import datetime, timedelta, timezone
from typing import NamedTuple
from urllib.parse import urlencode
from authlib.integrations.starlette_client import OAuth, OAuthError
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
@@ -16,6 +18,7 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import email as email_mod
from app import mailtext
from app import sysconfig
from app.auth import (
_YOUTUBE_SCOPES,
@@ -25,6 +28,8 @@ from app.auth import (
READ_SCOPES,
WRITE_SCOPES,
current_user,
env_admin_canonicals,
find_user_by_email,
get_or_create_demo_user,
google_enabled,
is_allowed,
@@ -37,27 +42,53 @@ from app.db import SessionLocal, get_db
from app.models import AuthToken, DemoWhitelist, Invite, OAuthToken, User
from app.ratelimit import RateLimiter
from app.security import encrypt, hash_password, hash_token, verify_password
from app.utils import valid_email
from app.utils import canonical_email, valid_email
VERIFY_TTL = timedelta(hours=24)
# A confirmation link that rides an APPROVAL or an invitation lives longer than one the person is
# waiting for. Nobody sits refreshing their inbox for an approval — it can easily be read on Monday
# after a weekend, and if it has expired by then the one-mail-one-click promise is spent and they are
# back to two messages. The risk is small: this token only ever confirms an address. It signs nobody
# in and sets no password.
APPROVAL_VERIFY_TTL = timedelta(days=7)
RESET_TTL = timedelta(hours=1)
# An admin-issued set-password link (E12 S5) travels out of band — the admin has to actually reach the
# person and hand it over — so the one-hour reset window is far too short. A day matches VERIFY_TTL and
# the affordance the admin was shown.
ADMIN_SETPW_TTL = timedelta(hours=24)
# Both kinds set a password from a link, and the SIGNED-OUT card consumes them through the same
# `#reset=` entry, so the user-facing flow is one flow. They stay separate kinds for one reason: a
# `reset` token proves control of the MAILBOX (it was mailed there) and therefore confirms the address;
# an admin-issued one proves nothing about the mailbox — the admin could hand it to anyone — so it must
# not silently satisfy the email gate. See password_reset_confirm.
_PASSWORD_TOKEN_KINDS = ("reset", "setpw")
_DECOY_PASSWORD_HASH = hash_password(secrets.token_urlsafe(16))
_register_limiter = RateLimiter(max_events=5, window_seconds=300)
_login_limiter = RateLimiter(max_events=10, window_seconds=300)
_reset_limiter = RateLimiter(max_events=5, window_seconds=300)
# Separate from _reset_limiter on purpose: opening a link must not eat the budget for REQUESTING one.
_reset_check_limiter = RateLimiter(max_events=10, window_seconds=300)
_verify_resend_limiter = RateLimiter(max_events=5, window_seconds=300)
_request_access_limiter = RateLimiter(max_events=5, window_seconds=300)
_suspend_email_limiter = RateLimiter(max_events=1, window_seconds=3600)
def _notify_suspended(background: BackgroundTasks, email: str) -> None:
def _notify_suspended(background: BackgroundTasks, user: User) -> None:
"""Schedule the one-per-hour 'account suspended' email to a user whose (otherwise valid)
sign-in was just blocked. Gated by verified credentials at the call site, so this can't be
triggered by someone who doesn't control the account."""
if _suspend_email_limiter.allow(email):
background.add_task(email_mod.send_account_suspended, email, operator_contact())
triggered by someone who doesn't control the account.
Takes the user rather than the address because the mail needs their language, and the send runs
as a background task — after this request's session, and the row, are out of reach."""
if _suspend_email_limiter.allow(user.email):
background.add_task(
email_mod.send_account_suspended,
user.email,
mailtext.lang_for_user(user),
operator_contact(),
)
_demo_limiter = RateLimiter(max_events=5, window_seconds=60)
@@ -90,18 +121,43 @@ def google_oauth(db: Session):
return _oauth.google
def admin_notify_emails(db: Session) -> list[str]:
"""Who to email about admin events (e.g. a new access request). The ACTUAL admins — every active,
non-suspended `role=admin` user — UNIONED with the env ADMIN_EMAILS bootstrap list. Using the DB
roles means an admin promoted through the UI is notified too (env alone would miss them, yet they
CAN approve requests — the approve UI is role-gated, so the notify set must match); the env union
keeps a freshly-seeded instance working before any DB admin exists."""
db_admins = db.execute(
select(User.email).where(
def admin_notify_groups(db: Session) -> dict[str, list[str]]:
"""Who to email about admin events (e.g. a new access request), grouped by the language each of
them reads. The ACTUAL admins — every active, non-suspended `role=admin` user — UNIONED with the
env ADMIN_EMAILS bootstrap list. Using the DB roles means an admin promoted through the UI is
notified too (env alone would miss them, yet they CAN approve requests — the approve UI is
role-gated, so the notify set must match); the env union keeps a freshly-seeded instance working
before any DB admin exists.
Grouped rather than one blast to everyone, because these recipients are people with their own UI
language: a single message would have to pick one of them and be wrong for the rest. An env
address with no admin account still gets looked up, so an operator listed in ADMIN_EMAILS whose
account is (say) suspended is still written to in the language they chose."""
rows = db.execute(
select(User.email, User.preferences).where(
User.role == "admin", User.is_active.is_(True), User.is_suspended.is_(False)
)
).scalars()
return sorted({e.lower() for e in db_admins if e} | settings.admin_email_set)
).all()
by_email = {e.lower(): mailtext.normalize((p or {}).get("language")) for e, p in rows if e}
for email in settings.admin_email_set:
if email not in by_email:
by_email[email] = mailtext.lang_for_email(db, email)
groups: dict[str, list[str]] = {}
for email in sorted(by_email):
groups.setdefault(by_email[email], []).append(email)
return groups
def notify_admins_of_request(
db: Session, background: BackgroundTasks | None, requester: str
) -> None:
"""Tell every admin that someone is waiting — one mail per language group. Sends inline when
there is no BackgroundTasks to hand (the registration path already runs outside a response)."""
for lang, recipients in admin_notify_groups(db).items():
if background is None:
email_mod.send_admin_new_request(recipients, lang, requester)
else:
background.add_task(email_mod.send_admin_new_request, recipients, lang, requester)
def is_demo_allowed(db: Session, email: str) -> bool:
@@ -147,14 +203,64 @@ def _client_ip(request: Request) -> str:
return peer
def upsert_pending_invite(db: Session, email: str) -> Invite | None:
"""Idempotently record an access request. Returns the Invite if a *new* pending row
was created (so the caller can notify admins), else None for already-known emails."""
class InviteRequest(NamedTuple):
"""What an access request settled on.
`invite` is the row the request now lives under — which is NOT always the address that was just
typed: one Gmail mailbox spelled two ways settles on whichever row already exists, and that row's
address is where mail will go. Callers that tell the user what happened need it (E9.5 S4).
`is_new` is the old boolean return in a name: true only when a row was created or a denied one
reopened, i.e. exactly when admins should hear about it. A repeat request must not re-spam them.
"""
invite: Invite | None
is_new: bool
def upsert_pending_invite(db: Session, email: str, lang: str | None = None) -> InviteRequest:
"""Idempotently record an access request, and the language it was made in.
"Already known" is decided on the CANONICAL form: `npeter83.temu@` and `npeter83temu@` are one
Gmail mailbox, and treating them as two is what put a second, silent access request in the admin
queue on 2026-09-03 for an already-approved user. An approved sibling wins — re-requesting must
not reopen a decision that has already been made for that mailbox. The row is still STORED under
the address as typed, because that is where mail has to go.
`lang` is the language of the surface the request came from — the sign-in page, or the locale
Google reported for a refused sign-in. The latest one wins: someone who asks again from a
Hungarian page is telling us something newer than last time. Anything unsupported is ignored
rather than stored, so a forged body cannot put junk in the column.
The commit contract is unchanged: this still commits ONLY when it writes something. Callers rely
on that — `_register_account` persists its own flushed user precisely because a no-op here
commits nothing (E9.5 S6).
"""
email = (email or "").lower()
if not valid_email(email):
return None
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
return InviteRequest(None, False)
lang = lang if lang in mailtext.SUPPORTED else None
canon = canonical_email(email)
rows = (
db.execute(select(Invite).where(Invite.email_canonical == canon).order_by(Invite.id))
.scalars()
.all()
)
# An approved row wins, so a denied duplicate can't mask an approval. Then a row that is not
# DENIED, preferring the exact spelling — a denied sibling must never be picked while a pending
# row exists, or the reopen branch below would fire and put one person in the admin's queue
# twice. Only when every row is denied does the spelling decide which one reopens.
undecided = [r for r in rows if r.status != "denied"]
inv = (
next((r for r in rows if r.status == "approved"), None)
or next((r for r in undecided if r.email == email), None)
or next(iter(undecided), None)
or next((r for r in rows if r.email == email), None)
or next(iter(rows), None)
)
if inv is not None:
relearned = lang is not None and inv.lang != lang
if relearned:
inv.lang = lang
# Let a previously-denied person ask again; don't disturb approved/pending rows.
if inv.status == "denied":
inv.status = "pending"
@@ -162,12 +268,39 @@ def upsert_pending_invite(db: Session, email: str) -> Invite | None:
inv.decided_at = None
inv.decided_by = None
db.commit()
return inv
return None
inv = Invite(email=email, status="pending")
return InviteRequest(inv, True)
if relearned:
db.commit()
return InviteRequest(inv, False)
inv = Invite(email=email, status="pending", lang=lang)
db.add(inv)
db.commit()
return inv
return InviteRequest(inv, True)
def _access_notice_url(kind: str, email: str = "", inv: Invite | None = None) -> str:
"""The landing URL for a Google sign-in that did not reach an account: `requested` (we recorded
the request) or `denied` (nothing to record).
It rides the FRAGMENT, not the query string, for the same reason the verify/reset tokens do
(SB3): a fragment never reaches the server, a proxy access log or a Referer header — and this
notice now carries the user's address. Naming it is the point of E9.5 S4: on 2026-09-03 a
registration as `npeter83.temu@` met an OAuth claim of `npeter83temu@`, and a screen that said
"request recorded" without saying FOR WHAT hid the mismatch for a day.
`linked` names the address the request actually lives under, and ONLY when that differs from the
one Google just returned — the honest version of "we didn't open a second request". Same
spelling twice is just a repeat, and saying anything there would be noise.
urlencode (not an f-string): a `+tag` address must arrive as %2B, or the SPA's URLSearchParams
would read it back as a space and mangle exactly the address this epic exists to protect.
"""
params = {"access": kind}
if email:
params["email"] = email
if inv is not None and inv.email != email:
params["linked"] = inv.email
return "/#" + urlencode(params)
_DEV_OAUTH_HOSTS = frozenset(
@@ -201,6 +334,12 @@ class EmailIn(BaseModel):
class EmailPasswordIn(BaseModel):
email: str | None = None
password: str | None = None
# The language the sign-up page was in. Captured at registration because it is the ONLY moment
# the choice exists anywhere the server can see: it lives in the browser until an account does,
# and a preference nobody recorded cannot be backfilled later (the lesson `google_email` taught
# in E9.5 S3 — those rows still read NULL until each user signs in again). Nothing reads it yet;
# the bilingual mail templates are E9.5 S7.
lang: str | None = None
class TokenIn(BaseModel):
@@ -283,26 +422,37 @@ async def callback(
# A denied Google login doubles as an access request: record it for the admin and
# land the user on a friendly "request received" screen instead of a raw 403.
if email:
inv = upsert_pending_invite(db, email)
admins = admin_notify_emails(db)
if inv is not None and admins:
background.add_task(email_mod.send_admin_new_request, admins, email)
return RedirectResponse(url="/?access=requested")
return RedirectResponse(url="/?access=denied")
# Google just told us this person's locale — the only language signal that will ever
# exist for an address with no account, and the decision mail is written from it.
req = upsert_pending_invite(db, email, (userinfo.get("locale") or "").split("-")[0])
if req.is_new:
notify_admins_of_request(db, background, email)
return RedirectResponse(url=_access_notice_url("requested", email, req.invite))
return RedirectResponse(url=_access_notice_url("denied"))
user = db.query(User).filter(User.google_sub == sub).one_or_none()
if user is None:
# No account carries this Google identity yet. If one already exists for this email —
# No account carries this Google identity yet. If one already exists for this MAILBOX —
# e.g. an email+password registration — adopt the Google identity onto it instead of
# inserting a duplicate (which would collide on the unique email and 500). Google has
# verified the address, so this safely unifies password + Google sign-in on one account.
existing = db.query(User).filter(User.email == email).one_or_none()
# inserting a duplicate. Google has verified the address, so this safely unifies password +
# Google sign-in on one account.
#
# Matched canonically, and that is not cosmetic: on 2026-09-03 a registration as
# `npeter83.temu@gmail.com` met an OAuth claim of `npeter83temu@gmail.com`, the literal
# lookup found nothing, and prod ended up with TWO active user rows for one Gmail mailbox
# (ids 13 and 14). Exact spelling still wins when both exist, so an account that already
# matches literally is never passed over for a canonical sibling.
canon = canonical_email(email)
candidates = db.query(User).filter(User.email_canonical == canon).order_by(User.id).all()
existing = next((u for u in candidates if u.email == email), None) or next(
iter(candidates), None
)
if existing is not None:
if existing.google_sub is not None and existing.google_sub != sub:
# A different Google identity is already bound to this email — refuse rather than
# hijack it. (Near-impossible with real Google accounts: email↔sub is stable.)
log.warning("Google login email collision (different sub): %s", email)
return RedirectResponse(url="/?access=denied")
return RedirectResponse(url=_access_notice_url("denied"))
if not userinfo.get("email_verified", False):
# Adopting (and activating) a pre-existing password account requires Google to
# actually attest the address — otherwise an unverified Google email that merely
@@ -311,7 +461,7 @@ async def callback(
"Google login: unverified email, refusing to adopt account: %s",
email,
)
return RedirectResponse(url="/?access=denied")
return RedirectResponse(url=_access_notice_url("denied"))
user = existing
user.google_sub = sub
# SEC-10: drop any password pre-set on this account while its email was still unverified,
@@ -328,22 +478,23 @@ async def callback(
# tell the (verified) owner why — a successful Google auth proves they control this account.
if user.is_suspended:
log.info("Suspended account blocked at Google login: %s", email)
_notify_suspended(background, user.email)
_notify_suspended(background, user)
return RedirectResponse(url="/?login=suspended")
# Keep the email in sync with Google, but never overwrite it with one another account already
# owns — that would hit the unique constraint and 500, wedging this user's sign-in (rare: a
# Google-side email change to a value that collides with a different local account).
if user.email != email:
clash = db.query(User).filter(User.email == email, User.id != user.id).one_or_none()
if clash is None:
user.email = email
# Record what Google actually answers with, beside the address the user registered with. The two
# are the same string for most people; where they differ (a deliberate `+tag`, a different dot
# pattern) the UI shows both, so nobody has to guess which one signs them in.
if email:
user.google_email = email
_sync_google_email(db, user, email)
user.display_name = userinfo.get("name")
user.avatar_url = userinfo.get("picture")
# ADMIN_EMAILS (env) is the bootstrap admin list and always wins, so the configured admin can't
# be locked out. Everyone else's role is managed in the admin UI and stored in the DB, so a
# routine login must NOT reset it (that would clobber a UI promotion/demotion). Only seed a
# default for a brand-new account (its role attribute is still unset before flush).
if email in settings.admin_email_set:
# Canonical, like every other read of the env lists: a dotted or `+tag`ged ADMIN_EMAILS entry
# would otherwise never bootstrap its admin, because Google answers with the plain primary.
if canonical_email(email) in env_admin_canonicals():
user.role = "admin"
elif not user.role:
user.role = "user"
@@ -355,8 +506,7 @@ async def callback(
# picked one yet); unsupported locales fall back to English.
prefs = dict(user.preferences or {})
if not prefs.get("language"):
loc = (userinfo.get("locale") or "").split("-")[0].lower()
prefs["language"] = loc if loc in {"en", "hu"} else "en"
prefs["language"] = mailtext.normalize((userinfo.get("locale") or "").split("-")[0].lower())
user.preferences = prefs
db.flush()
@@ -375,6 +525,30 @@ async def callback(
return RedirectResponse(url="/")
def _sync_google_email(db: Session, user: User, email: str) -> None:
"""Follow a genuine Google-side email change — and ONLY that.
A different SPELLING of the same mailbox is the user's own choice and must survive. People
register as `me+siftlode@gmail.com` precisely so a later leak from this site is identifiable, and
silently rewriting that to `me@gmail.com` destroys the thing it was for. It also locked them out
of the spelling they chose, because sign-in matches the stored address first. Google returns the
account's PRIMARY address, which need not be what they registered with — and can never contain a
`+` at all — so without this guard every plus-tagged registration lost its tag on the first Google
sign-in. (E9.5 S3 made that reachable by matching adoption across spellings; the rewrite itself is
older.)
Never overwrite with an address another account already owns either: that would hit the unique
constraint and 500, wedging this user's sign-in.
"""
if not email or user.email == email:
return
if canonical_email(user.email) == canonical_email(email):
return # same mailbox, their spelling — leave it alone
clash = db.query(User).filter(User.email == email, User.id != user.id).one_or_none()
if clash is None:
user.email = email
def _store_token(db: Session, user: User, token: dict) -> None:
"""Persist the OAuth grant on the user's single token row, creating it if absent. Google
only returns a refresh_token on (re)consent, so the previous one is kept otherwise.
@@ -435,6 +609,9 @@ def _complete_link(
if user.google_sub is not None and user.google_sub != sub:
return RedirectResponse(url="/?link=mismatch")
user.google_sub = sub
linked_email = (userinfo.get("email") or "").strip().lower()
if linked_email:
user.google_email = linked_email # see the sign-in callback: the other half of the identity
if not user.display_name:
user.display_name = userinfo.get("name")
if not user.avatar_url:
@@ -468,15 +645,23 @@ async def logout(request: Request):
return JSONResponse({"ok": True, "switched": False})
class AccessRequestIn(EmailIn):
# The language of the page this request was made from. Same reasoning as registration's `lang`
# (and the same shape): it is the ONLY moment this preference exists anywhere the server can
# see, since the requester has no account to carry it, and it decides what language the approval
# or denial mail is written in.
lang: str | None = None
@router.post("/request-access")
async def request_access(
body: EmailIn,
body: AccessRequestIn,
request: Request,
background: BackgroundTasks,
db: Session = Depends(get_db),
) -> dict:
"""Public: ask for access. Idempotent — repeat requests for the same email don't
duplicate or re-spam admins (upsert returns None for an already-pending row).
duplicate or re-spam admins (upsert answers `is_new=False` for an already-pending row).
Answers a uniform ``"pending"`` for EVERY outcome — already-allowed, newly-pending,
already-pending, and rate-limited alike. The old ``"approved"`` reply for an allow-listed
@@ -491,10 +676,9 @@ async def request_access(
return {"status": "pending"}
if is_allowed(db, email):
return {"status": "pending"} # already allowed — don't reveal it
inv = upsert_pending_invite(db, email)
admins = admin_notify_emails(db)
if inv is not None and admins:
background.add_task(email_mod.send_admin_new_request, admins, email)
req = upsert_pending_invite(db, email, body.lang)
if req.is_new:
notify_admins_of_request(db, background, email)
return {"status": "pending"}
@@ -551,6 +735,29 @@ def _issue_token(db: Session, user: User, kind: str, ttl: timedelta) -> str:
return raw
def _peek_token(db: Session, raw: str | None, kind: str) -> User | None:
"""Is this token still usable — WITHOUT spending it. Same four conditions as _consume_token, and
the same "does its user still exist" answer, so the two can never disagree about a given token.
Deliberately NOT the read half of a read-check-write: _consume_token stays one atomic UPDATE, and
this never feeds it. A peek is only ever an early warning — the confirm endpoint remains the
authority, because a token can expire, be spent, or lose its account between the two calls.
"""
if not raw:
return None
row = db.execute(
select(AuthToken).where(
AuthToken.token_hash == hash_token(raw),
AuthToken.kind == kind,
AuthToken.used_at.is_(None),
AuthToken.expires_at >= datetime.now(timezone.utc),
)
).scalar_one_or_none()
# db.get, exactly as _consume_token does: a deleted account cascades its tokens away, so this is
# belt-and-braces — but the two must agree, and agreeing by construction beats agreeing by luck.
return db.get(User, row.user_id) if row is not None else None
def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
"""Validate + burn a token atomically. Returns its user, or None if missing/expired/used/wrong
kind. The burn is a single conditional UPDATE (…WHERE used_at IS NULL…RETURNING) rather than a
@@ -576,6 +783,35 @@ def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
return db.get(User, row.user_id)
def _consume_password_token(db: Session, raw: str | None) -> tuple[User, str] | None:
"""Spend a password-setting token of either kind; returns (user, kind). The kind is returned, not
swallowed, because the caller has to treat them differently — only a mailed reset attests the
address."""
for kind in _PASSWORD_TOKEN_KINDS:
user = _consume_token(db, raw, kind)
if user is not None:
return user, kind
return None
def issue_set_password_link(db: Session, user: User) -> tuple[str, datetime]:
"""An admin-issued, single-use link that lets `user` set a password without any email (E12 S5).
One mechanism answers three things: SEC-15 (on a no-SMTP instance every registration is
auto-verified, so a pre-empted address cannot be recovered by mail), SEC-16 (a pre-empted address
stays takeable until its owner resets, and password-reset needs working mail), and the everyday
"no SMTP configured and someone forgot their password".
Issuing does NOT revoke the current password: an admin misclick must not lock anyone out, and the
account may be perfectly healthy. USING the link replaces the hash and bumps the session epoch,
which is what evicts a squatter — see password_reset_confirm."""
expires_at = datetime.now(timezone.utc) + ADMIN_SETPW_TTL
raw = _issue_token(db, user, "setpw", ADMIN_SETPW_TTL)
# The same `#reset=` fragment the mailed link uses: the signed-out card already knows how to
# consume it, and a second entry shape would be a second thing to keep correct.
return f"{_app_base()}/#reset={raw}", expires_at
@router.post("/register")
def register(
body: EmailPasswordIn,
@@ -606,17 +842,59 @@ def register(
# session). The create path hashes the password + writes several rows + schedules emails; doing it
# inline would make a NEW email respond measurably slower than an already-registered one (which
# skips all that) — a timing oracle for enumeration. Off-path, any valid email responds the same.
background.add_task(_register_account, email, password)
background.add_task(_register_account, email, password, body.lang)
return {"status": "ok"}
def _register_account(email: str, password: str) -> None:
def _register_account(email: str, password: str, lang: str | None = None) -> None:
"""Background worker for /register: create the pending account (+ verification email + admin
notice) for a genuinely new email; no-op for an already-registered one. Runs after the response
with its own DB session, so the request's timing never reveals whether the email already exists."""
with SessionLocal() as db:
if db.execute(select(User).where(User.email == email)).scalar_one_or_none() is not None:
return # already registered — nothing to do (the uniform "ok" was already returned)
# Canonical, so a second spelling of one Gmail mailbox cannot open a SECOND account. Prod
# already carries such a pair (user ids 13 and 14, both for npeter83temu@gmail.com); this is
# the end that stops new ones appearing, while the Google callback copes with the existing.
candidates = (
db.execute(
select(User).where(User.email_canonical == canonical_email(email)).order_by(User.id)
)
.scalars()
.all()
)
# Exact spelling wins when a mailbox carries two rows, same as every other canonical lookup
# in this epic (`find_user_by_email`, `_activate_user_by_email`, the Google callback). Without
# it `.first()` handed back whichever row Postgres felt like, so a re-registration of the
# dotted spelling could resolve to the undotted, already-confirmed twin and send nothing.
existing = next((u for u in candidates if u.email == email), None) or next(
iter(candidates), None
)
if existing is not None:
# Already registered. This used to return in silence, and that silence is the most
# likely explanation for the 2026-09-03 report that a confirmation mail "never arrived":
# nothing was ever SENT. The person re-registered, got the same cheerful "check your
# inbox", and waited for a message that did not exist.
#
# Re-sending the link instead leaks nothing new. The response and its timing are
# unchanged (uniform "ok", decided off the response path), the mail goes to the address
# ON THE ACCOUNT, and this is exactly what the public /auth/verify/resend already does
# for the same input — so it is not a new capability, just the end of a dead end. An
# account that is already confirmed still gets nothing: there is nothing to confirm.
#
# And a re-send is ALL it does: the submitted password is discarded. E9.5's closing
# review tried "while the address is unverified, the newest registration owns it"
# (replace the hash, kill the outstanding verify tokens) as a fix for address
# pre-emption, and round 2 showed it was worse than the problem — DO NOT REPEAT IT.
# The rule is symmetric, and an attacker can always be last: it hands an unauthenticated
# caller the power to overwrite a real registrant's chosen password and delete the
# confirmation link already in their inbox, and on a SEC-10-stripped row (password_hash
# NULL, password_revoked_at set) it would install a password where the whole point was
# that there is none. The owner's real recourse already exists and needs no new
# primitive: password-reset accepts this account, proves the mailbox, rewrites the hash
# and bumps the session epoch. The residual is logged as SEC-16.
user = existing
if not user.email_verified and not user.is_demo and email_mod.email_enabled():
send_verification_link(db, user)
return
# Without working SMTP we can't deliver a verification link, so email ownership can't gate
# sign-in. Admin approval stays the real gate (is_active=False); mark the account verified so
# the flow still completes on a no-SMTP self-host. See email.email_enabled().
@@ -626,6 +904,8 @@ def _register_account(email: str, password: str) -> None:
password_hash=hash_password(password),
is_active=False,
email_verified=not email_ok,
# Only a supported code is stored, so a forged body can't put junk in preferences.
preferences={"language": lang} if lang in mailtext.SUPPORTED else {},
)
db.add(user)
try:
@@ -636,28 +916,81 @@ def _register_account(email: str, password: str) -> None:
# no-op as the pre-check hit — roll back and stop.
db.rollback()
return
upsert_pending_invite(db, email) # admin-approval gate (commits)
upsert_pending_invite(db, email) # admin-approval gate
# Persist the account ourselves. upsert_pending_invite commits only when it actually writes an
# invite row — not when this address already has a pending or approved one — and _issue_token
# below only runs with SMTP on. So for an address the admin had already approved (or one whose
# request was still pending), on an instance with no working SMTP, NOTHING along this path
# committed and the flushed user was silently rolled back when the session closed: the
# would-be user got the uniform "ok" and no account, and re-registering did the same again.
# Commit unconditionally; it is a no-op when upsert already committed the same transaction.
db.commit()
if email_ok:
raw = _issue_token(db, user, "verify", VERIFY_TTL)
# Fragment (#), not a query token: keeps the token out of proxy/access logs + Referer (SB3).
email_mod.send_verify_email(email, f"{_app_base()}/#verify={raw}")
admins = admin_notify_emails(db)
if admins:
email_mod.send_admin_new_request(admins, email)
send_verification_link(db, user)
# Fired unconditionally, unlike the other two callers which gate on a fresh invite row: for
# an address the admin already approved, nothing lands in the Access-requests queue, so this
# is the only place the account's existence is announced at all. Delivery is a separate
# matter — with no working SMTP `_send` drops it, which is why the pre-approved registrant
# still needs the admin to notice them (roadmap UX-25).
notify_admins_of_request(db, None, email)
def issue_verification_link(db: Session, user: User, ttl: timedelta | None = None) -> str:
"""A fresh confirmation link for `user`, returned rather than sent — for a mail that carries the
link inside a larger message (the branching approval/invitation mails, E9.5 S6).
Same token machinery and the same fragment (SB3) as the standalone confirmation mail, so the two
kinds of link cannot drift apart in shape or in how they are consumed."""
raw = _issue_token(db, user, "verify", ttl or VERIFY_TTL)
return f"{_app_base()}/#verify={raw}"
def send_verification_link(db: Session, user: User) -> bool:
"""Issue a fresh verification token for `user` and mail them the link.
Always to the address ON THE ACCOUNT, never a spelling typed into some form: the account's
address is the one its owner chose to be reachable at, and a `+tag` there is usually deliberate.
The token rides the URL FRAGMENT (#) so it can never reach a proxy/access log or a Referer (SB3).
One definition, three callers — registration, the self-service resend, and the admin-side resend
(E9.5 S5). The link's shape and TTL must not drift between them; a verification mail that differs
by who triggered it is a bug waiting to be discovered by exactly one user.
Returns whether the mail actually went out, so a caller with a human waiting on the answer (the
admin-side resend) can say so instead of reporting success for a message that was dropped.
"""
return email_mod.send_verify_email(
user.email,
mailtext.lang_for_user(user),
issue_verification_link(db, user),
hours=int(VERIFY_TTL.total_seconds() // 3600),
)
@router.post("/verify")
def verify_email_confirm(body: TokenIn, db: Session = Depends(get_db)) -> dict:
"""Confirm an email-verification token the SPA read from the URL fragment (SB3). Uniform 400
on any invalid/expired/used token."""
on any invalid/expired/used token.
`password_reset_required` tells the SPA to send the user straight to the reset flow: SEC-10 had
dropped the password this account registered with, so sign-in would answer with the deliberately
uniform "Invalid email or password" — true, but unreadable to the person who just proved they own
the mailbox. Saying so leaks nothing: holding a valid single-use verify token for this account
already proves that much. Since E9.5 S1 the common source of that state is gone (admin approval no
longer revokes), leaving accounts stripped before that change and the Google-adoption path, so this
flag is rare now — but it must stay, because those users have no other way back in."""
user = _consume_token(db, body.token, "verify")
if user is None:
raise HTTPException(
status_code=400, detail="This verification link is invalid or has expired."
)
raise HTTPException(status_code=400, detail={"code": "verify", "reason": "invalid"})
user.email_verified = True
needs_password = user.password_hash is None and user.password_revoked_at is not None
email = user.email
db.commit()
return {"ok": True}
# Naming the address lets the SPA say WHICH mailbox it just confirmed — the same reasoning as
# password_reset_required above: holding a valid single-use verify token for this account already
# proves that much, so echoing it back leaks nothing. It matters in a browser that holds a session
# for a different account, where an unqualified "email confirmed" would be actively misleading.
return {"ok": True, "password_reset_required": needs_password, "email": email}
@router.post("/verify/resend")
@@ -687,22 +1020,36 @@ def _resend_verify_if_eligible(email: str) -> None:
if not email_mod.email_enabled():
return
with SessionLocal() as db:
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
user = find_user_by_email(db, email)
if user is not None and not user.email_verified and not user.is_demo:
raw = _issue_token(db, user, "verify", VERIFY_TTL)
email_mod.send_verify_email(email, f"{_app_base()}/#verify={raw}")
send_verification_link(db, user)
@router.get("/verify")
def verify_email(token: str, db: Session = Depends(get_db)):
"""Legacy query-token verification link (pre-SB3 emails still in flight). New emails use the
fragment + POST flow above; this bounces to a friendly status for older links until they expire."""
user = _consume_token(db, token, "verify")
if user is None:
return RedirectResponse(url="/?verify=invalid")
user.email_verified = True
db.commit()
return RedirectResponse(url="/?verify=ok")
def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
"""Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
shape so the SPA can say it in the user's own language (the old bare English strings were
unreadable under a bilingual UI, and unusable as anything but a one-line error).
The account-shaped reasons carry the address ON THE ACCOUNT plus both gate flags. That is what
turns a flat 403 into an honest screen — it names the mailbox and says which of the two gates is
still shut. It is not an enumeration leak: this branch is only reachable once the password has
been proven, so the caller already knows the account exists. The two reasons that answer BEFORE
that proof (`invalid`, `rate_limited`) carry nothing at all, so they stay byte-identical for a
real account and an unknown one.
Since E9.5 S1 the two gates are order-independent, so `approved` really can be true while
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
rather than assuming a sequence.
"""
detail: dict = {"code": "login", "reason": reason}
if user is not None:
detail |= {
"email": user.email,
"email_verified": user.email_verified,
"approved": user.is_active,
}
detail |= {k: v for k, v in extra.items() if v is not None}
return HTTPException(status_code=status, detail=detail)
@router.post("/password-login")
@@ -716,29 +1063,23 @@ def password_login(
Once the password is proven correct, the owner gets a specific reason if their account is
still pending or suspended — that's not an enumeration leak (they already hold the password)."""
if not _login_limiter.allow(_client_ip(request)):
raise HTTPException(status_code=429, detail="Too many attempts. Try again shortly.")
raise _login_refused(429, "rate_limited")
email = (body.email or "").strip().lower()
password = body.password or ""
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
user = find_user_by_email(db, email)
# Always run the KDF (against a decoy hash for an unknown / passwordless / demo account) so the
# response time is the same whether or not the email exists — no timing-based enumeration.
candidate_hash = user.password_hash if (user and not user.is_demo) else None
password_ok = verify_password(password, candidate_hash or _DECOY_PASSWORD_HASH)
if user is None or user.is_demo or not password_ok:
raise HTTPException(status_code=401, detail="Invalid email or password.")
raise _login_refused(401, "invalid")
if user.is_suspended:
_notify_suspended(background, user.email)
op = operator_contact()
detail = "Your account has been suspended."
if op:
detail += f" If you have questions, contact the operator at {op}."
raise HTTPException(status_code=403, detail=detail)
_notify_suspended(background, user)
raise _login_refused(403, "suspended", user, operator=operator_contact() or None)
if not user.email_verified:
raise HTTPException(
status_code=403, detail="Please verify your email first (check your inbox)."
)
raise _login_refused(403, "email_unverified", user)
if not user.is_active:
raise HTTPException(status_code=403, detail="Your account is awaiting admin approval.")
raise _login_refused(403, "pending_approval", user)
_establish_session(request, user)
log.info("Password login: %s (id=%s, role=%s)", email, user.id, user.role)
return {"ok": True}
@@ -767,12 +1108,43 @@ def _send_reset_if_eligible(email: str) -> None:
"""Background worker for password_reset_request: issue a reset token + email the link, but ONLY
for a real (non-demo) password account. Runs after the response with its own DB session, so the
request's timing never reveals whether the account exists. Token rides the URL FRAGMENT (#) so it
can't leak into proxy/access logs or a Referer (SB3)."""
can't leak into proxy/access logs or a Referer (SB3).
"Password account" is `password_hash OR password_revoked_at` — an account SEC-10 stripped had a
password and must be able to get a new one, or it is locked out permanently (no password to sign in
with, and formerly no reset link either). A Google-only account that never had a password stays
ineligible on purpose: its owner adds one from Settings (/auth/set-password) while signed in, which
proves identity through the session rather than by mail."""
with SessionLocal() as db:
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
if user is not None and user.password_hash and not user.is_demo:
user = find_user_by_email(db, email)
if user is None or user.is_demo:
return
if user.password_hash or user.password_revoked_at:
raw = _issue_token(db, user, "reset", RESET_TTL)
email_mod.send_password_reset(email, f"{_app_base()}/#reset={raw}")
# The account's own address, as above — never the spelling that was typed into the form.
email_mod.send_password_reset(
user.email, mailtext.lang_for_user(user), f"{_app_base()}/#reset={raw}"
)
@router.post("/password-reset/check")
def password_reset_check(body: TokenIn, request: Request, db: Session = Depends(get_db)) -> dict:
"""Is the reset link the SPA just opened still good? Answers without spending the token.
Why this exists: the card used to render "set a new password" purely because a `#reset=` fragment
was present, never asking whether it could still succeed. Reported on 2026-09-05 after deleting an
account and then opening its reset link — the user types a new password into a form that answers
400. Opening a dead reset link is not an edge case: the token lives one hour, mail is slow or lands
in spam, and whoever got two mails usually clicks the older one.
Not an enumeration oracle: probing requires already holding a 43-character random, and the answer
is exactly what submitting it would reveal. Throttled per IP anyway — and a throttled caller is
told `true`, never `false`: this check may only ever save a user a wasted form, so when it cannot
answer it must degrade to the old behaviour instead of declaring a live link dead.
"""
if not _reset_check_limiter.allow(_client_ip(request)):
return {"ok": True}
return {"ok": any(_peek_token(db, body.token, k) is not None for k in _PASSWORD_TOKEN_KINDS)}
@router.post("/password-reset/confirm")
@@ -783,17 +1155,25 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
if len(new_password) < PASSWORD_MIN_LEN:
raise HTTPException(
status_code=400,
detail=f"Password must be at least {PASSWORD_MIN_LEN} characters.",
detail={"code": "reset", "reason": "weak_password", "min": PASSWORD_MIN_LEN},
)
user = _consume_token(db, token, "reset")
if user is None:
raise HTTPException(status_code=400, detail="This reset link is invalid or has expired.")
spent = _consume_password_token(db, token)
if spent is None:
raise HTTPException(status_code=400, detail={"code": "reset", "reason": "invalid"})
user, kind = spent
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # the account has a password again — SEC-10's trace is spent
# A valid reset link proves control of the mailbox, so completing a reset also verifies the
# email — this rescues an account that registered but never clicked the verification link (it
# can then sign in). The separate admin-approval gate (is_active) is deliberately NOT touched
# here: a reset proves email ownership, not that an admin approved the account.
user.email_verified = True
#
# An admin-issued `setpw` link proves NOTHING about the mailbox — the admin hands it over by
# whatever means they choose — so it must not satisfy the email gate. Claiming otherwise would let
# an admin confirm an address nobody ever proved, which is the same false attestation SEC-15 is
# about. The person still confirms their address the normal way.
if kind == "reset":
user.email_verified = True
bump_session_epoch(
user
) # SA4: a reset (often a compromise response) kills all existing sessions.
@@ -801,7 +1181,7 @@ def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db))
for row in db.execute(
select(AuthToken).where(
AuthToken.user_id == user.id,
AuthToken.kind == "reset",
AuthToken.kind.in_(_PASSWORD_TOKEN_KINDS),
AuthToken.used_at.is_(None),
)
).scalars():
@@ -828,12 +1208,33 @@ def bump_session_epoch(user: User) -> None:
def revoke_preset_password_if_unverified(user: User) -> None:
"""SEC-10 (account pre-emption): a password set on an account whose email was NEVER verified is
untrusted — whoever registered the address (to pre-empt the real owner) could have chosen it. When
ownership is finally attested (Google adoption, or admin approval), drop that password and bump the
epoch to revoke any session it minted; the owner re-sets one via password-reset. No-op for an
already-verified account, so a legit no-SMTP registration is unaffected. The single source for this
rule — both the Google-callback and admin-activation paths call it so they can't drift."""
the real owner attests ownership through Google, drop that password and bump the epoch to revoke any
session it minted; the owner re-sets one via password-reset. No-op for an already-verified account,
so a legit no-SMTP registration is unaffected.
The Google-adoption callback is the ONLY caller (E9.5 S1). Admin approval used to call this too, and
dropping it was right for the reason S1 gave — the revoke destroyed the password a legitimate
registrant had just chosen, and since the admin normally decides before the user reads their mail,
that made the "your password was cleared" detour the DEFAULT path rather than an edge case.
S1 also claimed the revoke was REDUNDANT there, because `password_login` refuses an unverified
account before it reads `is_active`. That part was wrong, and the epic's own review caught it: the
gate holds only while the address is unverified, and S6's approval mail exists to get the real
owner to end that state — at which point a pre-emptor's password works. Nothing replaced the
revoke, and that is deliberate. The review tried "the newest registration owns an unverified
account" and reverted it: the rule is symmetric, so an attacker can simply register last, and it
would have handed anonymous callers a write onto a real account's credential. What the owner
actually has is password-reset, which proves the mailbox, rewrites the hash and bumps the epoch.
The residual is logged as SEC-16; re-adding a caller here would put "your password was cleared"
back in front of every legitimate registrant, which is the whole reason S1 removed it.
`password_revoked_at` is what makes "the owner re-sets one via password-reset" actually true. The
revoke leaves `password_hash IS NULL`, which is also what a Google-only account looks like — and
_send_reset_if_eligible keyed off exactly that, so a revoked account silently got no reset link and
was locked out for good (it could not sign in either). The stamp separates the two cases."""
if not user.email_verified:
user.password_hash = None
user.password_revoked_at = datetime.now(timezone.utc)
bump_session_epoch(user)
@@ -845,7 +1246,7 @@ def logout_others(
are rejected, then re-stamp THIS cookie so the caller stays signed in. For "I left myself logged
in somewhere" / a suspected session compromise. The demo account (shared) is excluded."""
if user.is_demo:
raise HTTPException(status_code=403, detail="Not available in the demo account.")
raise HTTPException(status_code=403, detail={"code": "demo"})
# Commit the bump BEFORE re-stamping this cookie, so a failed commit can't strand the current
# session at a newer epoch than the DB (see set_password).
bump_session_epoch(user)
@@ -892,7 +1293,7 @@ def set_password(
account) needs no current password — the session already proves identity. Changing an
existing one requires the current password, so a hijacked session can't silently rotate it."""
if user.is_demo:
raise HTTPException(status_code=403, detail="Not available in the demo account.")
raise HTTPException(status_code=403, detail={"code": "demo"})
new_password = body.password or ""
if len(new_password) < PASSWORD_MIN_LEN:
raise HTTPException(
@@ -903,6 +1304,7 @@ def set_password(
if not verify_password(body.current_password or "", user.password_hash):
raise HTTPException(status_code=403, detail="Current password is incorrect.")
user.password_hash = hash_password(new_password)
user.password_revoked_at = None # same as the reset path: a password exists again
# SA4: a password change logs out every OTHER session; re-stamp this cookie at the new epoch so
# the person making the change stays signed in here. Commit BEFORE re-stamping so a failed commit
# can't leave the cookie at a newer epoch than the DB (which would wrongly 401 THIS session).
+13 -17
View File
@@ -26,7 +26,7 @@ from app.models import (
User,
Video,
)
from app.routes._common import owned_or_404
from app.routes._common import get_or_404, not_found, owned_or_404
from app.sync.explore import explore_ingest_page
from app.sync.runner import backfill_channel_now, run_recent_backfill
from app.sync.subscriptions import apply_channel_details
@@ -229,6 +229,7 @@ def _channel_detail_dict(
count_live: int,
deep_requested: bool,
priority: int,
hidden: bool,
tag_ids: list[int],
) -> dict:
return {
@@ -262,6 +263,9 @@ def _channel_detail_dict(
# This user's per-subscription feed priority (0 when not subscribed) so the channel page can
# offer the same priority stepper as the manager. Editable only while subscribed.
"priority": priority,
# This user's per-subscription "hide from feed" flag (False when not subscribed), so the
# channel page can offer the same Unhide toggle the manager has. Editable only while subscribed.
"hidden": hidden,
"backfill_done": channel.backfill_done,
"from_explore": channel.from_explore,
"details_synced": channel.details_synced_at is not None,
@@ -283,9 +287,7 @@ def channel_detail(
needed — a stub, or a channel last enriched before those fields existed (published_at is a
reliable sentinel: every real channel has a creation date). The enrich uses the API key
(1 unit, cached after), so it's skipped for the quota-less demo account."""
channel = db.get(Channel, channel_id)
if channel is None:
raise HTTPException(status_code=404, detail="Unknown channel")
channel = get_or_404(db, Channel, channel_id, what="channel")
if channel.published_at is None and not user.is_demo:
try:
with (
@@ -344,6 +346,7 @@ def channel_detail(
count_live=count_live,
deep_requested=bool(sub.deep_requested) if sub else False,
priority=int(sub.priority) if sub else 0,
hidden=bool(sub.hidden) if sub else False,
tag_ids=tag_ids,
)
@@ -361,9 +364,7 @@ def explore_channel(
it, then ingests one page of its uploads (newest-first) marked via_explore + enriched.
Returns the next page token so the page can "load more". Guarded against the shared
backfill quota reserve; blocked for the demo account (it spends shared quota)."""
channel = db.get(Channel, channel_id)
if channel is None:
raise HTTPException(status_code=404, detail="Unknown channel")
channel = get_or_404(db, Channel, channel_id, what="channel")
if _is_blocked(db, user, channel_id):
raise HTTPException(status_code=403, detail="You've blocked this channel.")
if quota.backfill_reserve_reached():
@@ -421,7 +422,7 @@ def _user_subscription(db: Session, user: User, channel_id: str) -> Subscription
)
).scalar_one_or_none()
if sub is None:
raise HTTPException(status_code=404, detail="Not subscribed to this channel")
raise not_found("subscription")
return sub
@@ -507,9 +508,7 @@ def reset_backfill(
state (a "reset" trigger). Clears the channel's backfill markers, opts it back into deep
backfill, and re-runs its recent pull immediately; the deep scheduler re-pages the full
back-catalog on its next run. Idempotent — videos upsert by id, so nothing duplicates."""
channel = db.get(Channel, channel_id)
if channel is None:
raise HTTPException(status_code=404, detail="Unknown channel")
channel = get_or_404(db, Channel, channel_id, what="channel")
channel.backfill_done = False
channel.backfill_cursor = None
channel.recent_synced_at = None
@@ -539,9 +538,7 @@ def subscribe(
local subscription with YouTube's returned resource id and enriches the channel's metadata
if it's still a stub, so the new row renders properly in the manager. It does NOT pull the
channel's videos — the scheduler picks those up on its next run, like any subscribed channel."""
channel = db.get(Channel, channel_id)
if channel is None:
raise HTTPException(status_code=404, detail="Unknown channel")
channel = get_or_404(db, Channel, channel_id, what="channel")
existing = db.execute(
select(Subscription).where(
Subscription.user_id == user.id, Subscription.channel_id == channel_id
@@ -658,8 +655,7 @@ def block_channel(
"""Block a channel for this user: its videos won't appear in their live search (and won't be
ingested for them) and are hidden from their feed / explore. Idempotent. The now-hidden
un-kept search/explore videos are reclaimed by the discovery-cleanup job in due course."""
if db.get(Channel, channel_id) is None:
raise HTTPException(status_code=404, detail="Unknown channel")
get_or_404(db, Channel, channel_id, what="channel")
if not _is_blocked(db, user, channel_id):
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
# Stop any active exploration of it so it can be cleaned up.
@@ -701,7 +697,7 @@ def attach_tag(
_user_subscription(db, user, channel_id)
# Only the user's own tags may be attached as personal links (owned_or_404 → 404 if the tag
# is missing OR belongs to someone else, without leaking which).
owned_or_404(db, Tag, body.tag_id, user, detail="Unknown tag")
owned_or_404(db, Tag, body.tag_id, user, what="tag")
exists = db.execute(
select(ChannelTag).where(
ChannelTag.channel_id == channel_id,
+2 -1
View File
@@ -10,6 +10,7 @@ from sqlalchemy.orm import Session
from app import audit
from app import email as email_mod
from app import mailtext
from app import sysconfig
from app.audit import AuditAction
from app.auth import admin_user
@@ -115,7 +116,7 @@ def send_test_email(
lets the admin verify the settings actually work."""
if not email_mod.email_enabled():
raise HTTPException(status_code=400, detail="SMTP is not configured.")
sent = email_mod.send_test(user.email)
sent = email_mod.send_test(user.email, mailtext.lang_for_user(user))
if not sent:
raise HTTPException(status_code=422, detail="SMTP send failed — check the settings.")
return {"sent": True, "to": user.email}
+197
View File
@@ -0,0 +1,197 @@
"""E24 · the Home surface's HTTP shell.
Thin by construction: every decision — what counts as in progress, what fits a time budget, which
tier the hero comes from — lives in `app/discover/`, so the scheduler, a future CLI or a test can ask
the same questions without a request. This file validates, gates and serializes.
The consolidated `GET /api/discover` mirrors the Plex Discover route's one-payload shape (E5 S4): a
landing page that fires four requests to answer one question would spend its whole budget on the
thing it exists to remove.
"""
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.auth import current_user, require_human
from app.db import get_db
from app.discover import hero as hero_mod
from app.discover import pins, rails
from app.discover.budget import Budget, DEFAULT_BUDGET
from app.models import ComfortPin, User
from app.routes._common import owned_or_404
router = APIRouter(prefix="/api/discover", tags=["discover"])
class PinIn(BaseModel):
# From `pins.PIN_KINDS`, not a second hand-written list: the vocabulary, the schema columns and
# this request contract have to move together, and re-typing them here is how they would not.
kind: Literal[*pins.PIN_KINDS]
# The id the rest of the API speaks for that kind: a video id, a channel id, a Plex rating_key,
# or a playlist id. `pins.resolve_ref` does the translation to the row's internal key.
ref: str
class ReorderIn(BaseModel):
ids: list[int]
@router.get("")
def get_discover(
budget: Budget = DEFAULT_BUDGET,
seed: int = 0,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Everything the Home page renders, in one pass.
`resume` and `shelf` are built once and handed to the hero, so the two rails the hero draws from
are not queried twice for a single page load."""
resume = rails.continue_watching(db, user)
shelf = pins.shelf(db, user)
return {
"budget": budget,
"hero": hero_mod.pick(db, user, budget, seed, resume=resume, shelf=shelf),
"continue_watching": resume,
"comfort": shelf,
}
# How many already-shown cards a reroll may carry, matched to the client's own rolling window
# (`SEEN_WINDOW` in `Home.tsx`). A cheap sanity clamp, and deliberately no more than that: `exclude`
# never reaches SQL — `hero.pick` uses it only as set membership over an already-bounded candidate
# pool (the resume rail's `RAIL_LIMIT` rows, the shelf, `SURPRISE_DRAW` surprise draws) — so there is
# no scan for a crafted URL to unbound, and the set is built in full before this slice anyway. The
# comment that claimed otherwise was corrected in round 2; the request line's own length is what
# actually bounds a hostile `exclude`.
MAX_EXCLUDE = 32
@router.get("/pick")
def get_pick(
budget: Budget = DEFAULT_BUDGET,
seed: int = 0,
exclude: str = "",
start: int = 0,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The "Mást kérek" reroll: a hero on its own, for a new seed and/or a new budget.
One seed drives BOTH sources (see `discover/queries.py`) — otherwise a reroll would re-draw the
Plex half while the YouTube half stood still, and the button would look broken half the time.
`exclude` is a comma-separated list of `<source>:<id>` keys the user has already been shown. It
is what makes the button honest: a seed alone only moves the surprise tier, so a user with
something half-finished got the same card back every press.
`start` is the press count: it rotates which tier answers first, so consecutive presses offer a
different KIND of answer instead of walking down the resume rail. Taken modulo the tier count in
`hero.pick`, so any integer is valid and no value can 422 a button."""
skip = {p for p in exclude.split(",") if p}
return {
"budget": budget,
"hero": hero_mod.pick(
db, user, budget, seed, exclude=set(list(skip)[:MAX_EXCLUDE]), start=start
),
}
@router.get("/pins")
def list_pins(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The shelf on its own — for the shelf editor, which needs it without the rest of the page."""
return {"items": pins.shelf(db, user)}
@router.post("/pins")
def add_pin(
body: PinIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Pin something to the comfort shelf.
`require_human`, like every other personal-state write: the shared demo account must not be able
to curate a communal shelf. Re-pinning something already on the shelf is a no-op that returns the
existing pin rather than a 409 — the gesture is "make sure this is on my shelf", and the unique
constraint is the mechanism, not the message."""
resolved = pins.resolve_ref(db, user, body.kind, body.ref)
if resolved is None:
raise HTTPException(status_code=404, detail="Unknown item")
column, value = resolved
existing = db.execute(
select(ComfortPin).where(
ComfortPin.user_id == user.id,
getattr(ComfortPin, column) == value,
)
).scalar_one_or_none()
if existing is not None:
return {"ok": True, "id": existing.id, "created": False}
pin = ComfortPin(user_id=user.id, position=pins.next_position(db, user), **{column: value})
db.add(pin)
try:
db.commit()
except IntegrityError:
# A concurrent pin of the same thing from a second tab. The unique constraint did its job;
# the user's intent ("be on my shelf") is satisfied either way, so report the winner rather
# than a 500 the user can do nothing with.
db.rollback()
winner = db.execute(
select(ComfortPin).where(
ComfortPin.user_id == user.id,
getattr(ComfortPin, column) == value,
)
).scalar_one_or_none()
if winner is None:
# Not the unique constraint, then: the only other way in is the target row being pruned
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
# item), which trips the FK. Semantically that is the same "no such thing" the pre-check
# returns above — re-raising handed the user a 500 they could do nothing with.
raise HTTPException(status_code=404, detail="Unknown item")
return {"ok": True, "id": winner.id, "created": False}
return {"ok": True, "id": pin.id, "created": True}
@router.delete("/pins/{pin_id}")
def remove_pin(
pin_id: int,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
pin = owned_or_404(db, ComfortPin, pin_id, user, what="pin")
db.delete(pin)
db.commit()
return {"deleted": pin_id}
@router.post("/pins/reorder")
def reorder_pins(
body: ReorderIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Rewrite the shelf order from a full list of pin ids.
Whole-list rather than a move-one delta: the shelf is short, and a full rewrite cannot leave two
tiles claiming the same slot the way an interrupted sequence of single moves can. Ids that are not
this user's are ignored rather than 404'd — a stale tab replaying an order that includes a pin
deleted meanwhile should reorder what still exists, not fail wholesale."""
owned = {p.id for p in pins.list_pins(db, user)}
for position, pin_id in enumerate(i for i in body.ids if i in owned):
db.execute(
update(ComfortPin)
.where(ComfortPin.id == pin_id, ComfortPin.user_id == user.id)
.values(position=position)
)
db.commit()
return {"ok": True}
+182 -12
View File
@@ -6,23 +6,29 @@ worker is a separate process); this module never downloads — it enqueues, mana
serves finished files (range-aware, with the user's custom display name), and shares them.
"""
import os
import re
import tempfile
from datetime import datetime, timedelta, timezone
from pathlib import Path
from urllib.parse import parse_qs, urlparse
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import FileResponse
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app import sysconfig
from app import audit, sysconfig
from app.auth import admin_user, require_human
from app.config import settings
from app.db import get_db
from app.downloads import credentials as dlcreds
from app.downloads import edit as editmod
from app.downloads import links as linksmod
from app.downloads import quota, service, storage
from app.downloads import upload as uploadmod
from app.downloads import urlguard
from app.models import (
DownloadJob,
DownloadLink,
@@ -32,6 +38,7 @@ from app.models import (
MediaAsset,
User,
)
from app.routes._common import get_or_404, not_found
from app.security import hash_password
from app.userscope import is_messageable_user, messageable_clauses
from app.utils import iso
@@ -60,7 +67,18 @@ def resolve_source(raw: str) -> tuple[str, str]:
if _VIDEO_ID.match(raw):
return "youtube", raw
if raw.startswith(("http://", "https://")):
u = urlparse(raw)
# `urlparse` raises ValueError on a malformed authority ("http://[::1/x"), and this runs
# BEFORE the urlguard check that would otherwise turn it into a structured refusal — so
# unguarded, that typo escapes as a 500. Answered in urlguard's own shape, because it is the
# same typo class: ":99999" is caught there and localized, and one of the two arriving as a
# bare English string would be exactly the split the canonical-patterns table forbids.
try:
u = urlparse(raw)
except ValueError as exc:
raise HTTPException(
status_code=422,
detail={"code": "url_blocked", "reason": urlguard.REASON_MALFORMED},
) from exc
host = u.netloc.lower()
if "youtube.com" in host:
if u.path.startswith(("/shorts/", "/embed/", "/live/")):
@@ -122,6 +140,9 @@ def _serialize(job: DownloadJob, asset: MediaAsset | None) -> dict:
"width": asset.width,
"height": asset.height,
"container": asset.container,
# "video" | "audio" | "file" — the library gates play/edit on this (media only); a
# generic uploaded file is download-only. Null while still in flight (set at finalize).
"media_kind": asset.media_kind,
"uploader_url": asset.uploader_url,
"thumbnail_url": asset.thumbnail_url,
"expires_at": iso(asset.expires_at),
@@ -247,6 +268,65 @@ def delete_profile(
return {"deleted": profile_id}
# --- per-user download auth (cookies) ------------------------------------------------------
# Write-only surface: the stored blob is NEVER returned — only the supported hosts + whether each
# is configured + when. See downloads.credentials for the host-scope + cache-privacy guarantees.
@router.get("/credentials")
def list_credentials(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
return {"hosts": dlcreds.status_list(db, user.id)}
class SetCredentialIn(BaseModel):
cookies: str
@router.put("/credentials/{host_key}")
def set_credential(
host_key: str,
body: SetCredentialIn,
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Store (encrypted) this user's cookies.txt for one supported host. Write-only: the value is
never echoed back, logged, or audited — only the fact that it changed."""
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)})
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_CREDENTIAL_SET,
target_type="download_credential",
target_id=host_key,
)
db.commit()
return {"ok": True, "hosts": dlcreds.status_list(db, user.id)}
@router.delete("/credentials/{host_key}")
def delete_credential(
host_key: str, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
if not dlcreds.is_supported_host(host_key):
raise HTTPException(status_code=404, detail="Unknown host")
if dlcreds.delete_credential(db, user.id, host_key):
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_CREDENTIAL_DELETE,
target_type="download_credential",
target_id=host_key,
)
db.commit()
return {"ok": True, "hosts": dlcreds.status_list(db, user.id)}
def _resolve_spec(
db: Session, user: User, profile_id: int | None, spec: dict | None
) -> tuple[dict, int | None]:
@@ -285,6 +365,17 @@ def enqueue_download(
body: EnqueueDownloadIn, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
source_kind, source_ref = resolve_source(body.source)
# SEC-12: a non-YouTube source is a raw URL the worker will fetch server-side, so refuse the
# ones pointing back into our own network BEFORE a job exists. The worker re-checks (DNS can
# move between here and the download); see downloads/urlguard.py for what that does and does
# not cover.
if source_kind == "url":
try:
urlguard.check_url(source_ref)
except urlguard.BlockedUrl as e:
raise HTTPException(
status_code=422, detail={"code": "url_blocked", "reason": e.reason}
) from e
spec, profile_id = _resolve_spec(db, user, body.profile_id, body.spec)
display_name = (body.display_name or "").strip() or None
try:
@@ -294,6 +385,61 @@ def enqueue_download(
return _serialize_job(db, job)
# Read the upload in 1 MiB chunks so a large file never sits fully in memory, and the size cap is
# enforced on the ACTUAL bytes read (a client-set Content-Length can't be trusted to gate it).
_UPLOAD_CHUNK = 1024 * 1024
@router.post("/upload")
def upload_download(
file: UploadFile = File(...),
display_name: str | None = Form(None),
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> dict:
"""Ingest a local file into the caller's library (E7 S5). Any file type is accepted; media
(video/audio) gets the full function set, a generic file is download-only. The file is streamed
to a temp path under the download volume (enforcing the per-file cap as it goes), then handed to
the non-yt-dlp ingest which probes + places it and returns an already-`done` job.
A sync (threadpool) handler on purpose: the ingest does blocking work — an ffprobe of up to 30s
plus sync DB writes — so it must NOT run on the event loop. `file.file` is the sync file object
Starlette spools the multipart body into."""
cap = sysconfig.get_int(db, "download_upload_max_bytes")
staging = Path(settings.download_root) / ".staging" / "uploads"
staging.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(staging))
tmp_path = Path(tmp)
try:
size = 0
with os.fdopen(fd, "wb") as out:
while chunk := file.file.read(_UPLOAD_CHUNK):
size += len(chunk)
if cap and size > cap:
raise HTTPException(
status_code=422, detail={"code": "upload_too_large", "limit": cap}
)
out.write(chunk)
if size == 0:
raise HTTPException(status_code=400, detail={"code": "upload_empty"})
try:
job = uploadmod.ingest_upload(
db,
user.id,
filename=file.filename or "upload",
src_path=tmp_path,
size=size,
display_name=(display_name or "").strip() or None,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
return _serialize_job(db, job)
finally:
# Ingest MOVES the temp file into place on success (unlink is then a no-op); on any early exit
# — cap hit, empty, quota, probe/move error — this removes the abandoned staging file.
tmp_path.unlink(missing_ok=True)
class EnqueueEditIn(BaseModel):
source_job_id: int = 0
edit_spec: dict
@@ -410,7 +556,12 @@ def _clean_url(raw) -> str | None:
return None
if not url.startswith(("http://", "https://")):
return None
parsed = urlparse(url)
# The third site in this module handing user input to `urlparse`, which raises on a malformed
# authority. This one already has a "not a plausible URL" answer, so the typo belongs in it.
try:
parsed = urlparse(url)
except ValueError:
return None
if not parsed.netloc:
return None
return url[:2048]
@@ -484,8 +635,10 @@ def redownload_download(
never deletes a file another job/user still shares: the old asset is only removed if THIS job
was its last holder."""
job = _own_job(db, user, job_id)
if job.job_kind == "edit":
raise HTTPException(status_code=409, detail="Editor clips can't be re-downloaded.")
if job.job_kind in ("edit", "upload"):
# Neither has a re-fetchable source: an editor clip is derived locally, an upload IS the
# source. Re-running yt-dlp against source_kind='upload'/'edit' would just fail.
raise HTTPException(status_code=409, detail="This item can't be re-downloaded.")
if job.status in ("queued", "running", "paused"):
raise HTTPException(status_code=409, detail="This download is still in progress.")
service.redownload(db, job)
@@ -550,8 +703,21 @@ def download_file(
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
# Starlette's FileResponse honours the Range header (206 partial content) for seek/resume.
return FileResponse(path, filename=filename)
# The served Content-Type comes from the ffprobe-verified media_kind + a container allow-list
# (storage.served_media_type), NEVER guessed from the file's own extension. A generic 'file' (and
# any media whose container we don't recognise — e.g. a media/HTML polyglot named "evil.html")
# returns None → a neutral force-download attachment, so uploaded HTML/SVG can't execute inline on
# this origin. Recognised media is served with its real type so the in-app player streams it. Both
# are range-aware (FileResponse → 206).
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
return FileResponse(path, filename=filename, media_type=media_type)
# --- editor filmstrip (scrub track) --------------------------------------------------------
@@ -636,7 +802,7 @@ def share_download(
# Uniform "no user" for a missing OR non-messageable (demo/suspended/deactivated) target, so
# sharing can't be used to probe which addresses exist or to reach a suspended account.
if not is_messageable_user(recipient):
raise HTTPException(status_code=404, detail="No user with that email.")
raise not_found("recipient")
if recipient.id == user.id:
raise HTTPException(status_code=400, detail="That's already your download.")
exists = db.execute(
@@ -763,7 +929,12 @@ def list_links(
class CreateLinkIn(BaseModel):
password: str = ""
allow_download: bool = False
# D1 (E7 S3): a newly shared download is downloadable by default — the common intent is "let the
# recipient have the file", not stream-only. Forward-only: this only sets the default for links
# minted from now on; existing links keep whatever `allow_download` they were created with (the
# column default stays False, so no back-catalogue link silently flips). The owner can still
# uncheck it in the Share dialog, and toggle any link stream-only afterwards.
allow_download: bool = True
expires_days: int | None = None
@@ -921,8 +1092,7 @@ def admin_set_quota(
admin: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
if db.get(User, user_id) is None:
raise HTTPException(status_code=404, detail="Unknown user")
get_or_404(db, User, user_id, what="user")
cur = quota.resolve(db, user_id)
row = db.get(DownloadQuota, user_id)
if row is None:
+12 -7
View File
@@ -1,7 +1,7 @@
from datetime import date, datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import and_, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
@@ -22,12 +22,14 @@ from app.feedquery import (
sort_keys,
)
from app.models import (
POSITION_MAX_S,
ChannelTag,
Tag,
User,
Video,
VideoState,
)
from app.routes._common import get_or_404
from app.sync.videos import parse_iso8601_duration
from app.utils import iso
from app.youtube.client import YouTubeClient, YouTubeError
@@ -234,8 +236,7 @@ def set_video_state(
status = body.status
if status not in VALID_STATES:
raise HTTPException(status_code=400, detail=f"status must be one of {VALID_STATES}")
if db.get(Video, video_id) is None:
raise HTTPException(status_code=404, detail="Unknown video")
get_or_404(db, Video, video_id, what="video")
now = datetime.now(timezone.utc)
@@ -303,11 +304,16 @@ def clear_video_state(
class VideoProgressIn(BaseModel):
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
# one into a 500.
model_config = ConfigDict(allow_inf_nan=False)
# Accept a float and truncate (below) — the player may post a raw currentTime (a float), which
# the old `int(payload.get(...))` silently truncated. A plain `int` field would 422 that on the
# 10s progress hot path, losing the resume position; `float` keeps the old tolerance.
position_seconds: float = 0
duration_seconds: float = 0
# Bounded above by the int4 column (see POSITION_MAX_S): past it the flush was a 500.
position_seconds: float = Field(0, le=POSITION_MAX_S)
duration_seconds: float = Field(0, le=POSITION_MAX_S)
@router.post("/videos/{video_id}/progress")
@@ -326,8 +332,7 @@ def set_video_progress(
if position < 0:
position = 0
if db.get(Video, video_id) is None:
raise HTTPException(status_code=404, detail="Unknown video")
get_or_404(db, Video, video_id, what="video")
# Decide whether this position is worth keeping (else clear it).
near_end = duration > 0 and position > duration - FINISH_MARGIN_SECONDS
+28 -1
View File
@@ -5,6 +5,7 @@ from sqlalchemy.orm import Session
from app.auth import (
count_admins,
count_pending_activations,
current_user,
google_enabled,
has_read_scope,
@@ -13,6 +14,7 @@ from app.auth import (
optional_current_user,
purge_user,
)
from app import email as email_mod
from app import sysconfig
from app.db import get_db
from app.models import Invite, User
@@ -98,11 +100,21 @@ def get_me(
if user is None:
return {"authenticated": False}
pending_invites = 0
pending_activations = 0
# Admin-only, because it drives an operator warning: email is fail-soft, so a half-filled SMTP
# config costs verification links, password resets and access-request notices with nothing
# visible anywhere. Surfacing it here puts the warning on the screens where onboarding happens.
email_configured = True
if user.role == "admin":
pending_invites = (
db.scalar(select(func.count()).select_from(Invite).where(Invite.status == "pending"))
or 0
)
# Counted separately from pending_invites rather than folded into it: the two are different
# facts (a request awaiting a decision vs a decided address whose sign-up never got let in),
# and quietly widening one number would make an existing field mean something new.
pending_activations = count_pending_activations(db)
email_configured = email_mod.email_enabled(db) # this request's session; no second connect
return {
"authenticated": True,
"id": user.id,
@@ -112,6 +124,12 @@ def get_me(
"role": user.role,
"is_demo": user.is_demo,
"has_google": user.google_sub is not None,
# The address Google answers with, when it DIFFERS from the account's own. Same string for
# most people, and then this is None so no surface has to decide whether to show it — the
# comparison lives here, once, instead of in every consumer.
"google_email": (
user.google_email if user.google_email and user.google_email != user.email else None
),
"has_password": user.password_hash is not None,
# Instance-wide: whether Google sign-in / YouTube connect is available at all (the UI hides
# YouTube-access affordances and the onboarding nudge when it isn't configured).
@@ -121,6 +139,10 @@ def get_me(
"can_read": has_read_scope(user),
"can_write": has_write_scope(user),
"pending_invites": pending_invites,
# Sign-ups an admin still has to let in (roadmap UX-25): approved BEFORE they registered, so
# nothing ever reached the Access-requests queue and the badge used to read zero.
"pending_activations": pending_activations,
"email_configured": email_configured,
"preferences": user.preferences or {},
}
@@ -137,8 +159,13 @@ def delete_my_account(
which all cascade on the users row. Quota-audit events are kept but anonymised (FK SET NULL).
The shared demo account can't be deleted, and the last remaining admin can't delete itself
(that would lock everyone out of the admin surfaces)."""
# Structured, like the other three demo refusals: `req` leaves 403 to the caller and
# `notifyActionError` discards a plain-string detail in favour of the caller's translated
# fallback, so a hand-written English sentence here reaches a Hungarian UI as a generic
# "couldn't do that" — or, on this affordance, as nothing at all. `reason` distinguishes it
# from the other sites' bare `{code: "demo"}` so the UI can say what specifically is refused.
if user.is_demo:
raise HTTPException(status_code=403, detail="The shared demo account can't be deleted.")
raise HTTPException(status_code=403, detail={"code": "demo", "reason": "delete_account"})
if user.role == "admin" and count_admins(db) <= 1:
raise HTTPException(
status_code=400,
+1 -1
View File
@@ -46,7 +46,7 @@ WELCOME = {
"This is your private message inbox. Conversations with other members are end-to-end "
"encrypted — only you and the person you're chatting with can read them, not even an "
"admin. To start messaging someone, set up secure messaging with a passphrase first.\n\n"
"Enjoy your feed!"
"Enjoy your Feed!"
),
"hu": (
"Üdv a Siftlode-on! 👋\n\n"
+135 -44
View File
@@ -3,6 +3,8 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import func, select, and_
@@ -14,22 +16,56 @@ from app.db import get_db
from app.feedquery import feed_columns, serialize
from app.models import Channel, Playlist, PlaylistItem, User, Video, VideoState
from app.sync.playlists import (
plan_push,
push_playlist,
merge_playlist,
plan_merge,
recompute_dirty,
repull_playlist,
sync_user_playlists,
)
from app.ratelimit import RateLimiter
from app.routes._common import get_or_404, owned_or_404
from app.youtube.client import YouTubeClient, YouTubeError
router = APIRouter(prefix="/api/playlists", tags=["playlists"])
# SEC-14, same reasoning as the sync module's limiter: a playlist pull lists every playlist the
# caller owns and then every item in each, against the instance-wide YouTube budget. Its own
# limiter rather than a shared one — spending the manual subscription-import allowance must not
# also close the playlist button, and vice versa.
#
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
# legitimate work. Their exposure is tracked as SEC-18 instead.
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
def _check_pull_cooldown(user: User) -> None:
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
if not _playlist_pull_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
def _own_playlist(db: Session, user: User, playlist_id: int) -> Playlist:
pl = db.get(Playlist, playlist_id)
if pl is None or pl.user_id != user.id:
raise HTTPException(status_code=404, detail="Unknown playlist")
return pl
return owned_or_404(db, Playlist, playlist_id, user, what="playlist")
def _refused(status: int, reason: str) -> HTTPException:
"""A playlist action the server refuses, as a reason the UI translates
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
def _touch(pl: Playlist) -> None:
"""Bump the playlist's `updated_at` for an edit that only writes `playlist_items`.
`UpdatedAtMixin`'s `onupdate` fires on an UPDATE of the PLAYLIST row, and add / remove /
reorder / move write the item rows instead — so without this the rail's "recently changed"
sort would only ever see renames. The assignment is what makes the column mean "last time
this list changed", which is what the rail promises the reader."""
pl.updated_at = datetime.now(timezone.utc)
def _next_playlist_position(db: Session, user_id: int) -> int:
@@ -71,6 +107,7 @@ def _add_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) ->
)
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.commit()
return True
@@ -88,6 +125,7 @@ def _remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool)
db.delete(item)
if mark_dirty:
recompute_dirty(db, pl)
_touch(pl)
db.commit()
return True
@@ -109,6 +147,9 @@ def _summary(
"item_count": count,
"total_duration_seconds": total_duration,
"cover_thumbnail": cover,
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
# "recently changed first" sort reads it. See _touch for why it isn't automatic.
"updated_at": pl.updated_at,
}
if has_video is not None:
out["has_video"] = has_video
@@ -219,7 +260,7 @@ def create_playlist(
) -> dict:
name = body.name.strip()
if not name:
raise HTTPException(status_code=400, detail="name is required")
raise _refused(400, "name_required")
pl = Playlist(user_id=user.id, name=name, position=_next_playlist_position(db, user.id))
db.add(pl)
db.commit()
@@ -251,8 +292,7 @@ def add_watch_later(
) -> dict:
"""Bookmark shortcut: add a video to the built-in Watch later playlist."""
video_id = body.video_id
if db.get(Video, video_id) is None:
raise HTTPException(status_code=404, detail="Unknown video")
get_or_404(db, Video, video_id, what="video")
pl = _watch_later(db, user)
_add_item(db, pl, video_id, mark_dirty=False)
return {"saved": True, "playlist_id": pl.id}
@@ -280,10 +320,8 @@ def sync_youtube(
"""Pull the user's YouTube playlists into local mirrors now (read-only direction).
Also runs automatically on the scheduler; this is the manual 'sync now'."""
if not has_read_scope(user):
raise HTTPException(
status_code=403,
detail="Connect YouTube (read access) to sync your playlists.",
)
raise _refused(403, "youtube_read_scope")
_check_pull_cooldown(user)
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return sync_user_playlists(db, user)
@@ -299,17 +337,16 @@ def revert_youtube(
once the in-session undo history is gone."""
pl = _own_playlist(db, user, playlist_id)
if not pl.yt_playlist_id:
raise HTTPException(status_code=400, detail="This playlist isn't linked to YouTube.")
raise _refused(400, "not_on_youtube")
if not has_read_scope(user):
raise HTTPException(
status_code=403, detail="Connect YouTube (read access) to sync your playlists."
)
raise _refused(403, "youtube_read_scope")
_check_pull_cooldown(user)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return repull_playlist(db, user, pl)
except YouTubeError:
db.rollback()
raise HTTPException(status_code=422, detail="Couldn't refresh from YouTube.")
raise _refused(422, "youtube_failed")
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
@@ -317,9 +354,7 @@ def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
Watch later and read-only YouTube mirrors are never pushable."""
pl = _own_playlist(db, user, playlist_id)
if pl.kind == "watch_later":
raise HTTPException(
status_code=400, detail="The Watch later list can't be sent to YouTube."
)
raise _refused(400, "watch_later_not_pushable")
return pl
@@ -329,15 +364,15 @@ def push_plan(
user: User = Depends(require_write_scope),
db: Session = Depends(get_db),
) -> dict:
"""Dry run: what 'Sync to YouTube' would do + the quota estimate, so the UI can show a
confirm. For a linked playlist this reads the live YouTube state (cheap, 1 unit/page).
Write scope is required (require_write_scope) — syncing is a YouTube write."""
"""Dry run: what a two-way 'Sync to YouTube' merge would do (what each side gains/loses) +
the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
pl = _pushable(db, user, playlist_id)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_push(db, user, pl)
plan = plan_merge(db, user, pl)
except YouTubeError:
raise HTTPException(status_code=422, detail="Couldn't read the playlist on YouTube.")
raise _refused(422, "youtube_failed")
plan["remaining_today"] = quota.remaining_today(db)
plan["affordable"] = plan["units_estimate"] <= plan["remaining_today"]
return plan
@@ -349,14 +384,14 @@ def push(
user: User = Depends(require_write_scope),
db: Session = Depends(get_db),
) -> dict:
"""Push a local playlist to YouTube (create it on first push, else reconcile to match
local — local wins). Refuses if the estimated quota exceeds what's left today, so a
big playlist can't strand half-pushed. Write scope required (require_write_scope)."""
"""Two-way sync a local playlist with YouTube (create it on first push, else 3-way merge so
neither side loses an item). Refuses if the estimated quota exceeds what's left today, so a
big sync can't strand half-applied. Write scope required (require_write_scope)."""
pl = _pushable(db, user, playlist_id)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
# Read the live playlist ONCE (linked playlists only) and share it with both the
# plan and the push, halving read-quota and closing the second read→write TOCTOU
# plan and the merge, halving read-quota and closing the second read→write TOCTOU
# window. A fresh export (no yt_playlist_id) reads nothing.
live = None
if pl.yt_playlist_id:
@@ -365,16 +400,13 @@ def push(
list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id)),
yt.get_playlist_snippet(pl.yt_playlist_id),
)
plan = plan_push(db, user, pl, live=live)
plan = plan_merge(db, user, pl, live=live)
if plan["units_estimate"] > quota.remaining_today(db):
raise HTTPException(
status_code=429,
detail="Not enough YouTube quota left today for this sync.",
)
return push_playlist(db, user, pl, live=live)
raise _refused(429, "quota_exceeded")
return merge_playlist(db, user, pl, live=live)
except YouTubeError:
db.rollback()
raise HTTPException(status_code=422, detail="YouTube playlist sync failed.")
raise _refused(422, "youtube_failed")
@router.get("/{playlist_id}")
@@ -419,7 +451,7 @@ def rename_playlist(
if "name" in body.model_fields_set:
name = (body.name or "").strip()
if not name:
raise HTTPException(status_code=400, detail="name cannot be empty")
raise _refused(400, "name_required")
if name != pl.name:
pl.name = name
recompute_dirty(db, pl)
@@ -454,18 +486,18 @@ def delete_playlist(
) -> dict:
pl = _own_playlist(db, user, playlist_id)
if pl.kind == "watch_later":
raise HTTPException(status_code=400, detail="The Watch later list can't be deleted.")
raise _refused(400, "watch_later_undeletable")
if on_youtube:
if not pl.yt_playlist_id:
raise HTTPException(status_code=400, detail="This playlist isn't on YouTube.")
raise _refused(400, "not_on_youtube")
if not has_write_scope(user):
raise HTTPException(status_code=403, detail="Enable YouTube editing in Settings first.")
raise _refused(403, "youtube_editing_off")
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_DELETE):
with YouTubeClient(db, user) as yt:
yt.delete_playlist(pl.yt_playlist_id)
except YouTubeError:
raise HTTPException(status_code=422, detail="YouTube delete failed.")
raise _refused(422, "youtube_failed")
db.delete(pl) # items cascade
db.commit()
return {"deleted": playlist_id}
@@ -480,8 +512,7 @@ def add_item(
) -> dict:
pl = _own_playlist(db, user, playlist_id)
video_id = body.video_id
if db.get(Video, video_id) is None:
raise HTTPException(status_code=404, detail="Unknown video")
get_or_404(db, Video, video_id, what="video")
_add_item(db, pl, video_id, mark_dirty=True)
return {"playlist_id": pl.id, "video_id": video_id}
@@ -498,6 +529,65 @@ def remove_item(
return {"playlist_id": pl.id, "video_id": video_id}
class MoveItemIn(BaseModel):
target_playlist_id: int
@router.post("/{playlist_id}/items/{video_id}/move")
def move_item(
playlist_id: int,
video_id: str,
body: MoveItemIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Move one video from a playlist into another of the caller's playlists, atomically.
ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
from the source — leaves the video in BOTH lists whenever the second call fails, which is
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
case. Idempotent: if the video is already in the target, the source row is simply dropped
(moving into a list that already has it is a removal, not a duplicate)."""
src = _own_playlist(db, user, playlist_id)
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
# the list on screen — which is still there.
dst = owned_or_404(db, Playlist, body.target_playlist_id, user, what="move_target")
if src.id == dst.id:
raise HTTPException(
status_code=400, detail={"code": "playlist_move", "reason": "same_playlist"}
)
item = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == src.id, PlaylistItem.video_id == video_id
)
)
if item is None:
# A concurrent edit (another tab, or a YouTube re-pull) already took it out.
raise HTTPException(
status_code=404, detail={"code": "playlist_move", "reason": "not_in_playlist"}
)
already = db.scalar(
select(PlaylistItem).where(
PlaylistItem.playlist_id == dst.id, PlaylistItem.video_id == video_id
)
)
if already is not None:
db.delete(item) # the uq_playlist_video constraint would reject a second row anyway
else:
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
# survives the move, and there is no window where the video belongs to neither list.
item.playlist_id = dst.id
item.position = _next_item_position(db, dst.id)
# Both sides changed, so both get the dirty recompute and the timestamp — the asymmetric
# version of this (touching only the source) is the recurring smell CLAUDE.md warns about.
recompute_dirty(db, src)
recompute_dirty(db, dst)
_touch(src)
_touch(dst)
db.commit()
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
class ReorderItemsIn(BaseModel):
video_ids: list[str] = []
@@ -533,5 +623,6 @@ def reorder_items(
it.position = pos
pos += 1
recompute_dirty(db, pl)
_touch(pl)
db.commit()
return {"playlist_id": pl.id, "count": pos}
+1 -23
View File
@@ -3,13 +3,9 @@
import logging
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models import (
PlexItem,
PlexLibrary,
)
from app.models import PlexItem
from app.plex.lookups import by_rating_key
log = logging.getLogger("siftlode.plex")
@@ -25,21 +21,3 @@ def item_or_404(db: Session, rating_key: str) -> PlexItem:
def _norm_lang(lang: str | None) -> str:
"""The overlay has an EN base + an optional HU twin; anything other than `hu` serves EN."""
return "hu" if lang == "hu" else "en"
def _content_libs(db: Session) -> tuple[list[int], list[int]]:
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5)."""
libs = (
db.execute(
select(PlexLibrary).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
)
.scalars()
.all()
)
movie_lib_ids = [lb.id for lb in libs if lb.kind == "movie"]
show_lib_ids = [lb.id for lb in libs if lb.kind == "show"]
return movie_lib_ids, show_lib_ids
+70 -11
View File
@@ -2,6 +2,7 @@
from fastapi import APIRouter, Depends, Query
from sqlalchemy import (
Boolean,
Integer,
String,
and_,
@@ -27,15 +28,19 @@ from app.plex import rollup
from app.plex.cards import episode_card
from app.plex.query import (
FIELD_COLUMNS,
PLAYBACK_CLASSES,
TS_CONFIG,
LibraryFilters,
apply_meta_filters,
apply_movie_state_filter,
apply_show_state_filter,
content_libs,
field_recheck,
genres_col,
meta_filter_conds,
movie_status_case,
playable_conds,
playback_class_expr,
show_status_case,
split_csv,
summary_col,
@@ -43,12 +48,28 @@ from app.plex.query import (
vector_col,
vector_match,
)
from app.routes.plex._shared import _content_libs, _norm_lang
from app.routes.plex._shared import _norm_lang
from app.utils import to_tsquery_str
router = APIRouter(prefix="/api/plex", tags=["plex"])
def sort_column(u, sort: str):
"""The browse grid's sort key -> orderable column on the union subquery. Module-level (not inline
in the handler) so a test can pin its vocabulary to `PLEX_SORT_KEYS` — a key the Sort control
offers but this cannot order would quietly fall back to `added_at` and look like a broken control.
An unknown key (a stale bookmark, a hand-edited URL) deliberately keeps that fallback."""
return {
"added": u.c.added_at,
"title": func.lower(u.c.title),
"year": u.c.year,
"rating": u.c.rating,
"duration": u.c.duration_s,
"release": u.c.rel,
"lastwatched": u.c.last_watched,
}.get(sort, u.c.added_at)
@router.get("/libraries")
def list_libraries(user: User = Depends(current_user), db: Session = Depends(get_db)) -> dict:
"""The mirrored libraries the user can browse (drives the Plex scope selector). HU overlay
@@ -105,7 +126,7 @@ def unified_library(
offset = max(0, offset)
lang = _norm_lang(lang)
p = f.as_dict()
movie_lib_ids, show_lib_ids = _content_libs(db)
movie_lib_ids, show_lib_ids = content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
@@ -135,9 +156,13 @@ def unified_library(
PlexItem.originally_available_at.label("rel"),
PlexItem.duration_s.label("duration_s"),
PlexItem.playable.label("playable"),
PlexItem.transcode_hw.label("transcode_hw"),
cast(null(), Integer).label("season_count"),
func.coalesce(st.position_seconds, 0).label("position_seconds"),
m_status.label("status"),
# "last watched" sort key: when the movie was last resumed, else when it was marked
# watched. NULL for never-touched movies → nullslast sinks them (see the sort dict).
func.coalesce(st.progress_updated_at, st.watched_at).label("last_watched"),
(
func.ts_rank(vector_col(PlexItem, lang), tsq)
if tsq is not None
@@ -153,13 +178,15 @@ def unified_library(
mq = mq.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
mq = mq.where(PlexItem.duration_s <= f.duration_max)
# Playback-class filter (movie-only — a show is an aggregate with no single class).
mq = mq.where(*playable_conds(split_csv(f.playable)))
if tsq is not None:
mq = mq.where(vector_match(PlexItem, lang, tsq))
if recheck is not None:
mq = mq.where(recheck(PlexItem, field, tsq, lang))
selects.append(mq)
if want_shows and grid_field:
agg = rollup.agg_subquery(db, user.id).subquery()
agg = rollup.agg_subquery(db, user.id, include_resumed=True).subquery()
tot = func.coalesce(agg.c.total, 0)
wat = func.coalesce(agg.c.watched, 0)
inp = func.coalesce(agg.c.inprog, 0)
@@ -176,9 +203,13 @@ def unified_library(
PlexShow.originally_available_at.label("rel"),
cast(null(), Integer).label("duration_s"),
cast(null(), String).label("playable"),
cast(null(), Boolean).label("transcode_hw"),
PlexShow.child_count.label("season_count"),
literal(0).label("position_seconds"),
s_status.label("status"),
# Show leg's "last watched": max(resume/watched time) across the show's episodes (see
# rollup.agg_subquery include_resumed). NULL for a never-watched show → nullslast.
agg.c.resumed_at.label("last_watched"),
(
func.ts_rank(vector_col(PlexShow, lang), tsq)
if tsq is not None
@@ -206,13 +237,7 @@ def unified_library(
if tsq is not None:
order = [u.c.rank.desc()]
else:
col = {
"title": func.lower(u.c.title),
"year": u.c.year,
"rating": u.c.rating,
"duration": u.c.duration_s,
"release": u.c.rel,
}.get(sort, u.c.added_at)
col = sort_column(u, sort)
order = [(col.asc() if sort_dir == "asc" else col.desc()).nullslast()]
order.append(u.c.rk.desc())
rows = db.execute(select(u).order_by(*order).offset(offset).limit(limit)).all()
@@ -230,6 +255,7 @@ def unified_library(
"duration_seconds": r.duration_s,
"thumb": f"/api/plex/image/{r.rk}",
"playable": r.playable,
"transcode_hw": r.transcode_hw,
"status": r.status,
"position_seconds": r.position_seconds,
}
@@ -305,13 +331,14 @@ def facets(
"genres": [],
"content_ratings": [],
"studios": [],
"playable": [],
"year_min": None,
"year_max": None,
"rating_max": None,
"duration_min": None,
"duration_max": None,
}
all_movie_ids, all_show_ids = _content_libs(db)
all_movie_ids, all_show_ids = content_libs(db)
movie_ids = all_movie_ids if scope in ("movie", "both") else []
show_ids = all_show_ids if scope in ("show", "both") else []
if not movie_ids and not show_ids:
@@ -331,6 +358,9 @@ def facets(
query = query.where(model.duration_s >= pp["duration_min"])
if with_dur and pp.get("duration_max") is not None:
query = query.where(model.duration_s <= pp["duration_max"])
# Playback-class filter (movie-only) — applied for the OTHER groups so an active class
# narrows them; the playable facet itself passes pp with playable=None (self-exclusion).
query = query.where(*playable_conds(split_csv(pp.get("playable"))))
# Join watch-state and reuse the grid's exact movie state filter (unified_library) —
# hidden movies never appear in the grid, so they must not inflate facet counts/bounds either.
st = aliased(PlexState)
@@ -349,6 +379,7 @@ def facets(
genre_counts: dict[str, int] = {}
cr_counts: dict[str, int] = {}
studio_counts: dict[str, int] = {}
playable_counts: dict[str, int] = {}
years: list[int] = []
ratings: list[float] = []
durs: list[int] = []
@@ -413,6 +444,23 @@ def facets(
)
for s, c in db.execute(stq).all():
studio_counts[s] = studio_counts.get(s, 0) + c
# Playback class (movie-only) — exclude its own so each class stays an addable OR option.
if kind_movie:
pcls = playback_class_expr()
pcq = (
wsq(
select(pcls.label("cls"), func.count()),
model,
ids,
{**p, "playable": None},
kind_movie,
True,
)
.where(pcls.isnot(None))
.group_by(pcls)
)
for cls, c in db.execute(pcq).all():
playable_counts[cls] = playable_counts.get(cls, 0) + c
# Year bounds — exclude the year filter.
by = db.execute(
wsq(
@@ -470,6 +518,11 @@ def facets(
# Same for the actively-selected studios, so a selected studio stays removable even at zero matches.
for s in split_csv(f.studios):
studio_counts.setdefault(s, 0)
# Keep an actively-selected playback class visible even at zero matches (the sidebar hides the whole
# section when its facet list is empty, which would trap a zero-result selection with no way to undo).
for c in split_csv(f.playable):
if c in PLAYBACK_CLASSES:
playable_counts.setdefault(c, 0)
return {
"genres": [
{"value": g, "count": c} for g, c in sorted(genre_counts.items(), key=lambda kv: kv[0])
@@ -481,6 +534,12 @@ def facets(
{"value": s, "count": c}
for s, c in sorted(studio_counts.items(), key=lambda kv: (-kv[1], kv[0]))
],
# Fixed good→heavy order (not by count), so the class list reads as a severity scale.
"playable": [
{"value": c, "count": playable_counts[c]}
for c in PLAYBACK_CLASSES
if c in playable_counts
],
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"rating_max": max(ratings) if ratings else None,
+136 -17
View File
@@ -1,5 +1,9 @@
"""Plex collections: browse mirrored collections + the admin manual-collection editor."""
from collections.abc import Callable
from datetime import date, datetime
from typing import Any, NamedTuple
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import func, select
@@ -12,6 +16,7 @@ from app.models import (
PlexItem,
PlexLibrary,
PlexShow,
PlexState,
User,
)
from app.plex import rollup
@@ -19,8 +24,7 @@ from app.plex import sync as plex_sync
from app.plex.cards import movie_card, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.plex.query import like_escape, runtime_by_key, top_art_keys
from app.routes.plex._shared import _content_libs
from app.plex.query import content_libs, like_escape, runtime_by_key, top_art_keys
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
# members (Plex shows one cert on the collection; we don't store it, so derive it).
@@ -84,13 +88,20 @@ def _collection_card(c: PlexCollection) -> dict:
def collections(
library: str | None = None,
q: str | None = None,
keys: str | None = None,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Mirrored collections as cards (most-populated first), optionally name-filtered by `q`. With a
`library` plex_key → that one library's collections (used by the collection editor). WITHOUT one →
the UNION across all enabled libraries (the unified-scope sidebar picker, which has no single
library selected)."""
library selected).
`keys` (CSV of rating_keys) narrows to just those. It exists because a Plex link carries the
collection FILTER as bare rating_keys — nothing in the URL says what "17045" is called — so the
sidebar token and the applied-filter chip had nothing to render but the raw id. Resolving them
through this endpoint costs one small row each; pulling the whole list to find two titles is
~200 KB."""
if library is not None:
lib = (
db.execute(select(PlexLibrary).where(PlexLibrary.plex_key == str(library)))
@@ -114,6 +125,13 @@ def collections(
term = (q or "").strip()
if term:
query = query.where(PlexCollection.title.ilike(f"%{like_escape(term)}%"))
if keys is not None:
wanted = [k for k in (keys or "").split(",") if k]
# An explicit but empty `keys=` asks for nothing — answer nothing, rather than falling through
# to "every collection" (which is what an unguarded empty IN () would have to become).
if not wanted:
return {"collections": []}
query = query.where(PlexCollection.rating_key.in_(wanted))
rows = (
db.execute(
query.order_by(
@@ -128,7 +146,7 @@ def collections(
# members (a collection lives in one library, so exactly one side has art) — a show collection would
# otherwise render blank; the collection's own poster is the next fallback, then the colour hash.
# Runtime is movie-only (shows have no single duration).
movie_ids, show_ids = _content_libs(db)
movie_ids, show_ids = content_libs(db)
keys = [c.rating_key for c in rows]
art_m = top_art_keys(db, PlexItem, movie_ids, PlexItem.collection_keys, keys, per=2)
art_s = top_art_keys(db, PlexShow, show_ids, PlexShow.collection_keys, keys, per=2)
@@ -150,14 +168,69 @@ def collections(
}
class Member(NamedTuple):
"""One collection member flattened to everything the Sort control can order by. The page's two
legs are different tables (movies and shows), so there is no single query to ORDER BY — the
ordering is a pure list sort over these, which is also what makes it unit-testable."""
kind: str # "movie" | "show"
obj: Any # PlexItem | PlexShow, carried through to the card serializer
title: str # lower-cased; never NULL (the column is not nullable)
year: int | None
rating: float | None
added: datetime | None
release: date | None
duration: int | None # movies only — a show has no single runtime
last_watched: datetime | None
# Sort key -> what to read off a Member. Pinned to `PLEX_SORT_KEYS` by a test: the collection page
# shows the SAME Sort control as the browse grid, so a key this map is missing would be an option
# that silently does nothing here.
MEMBER_SORT_KEYS: dict[str, Callable[[Member], Any]] = {
"added": lambda m: m.added,
"lastwatched": lambda m: m.last_watched,
"release": lambda m: m.release,
"year": lambda m: m.year,
"rating": lambda m: m.rating,
"duration": lambda m: m.duration,
"title": lambda m: m.title,
}
# What the page ordered by before it had a control, and still its default: a film series wants to be
# read in release order, not by whenever Plex happened to ingest it.
DEFAULT_MEMBER_SORT = "year"
def sort_members(members: list[Member], sort: str, sort_dir: str) -> list[Member]:
"""Order a collection's members by one of `PLEX_SORT_KEYS`.
Two properties worth stating because they are easy to get wrong with a plain `reverse=True`:
members with no value for the key sink to the END in BOTH directions (matching the browse grid's
`nullslast()` — a movie with no rating is not "the worst rated"), and ties always resolve
title-ascending, so flipping the direction reverses the ranking rather than the alphabet inside
it. An unknown key falls back to the page's default instead of returning an arbitrary order."""
key = MEMBER_SORT_KEYS.get(sort, MEMBER_SORT_KEYS[DEFAULT_MEMBER_SORT])
ordered = sorted(members, key=lambda m: m.title) # stable secondary, applied first
have = [m for m in ordered if key(m) is not None]
missing = [m for m in ordered if key(m) is None]
have.sort(key=key, reverse=sort_dir != "asc")
return have + missing
@router.get("/collections/{rating_key}")
def collection_detail(
rating_key: str,
sort: str = DEFAULT_MEMBER_SORT,
sort_dir: str = "asc",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""One collection's hero data + its members (movies + shows), ordered by release year then title —
the Collection info page. Read-only here; edits go through the admin mutation endpoints below."""
"""One collection's hero data + its members (movies + shows) in the requested order — the
Collection info page. The sort vocabulary is the browse grid's (`PLEX_SORT_KEYS`) so the same Sort
control means the same thing on both surfaces; the default is release year ascending, which is
what this page did before it had a control. Read-only here; edits go through the admin mutation
endpoints below."""
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
@@ -180,19 +253,63 @@ def collection_detail(
.all()
)
mstate = rollup.item_state_map(db, user.id, [m.id for m in movies])
sstate = rollup.state_map(db, user.id, [s.id for s in shows])
# Merge the two type-homogeneous lists into ONE member list ordered by (year, title).
tagged = [(m.year, (m.title or "").lower(), "movie", m) for m in movies] + [
(s.year, (s.title or "").lower(), "show", s) for s in shows
# ONE rollup pass over the show members, for both the aggregate badge status and `resumed_at` —
# the browse grid's "last watched" for a show (the newest resume/watched across its episodes). The
# cheaper `rollup.state_map` gives only the status, which would leave `lastwatched` unorderable on
# a show collection while the control still offered it.
show_rows = (
db.execute(
rollup.agg_subquery(db, user.id, include_resumed=True).where(
PlexItem.show_id.in_([s.id for s in shows])
)
).all()
if shows
else []
)
sstate = {r.show_id: rollup.show_status(r.total, r.watched, r.inprog) for r in show_rows}
sresumed = {r.show_id: r.resumed_at for r in show_rows}
def _movie_last_watched(st: PlexState | None) -> datetime | None:
# Same definition as the browse grid's: when it was last resumed, else when it was marked
# watched (coalesce(progress_updated_at, watched_at)).
return (st.progress_updated_at or st.watched_at) if st else None
# Merge the two type-homogeneous lists into ONE member list, then order it.
members = [
Member(
kind="movie",
obj=m,
title=(m.title or "").lower(),
year=m.year,
rating=m.rating,
added=m.added_at,
release=m.originally_available_at,
duration=m.duration_s,
last_watched=_movie_last_watched(mstate.get(m.id)),
)
for m in movies
] + [
Member(
kind="show",
obj=sh,
title=(sh.title or "").lower(),
year=sh.year,
rating=sh.rating,
added=sh.added_at,
release=sh.originally_available_at,
duration=None,
last_watched=sresumed.get(sh.id),
)
for sh in shows
]
tagged.sort(key=lambda t: (t[0] is None, t[0] or 0, t[1]))
members = sort_members(members, sort, sort_dir)
items = [
movie_card(o, mstate.get(o.id))
if kind == "movie"
else show_card(o, sstate.get(o.id, "new"))
for _, _, kind, o in tagged
movie_card(m.obj, mstate.get(m.obj.id))
if m.kind == "movie"
else show_card(m.obj, sstate.get(m.obj.id, "new"))
for m in members
]
years = [y for y, _, _, _ in tagged if y is not None]
years = [m.year for m in members if m.year is not None]
certs = [m.content_rating for m in movies] + [s.content_rating for s in shows]
return {
**_collection_card(col),
@@ -230,7 +347,9 @@ def _reject_duplicate_name(
.first()
)
if dup is not None and dup.rating_key != exclude_rk:
raise HTTPException(status_code=409, detail="A collection with this name already exists")
# Structured, so the bilingual UI says it in the reader's language (it was toasted in
# Hungarian beside an English error dialog saying the same thing).
raise HTTPException(status_code=409, detail={"code": "collection", "reason": "name_taken"})
def _write_or_pending_error(wrote: bool, e: Exception) -> HTTPException:
+4
View File
@@ -411,6 +411,10 @@ def item_detail(
"year": it.year,
"duration_seconds": it.duration_s,
"playable": it.playable,
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
"transcode_hw": it.transcode_hw,
"codec_video": it.codec_video,
"video_bit_depth": it.video_bit_depth,
"thumb": f"/api/plex/image/{it.rating_key}",
"art": f"/api/plex/image/{it.rating_key}?variant=art",
"library": _lib_key(db, it.library_id),
+9 -7
View File
@@ -27,12 +27,13 @@ from app.plex.query import (
LibraryFilters,
apply_meta_filters,
apply_show_state_filter,
content_libs,
genres_col,
meta_filter_conds,
top_art_keys,
)
from app.plex.tmdb import resolve_recs
from app.routes.plex._shared import _content_libs, _norm_lang
from app.routes.plex._shared import _norm_lang
router = APIRouter(prefix="/api/plex", tags=["plex"])
@@ -43,7 +44,7 @@ SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows
def _scope_libs(db: Session, scope: str) -> tuple[list[int], list[int]]:
"""The movie/show library ids visible for this scope (empty list = out of scope)."""
movie_ids, show_ids = _content_libs(db)
movie_ids, show_ids = content_libs(db)
return (
movie_ids if scope in ("movie", "both") else [],
show_ids if scope in ("show", "both") else [],
@@ -112,10 +113,11 @@ def _recently_added(
else []
)
# Merge the two type-homogeneous lists by added_at, then serialize each side in one batch so the
# per-card state lookups stay batched (no N+1). The `(is not None, value)` key sorts a missing
# timestamp last under reverse.
# per-card state lookups stay batched (no N+1). Same `or NO_TS` key as the two rails below — the
# E24 review found this one still on the older `(is not None, value)` form, i.e. one rule in two
# spellings, with the two comments explaining it differently.
tagged = [(m.added_at, "movie", m) for m in movies] + [(s.added_at, "show", s) for s in shows]
tagged.sort(key=lambda t: (t[0] is not None, t[0]), reverse=True)
tagged.sort(key=lambda t: t[0] or rollup.NO_TS, reverse=True)
top = tagged[:RAIL_LIMIT]
# Key the serialized cards back by rating_key (globally unique in Plex — a movie and a show never
# share one), so the merge doesn't depend on object identity staying alive.
@@ -170,7 +172,7 @@ def _recently_watched(
)
for ep, st, sh_title, sh_rk in db.execute(eq).all():
rows.append((st.watched_at, leaf_card(ep, st, show_title=sh_title, show_rk=sh_rk)))
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
@@ -234,7 +236,7 @@ def _continue_watching(
for sh, resumed_at in srows:
rows.append((resumed_at, show_card(sh, "in_progress")))
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
rows.sort(key=lambda r: r[0] or rollup.NO_TS, reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
+3 -4
View File
@@ -19,8 +19,7 @@ from app.models import (
from app.plex import rollup
from app.plex import tmdb as plex_tmdb
from app.plex.cards import movie_card, show_card
from app.plex.query import split_csv
from app.routes.plex._shared import _content_libs
from app.plex.query import content_libs, split_csv
router = APIRouter(prefix="/api/plex", tags=["plex"])
log = logging.getLogger("siftlode.plex")
@@ -71,7 +70,7 @@ def people_cards(
wanted = split_csv(names)[:16]
if not wanted:
return {"people": []}
movie_lib_ids, show_lib_ids = _content_libs(db)
movie_lib_ids, show_lib_ids = content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
photos = {
@@ -185,7 +184,7 @@ def person_detail(
and _should_schedule_enrich(person.id)
):
background_tasks.add_task(_enrich_person_bg, person.id)
movie_lib_ids, show_lib_ids = _content_libs(db)
movie_lib_ids, show_lib_ids = content_libs(db)
movies: list[dict] = []
if movie_lib_ids:
+255 -99
View File
@@ -1,9 +1,8 @@
"""Plex playback: image/subtitle proxies, HLS streaming, and watch-state writes."""
import hashlib
import os
import tempfile
from datetime import datetime, timezone
import threading
from datetime import datetime, timedelta, timezone
import httpx
from fastapi import (
@@ -11,17 +10,20 @@ from fastapi import (
BackgroundTasks,
Depends,
HTTPException,
Path,
Query,
Request,
Response,
)
from fastapi.responses import FileResponse
from pydantic import BaseModel
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.auth import current_user, resolved_user_id
from app.db import SessionLocal, get_db
from app.models import (
POSITION_MAX_S,
PlexItem,
PlexSeason,
PlexShow,
@@ -88,62 +90,70 @@ def image(rating_key: str, request: Request, variant: str = "thumb") -> Response
return imgcache.response(data, ct)
# Both subtitle routes authenticate with the full `current_user`, not the cookie-only
# `_require_session` the image proxies use. That lighter gate exists for bursts of low-sensitivity
# poster fetches and skips the is_active / is_suspended / session-epoch checks — acceptable for a
# poster, not for a route that starts reads of the media library: a suspended or "log out
# everywhere"-revoked cookie kept scheduling them. They are not bursty (one poll a second, one fetch
# per track), so the DB user-load costs nothing that matters.
#
# Both path parameters are bounded, the same way on both routes: they become a cache FILE name
# (`sub_<rating_key>_<ord>`), and an absurd one — a 250-digit key or ordinal — outgrew the OS's name
# limit and turned the status poll into a 500. A Plex rating_key is a short number; the bound only
# has to be generous.
_RatingKey = Path(max_length=32)
_SubOrd = Path(ge=0, le=9999)
@router.get("/subtitle/{rating_key}/{sub_ord}/status")
def subtitle_status(
rating_key: str = _RatingKey,
sub_ord: int = _SubOrd,
user: User = Depends(current_user),
# Bounded both ways: a non-finite position used to reach the scheduler's arithmetic and 500, and
# an absurd one reached ffmpeg's `-ss`, which refused it.
at: float = Query(0.0, ge=0.0, le=subtitles.MAX_POSITION_S, allow_inf_nan=False),
) -> dict:
"""Is this subtitle track showable yet — and if not, get it moving. Extraction runs in the
background (see plex/subtitles.py for the measurements that forced it there), so the player polls
this while it waits instead of hanging on a <track> that will not load for a minute or more.
`at` is the viewer's absolute position: the extractor windows around it, so what gets read first
is the part actually about to be watched."""
return subtitles.ensure(rating_key, sub_ord, at)
@router.get("/subtitle/{rating_key}/{sub_ord}")
def subtitle(rating_key: str, sub_ord: int, request: Request, offset: float = 0.0) -> Response:
def subtitle(
rating_key: str = _RatingKey,
sub_ord: int = _SubOrd,
user: User = Depends(current_user),
# Finite, like every float this router takes: `-inf` crashed the shift (a 500), `nan` stacked
# every cue at 0. It can legitimately be negative (a subtitle nudged later than the session start).
offset: float = Query(0.0, allow_inf_nan=False),
) -> Response:
"""Serve the ord-th subtitle track as WebVTT for the browser's <track> — so choosing a subtitle
overlays it directly (no HLS re-mux / session restart). Handles EXTERNAL sidecar subs (fetched
from Plex via the stream `key` — the old `-map 0:s` broke on these since they're not in the file)
AND embedded text subs (extracted from the local file). Image subs (PGS/VobSub) → 415. Disk-cached."""
_require_session(request)
# Cache the ABSOLUTE-time VTT once (per rk+ord); the per-session cue-shift is applied on the way
# out, so a resume/seek to any offset reuses the same cached track (no cache-per-offset explosion).
cache_key = f"sub_{rating_key}_{sub_ord}"
hit = subtitles.cache_read(cache_key)
if hit is not None:
return Response(subtitles.shift_vtt(hit, offset), media_type="text/vtt")
try:
with SessionLocal() as db:
it = item_or_404(db, rating_key)
with PlexClient(db) as plex:
part = ((plex.metadata(it.rating_key) or {}).get("Media") or [{}])[0].get(
"Part"
) or [{}]
subs = [s for s in (part[0].get("Stream") or []) if s.get("streamType") == 3]
if sub_ord < 0 or sub_ord >= len(subs):
raise HTTPException(status_code=404, detail="No such subtitle")
s = subs[sub_ord]
codec = (s.get("codec") or s.get("format") or "").lower()
if codec in subtitles.BITMAP_SUBS:
raise HTTPException(
status_code=415,
detail="Image-based subtitle can't be shown as text",
)
if s.get("key"): # external sidecar → fetch from Plex, convert to VTT
raw = plex.raw_get(s["key"])
if codec in ("srt", "subrip", ""):
vtt = subtitles.srt_to_vtt(raw.decode("utf-8", "replace"))
else:
fd, tmp = tempfile.mkstemp(suffix="." + (codec or "sub"))
try:
os.write(fd, raw)
os.close(fd)
vtt = subtitles.ffmpeg_to_vtt(tmp, None)
finally:
os.unlink(tmp)
elif s.get("index") is not None: # embedded → extract from the local file
src = plex_paths.local_media_path(db, it.file_path)
if src is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
emb_ord = sum(1 for x in subs[:sub_ord] if x.get("index") is not None)
vtt = subtitles.ffmpeg_to_vtt(str(src), emb_ord)
else:
raise HTTPException(status_code=404, detail="Subtitle not fetchable")
except PlexNotConfigured as e:
raise HTTPException(status_code=400, detail=str(e))
except PlexError as e:
raise HTTPException(status_code=502, detail=f"Subtitle fetch failed: {e}")
subtitles.cache_write(cache_key, vtt)
return Response(subtitles.shift_vtt(vtt, offset), media_type="text/vtt")
AND embedded text subs (extracted from the local file). Image subs (PGS/VobSub) → 415. Disk-cached.
A pure READ: it serves whatever is cached and schedules nothing — the status poll owns that,
because only it knows where the viewer is (see `subtitles.cached_track`). The player mounts the
<track> only once /status says there is something to show, so in practice this is only ever hit
warm; on a cache eviction race the 202 sends the player back to polling."""
# The cached VTT holds ABSOLUTE cue times (per rk+ord); the per-session cue-shift is applied on
# the way out, so a resume/seek to any offset reuses the same track (no cache-per-offset explosion).
hit, code = subtitles.cached_track(rating_key, sub_ord)
if hit is None and code:
raise HTTPException(
status_code=subtitles.UNAVAILABLE_STATUS.get(code, 422),
detail={"code": "subtitle", "reason": code},
)
if hit is None:
return Response(
status_code=202, media_type="application/json", content='{"state":"preparing"}'
)
return Response(subtitles.shift_vtt(hit, offset), media_type="text/vtt")
# Resume/watched thresholds. _PROGRESS_MIN_S matches the feed's PROGRESS_MIN_SECONDS (5), but
@@ -155,6 +165,47 @@ _PROGRESS_MIN_S = 5
_FINISH_MARGIN_S = 30
# S2 write-back throttle: the resume position is now pushed to Plex on the periodic (non-final)
# checkpoint too — not only on the final pause/unmount flush — so Plex (the source of truth) stays
# current in near-real-time. To keep a single watch from spraying `/:/timeline` calls, a push is
# rate-limited to at most one per item per user every _PUSH_THROTTLE_S seconds; a `final` checkpoint
# (pause / seek / stop) always pushes immediately and resets the window. This is an in-memory, single-
# process rate-limiter (the deployment is one FastAPI process): after a restart the map is empty, so at
# most one extra push happens on the first checkpoint — harmless. It is NOT durable domain state.
_PUSH_THROTTLE_S = 10
_last_push_at: dict[tuple[int, int], datetime] = {}
_push_throttle_lock = threading.Lock()
def _should_push_resume(user_id: int, item_id: int, now: datetime, final: bool) -> bool:
"""Whether to mirror this resume checkpoint to Plex now. `final` always pushes; a periodic tick
pushes only when ≥_PUSH_THROTTLE_S has elapsed since the last push for this (user, item). Records
`now` as the last-push time whenever it returns True so the window is measured from the real push
(and so `_resume_push_superseded` can drop an older, still-pending push — see there)."""
with _push_throttle_lock:
last = _last_push_at.get((user_id, item_id))
if final or last is None or (now - last).total_seconds() >= _PUSH_THROTTLE_S:
_last_push_at[(user_id, item_id)] = now
# Opportunistic eviction so the map can't grow unbounded over a long-lived process (one
# entry per item ever checkpointed). Anything older than a few windows is dead — a push
# completes in seconds, so a stale entry protects nothing.
if len(_last_push_at) > 2048:
cutoff = now - timedelta(seconds=_PUSH_THROTTLE_S * 6)
for k in [k for k, t in _last_push_at.items() if t < cutoff]:
del _last_push_at[k]
return True
return False
def _resume_push_superseded(user_id: int, item_id: int, intent: datetime) -> bool:
"""True if a newer resume push for this (user, item) was scheduled after `intent`. Called at the
HTTP boundary of a background resume push: because periodic and seek/pause pushes are independent
background tasks with no ordering guard, an older still-pending periodic push could otherwise land
at Plex AFTER a newer backward-seek push and — with the Plex-authoritative pull — durably revive the
stale position. Skipping a superseded push keeps the last-scheduled position the one Plex settles on."""
with _push_throttle_lock:
return _last_push_at.get((user_id, item_id), intent) > intent
@router.get("/person-image")
def person_image(u: str, request: Request) -> Response:
@@ -178,6 +229,26 @@ def person_image(u: str, request: Request) -> Response:
return imgcache.response(r.content, ct)
def _push_resume_guarded(
user_id: int,
item_id: int,
rating_key: str,
time_ms: int,
duration_ms: int,
expect_ts: datetime | None,
intent: datetime,
) -> None:
"""Background entrypoint for a resume push that drops itself if a newer resume push for the same
(user, item) was scheduled after it (see `_resume_push_superseded`), then delegates to the real
push. This narrows the periodic-vs-seek write-ordering race to the (rare) case where the older push
has already left the process before the newer one is scheduled."""
if _resume_push_superseded(user_id, item_id, intent):
return
plex_watch.push_state_to_plex(
user_id, item_id, rating_key, "resume", time_ms, duration_ms, expect_ts
)
def _push_watch(
background: BackgroundTasks,
db: Session,
@@ -187,10 +258,14 @@ def _push_watch(
action: str,
time_ms: int = 0,
duration_ms: int = 0,
intent: datetime | None = None,
) -> None:
"""Schedule a best-effort Siftlode→Plex watch push, but only for a user whose owner link has
push enabled — non-owner users never touch Plex. Guarding here (with the request session) avoids
spinning up a throwaway background session on every state change for the common no-link case."""
spinning up a throwaway background session on every state change for the common no-link case.
A `resume` push with an `intent` timestamp is routed through `_push_resume_guarded` so a superseded
(older) push can't land out of order; `watched`/`unwatched` scrobbles go straight through."""
if plex_watch.link_for_push(db, user_id) is None:
return
# Capture the row's freshest timestamp NOW so the background push can tell if a newer local edit
@@ -202,6 +277,12 @@ def _push_watch(
.scalars()
.first()
)
marker = plex_watch._state_marker(st)
if action == "resume" and intent is not None:
background.add_task(
_push_resume_guarded, user_id, item_id, rating_key, time_ms, duration_ms, marker, intent
)
return
background.add_task(
plex_watch.push_state_to_plex,
user_id,
@@ -210,15 +291,20 @@ def _push_watch(
action,
time_ms,
duration_ms,
plex_watch._state_marker(st),
marker,
)
class ItemProgressIn(BaseModel):
# Finite only: JSON parsing here accepts `Infinity`/`NaN`, and the handler's `int()` turned
# one into a 500.
model_config = ConfigDict(allow_inf_nan=False)
# float, truncated in the handler — the player may post a raw currentTime (a float); a plain
# `int` field would 422 it on the periodic progress checkpoint (the old `int(...)` truncated).
position_seconds: float = 0
duration_seconds: float = 0
# Bounded above by the int4 column (see POSITION_MAX_S): past it the flush was a 500.
position_seconds: float = Field(0, le=POSITION_MAX_S)
duration_seconds: float = Field(0, le=POSITION_MAX_S)
final: bool = False
@@ -232,10 +318,19 @@ def item_progress(
) -> dict:
"""Checkpoint the resume position while playing (near the end → mark watched + clear it).
`final` (sent on the pause/pagehide/unmount flush, not the periodic 10s checkpoint) is what
debounces the Plex resume push: we mirror the settled position to Plex only on a final flush, so
a single watch doesn't spray `/:/timeline` calls at the server every 10 seconds."""
S2 write-back: the resume position is mirrored to Plex (the source of truth) on the periodic
checkpoint too, not only on the final flush — throttled to ≥_PUSH_THROTTLE_S per item (see
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
it = item_or_404(db, rating_key)
# A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
# that such a viewer still wants their HLS session: the one thing `last_access` could not see,
# since it moved only on a file fetch and a paused player fetches nothing (see
# stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
# finish latch is up, or before the first position is known. Those are the states in which
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
# the same 404 → reclaim the player already handles.
plex_stream.touch_session(str(rating_key))
position = max(0, int(body.position_seconds))
duration = int(body.duration_seconds)
final = body.final
@@ -259,6 +354,10 @@ def item_progress(
st.position_seconds = 0
st.progress_updated_at = now
st.synced_to_plex = False
# Terminal state for resume write-back: drop the throttle/ordering entry so the map doesn't
# retain a dead key for every finished item.
with _push_throttle_lock:
_last_push_at.pop((user.id, it.id), None)
db.commit()
# Immediate on the watched transition; skip a repeat scrobble if it was already watched
# (a later checkpoint of the same finished item would otherwise bump Plex's viewCount again).
@@ -280,10 +379,13 @@ def item_progress(
db.add(st)
st.position_seconds = position
st.progress_updated_at = now
if final:
push_now = _should_push_resume(user.id, it.id, now, final)
if push_now:
# Mark dirty before the (background) push so the reconcile knows to re-push if it fails, and so
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
st.synced_to_plex = False
db.commit()
if final:
if push_now:
_push_watch(
background,
db,
@@ -293,6 +395,7 @@ def item_progress(
"resume",
position * 1000,
duration * 1000,
intent=now,
)
return {"position_seconds": position}
@@ -325,10 +428,20 @@ def item_state(
)
prev_status = st.status if st is not None else "new"
if status == "new":
# Does Plex hold anything for this item that the un-mark has to clear? Two things qualify: a
# view count, and a resume point. An IN-PROGRESS item carries status "new" WITH a position —
# that is how the importer maps a bare `viewOffset` (`_plex_watch_to_state`) — so keying the
# push on the status alone silently skipped the single case "Clear resume point" exists for:
# the un-mark never reached Plex, the offset stayed, and the next pull imported it straight
# back (Protector, cleared twice by the user and back at 92s both times). `hidden` is
# Siftlode-only and still pushes nothing, exactly as this endpoint's docstring says.
mirrored_on_plex = st is not None and (
st.status == "watched" or (st.position_seconds or 0) > 0
)
if st is not None:
db.delete(st)
db.commit()
if prev_status != "new": # something → unwatched: mirror the un-mark to Plex
if mirrored_on_plex:
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
return {"status": "new"}
if st is None:
@@ -443,23 +556,21 @@ def season_state(
@router.post("/stream/{rating_key}/session")
def stream_session(
rating_key: str,
start: float = 0.0,
# Finite: `start=inf` reached `int()` in the session builder and came back as a 500.
start: float = Query(0.0, allow_inf_nan=False),
audio: int | None = None,
aoff: float = 0.0,
aoff: float = Query(0.0, allow_inf_nan=False),
multi: bool = False,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Prepare playback from `start` seconds. Direct-playable files are served raw; remux files
(re)start an HLS session at the offset (seek-restart); transcode (HEVC/VP9) is P3. Selecting a
non-default audio track (`audio` = stream ordinal) forces the HLS path so the chosen audio can be
muxed in. Subtitles are independent WebVTT tracks (GET /subtitle) — no session restart."""
plex_stream.reap_idle() # opportunistic cleanup of idle sessions on each new playback
(re)start an HLS session at the offset (seek-restart, video stream-copied); transcode files (non-h264:
HEVC/VP9/AV1/…) take the same HLS path but with the video re-encoded to H.264 (VAAPI or libx264).
Selecting a non-default audio track (`audio` = stream ordinal) forces the HLS path so the chosen audio
can be muxed in. Subtitles are independent WebVTT tracks (GET /subtitle) — no session restart."""
it = item_or_404(db, rating_key)
if it.playable == "transcode":
raise HTTPException(
status_code=501, detail="This format needs transcoding (a later phase)."
)
transcode = it.playable == "transcode"
# `multi` (item has ≥2 audio tracks) forces the HLS path so ALL audio tracks ship as renditions and
# the client can switch audio without a restart — even for otherwise direct-playable files.
if it.playable == "direct" and audio is None and not aoff and not multi:
@@ -470,15 +581,23 @@ def stream_session(
}
if plex_paths.local_media_path(db, it.file_path) is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
s = plex_stream.start_session(db, it, start, audio, aoff, multi)
s = plex_stream.start_session(db, it, start, audio, aoff, multi, transcode=transcode)
if s is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
# Report the REAL media start (the keyframe ffmpeg landed on, which hls.js zero-bases to), not the
# requested offset — the client's absolute clock and the subtitle cue-shift both key off this.
# `transcode`/`hw`/`tonemap` let the player show an honest cold-start ("Preparing stream…") and
# surface the active mode + encoder in the stats overlay (`tonemap` = the HW HDR→SDR path); both
# segment streams are otherwise identical.
return {
"mode": "hls",
"start_s": s.media_start_s,
"url": f"/api/plex/stream/{it.rating_key}/hls/{s.entry}",
# The session id is part of the path (not a query param) so the playlist's RELATIVE segment
# names resolve back into the same session — see `stream_hls_file` for why it has to.
"url": f"/api/plex/stream/{it.rating_key}/hls/{s.id}/{s.entry}",
"transcode": s.transcode,
"hw": s.hw,
"tonemap": s.tonemap,
}
@@ -507,45 +626,82 @@ def stream_file(
return FileResponse(str(src)) # Starlette honours the Range header (206)
# What a session directory can contain, and nothing else. No `.vtt`: ffmpeg runs with `-sn` and
# subtitles are independent tracks served by GET /subtitle, so a WebVTT name here could only ever be
# a name that will not arrive — and accepting it cost a 20 s parked worker per request.
_HLS_CT = {
".m3u8": "application/vnd.apple.mpegurl",
".ts": "video/mp2t",
".vtt": "text/vtt",
}
# A real artifact name is ~20 characters. The bound is what keeps an absurd one out of the parser:
# `int()` on an unbounded digit run raises ValueError past 4300 digits (CPython's
# `sys.get_int_max_str_digits`), which reached the client as a 500 — the same shape of bug the
# subtitle routes were bounded for (`_RatingKey`, above).
_HLS_NAME_MAX = 128
@router.get("/stream/{rating_key}/hls/{filename}")
@router.get("/stream/{rating_key}/hls/{session_id}/{filename}")
def stream_hls_file(
rating_key: str,
session_id: str,
filename: str,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> Response:
"""Serve a file (playlist / segment / subtitle) from the current HLS session directory. The
playlist references its segments/subtitles by relative name, all resolving back here."""
"""Serve a file (playlist / segment / subtitle) from ONE HLS session's directory. The playlist
references its segments/subtitles by relative name, all resolving back here.
The session id in the path is what makes "back here" mean the same session. Sessions are keyed by
rating_key alone and a seek-restart or a second viewer of the same item replaces the registered
one, while the DIRECTORY is per (key, offset, tag) — so resolving by rating_key handed the old
viewer the new session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
bytes with a 200, which the player has no way to notice (deep-review round 5). A request for a
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
That reclaim is also why a request with no live session is no longer auto-started here: the
auto-start could only ever begin at 0, silently throwing away the viewer's position."""
# Filename must be a plain hls artifact — no path separators, only the expected extensions.
ext = "." + filename.rsplit(".", 1)[-1] if "." in filename else ""
if ext not in _HLS_CT or "/" in filename or "\\" in filename or ".." in filename:
if (
len(filename) > _HLS_NAME_MAX
or ext not in _HLS_CT
or "/" in filename
or "\\" in filename
or ".." in filename
):
raise HTTPException(status_code=404, detail="Not found")
s = plex_stream.current_session(str(rating_key))
if s is None:
# No session yet (e.g. reaped): the ENTRY playlist may auto-start one at 0 — master.m3u8 for a
# multi-audio item, index.m3u8 for a single-audio remux item.
it = item_or_404(db, rating_key)
if filename == "master.m3u8":
s = plex_stream.start_session(db, it, 0.0, multi=True)
elif filename == "index.m3u8" and it.playable == "remux":
s = plex_stream.start_session(db, it, 0.0)
else:
raise HTTPException(status_code=404, detail="No active playback session")
if s is None:
raise HTTPException(status_code=404, detail="Media file not found on disk")
if s is None or s.id != session_id:
raise HTTPException(status_code=404, detail="No active playback session")
path = s.dir / filename
timeout = 30 if ext == ".m3u8" else 20
if not plex_stream.wait_for(path, timeout):
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
# in order, so if it is not there this one is not next). The session is passed, not just its
# directory, because the single- and multi-audio argv branches spell their names differently and
# neither can produce the other's. Answering now instead of waiting it out is what keeps a seek
# past the produced region — or a deliberate flood of hopeless names — from parking threadpool
# workers and their DB connections (see `artifact_may_arrive`).
if not plex_stream.artifact_may_arrive(s, filename):
raise HTTPException(status_code=404, detail="Not available yet")
timeout = 30 if ext == ".m3u8" else 20
# `proc` so a session whose ffmpeg has already exited answers at once instead of polling out the
# whole timeout: this is a SYNC route on the ~40-slot threadpool, holding a pooled DB connection
# with it, so every such wait is a worker (and a connection) parked for 20 s — an ordinary seek
# past the produced region was enough to do it (round 5).
if not plex_stream.wait_for(path, timeout, proc=s.proc):
raise HTTPException(status_code=404, detail="Not available yet")
try:
content = path.read_bytes()
except OSError:
# The session was torn down between the wait and the read — `_kill` rmtrees the whole
# directory, and nothing holds `_lock` across those two steps. It has to present as the same
# 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
# instance's in-flight segment request races the rmtree on every seek-restart.
raise HTTPException(status_code=404, detail="No active playback session") from None
return Response(
content=path.read_bytes(),
content=content,
media_type=_HLS_CT[ext],
headers={"Cache-Control": "no-store"},
)
+1 -1
View File
@@ -25,7 +25,7 @@ router = APIRouter(prefix="/api/plex", tags=["plex"])
def _own_playlist_or_404(db: Session, pid: int, user: User) -> PlexPlaylist:
return owned_or_404(db, PlexPlaylist, pid, user, detail="Playlist not found")
return owned_or_404(db, PlexPlaylist, pid, user, what="playlist")
def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
+18 -2
View File
@@ -122,9 +122,25 @@ def watch_file(token: str, g: str | None = None, db: Session = Depends(get_db)):
raise HTTPException(status_code=404, detail="This video is no longer available.")
filename = storage.download_filename(asset.title or "video", asset.container, path)
# The served Content-Type comes from the asset's ffprobe-verified media_kind + a container
# allow-list (storage.served_media_type) — NEVER guessed from the file's own extension. A generic
# 'file' (and any media whose container we don't positively recognise, e.g. a media/HTML polyglot
# named "evil.html") returns None → a neutral force-download attachment, so arbitrary uploaded
# HTML/SVG can't execute inline on this login-free public origin. A recognised media container is
# served with its real audio/video type, inline unless the link forces a download so it can play
# in the watch page. Both are range-aware (FileResponse → 206) for seeking/resume.
media_type = storage.served_media_type(asset.media_kind, asset.container)
if media_type is None:
return FileResponse(
path,
filename=filename,
media_type="application/octet-stream",
content_disposition_type="attachment",
)
disposition = "attachment" if link.allow_download else "inline"
# Starlette's FileResponse honours the Range header (206) for seeking/scrubbing.
return FileResponse(path, filename=filename, content_disposition_type=disposition)
return FileResponse(
path, filename=filename, media_type=media_type, content_disposition_type=disposition
)
@router.get("/watch/{token}/poster.jpg")
+1 -1
View File
@@ -91,7 +91,7 @@ def create_view(
def _own_view(db: Session, user: User, view_id: int) -> SavedView:
return owned_or_404(db, SavedView, view_id, user, detail="Unknown view")
return owned_or_404(db, SavedView, view_id, user, what="view")
class UpdateViewIn(BaseModel):
+3 -2
View File
@@ -23,6 +23,7 @@ from app.scheduler import (
trigger_job,
)
from app.sync.runner import estimate_deep_backfill
from app.routes._common import not_found
router = APIRouter(prefix="/api/admin/scheduler", tags=["admin"])
@@ -41,7 +42,7 @@ def set_job_interval(
"""Change how often a scheduler job runs (minutes). Persisted as an override and applied
to the live scheduler immediately."""
if job_id not in JOB_INTERVALS:
raise HTTPException(status_code=404, detail="Unknown job")
raise not_found("job")
minutes = body.interval_minutes
if not (MIN_INTERVAL <= minutes <= MAX_INTERVAL):
raise HTTPException(
@@ -73,7 +74,7 @@ def run_job_now(
try:
result = trigger_job(job_id, actor_id=user.id)
except KeyError:
raise HTTPException(status_code=404, detail="Unknown job")
raise not_found("job")
if result == "already_running":
raise HTTPException(status_code=409, detail="Job is already running")
return {"id": job_id, "started": True}
+13 -2
View File
@@ -14,7 +14,9 @@ from sqlalchemy import select
from sqlalchemy.orm import Session
from app import email as email_mod
from app import mailtext
from app import state, sysconfig
from app.utils import canonical_email
from app.auth import PASSWORD_MIN_LEN
from app.db import get_db
from app.models import User
@@ -69,7 +71,13 @@ def setup_admin(body: SetupAdminIn, db: Session = Depends(require_setup)) -> dic
raise HTTPException(
status_code=400, detail=f"Password must be at least {PASSWORD_MIN_LEN} characters."
)
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
# Resolve canonically, not by the literal string (DX-17). The wizard can be re-run, and E9.5's
# whole point is that two spellings of one Gmail mailbox are one person — matching literally here
# would let a second run under the other spelling create the DUPLICATE admin row the epic exists
# to prevent. Every other lookup went canonical in S3; this one was outside that diff.
user = db.execute(
select(User).where(User.email_canonical == canonical_email(email))
).scalar_one_or_none()
if user is None:
user = User(email=email, role="admin")
db.add(user)
@@ -103,6 +111,9 @@ def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_set
class SetupTestEmailIn(BaseModel):
to: str = ""
# The wizard's page language. No account exists yet on a fresh instance, so this is the only
# thing that can say what language to write in; anything unsupported falls back to English.
lang: str | None = None
@router.post("/test-email")
@@ -111,7 +122,7 @@ def setup_test_email(body: SetupTestEmailIn, db: Session = Depends(require_setup
to = body.to.strip()
if "@" not in to:
raise HTTPException(status_code=400, detail="Enter a valid email address.")
if not email_mod.send_test(to):
if not email_mod.send_test(to, mailtext.normalize(body.lang)):
raise HTTPException(status_code=400, detail="Couldn't send — check the SMTP settings.")
return {"ok": True}
+10
View File
@@ -7,12 +7,20 @@ from app.audit import AuditAction
from app.auth import admin_user, current_user, has_read_scope, require_human
from app.db import get_db
from app.models import Channel, Subscription, User, Video
from app.ratelimit import RateLimiter
from app.sync.runner import estimate_deep_backfill
from app.sync.subscriptions import import_subscriptions
from app.scheduler import running_job_ids
router = APIRouter(prefix="/api/sync", tags=["sync"])
# SEC-14: the YouTube API budget is SHARED across the whole instance, and a subscription import
# walks every page of the caller's subscriptions — so one user holding down a manual sync button can
# spend the day's units for everyone. Keyed by user id, not client IP: the cost belongs to the
# account, and the same person on a second device is the same drain. 3 per 10 minutes leaves manual
# re-syncing entirely usable (the scheduler does the routine work anyway) while removing the loop.
_sync_limiter = RateLimiter(max_events=3, window_seconds=600)
@router.post("/subscriptions")
def sync_subscriptions(user: User = Depends(require_human), db: Session = Depends(get_db)) -> dict:
@@ -22,6 +30,8 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
status_code=403,
detail="Connect your YouTube account (read access) to import subscriptions.",
)
if not _sync_limiter.allow(str(user.id)):
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
with quota.measured(db, user.id, quota.QuotaAction.SUBSCRIPTIONS_PULL) as m:
result = import_subscriptions(db, user)
result["quota_used_estimate"] = m.used
+1 -1
View File
@@ -80,7 +80,7 @@ def create_tag(
def _own_tag(db: Session, user: User, tag_id: int) -> Tag:
# System tags (user_id NULL) are not user-editable → owned_or_404 404s them too.
return owned_or_404(db, Tag, tag_id, user, detail="Unknown tag")
return owned_or_404(db, Tag, tag_id, user, what="tag")
class TagUpdateIn(BaseModel):
+240 -71
View File
@@ -1,14 +1,16 @@
"""YouTube playlist sync, both directions. Read: mirror each user's own YouTube playlists
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: push local
edits out to YouTube (`plan_push` / `push_playlist`), behind the optional write scope.
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: two-way sync
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope.
YouTube's special Watch Later / History playlists are not exposed by the Data API and are
never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and
ingested (with a stub channel) so the mirror is faithful."""
import hashlib
import json
import logging
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TypeVar
@@ -42,6 +44,57 @@ def current_fingerprint(db: Session, pl: Playlist) -> str:
return fingerprint(pl.name, _local_video_ids(db, pl))
def _dump_ids(ids: list[str]) -> str:
"""Serialize the ordered id list stored as the 3-way-merge baseline (`synced_video_ids`)."""
return json.dumps(ids)
def _load_baseline(pl: Playlist) -> tuple[list[str] | None, set[str]]:
"""Parse `synced_video_ids` into (ordered id set, un-ingestable id set). Two on-disk forms:
a plain JSON list (legacy / no un-ingestable ids) or `{"ids": [...], "unavailable": [...]}`.
`unavailable` = YouTube-only videos that exist on YouTube but the Data API won't return
(private/deleted/region-locked), so they never became local rows — tracked so the merge can
skip them instead of forever re-classifying them as a YouTube-add (they'd never converge)."""
if not pl.synced_video_ids:
return None, set()
try:
val = json.loads(pl.synced_video_ids)
except (ValueError, TypeError):
return None, set()
if isinstance(val, list):
return [str(v) for v in val], set()
if isinstance(val, dict):
raw_ids = val.get("ids")
ids = [str(v) for v in raw_ids] if isinstance(raw_ids, list) else None
raw_un = val.get("unavailable")
un = {str(v) for v in raw_un} if isinstance(raw_un, list) else set()
return ids, un
return None, set()
def parse_baseline(pl: Playlist) -> list[str] | None:
"""The last-synced ordered id set, or None when no baseline has been recorded yet (a fresh
export, or a link created before S7). None makes the first merge union losslessly."""
return _load_baseline(pl)[0]
def parse_unavailable(pl: Playlist) -> set[str]:
"""The last-synced set of un-ingestable-but-live YouTube ids (see `_load_baseline`)."""
return _load_baseline(pl)[1]
def _set_baseline(pl: Playlist, ids: list[str], unavailable: list[str] | None = None) -> None:
"""Record `ids` as both the hash (drives `dirty`, over the local ids only) and the raw ordered
set (drives merge). `unavailable` = un-ingestable-but-live YouTube ids to carry forward so the
next merge skips them; stored only when non-empty (keeps the common case a plain list)."""
pl.synced_fingerprint = fingerprint(pl.name, ids)
if unavailable:
pl.synced_video_ids = json.dumps({"ids": ids, "unavailable": list(unavailable)})
else:
pl.synced_video_ids = _dump_ids(ids)
pl.dirty = False
def recompute_dirty(db: Session, pl: Playlist) -> None:
"""Set `dirty` by comparing the current state to the synced baseline. Unlinked lists
(and Watch later) are never dirty; a missing baseline counts as dirty (unknown YT state)."""
@@ -116,12 +169,18 @@ def _replace_items_from_youtube(
if v in present and v not in seen:
seen.add(v)
ordered.append(v)
_set_local_items(db, pl, ordered)
# After a pull/reset local == YouTube, so this ordered set IS the new merge baseline.
_set_baseline(pl, ordered)
return ordered
def _set_local_items(db: Session, pl: Playlist, ordered: list[str]) -> None:
"""Authoritatively replace `pl`'s local items with `ordered` (positions 0..n). The caller
is responsible for having ensured every id exists in the catalog."""
db.execute(delete(PlaylistItem).where(PlaylistItem.playlist_id == pl.id))
for pos, vid in enumerate(ordered):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=pos))
pl.synced_fingerprint = fingerprint(pl.name, ordered)
pl.dirty = False
return ordered
def sync_user_playlists(db: Session, user: User) -> dict:
@@ -216,22 +275,91 @@ def _reorder_moves(current: list[str], desired: list[str]) -> int:
return moves
def plan_push(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what pushing this local playlist to YouTube would do:
create vs update, how many inserts/deletes/reorders, the quota estimate, and whether
YouTube has diverged (items present there that local will remove). Costs read units
(1/page) for a linked playlist; nothing for a fresh export. `live` = a pre-fetched
(current_items, snippet) tuple so the caller can share one read with push_playlist."""
desired = _local_video_ids(db, pl)
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what a two-way sync of this playlist would do: create vs
update, how many items each side gains/loses, reorders, and the quota estimate (YouTube
writes only — local edits are free). Reads live YouTube state (1 unit/page) for a linked
playlist; nothing for a fresh export. `live` = a pre-fetched (current_items, snippet) tuple
so the caller can share one read with merge_playlist."""
local = _local_video_ids(db, pl)
if not pl.yt_playlist_id:
ops = 1 + len(desired) # create the playlist + one insert per video
ops = 1 + len(local) # create the playlist + one insert per video
return {
"action": "create",
"to_insert": len(desired),
"to_delete": 0,
"add_yt": len(local),
"add_local": 0,
"remove_yt": 0,
"remove_local": 0,
"to_reorder": 0,
"rename": False,
"yt_extra": 0,
"units_estimate": ops * WRITE_UNIT_COST,
}
if live is not None:
@@ -241,23 +369,17 @@ def plan_push(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
rename = bool(snippet) and (snippet.get("title") or "") != pl.name
cur_vids = [it["video_id"] for it in current]
cur_set = set(cur_vids)
desired_set = set(desired)
to_insert = [v for v in desired if v not in cur_set]
to_delete = [v for v in cur_vids if v not in desired_set]
# After membership reconciliation, the YT order is current-minus-deletes plus inserts
# appended; the reorder pass then moves items into the desired order.
after_membership = [v for v in cur_vids if v in desired_set] + to_insert
reorders = _reorder_moves(after_membership, desired)
ops = len(to_insert) + len(to_delete) + reorders + (1 if rename else 0)
yt_ids = [it["video_id"] for it in current]
mp = merge_plan(parse_baseline(pl), local, yt_ids, parse_unavailable(pl))
ops = len(mp.add_to_yt) + len(mp.remove_from_yt) + mp.reorders + (1 if rename else 0)
return {
"action": "update",
"to_insert": len(to_insert),
"to_delete": len(to_delete),
"to_reorder": reorders,
"add_yt": len(mp.add_to_yt),
"add_local": len(mp.add_to_local),
"remove_yt": len(mp.remove_from_yt),
"remove_local": len(mp.remove_from_local),
"to_reorder": mp.reorders,
"rename": rename,
"yt_extra": len(to_delete),
"units_estimate": ops * WRITE_UNIT_COST,
}
@@ -327,13 +449,16 @@ class _Breaker:
return out
def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Push a local playlist to YouTube: create it if needed, then reconcile membership
and order so YouTube matches local (local wins). Marks the playlist clean on success.
def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Two-way sync a local playlist with YouTube: create it on first push, else 3-way MERGE
both sides so an item added (or deleted) on either side survives — no side wins wholesale.
Local edits are applied directly; YouTube edits go through the failure breaker. On full
success the merged set becomes the new baseline and the playlist is marked clean; on any
YouTube failure the baseline is left untouched so the next sync retries and converges.
Caller must have checked the write scope and the quota budget. `live` = a pre-fetched
(current_items, snippet) tuple (from plan_push's read) to avoid a second live page-through."""
desired = _local_video_ids(db, pl)
inserted = deleted = reordered = created = 0
(current_items, snippet) tuple (from plan_merge's read) to avoid a second page-through."""
local = _local_video_ids(db, pl)
inserted = deleted = reordered = created = pulled = removed_local = 0
failures: list[str] = []
breaker = _Breaker()
with YouTubeClient(db, user) as yt:
@@ -341,8 +466,8 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
pl.yt_playlist_id = yt.create_playlist(pl.name)
db.commit()
created = 1
# Brand-new playlist: just append every video in order.
for vid in desired:
# Brand-new playlist: just append every local video in order.
for vid in local:
added = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
@@ -350,60 +475,95 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
if not isinstance(added, _Failed):
inserted += 1
if not failures:
pl.synced_fingerprint = fingerprint(pl.name, desired)
pl.dirty = False
_set_baseline(pl, local)
db.commit()
return {
"created": created,
"inserted": inserted,
"deleted": deleted,
"reordered": reordered,
"pulled": pulled,
"removed_local": removed_local,
"failures": failures,
"aborted": breaker.tripped,
"yt_playlist_id": pl.yt_playlist_id,
}
# Existing link: diff against the live YouTube state (reuse plan_push's read if given).
# Existing link: 3-way merge against the live YouTube state (reuse plan_merge's read).
if live is not None:
current, snippet = live
else:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
yt_ids = [it["video_id"] for it in current]
# Re-attempt ingestion of ids previously marked un-ingestable that are still on YouTube: a
# video that has since become available (e.g. private→public) should re-enter the merge, not
# stay skipped forever. Those that still won't ingest remain filtered (`effective_unavailable`).
prev_unavailable = parse_unavailable(pl)
still_listed = [v for v in yt_ids if v in prev_unavailable]
if still_listed:
_ensure_videos(db, yt, still_listed)
healed = set(
db.execute(select(Video.id).where(Video.id.in_(still_listed))).scalars().all()
)
effective_unavailable = prev_unavailable - healed
else:
effective_unavailable = prev_unavailable
mp = merge_plan(parse_baseline(pl), local, yt_ids, effective_unavailable)
# --- Siftlode side (free, direct): pull YouTube-adds in, drop YouTube-deletes, land the
# merged order authoritatively. Do this first so local reflects the merge even if a later
# YouTube write trips the breaker.
_ensure_videos(db, yt, mp.add_to_local)
# Only the ids that actually exist in the catalog can become local rows (a private/deleted
# YouTube video won't ingest); keep the merged order, dropping any that didn't materialize.
present = (
set(db.execute(select(Video.id).where(Video.id.in_(mp.final))).scalars().all())
if mp.final
else set()
)
final_local = [v for v in mp.final if v in present]
pulled = len([v for v in mp.add_to_local if v in present])
removed_local = len(mp.remove_from_local)
_set_local_items(db, pl, final_local)
db.commit()
# --- YouTube side (metered, breaker-guarded).
if snippet is not None and (snippet.get("title") or "") != pl.name:
try:
yt.update_playlist_title(pl.yt_playlist_id, pl.name)
except YouTubeError:
failures.append("title")
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
cur_vids = [it["video_id"] for it in current]
desired_set = set(desired)
# Delete extras (local is authoritative).
for it in current:
if it["video_id"] not in desired_set:
dropped = breaker.run(
lambda i=it: yt.delete_playlist_item(i["item_id"]),
lambda i=it: failures.append(i["video_id"]),
)
if not isinstance(dropped, _Failed):
deleted += 1
# Insert missing (append; the reorder pass below fixes positions).
cur_set = set(cur_vids)
for vid in desired:
if vid not in cur_set:
item_id = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
)
if not isinstance(item_id, _Failed):
item_id_by_vid[vid] = item_id
inserted += 1
# Reorder to match `desired` via insertion-sort moves over a local model.
model = [v for v in cur_vids if v in desired_set] + [
v for v in desired if v not in cur_set and v in item_id_by_vid
# Propagate genuine local deletes.
for vid in mp.remove_from_yt:
item_id = item_id_by_vid.get(vid)
if item_id is None:
continue
dropped = breaker.run(
lambda i=item_id: yt.delete_playlist_item(i),
lambda v=vid: failures.append(v),
)
if not isinstance(dropped, _Failed):
deleted += 1
item_id_by_vid.pop(vid, None)
# Insert local-adds (append; the reorder pass below fixes positions).
for vid in mp.add_to_yt:
item_id = breaker.run(
lambda v=vid: yt.add_playlist_item(pl.yt_playlist_id, v),
lambda v=vid: failures.append(v),
)
if not isinstance(item_id, _Failed):
item_id_by_vid[vid] = item_id
inserted += 1
# Reorder YouTube to match `final` via insertion-sort moves over a local model.
final_set = set(mp.final)
model = [v for v in yt_ids if v in final_set and v in item_id_by_vid] + [
v for v in mp.add_to_yt if v in item_id_by_vid
]
for i, want in enumerate(desired):
for i, want in enumerate(mp.final):
if want not in item_id_by_vid:
continue # failed insert; skip
continue # deleted-from-YT or failed insert; skip
if i < len(model) and model[i] == want:
continue
moved = breaker.run(
@@ -419,14 +579,23 @@ def push_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
model.pop(j)
model.insert(i, want)
if not failures:
pl.synced_fingerprint = fingerprint(pl.name, desired)
pl.dirty = False
# Carry forward un-ingestable-but-live YouTube ids so the next merge skips them
# (see merge_plan): this run's failed pulls, plus prior ones still on YouTube, minus
# any that have since left YouTube. Recomputed each success → self-healing.
yt_id_set = set(yt_ids)
unavailable = sorted(
(effective_unavailable | {v for v in mp.add_to_local if v not in present})
& yt_id_set
)
_set_baseline(pl, final_local, unavailable)
db.commit()
return {
"created": created,
"inserted": inserted,
"deleted": deleted,
"reordered": reordered,
"pulled": pulled,
"removed_local": removed_local,
"failures": failures,
"aborted": breaker.tripped,
"yt_playlist_id": pl.yt_playlist_id,
+14
View File
@@ -127,6 +127,10 @@ SPECS: tuple[ConfigSpec, ...] = (
min=1,
max=100_000,
),
# min=1, NOT 0: the per-file cap is enforced as `if cap and size > cap`, so a stored 0 would read
# as falsy and SILENTLY disable the limit (an unbounded streamed write to the volume). Floor it at
# 1 so the cap is always active; the overall footprint is separately bounded by the byte quota.
ConfigSpec("download_upload_max_bytes", "int", "downloads", "download_upload_max_bytes", min=1),
ConfigSpec(
"download_retention_days", "int", "downloads", "download_retention_days", min=0, max=3_650
),
@@ -155,6 +159,16 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
# Transcode re-encode tuning (E6 S1). hw = "auto"|"on"|"off" (free-text, like search_source); the
# libx264 knobs apply only to the software fallback. crf 0–51 (x264 range); max_height 0 = source.
ConfigSpec("plex_transcode_hw", "str", "plex", "plex_transcode_hw"),
ConfigSpec("plex_transcode_preset", "str", "plex", "plex_transcode_preset"),
ConfigSpec("plex_transcode_crf", "int", "plex", "plex_transcode_crf", min=0, max=51),
ConfigSpec(
"plex_transcode_max_height", "int", "plex", "plex_transcode_max_height", min=0, max=4320
),
# Audio loudness normalisation on the HLS path (remux + transcode alike, not direct play).
ConfigSpec("plex_audio_loudnorm", "bool", "plex", "plex_audio_loudnorm"),
# NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
# (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
# --- TMDB enrichment (E5 S2; optional) — the API key is the only secret ---
+33
View File
@@ -22,6 +22,39 @@ def valid_email(value: str | None) -> bool:
return bool(value and _EMAIL_RE.match(value))
# Gmail (and its googlemail.com alias) delivers to one mailbox regardless of dots in the local part
# and of any `+tag`; nobody else is guaranteed to. So the canonical form below strips those for
# Gmail only, and merely lowercases everything else.
_GMAIL_DOMAINS = {"gmail.com", "googlemail.com"}
def canonical_email(value: str | None) -> str:
"""The form used to decide whether two addresses are the SAME MAILBOX. Never used for sending —
mail always goes to the address as the user typed it.
Why this exists (observed on prod, 2026-09-03): someone registered as `npeter83.temu@gmail.com`
and was approved, but Google's OAuth `email` claim came back as the account's stored primary,
`npeter83temu@gmail.com`. The invite gate only lowercased, found no approved row for that literal
string, and turned an approved user's sign-in into a silent SECOND access request — plus a second
User row. The `+tag` case is worse and fails every time: a Google account cannot contain `+`, so
OAuth can only ever return the base address, meaning anyone who registers with
`user+siftlode@gmail.com` (a common throwaway habit) is guaranteed to be denied after approval.
Scope is deliberately narrow. The dot rule does NOT hold for Google Workspace custom domains
unless the admin turned it on, and dots are significant on most providers, so anything outside
gmail.com/googlemail.com is only lowercased. Approving one Gmail spelling always did mean
approving that mailbox, so collapsing them is the honest reading of an invite, not a widening
of it.
"""
email = (value or "").strip().lower()
local, sep, domain = email.partition("@")
if not sep or domain not in _GMAIL_DOMAINS:
return email
local = local.partition("+")[0].replace(".", "")
# Normalise the alias domain too, so googlemail and gmail spellings meet.
return f"{local}@gmail.com" if local else email
def now_utc() -> datetime:
"""Current time as a timezone-aware UTC datetime."""
return datetime.now(timezone.utc)
+1
View File
@@ -43,6 +43,7 @@ pydantic-settings==2.15.0
pydantic==2.13.4
pydantic_core==2.46.4
python-dotenv==1.2.3
python-multipart==0.0.32
requests==2.34.2
starlette==1.6.0
typing-inspection==0.4.4
+3
View File
@@ -1,4 +1,7 @@
fastapi>=0.115,<1.0
# Multipart form parsing — required by Starlette/FastAPI to accept file uploads (UploadFile/Form).
# The download center's local-file Upload (E7) is the only multipart endpoint in the app.
python-multipart>=0.0.9,<1.0
uvicorn[standard]>=0.30,<1.0
sqlalchemy>=2.0,<2.1
psycopg[binary]>=3.2,<4.0
+21
View File
@@ -189,3 +189,24 @@ def as_user(db, override_dep, _configured):
return TestClient(app, headers={"x-test-user": str(user.id)})
return _as
@pytest.fixture(autouse=True)
def _reset_rate_limiters():
"""Every RateLimiter is a module-level singleton keyed by client IP, and TestClient always
presents the same host — so without this the whole suite's login/register attempts pool into one
300s window and, past the cap, some unrelated test starts failing with a 429 depending on order.
Cleared before each test; a test that wants to exercise throttling can still exhaust one itself.
Sweeps every loaded `app.*` module rather than a hand-kept list, so a limiter added in a new
module is covered the day it lands instead of silently pooling until something fails."""
import sys
from app.ratelimit import RateLimiter
for name, mod in list(sys.modules.items()):
if name != "app" and not name.startswith("app."):
continue
for obj in vars(mod).values():
if isinstance(obj, RateLimiter):
obj.reset()
+112
View File
@@ -0,0 +1,112 @@
"""E12 · S3 — the route-level halves of SEC-12 and SEC-14.
`test_urlguard.py` pins the address judgement itself; this pins that the ENQUEUE route actually
consults it and answers in the structured `{code, reason}` shape the frontend localizes, and that a
manual sync trigger stops spending the instance-wide YouTube budget when held down.
DB-backed (the `as_user` fixture needs real rows) — skipped when no database is reachable.
"""
from app.downloads import urlguard
def test_enqueueing_a_loopback_url_is_refused_with_a_structured_detail(
db, as_user, make_user, monkeypatch
):
user = make_user()
db.commit()
monkeypatch.setattr(
urlguard.socket, "getaddrinfo", lambda *a, **kw: [(None, None, None, "", ("127.0.0.1", 0))]
)
r = as_user(user).post("/api/downloads", json={"source": "http://redis.internal:6379/x"})
assert r.status_code == 422
# The shape matters as much as the refusal: a bare English string under a bilingual UI is the
# thing the canonical-patterns table forbids.
assert r.json()["detail"] == {"code": "url_blocked", "reason": urlguard.REASON_PRIVATE}
def test_a_public_url_gets_past_the_guard(db, as_user, make_user, monkeypatch):
# Mutation check for the test above: with a public answer the request must fail LATER (or
# succeed), never with url_blocked — otherwise the guard would be refusing everything and the
# first test would still pass.
user = make_user()
db.commit()
monkeypatch.setattr(
urlguard.socket,
"getaddrinfo",
lambda *a, **kw: [(None, None, None, "", ("93.184.216.34", 0))],
)
r = as_user(user).post("/api/downloads", json={"source": "https://example.com/v"})
detail = r.json().get("detail") if r.status_code >= 400 else None
assert not (isinstance(detail, dict) and detail.get("code") == "url_blocked")
def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeypatch):
# A YouTube job resolves to a fixed public host from an 11-char id, so there is no
# user-controlled destination — resolving one would be pure latency on every enqueue.
def boom(*a, **kw):
raise AssertionError("the URL guard resolved DNS for a YouTube source")
user = make_user()
db.commit()
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
as_user(user).post("/api/downloads", json={"source": "dQw4w9WgXcQ"})
def test_manual_subscription_sync_goes_on_cooldown(db, as_user, make_user, monkeypatch):
from app.routes import sync as syncroutes
user = make_user()
db.commit()
# Neither the YouTube grant nor the import itself is under test — the limiter is.
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
c = as_user(user)
codes = [c.post("/api/sync/subscriptions").status_code for _ in range(4)]
# 3 per 10 minutes: the first three pass, the fourth is refused.
assert codes[:3] == [200, 200, 200]
assert codes[3] == 429
assert c.post("/api/sync/subscriptions").json()["detail"] == {"code": "sync_cooldown"}
def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch):
# Keyed by user id on purpose: the shared YouTube budget is spent per account, and one user
# exhausting their allowance must not lock everyone else out of their own sync button.
from app.routes import sync as syncroutes
a, b = make_user(), make_user()
db.commit()
monkeypatch.setattr(syncroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
ca = as_user(a)
for _ in range(4):
ca.post("/api/sync/subscriptions")
assert ca.post("/api/sync/subscriptions").status_code == 429
assert as_user(b).post("/api/sync/subscriptions").status_code == 200
def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_user, monkeypatch):
# SEC-14, the sweep half: `revert-youtube` spends under the SAME QuotaAction.PLAYLISTS_PULL key
# as the bulk `sync-youtube`, so limiting only one of them left the identical drain open one
# route over. Both must draw on the one allowance.
from app.models import Playlist
from app.routes import playlists as plroutes
user = make_user()
pl = Playlist(user_id=user.id, name="Linked", source="youtube", yt_playlist_id="PL123")
db.add(pl)
db.commit()
monkeypatch.setattr(plroutes, "has_read_scope", lambda u: True)
monkeypatch.setattr(plroutes, "repull_playlist", lambda d, u, p: {"ok": True})
monkeypatch.setattr(plroutes, "sync_user_playlists", lambda d, u: {"ok": True})
c = as_user(user)
codes = [c.post(f"/api/playlists/{pl.id}/revert-youtube").status_code for _ in range(4)]
assert codes[:3] == [200, 200, 200]
assert codes[3] == 429
# One allowance, not one each: the bulk route is now on cooldown too.
r = c.post("/api/playlists/sync-youtube")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "sync_cooldown"}
+170
View File
@@ -0,0 +1,170 @@
"""Deleting a whitelist / access-request entry (admin).
Deny records an auditable decision and keeps the row; delete forgets the address outright, for an
approval added by mistake or one that is no longer wanted. Neither stops the person asking again —
`upsert_pending_invite` reopens a denied row as pending, and a deleted one comes back as a fresh
request. The one hard rule is that an entry backed by an account is NOT deletable here: this row is
that account's approval gate, and orphaning it would strand someone who could then neither sign in
nor be approved.
"""
from sqlalchemy import select
from app.models import Invite, User
def _invites(client):
r = client.get("/api/admin/invites")
assert r.status_code == 200
return {i["email"]: i for i in r.json()}
def test_admin_deletes_a_whitelist_entry_that_never_became_an_account(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="mistake@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)["mistake@example.com"]
assert listed["has_account"] is False # drives the delete affordance in the UI
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 200 and r.json()["deleted"] == listed["id"]
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == "mistake@example.com")).scalar_one_or_none()
is None
)
def test_deleting_an_entry_that_has_an_account_is_refused(db, make_user, as_user):
# The guard that matters: this row is what is_allowed() reads, so deleting it would revoke the
# user's access without touching the account or saying so.
admin = make_user(email="admin@example.com", role="admin")
member = make_user(email="member@example.com")
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)[member.email]
assert listed["has_account"] is True
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 409
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none()
is not None
)
def test_a_pending_request_from_a_registration_is_not_deletable(db, make_user, as_user):
# The shape the ordinary sign-up produces: _register_account writes the User (inactive) AND the
# pending Invite together. The invite is that account's approval gate, so deleting it would
# strand the person — they could neither sign in nor be approved, and re-registering is a silent
# no-op. Managed from Users & roles instead, which is what the UI and the 409 both say.
admin = make_user(email="admin@example.com", role="admin")
make_user(email="signup@example.com", is_active=False, email_verified=False)
db.add(Invite(email="signup@example.com", status="pending"))
db.commit()
client = as_user(admin)
listed = _invites(client)["signup@example.com"]
assert listed["has_account"] is True # the UI shows a disabled control, not a missing one
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 409
db.expire_all()
assert (
db.execute(select(Invite).where(Invite.email == "signup@example.com")).scalar_one_or_none()
is not None
)
def test_deleting_an_env_allow_listed_address_reports_that_it_kept_access(
db, make_user, as_user, monkeypatch
):
# Removing the row is NOT always a revoke: is_allowed falls back to the env bootstrap lists once
# no invite exists. The admin has to be told, or they think they removed access and did not.
from app.config import settings
# Patch the raw env field, not the derived property: that keeps the split/strip/lower parsing
# under test, and matches how the rest of the suite stubs settings.
monkeypatch.setattr(settings, "allowed_emails", " OPS@Example.com , someone@else.test ")
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="ops@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
listed = _invites(client)["ops@example.com"]
# The LIST carries it too, so the confirm dialog can warn before the delete happens.
assert listed["still_allowed_by_env"] is True
r = client.delete(f"/api/admin/invites/{listed['id']}")
assert r.status_code == 200
assert r.json()["still_allowed_by_env"] is True
def test_an_admin_emails_address_is_also_reported(db, make_user, as_user, monkeypatch):
# ADMIN_EMAILS is the sharper case: deleting the row not only keeps access, the next Google
# sign-in re-promotes the address to admin.
from app.config import settings
monkeypatch.setattr(settings, "admin_emails", "boss@example.com")
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="boss@example.com", status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
assert _invites(client)["boss@example.com"]["still_allowed_by_env"] is True
def test_deleting_a_pending_request_forgets_it(db, make_user, as_user):
# A pending row is deletable too — "drop this without recording a decision" is a different
# choice from Deny, which stores a decision (status, decided_by) and mails the requester.
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="asker@example.com", status="pending"))
db.commit()
client = as_user(admin)
r = client.delete(f"/api/admin/invites/{_invites(client)['asker@example.com']['id']}")
assert r.status_code == 200
assert r.json()["still_allowed_by_env"] is False # nothing in the env allow-lists
assert "asker@example.com" not in _invites(client)
def test_deleting_a_missing_invite_is_404(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
assert as_user(admin).delete("/api/admin/invites/9999").status_code == 404
def test_a_regular_user_cannot_delete_an_invite(db, make_user, as_user):
make_user(email="admin@example.com", role="admin")
plain = make_user(email="plain@example.com")
db.add(Invite(email="mistake@example.com", status="approved"))
db.commit()
inv_id = db.execute(select(Invite.id).where(Invite.email == "mistake@example.com")).scalar_one()
assert as_user(plain).delete(f"/api/admin/invites/{inv_id}").status_code == 403
db.expire_all()
assert db.get(Invite, inv_id) is not None
def test_deleting_the_user_also_clears_the_way_to_re_invite(db, make_user, as_user):
# The refusal above tells the admin to delete the account instead; purge_user takes the invite
# with it, so the address really is forgotten and a fresh request starts over as pending.
admin = make_user(email="admin@example.com", role="admin")
member = make_user(email="member@example.com")
db.add(Invite(email=member.email, status="approved", decided_by=admin.email))
db.commit()
client = as_user(admin)
assert client.delete(f"/api/admin/users/{member.id}").status_code == 200
db.expire_all()
assert db.execute(select(User).where(User.email == member.email)).scalar_one_or_none() is None
assert (
db.execute(select(Invite).where(Invite.email == member.email)).scalar_one_or_none() is None
)
+203
View File
@@ -0,0 +1,203 @@
"""The admin lifeline for a sign-up that is stuck outside (E9.5 S5, roadmap UX-25), DB-lane.
Approve an address BEFORE the person registers and their sign-up produces no queue row at all —
`_decide` only takes a still-pending invite — so the Access-requests screen stayed empty while they
sat locked out, and the badge read zero. The only working move was re-typing the address into the
whitelist box, which nobody would guess.
Two earlier attempts at this were reverted after review, and both moved AUTHORITY rather than
visibility: auto-activating on registration, and reopening the approved invite to pending. So the
tests here pin the boundary as much as the feature — an admin click, gated on an address that is
already allowed, and nothing an unauthenticated caller can reach.
"""
from sqlalchemy import select
from app.models import AuditLog, AuthToken, Invite, User
_PW = "correct horse battery"
def _pending_access(client):
r = client.get("/api/admin/pending-access")
assert r.status_code == 200
return {u["email"]: u for u in r.json()}
def _stuck_registrant(db, make_user, admin):
"""The UX-25 shape: the admin approved the address first, the account arrived afterwards."""
db.add(Invite(email="later@example.com", status="approved", decided_by=admin.email))
db.commit()
return make_user(email="later@example.com", password=_PW, is_active=False)
def test_a_pre_approved_registrant_is_listed_and_offers_activation(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
_stuck_registrant(db, make_user, admin)
row = _pending_access(as_user(admin))["later@example.com"]
assert row["is_active"] is False
assert (
row["can_activate"] is True
) # the address is already allowed — the click just finishes it
def test_an_ordinary_registrant_awaiting_a_decision_is_not_listed_twice(db, make_user, as_user):
# They already have a pending invite with Approve on it in the queue above, and their address is
# not allowed YET — the decision is the queue's to own. Caught in UAT on 2026-09-05: an earlier
# filter keyed on "unconfirmed" put every fresh registration in both lists at once.
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="normal@example.com", status="pending"))
db.commit()
make_user(email="normal@example.com", password=_PW, is_active=False)
assert "normal@example.com" not in _pending_access(as_user(admin))
def test_a_fresh_registration_stays_in_the_queue_and_out_of_this_list(db, make_user, as_user):
# The realistic version of the case above: a brand-new sign-up is inactive AND unconfirmed. It
# still belongs only to the queue — the admin's next act is the decision, not a re-send.
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="fresh@example.com", status="pending"))
db.commit()
make_user(email="fresh@example.com", password=_PW, is_active=False, email_verified=False)
assert "fresh@example.com" not in _pending_access(as_user(admin))
def test_activating_lets_them_in_and_is_recorded(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
stuck = _stuck_registrant(db, make_user, admin)
r = as_user(admin).post(f"/api/admin/users/{stuck.id}/activate")
assert r.status_code == 200 and r.json()["is_active"] is True
db.expire_all()
assert db.get(User, stuck.id).is_active is True
actions = db.execute(select(AuditLog.action)).scalars().all()
assert (
"user_activate" in actions
) # the admin acted on someone's access; it is not a silent read
def test_activating_an_address_that_is_not_allowed_is_refused(db, make_user, as_user):
# The guard that keeps this button from manufacturing an inconsistent account: password sign-in
# only checks is_active and would let them in, while their Google sign-in would be turned away
# and silently reopened as a fresh access request. Whitelisting is the move that says "let them
# in"; this one only finishes a decision already made.
admin = make_user(email="admin@example.com", role="admin")
nobody = make_user(email="nobody@example.com", password=_PW, is_active=False)
r = as_user(admin).post(f"/api/admin/users/{nobody.id}/activate")
assert r.status_code == 409
assert r.json()["detail"]["reason"] == "not_allowed"
db.expire_all()
assert db.get(User, nobody.id).is_active is False
def test_activating_twice_is_not_an_error(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
stuck = _stuck_registrant(db, make_user, admin)
client = as_user(admin)
assert client.post(f"/api/admin/users/{stuck.id}/activate").status_code == 200
assert client.post(f"/api/admin/users/{stuck.id}/activate").status_code == 200
def test_a_suspended_account_is_not_offered_as_stuck(db, make_user, as_user):
# It cannot sign in either, but that is a decision already made — not a person waiting on one.
admin = make_user(email="admin@example.com", role="admin")
db.add(Invite(email="banned@example.com", status="approved", decided_by=admin.email))
db.commit()
make_user(email="banned@example.com", password=_PW, is_active=False, is_suspended=True)
assert "banned@example.com" not in _pending_access(as_user(admin))
# --- the confirmation resend ---------------------------------------------------------------------
def test_an_unconfirmed_account_is_listed_and_can_be_sent_a_fresh_link(
db, make_user, as_user, monkeypatch
):
from app.routes import admin as admin_routes
from app.routes import auth as auth_routes
monkeypatch.setattr(admin_routes.email_mod, "email_enabled", lambda db=None: True)
sent: list[tuple[str, str]] = []
monkeypatch.setattr(
auth_routes.email_mod,
"send_verify_email",
lambda to, lang, link, **kw: sent.append((to, link)) is None,
)
admin = make_user(email="admin@example.com", role="admin")
# Active but unconfirmed — a real state since E9.5 S1 made the two gates order-independent. It is
# not the admin's to decide, but the mail may simply never have arrived, so it is theirs to help
# with. The approved invite is what puts them on this screen at all: the list is "the address may
# sign in, this account still can't", so an undecided request stays with the queue that owns it.
db.add(Invite(email="unconfirmed@example.com", status="approved", decided_by=admin.email))
db.commit()
u = make_user(email="unconfirmed@example.com", password=_PW, email_verified=False)
assert "unconfirmed@example.com" in _pending_access(as_user(admin))
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
assert r.status_code == 200 and r.json()["email"] == "unconfirmed@example.com"
assert [to for to, _ in sent] == ["unconfirmed@example.com"]
tokens = db.execute(select(AuthToken).where(AuthToken.user_id == u.id)).scalars().all()
assert [t.kind for t in tokens] == ["verify"]
def test_the_resend_says_so_when_the_instance_cannot_send_mail(db, make_user, as_user, monkeypatch):
from app.routes import admin as admin_routes
monkeypatch.setattr(admin_routes.email_mod, "email_enabled", lambda db=None: False)
admin = make_user(email="admin@example.com", role="admin")
u = make_user(email="unconfirmed@example.com", password=_PW, email_verified=False)
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
# Mail is fail-soft everywhere else, which is exactly what left the 0.78.5 sign-ups with nothing
# to fall back on. An admin pressing a button gets the failure, not a success toast.
assert r.status_code == 409 and r.json()["detail"]["reason"] == "email_off"
def test_resending_to_an_already_confirmed_account_is_refused(db, make_user, as_user):
admin = make_user(email="admin@example.com", role="admin")
u = make_user(email="fine@example.com", password=_PW)
r = as_user(admin).post(f"/api/admin/users/{u.id}/resend-verification")
assert r.status_code == 409 and r.json()["detail"]["reason"] == "not_applicable"
# --- the badge -----------------------------------------------------------------------------------
def test_the_badge_counts_stuck_sign_ups_but_not_ones_the_queue_already_shows(
db, make_user, client
):
# Signed in for real, not via the as_user override: /api/me resolves its user through
# optional_current_user, which that override does not cover (see test_me_email_flag).
admin = make_user(email="admin@example.com", password=_PW, role="admin")
_stuck_registrant(db, make_user, admin) # approved first → counts
db.add(Invite(email="normal@example.com", status="pending")) # already in the queue → doesn't
db.commit()
make_user(email="normal@example.com", password=_PW, is_active=False)
make_user(email="unconfirmed@example.com", password=_PW, email_verified=False) # not ours
assert (
client.post(
"/auth/password-login", json={"email": admin.email, "password": _PW}
).status_code
== 200
)
me = client.get("/api/me").json()
assert me["authenticated"] is True
# An unconfirmed account waits on its OWNER, not on the admin — listing it is help, badging it
# would be nagging about something the admin did not cause.
assert me["pending_activations"] == 1
assert me["pending_invites"] == 1 # unchanged meaning: still just the undecided requests
+442 -14
View File
@@ -97,28 +97,85 @@ def test_reset_confirm_is_single_use_and_kills_existing_sessions(client, make_us
assert replay.status_code == 400
def test_activating_an_unverified_account_drops_its_preset_password(db, make_user):
# SEC-10 (account pre-emption): an attacker registers victim@x with a password before the approved
# owner arrives. The address was never verified, so admin approval must NOT hand that pre-set
# password a working login — it is dropped and any session it minted is revoked.
def test_approving_an_unverified_account_keeps_its_password(db, make_user):
# E9.5 S1, inverting an earlier rule. Approval used to drop a password set while the email was
# unverified (SEC-10, account pre-emption). It was dropped because it destroyed the password a
# legitimate registrant had just chosen, in what is the NORMAL ordering — the admin decides
# before the user reads their mail — making "your password was cleared for security" the default
# experience.
#
# S1 ALSO argued the revoke was redundant, because password_login refuses an unverified account
# before it reads is_active. The epic's closing review refuted that: the gate holds only while
# the address is unverified, and S6's approval mail exists to get the real owner to end that
# state. Do not restore that reasoning from this comment — the residual is SEC-16, and round 2
# of the review tried and reverted a guard for it (see test_review_round.py).
from app.routes.admin import _activate_user_by_email
victim = make_user(
email="victim@example.com",
password="attacker-chosen",
user = make_user(
email="approved-first@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
epoch0 = victim.session_epoch or 0
_activate_user_by_email(db, "victim@example.com")
assert victim.is_active is True
assert victim.password_hash is None
assert (victim.session_epoch or 0) == epoch0 + 1
epoch0 = user.session_epoch or 0
_activate_user_by_email(db, "approved-first@example.com")
assert user.is_active is True
assert user.password_hash is not None
assert user.password_revoked_at is None
assert (user.session_epoch or 0) == epoch0 # nothing to revoke, so no epoch bump either
def test_an_approved_but_unverified_account_still_cannot_sign_in(client, make_user):
# The gate that stops a pre-emptor BEFORE the address is confirmed, whatever the admin decided.
# It is not the whole story — confirming the address opens it, which is SEC-16 — but it is the
# invariant this test owns, and a change that let an unverified account sign in would be a
# straightforwardly worse bug than the residual.
make_user(
email="notyet@example.com",
password="attacker-chosen",
email_verified=False,
is_active=True,
)
r = _login(client, "notyet@example.com", "attacker-chosen")
assert r.status_code == 403
assert r.json()["detail"]["reason"] == "email_unverified"
def test_verifying_afterwards_lets_the_original_password_in(client, db, make_user):
# The payoff for the legitimate registrant: the password chosen at registration still works once
# the mailbox is proven — no "your password was cleared for security" detour, no reset round-trip.
from app.routes.auth import _issue_token
user = make_user(
email="ordering@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
from app.routes.admin import _activate_user_by_email
_activate_user_by_email(db, "ordering@example.com") # admin decides FIRST — the common ordering
raw = _issue_token(db, user, "verify", timedelta(hours=1))
db.commit()
verified = client.post("/auth/verify", json={"token": raw})
assert verified.status_code == 200
assert verified.json() == {
"ok": True,
"password_reset_required": False,
"email": "ordering@example.com",
}
assert _login(client, "ordering@example.com", "chosen at registration").status_code == 200
def test_activating_a_verified_account_keeps_its_password(db, make_user):
# A legit account whose email IS verified (e.g. a no-SMTP registration) keeps its password on
# approval — the guard must not break the normal flow.
# The no-SMTP shape: registration marks such an account verified up front, so it was never in
# scope for the SEC-10 guard even before E9.5 S1. Kept as the other half of the pair.
from app.routes.admin import _activate_user_by_email
user = make_user(
@@ -127,3 +184,374 @@ def test_activating_a_verified_account_keeps_its_password(db, make_user):
_activate_user_by_email(db, "legit@example.com")
assert user.is_active is True
assert user.password_hash is not None
def _register_and_fetch(db, email: str):
"""Run the /register background worker, then read the users/invites rows back through a FRESH
query. The worker owns its own session, so anything it failed to commit is simply not there."""
from sqlalchemy import select as _select
from app.models import Invite, User
from app.routes.auth import _register_account
_register_account(email, _PW)
db.expire_all()
user = db.execute(_select(User).where(User.email == email)).scalar_one_or_none()
invite = db.execute(_select(Invite).where(Invite.email == email)).scalar_one_or_none()
return user, invite
def _no_smtp(monkeypatch):
"""Put the worker on the no-SMTP path and capture the admin notice instead of sending it."""
from app.routes import auth as auth_mod
notified: list[str] = []
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: False)
monkeypatch.setattr(
auth_mod.email_mod,
"send_admin_new_request",
lambda admins, lang, to: notified.append(to) or True,
)
return notified
def test_registration_of_a_pre_approved_email_persists_the_account_without_smtp(
db, make_user, monkeypatch
):
# THE regression. An admin whitelists an address, the person registers, and the account silently
# never existed: upsert_pending_invite leaves an already-approved row alone and returns WITHOUT
# committing, and the only other commit on this path (_issue_token) is gated on SMTP — so with no
# working mail the flushed user was rolled back at session close. The sign-up answered "ok",
# created nothing, and re-registering did the same again.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "roland@example.com")
assert user is not None, "pre-approved registration was rolled back — the account is gone"
assert user.password_hash is not None
assert user.email_verified is True # no SMTP => ownership can't be proven by link
assert (
user.is_active is False
) # approval is bound to the account by an admin, not by signing up
# The admin's standing decision is left exactly as it was — is_allowed() still reads "approved",
# so their Google sign-in keeps working and an anonymous sign-up cannot revoke access.
assert invite.status == "approved"
assert invite.decided_by == "admin@example.com"
# The notify path still fires for an already-approved address — nothing lands in the queue, so
# it is the only announcement there is. (Whether it is DELIVERED is separate: on a real no-SMTP
# instance `_send` drops it, which is the open half of the problem, roadmap UX-25.)
assert notified == ["roland@example.com"]
def test_registration_with_an_already_pending_request_persists_the_account(
db, make_user, monkeypatch
):
# The same rollback, reached the other way: someone uses Request access, then registers before
# the admin has decided. upsert_pending_invite leaves the pending row alone and does not commit.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="pia@example.com", status="pending"))
db.commit()
user, invite = _register_and_fetch(db, "pia@example.com")
assert user is not None, "registration was rolled back — the account is gone"
assert user.is_active is False
assert invite.status == "pending"
assert notified == ["pia@example.com"]
def test_an_admin_can_activate_a_pre_approved_registrant(db, monkeypatch):
# The recovery path for the account above, pinned so it cannot quietly break: re-entering the
# address in the Access-requests whitelist box (POST /api/admin/invites) runs
# _activate_user_by_email.
from app.models import Invite
from app.routes.admin import _activate_user_by_email
_no_smtp(monkeypatch)
db.add(Invite(email="roland@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, _ = _register_and_fetch(db, "roland@example.com")
_activate_user_by_email(db, "roland@example.com")
db.expire_all()
assert user.is_active is True
assert user.password_hash is not None # approval never touches the password (E9.5 S1)
def test_registration_of_an_unknown_email_still_persists_account_and_pending_invite(
db, make_user, monkeypatch
):
# Guard, not a regression test: this branch always worked (upsert INSERTs the invite and commits
# the same transaction). It must keep working — and it is the case that DOES notify the admin.
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
user, invite = _register_and_fetch(db, "stranger@example.com")
assert user is not None
assert user.is_active is False
assert invite is not None and invite.status == "pending"
assert notified == ["stranger@example.com"]
def test_registration_of_a_previously_denied_email_revives_the_request(db, make_user, monkeypatch):
# Guard: a denied person may ask again — the invite flips back to pending (and commits) so the
# admin sees a fresh request. The stale decision must not leak into the new account's state.
from app.models import Invite
notified = _no_smtp(monkeypatch)
make_user(email="admin@example.com", role="admin")
db.add(Invite(email="denis@example.com", status="denied", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "denis@example.com")
assert user is not None
assert user.is_active is False
assert invite.status == "pending"
assert invite.decided_at is None and invite.decided_by is None
assert notified == ["denis@example.com"]
def test_registration_of_a_pre_approved_email_persists_the_account_with_smtp(db, monkeypatch):
# Guard, not a regression test: with SMTP on, _issue_token's own commit already persisted the
# user, so this branch was never broken and must stay unbroken.
from app.models import Invite
from app.routes import auth as auth_mod
sent: list[str] = []
monkeypatch.setattr(auth_mod.email_mod, "email_enabled", lambda: True)
monkeypatch.setattr(
auth_mod.email_mod, "send_verify_email", lambda to, lang, link, **kw: sent.append(to)
)
monkeypatch.setattr(auth_mod.email_mod, "send_admin_new_request", lambda admins, lang, to: True)
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
user, invite = _register_and_fetch(db, "rita@example.com")
assert user is not None
assert user.email_verified is False # must confirm the address
assert user.is_active is False
assert invite.status == "approved"
assert sent == ["rita@example.com"]
def test_verifying_an_email_never_grants_access_on_its_own(client, db, make_user):
# Verification attests the address; it does not decide access. The two gates stay separate.
from app.models import Invite
from app.routes.auth import _issue_token
user = make_user(email="rita@example.com", password=_PW, email_verified=False, is_active=False)
db.add(Invite(email="rita@example.com", status="approved", decided_by="admin@example.com"))
db.commit()
raw = _issue_token(db, user, "verify", timedelta(hours=1))
assert client.post("/auth/verify", json={"token": raw}).status_code == 200
db.expire_all()
assert user.email_verified is True
assert user.is_active is False
assert _login(client, "rita@example.com", _PW).status_code == 403
def test_register_route_creates_the_account_end_to_end(client, db, monkeypatch):
# The tests above call _register_account directly, so nothing covered the ROUTE: its
# allow_registration gate, its rate limiter and the background.add_task wiring that makes the
# whole flow reachable at all. TestClient runs BackgroundTasks after the response, so a single
# POST exercises the real path end to end.
from sqlalchemy import select as _select
from app.models import User
_no_smtp(monkeypatch)
r = client.post("/auth/register", json={"email": "newbie@example.com", "password": _PW})
assert r.status_code == 200 and r.json() == {"status": "ok"}
db.expire_all()
user = db.execute(_select(User).where(User.email == "newbie@example.com")).scalar_one_or_none()
assert user is not None, "POST /auth/register answered ok but created nothing"
assert user.password_hash is not None
def test_register_route_is_refused_when_registration_is_closed(client, db, monkeypatch):
# The gate is an explicit 403 with nothing created — not a uniform "ok" that silently drops it.
from sqlalchemy import select as _select
from app import sysconfig
from app.models import User
_no_smtp(monkeypatch)
sysconfig.set_value(db, "allow_registration", False)
r = client.post("/auth/register", json={"email": "closed@example.com", "password": _PW})
assert r.status_code == 403
db.expire_all()
assert (
db.execute(_select(User).where(User.email == "closed@example.com")).scalar_one_or_none()
is None
)
# --- SEC-10 lockout: a revoked preset password must stay RECOVERABLE ---------------------------
#
# Regression pack for the prod incident of 2026-09-02. An account registered with a password, then
# stripped of it by the SEC-10 guard, ended up with password_hash = NULL. That is also what a
# Google-only account looks like, and reset eligibility keyed off `password_hash` alone — so the
# account could neither sign in (no password) nor obtain a reset link, and every response on the way
# was a deliberately uniform "ok"/401. `password_revoked_at` is the trace that tells the two apart;
# these pin each step of the way back in.
#
# E9.5 S1 removed the admin-approval caller, so Google adoption is the only path that still produces
# this state — plus every account stripped on prod before that change, which is why the whole recovery
# route below must keep working. These tests manufacture the state through the surviving producer.
def _revoke_as_google_adoption(db, user) -> None:
"""Put an account into the SEC-10-revoked state the way the ONLY remaining producer does: the
Google callback adopting an account whose email was never verified."""
from app.routes.auth import revoke_preset_password_if_unverified
revoke_preset_password_if_unverified(user)
db.commit()
def _capture_reset_links(monkeypatch) -> list[str]:
"""Patch the reset mailer and collect the recipients it is asked to write to."""
from app.routes import auth as auth_mod
sent: list[str] = []
monkeypatch.setattr(
auth_mod.email_mod, "send_password_reset", lambda to, lang, link: sent.append(to)
)
return sent
def test_the_google_adoption_revoke_stamps_password_revoked_at(db, make_user):
# The guard still fires where it earns its keep: the real owner has just proven ownership through
# Google, so a password set on the address while it was unverified must not survive.
victim = make_user(
email="stamped@example.com",
password="attacker-chosen",
email_verified=False,
is_active=False,
)
assert victim.password_revoked_at is None
epoch0 = victim.session_epoch or 0
_revoke_as_google_adoption(db, victim)
assert victim.password_hash is None
assert victim.password_revoked_at is not None # …so the reset flow can tell it HAD a password
assert (victim.session_epoch or 0) == epoch0 + 1 # any session that password minted is killed
def test_an_account_whose_password_was_revoked_can_still_get_a_reset_link(
db, make_user, monkeypatch
):
# THE lockout bug: without the stamp this sent nothing, and the account was unrecoverable.
from app.routes.auth import _send_reset_if_eligible
victim = make_user(
email="lockedout@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
_revoke_as_google_adoption(db, victim)
sent = _capture_reset_links(monkeypatch)
_send_reset_if_eligible("lockedout@example.com")
assert sent == ["lockedout@example.com"]
def test_a_google_only_account_is_still_not_reset_eligible(db, make_user, monkeypatch):
# The other side of the rule: an account that NEVER had a password gets no reset link — its owner
# adds one from Settings while signed in (/auth/set-password), which proves identity by session.
from app.routes.auth import _send_reset_if_eligible
make_user(email="googler@example.com", password=None, google_sub="sub-123")
sent = _capture_reset_links(monkeypatch)
_send_reset_if_eligible("googler@example.com")
assert sent == []
def test_completing_a_reset_clears_the_revocation_stamp(client, db, make_user):
from app.routes.auth import _issue_token
user = make_user(
email="recovered@example.com",
password="chosen at registration",
email_verified=False,
is_active=True, # already approved; reset-confirm verifies the email but never activates
)
_revoke_as_google_adoption(db, user)
raw = _issue_token(db, user, "reset", timedelta(hours=1))
db.commit()
ok = client.post(
"/auth/password-reset/confirm", json={"token": raw, "password": "a brand new one"}
)
assert ok.status_code == 200
db.expire_all()
assert user.password_hash is not None
assert user.password_revoked_at is None # spent — not a permanently reset-eligible account
assert _login(client, "recovered@example.com", "a brand new one").status_code == 200
def test_verify_tells_a_revoked_account_that_it_needs_a_new_password(client, db, make_user):
# The banner's input. Saying so leaks nothing: the caller just proved mailbox control by holding a
# valid single-use verify token. Without it the user meets only the uniform "Invalid email or
# password" and has no way to learn why the password they chose stopped working.
from app.routes.auth import _issue_token
user = make_user(
email="needspw@example.com",
password="chosen at registration",
email_verified=False,
is_active=False,
)
_revoke_as_google_adoption(db, user)
raw = _issue_token(db, user, "verify", timedelta(hours=1))
db.commit()
r = client.post("/auth/verify", json={"token": raw})
assert r.status_code == 200
assert r.json() == {
"ok": True,
"password_reset_required": True,
"email": "needspw@example.com",
}
def test_verify_reports_no_reset_needed_for_an_intact_account(client, db, make_user):
from app.routes.auth import _issue_token
user = make_user(email="intact@example.com", email_verified=False, is_active=False)
raw = _issue_token(db, user, "verify", timedelta(hours=1))
db.commit()
r = client.post("/auth/verify", json={"token": raw})
assert r.status_code == 200
assert r.json() == {
"ok": True,
"password_reset_required": False,
"email": "intact@example.com",
}
+43
View File
@@ -55,6 +55,49 @@ def test_channel_detail_returns_normal_shorts_live_counts(db, monkeypatch):
assert body["count_normal"] == 2
@pytest.mark.parametrize("hidden", [True, False])
def test_channel_detail_surfaces_subscription_hidden_flag(db, monkeypatch, hidden):
# S6: the detail GET must expose the per-subscription `hidden` flag so the channel page can offer
# the same Unhide toggle the manager has (the flag was previously omitted from the detail dict).
ch = Channel(id="UC_hid", title="Hidden?", published_at=_PUB)
db.add(ch)
me = User(email="hid@example.com")
db.add(me)
db.flush()
db.add(Subscription(user_id=me.id, channel_id=ch.id, hidden=hidden))
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: me
try:
resp = TestClient(app).get(f"/api/channels/{ch.id}")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
assert resp.json()["hidden"] is hidden
def test_channel_detail_hidden_defaults_false_when_not_subscribed(db, monkeypatch):
# No subscription row → the flag can't be True; it must serialize as False, not raise.
ch = Channel(id="UC_hid_ns", title="Unsub", published_at=_PUB)
me = User(email="hidns@example.com")
db.add_all([ch, me])
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: me
try:
resp = TestClient(app).get(f"/api/channels/{ch.id}")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
assert resp.json()["hidden"] is False
def _setup_channel_and_sub(db, *, email, is_demo=False):
ch = Channel(
id="UC_wire",
+90
View File
@@ -0,0 +1,90 @@
"""The shared demo account's refusal says WHY, in the reader's language (deep-review round 6).
`require_human` answered `403 "Not available in the demo account."` — a plain string. `req` leaves
403 to the caller, and round 5 taught `notifyActionError` to discard a plain-string detail in favour
of the caller's already-translated fallback, so from that commit on the demo user got a generic
"couldn't do that" and no hint that the demo account was the reason. Not every affordance behind
this gate is UI-gated on `is_demo` either — the comfort shelf is not, so the pin button is live.
A structured `{code: "demo"}` is what `localizeDetail` can turn into `errors.demo` in both locales.
Pinned at all FOUR sites: the shared dependency, the two auth routes that hand-roll the same check,
and the account-delete route in `routes/me.py` — which round 6 missed while its docstring claimed
all the sites were covered, so the next reader had no reason to look (round 7). An asymmetric fix is
the smell this repo bans, and a docstring asserting symmetry it does not have is worse than none.
The list is derived from the source, not typed out: `test_every_is_demo_refusal_is_on_the_list`
below fails when a fifth `is_demo` 403 appears anywhere in `app/`, so this file cannot fall behind
the code the way round 6's did. Needs the DB.
"""
import re
from pathlib import Path
from app.models import User
def test_every_demo_refusal_carries_the_same_structured_reason(db, make_user, as_user):
demo = make_user(email="demo@example.com", is_demo=True)
client = as_user(demo)
for method, url, body in [
# `require_human` itself, through an affordance the UI does NOT gate on is_demo.
("post", "/api/discover/pins", {"kind": "video", "ref": "v1"}),
# The three routes that repeat the check inline rather than depending on the gate.
("post", "/auth/logout-others", None),
("post", "/auth/set-password", {"password": "a long enough password"}),
("delete", "/api/me/account", None),
]:
r = getattr(client, method)(url, **({"json": body} if body is not None else {}))
assert r.status_code == 403, (url, r.text)
detail = r.json()["detail"]
# Every one of them is a dict whose `code` the frontend can localize; the delete route adds
# a `reason` so its refusal can be worded specifically.
assert isinstance(detail, dict), (url, detail)
assert detail["code"] == "demo", (url, detail)
def test_every_is_demo_refusal_is_on_the_list():
"""A fifth site must not be able to appear unnoticed.
Round 6 converted three of four `is_demo` 403s and wrote a docstring saying it had converted
them all; the fourth kept answering a bare English string for another round because nothing
counted them. So count them here: every `is_demo`-guarded 403 in `app/` has to raise a
structured detail, and the number of them has to match what this file exercises."""
app_dir = Path(__file__).resolve().parents[1] / "app"
sites = []
for path in sorted(app_dir.rglob("*.py")):
lines = path.read_text(encoding="utf-8").splitlines()
for i, line in enumerate(lines):
# `if user.is_demo:` / `if u.is_demo:` — the hand-rolled gate, not the ORM filters
# (`User.is_demo.is_(False)`) and not the column definition.
if not re.search(r"^\s*if\s+\w+\.is_demo\s*:", line):
continue
block = "\n".join(lines[i : i + 6])
if "403" not in block:
continue
sites.append(f"{path.relative_to(app_dir)}:{i + 1}")
assert 'detail={"code": "demo"' in block, (
f"{path.relative_to(app_dir)}:{i + 1} refuses the demo account with an "
f"unstructured detail — the frontend cannot localize it:\n{block}"
)
# auth.py's shared gate + routes/auth.py's two + routes/me.py's delete. A new one is welcome,
# but it has to be added to the loop above in the same commit.
assert len(sites) == 4, sites
def test_a_real_account_is_not_refused_by_the_gate(db, make_user, as_user):
"""The other half: the structured detail must not come from having broken the gate open the
other way. A non-demo account gets past `require_human` and fails (or succeeds) on its own
merits — anything but a 403 demo refusal."""
client = as_user(make_user(email="real@example.com"))
r = client.post("/auth/logout-others")
assert r.status_code == 200, r.text
assert not isinstance(r.json().get("detail"), dict)
def test_the_gate_reads_the_flag_and_not_the_email(db):
"""`is_demo` is a column, not a convention about the address — a real account that happens to
be called demo@ must not be refused."""
from app.auth import require_human
assert require_human(User(email="demo@example.com", is_demo=False)) is not None
+942
View File
@@ -0,0 +1,942 @@
"""E24 S1 — the Home surface's read-model.
Covers the four things that can silently rot here: the time-budget bands (pure), the cross-source
resume rail's merge order, the hero's tier ladder, and the comfort shelf (its schema-level one-of
invariant, its cascade, and the resolved `next` that makes one click enough).
DB lane (`db` fixture) except for the budget block — skipped without Postgres.
"""
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import delete, func, select, text
from sqlalchemy.exc import IntegrityError
from app.discover import hero, pins, queries, rails
from app.discover.budget import BANDS, fits, remaining_seconds
from app.plex import rollup
from app.plex.cards import leaf_card
from app.models import (
BlockedChannel,
Channel,
ComfortPin,
PlexItem,
PlexLibrary,
PlexShow,
PlexState,
Playlist,
PlaylistItem,
Subscription,
User,
Video,
VideoState,
)
NOW = datetime(2026, 6, 1, tzinfo=timezone.utc)
# ── budget (pure) ─────────────────────────────────────────────────────────────────────────────────
def test_bands_cover_every_runtime_and_overlap_deliberately():
"""Every runtime has a home in `any`, and the bands overlap so "30 perc" is not answered with a
two-minute Short just because it fits under the cap."""
assert fits(3, "any") and fits(10_000, "any")
assert fits(2 * 60, "short") and not fits(2 * 60, "medium") # a Short is not a 30-minute answer
assert fits(30 * 60, "medium") and not fits(30 * 60, "short")
assert fits(110 * 60, "long") and not fits(110 * 60, "medium")
def test_unknown_duration_only_fits_the_unbounded_band():
"""We cannot promise an unknown-length item finishes inside ten minutes, so it is not offered as
an answer to a bounded question — the one control on the page has to mean what it says."""
for budget in BANDS:
assert fits(None, budget) is (budget == "any")
def test_budget_is_matched_against_what_is_LEFT():
"""A 90-minute film with 8 minutes to go is a ten-minute answer."""
assert remaining_seconds(90 * 60, 82 * 60) == 8 * 60
assert fits(remaining_seconds(90 * 60, 82 * 60), "short")
assert not fits(remaining_seconds(90 * 60, 0), "short")
assert remaining_seconds(None, 30) is None
assert remaining_seconds(600, 900) == 0 # a position past the end never goes negative
# ── fixtures ──────────────────────────────────────────────────────────────────────────────────────
def _seed_yt(db) -> tuple[User, Channel]:
"""A subscribed channel with three videos of distinct lengths, so a budget can pick between them."""
user = User(email="discover-e24@example.com")
ch = Channel(id="UCe24", title="Comfort Channel", handle="@comfort")
db.add_all([user, ch])
db.flush()
db.add(Subscription(user_id=user.id, channel_id=ch.id))
db.add_all(
[
Video(
id="vshort",
channel_id=ch.id,
title="Ten minute video",
duration_seconds=9 * 60,
published_at=NOW - timedelta(days=1),
),
Video(
id="vmed",
channel_id=ch.id,
title="Half hour video",
duration_seconds=30 * 60,
published_at=NOW - timedelta(days=2),
),
Video(
id="vlong",
channel_id=ch.id,
title="Feature length video",
duration_seconds=95 * 60,
published_at=NOW - timedelta(days=3),
),
]
)
db.commit()
return user, ch
def _seed_plex(db, user: User) -> tuple[PlexItem, PlexShow, PlexItem]:
lib_m = PlexLibrary(plex_key="90", title="Movies", kind="movie", enabled=True)
lib_s = PlexLibrary(plex_key="91", title="Shows", kind="show", enabled=True)
db.add_all([lib_m, lib_s])
db.flush()
movie = PlexItem(
rating_key="pm1",
library_id=lib_m.id,
kind="movie",
title="A Movie",
duration_s=100 * 60,
playable="direct",
)
show = PlexShow(rating_key="ps1", library_id=lib_s.id, title="A Show", child_count=1)
db.add_all([movie, show])
db.flush()
ep1 = PlexItem(
rating_key="ps1e1",
library_id=lib_s.id,
kind="episode",
title="Ep 1",
show_id=show.id,
season_number=1,
episode_number=1,
duration_s=45 * 60,
)
ep2 = PlexItem(
rating_key="ps1e2",
library_id=lib_s.id,
kind="episode",
title="Ep 2",
show_id=show.id,
season_number=1,
episode_number=2,
duration_s=45 * 60,
)
db.add_all([ep1, ep2])
db.commit()
return movie, show, ep1
def _enable_plex(db):
from app import sysconfig
sysconfig.set_value(db, "plex_enabled", True)
db.commit()
# ── continue watching ─────────────────────────────────────────────────────────────────────────────
def test_continue_watching_merges_both_sources_by_recency(db):
"""One rail, not two lists side by side — and ordered by when the user last touched each thing,
across sources. Two side-by-side rails would just move the decision up a level."""
user, _ = _seed_yt(db)
movie, _, _ = _seed_plex(db, user)
_enable_plex(db)
db.add_all(
[
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW - timedelta(hours=5),
),
PlexState(
user_id=user.id,
item_id=movie.id,
status="new",
position_seconds=600,
progress_updated_at=NOW - timedelta(hours=1),
),
]
)
db.commit()
got = rails.continue_watching(db, user)
assert [(c["source"], c["id"]) for c in got] == [("plex", "pm1"), ("youtube", "vmed")]
assert got[0]["remaining_seconds"] == 100 * 60 - 600
def test_continue_watching_is_youtube_only_when_plex_is_off(db):
"""Plex is a deployment toggle; with it off the surface says nothing about Plex at all."""
user, _ = _seed_yt(db)
movie, _, _ = _seed_plex(db, user)
db.add_all(
[
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
),
PlexState(
user_id=user.id,
item_id=movie.id,
status="new",
position_seconds=600,
progress_updated_at=NOW,
),
]
)
db.commit()
assert {c["source"] for c in rails.continue_watching(db, user)} == {"youtube"}
def test_the_resume_rail_obeys_the_channel_blocklist(db):
"""Blocking a channel does not touch `video_states`, so a half-watched video from it survives
with its resume position — and this rail builds its own SELECT instead of going through
`filtered_query`. "Never show me this channel" has to outrank "you started it", or the landing
page answers "what should I watch" with the thing the user has just banned."""
user, ch = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
)
)
db.commit()
assert [c["id"] for c in rails.continue_watching(db, user)] == ["vmed"]
db.add(BlockedChannel(user_id=user.id, channel_id=ch.id))
db.commit()
assert rails.continue_watching(db, user) == []
# And so the hero, whose resume tier draws from that rail, has nothing to offer either.
assert hero.pick(db, user, "any", 0) is None
def test_a_null_timestamp_sorts_last(db):
"""A missing progress timestamp sorts last rather than 500ing the landing page.
Narrower than it was written to be, and the epic-end review corrected it: the previous key,
`(ts is not None, ts)`, does NOT raise on two nulls — tuple comparison finds the first UNEQUAL
element via `==`, so `None < None` is never reached, and this test passes under that key too.
What it does discriminate is the naive `key=lambda r: r[0]`, which really does raise. See
`plex.rollup.NO_TS` for the corrected rationale."""
user, _ = _seed_yt(db)
db.add_all(
[
VideoState(
user_id=user.id, video_id="vmed", status="new", position_seconds=1
), # no progress_updated_at
VideoState(
user_id=user.id, video_id="vlong", status="new", position_seconds=1
), # nor here
VideoState(
user_id=user.id,
video_id="vshort",
status="new",
position_seconds=1,
progress_updated_at=NOW,
),
]
)
db.commit()
got = rails.continue_watching(db, user)
assert got[0]["id"] == "vshort"
assert len(got) == 3
# ── hero ──────────────────────────────────────────────────────────────────────────────────────────
def test_hero_prefers_resume_and_labels_the_tier(db):
user, _ = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
)
)
db.commit()
got = hero.pick(db, user, "any", seed=1)
assert got["tier"] == "resume"
assert got["card"]["id"] == "vmed"
def test_hero_skips_a_resume_that_does_not_fit_the_budget(db):
""" "I have ten minutes" must not be answered with the 80 minutes left on a film, even though
that film is the most recent thing the user touched."""
user, _ = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vlong",
status="new",
position_seconds=60,
progress_updated_at=NOW,
)
)
db.commit()
got = hero.pick(db, user, "short", seed=1)
assert got["tier"] == "surprise"
assert got["card"]["duration_seconds"] <= BANDS["short"][1]
def test_hero_falls_back_to_the_shelf_before_guessing(db):
"""The user's own declared fallback beats anything we could infer."""
user, ch = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
db.commit()
got = hero.pick(db, user, "medium", seed=1)
assert got["tier"] == "comfort"
assert got["card"]["id"] == "vmed"
def test_hero_resolves_a_show_to_the_episode_that_would_play(db):
"""A show has no Play button. The hero of a partially-watched show is its next episode."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
_enable_plex(db)
db.add(
PlexState(
user_id=user.id,
item_id=ep1.id,
status="watched",
watched_at=NOW,
progress_updated_at=NOW,
)
)
db.commit()
got = hero.pick(db, user, "any", seed=1)
assert got["tier"] == "resume"
assert got["card"]["kind"] == "episode"
assert got["card"]["id"] == "ps1e2" # ep1 is finished → ep2 is what plays
def test_hero_is_none_when_nothing_fits(db):
"""An empty answer is an honest answer: the page renders its own empty state rather than being
handed a card that does not answer what was asked. Here the whole library is under 40 minutes,
so "egy este" has nothing to offer."""
user, ch = _seed_yt(db)
db.execute(text("DELETE FROM videos WHERE id = 'vlong'"))
db.commit()
assert hero.pick(db, user, "long", seed=1) is None
def test_surprise_is_reproducible_for_a_seed(db):
"""The reroll button depends on this: one seed drives both sources, so the same seed must give
the same answer and a different seed must be free to give another."""
user, _ = _seed_yt(db)
a = rails.surprise(db, user, "any", seed=7, limit=3)
b = rails.surprise(db, user, "any", seed=7, limit=3)
assert [c["id"] for c in a] == [c["id"] for c in b]
seeds = {tuple(c["id"] for c in rails.surprise(db, user, "any", s, 3)) for s in range(12)}
assert len(seeds) > 1 # the seed actually reaches the ordering
def test_surprise_never_offers_something_already_started(db):
"""The resume tier may have skipped an item because its REMAINING time did not fit; handing the
same item back as "Meglepetés" — where the full runtime does fit — would contradict the question
the user just answered."""
user, _ = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vlong",
status="new",
position_seconds=90 * 60,
progress_updated_at=NOW,
)
)
db.commit()
assert "vlong" not in {c["id"] for c in rails.surprise(db, user, "long", seed=3, limit=5)}
def test_the_seed_decides_which_source_leads(db):
"""With a pool of one, a fixed source order would mean the Plex library could never answer "egy
este" whenever a YouTube video also fit — the cross-source promise would quietly be a YouTube
promise."""
user, _ = _seed_yt(db)
_seed_plex(db, user)
_enable_plex(db)
leaders = {rails.surprise(db, user, "any", s, limit=1)[0]["source"] for s in range(6)}
assert leaders == {"youtube", "plex"}
def test_the_reroll_must_be_able_to_move_past_the_resume_tier(db):
"""A new seed alone only re-draws the SURPRISE tier — the resume tier deterministically returns
the most recently progressed thing that fits, so without an exclusion the reroll handed the same
card straight back and the button looked dead (found in the browser, 2026-09-06)."""
user, _ = _seed_yt(db)
db.add_all(
[
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
),
VideoState(
user_id=user.id,
video_id="vshort",
status="new",
position_seconds=60,
progress_updated_at=NOW - timedelta(hours=1),
),
]
)
db.commit()
first = hero.pick(db, user, "any", seed=1)
assert first["card"]["id"] == "vmed"
# Same seed, same everything — only the exclusion differs.
second = hero.pick(db, user, "any", seed=1, exclude={"youtube:vmed"})
assert second["card"]["id"] == "vshort"
def test_the_reroll_rotates_which_tier_answers(db):
"""Measured 2026-09-06: exclusion alone left the button walking down the resume rail (20 rows)
and cycling nine cards, every one of them `resume` — the shelf and the surprise tier were
unreachable by pressing it. `start` is the press count, so consecutive presses offer a different
KIND of answer."""
user, _ = _seed_yt(db)
# One of each: something half-finished, something pinned, and a third video for the draw.
db.add(
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
)
)
db.add(ComfortPin(user_id=user.id, video_id="vlong", position=0))
db.commit()
tiers = [hero.pick(db, user, "any", seed=1, start=n)["tier"] for n in range(4)]
assert tiers == ["resume", "comfort", "surprise", "resume"]
def test_the_rotation_still_falls_through_an_empty_tier(db):
"""A press must not come back empty just because the tier it happened to lead with has nothing.
Here there is no shelf at all, so the comfort-led press has to answer from somewhere else."""
user, _ = _seed_yt(db)
assert db.execute(select(func.count()).select_from(ComfortPin)).scalar_one() == 0
got = hero.pick(db, user, "any", seed=1, start=1)
assert got is not None and got["tier"] != "comfort"
def test_excluding_a_show_also_excludes_the_episode_it_resolves_to(db):
"""The user rerolled past the SHOW they were shown; handing back its next episode under a
different title would be the same suggestion wearing a hat."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
_enable_plex(db)
db.add(
PlexState(
user_id=user.id,
item_id=ep1.id,
status="new",
position_seconds=120,
progress_updated_at=NOW,
)
)
db.commit()
shown = hero.pick(db, user, "any", seed=1)
assert shown["card"]["id"] == "ps1e1" # the show resolved to its in-progress episode
# Excluding what the USER saw — the show — must move the tier on, not return the episode again.
after = hero.pick(db, user, "any", seed=1, exclude={"plex:ps1"})
assert after is None or after["card"]["id"] != "ps1e1"
# ── comfort shelf ─────────────────────────────────────────────────────────────────────────────────
def test_a_pin_must_reference_exactly_one_thing(db):
"""The union invariant lives in the schema, not in a comment at every insert site."""
user, ch = _seed_yt(db)
for kwargs in ({}, {"video_id": "vmed", "channel_id": ch.id}):
db.add(ComfortPin(user_id=user.id, position=0, **kwargs))
with pytest.raises(IntegrityError):
db.flush()
db.rollback()
def test_the_same_thing_cannot_be_pinned_twice(db):
"""The per-kind unique constraints are what make the route's concurrent-tab path safe (it catches
`IntegrityError` precisely because of them) — and the migration drift check filters constraint
diffs out as noise, so nothing else would notice them going missing. The route's own dedupe
returns early without ever reaching the DB, so only this test exercises them."""
user, _ = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
db.commit()
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=1))
with pytest.raises(IntegrityError):
db.flush()
db.rollback()
def test_deleting_the_pinned_row_removes_the_pin(db):
"""A film dropped from the library must not leave a dead tile on the one surface whose whole job
is to be reliable — which is why the shelf holds foreign keys, not strings."""
user, _ = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
db.commit()
db.execute(text("DELETE FROM videos WHERE id = 'vmed'"))
db.commit()
assert db.execute(select(func.count()).select_from(ComfortPin)).scalar() == 0
def test_shelf_resolves_a_channel_to_its_newest_unwatched_video(db):
""" "Put that channel on" has to be one click, so the tile carries what would play."""
user, ch = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, channel_id=ch.id, position=0))
db.commit()
(card,) = pins.shelf(db, user)
assert card["kind"] == "channel"
assert card["unwatched_count"] == 3
assert card["next"]["id"] == "vshort" # newest
def test_shelf_resolves_a_playlist_to_its_first_unwatched_item(db):
user, _ = _seed_yt(db)
pl = Playlist(user_id=user.id, name="Elalvós")
db.add(pl)
db.flush()
db.add_all(
[
PlaylistItem(playlist_id=pl.id, video_id="vshort", position=0),
PlaylistItem(playlist_id=pl.id, video_id="vmed", position=1),
]
)
db.add(VideoState(user_id=user.id, video_id="vshort", status="watched", watched_at=NOW))
db.add(ComfortPin(user_id=user.id, playlist_id=pl.id, position=0))
db.commit()
(card,) = pins.shelf(db, user)
assert card["kind"] == "playlist"
assert card["item_count"] == 2
assert card["next"]["id"] == "vmed"
def test_a_pinned_video_survives_leaving_the_feed_scope(db):
"""The shelf is the fallback surface. An entry silently vanishing because the user unsubscribed
from the channel is exactly the failure it must not have."""
user, ch = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
db.commit()
db.execute(delete(Subscription))
db.commit()
(card,) = pins.shelf(db, user)
assert card["id"] == "vmed"
assert card["next"]["id"] == "vmed"
def test_the_plex_surprise_obeys_the_time_budget(db):
"""The one control on the page has to mean what it says: a 100-minute film is not an answer to
"10 perc". The YouTube half of this rule has had its own test since S1; the Plex half's duration
bounds could be deleted with the whole suite still green, because every test that reached
`plex_surprise` passed budget="any"."""
user, _ = _seed_yt(db)
_seed_plex(db, user)
_enable_plex(db)
drawn = lambda budget: { # noqa: E731
(c["source"], c["id"]) for s in range(6) for c in rails.surprise(db, user, budget, s, 5)
}
assert ("plex", "pm1") not in drawn("short")
# …and it IS offered where it fits, so this cannot pass by the rail being empty.
assert ("plex", "pm1") in drawn("long")
def test_the_plex_surprise_never_offers_something_already_started(db):
"""A half-watched film is not a surprise, and the resume tier has already had its chance at it.
The mirror of `test_surprise_never_offers_something_already_started` for the other source — the
asymmetric half of one principle, which the house rules say to fix everywhere."""
user, _ = _seed_yt(db)
movie, _, _ = _seed_plex(db, user)
_enable_plex(db)
db.add(
PlexState(
user_id=user.id,
item_id=movie.id,
status="new",
position_seconds=600,
progress_updated_at=NOW,
)
)
db.commit()
drawn = {
(c["source"], c["id"]) for s in range(6) for c in rails.surprise(db, user, "long", s, 5)
}
assert ("plex", "pm1") not in drawn
def test_a_new_pin_lands_at_the_end_even_after_a_hole(db):
"""Positions are never renumbered on removal — `remove_pin` deletes the row, and every FK cascade
silently removes a pin when its target disappears — so the append position has to come from
`max(position)`, not from the row count. With a count, the first hole drops the next pin into the
MIDDLE of the shelf and the second makes two pins claim one slot."""
user, ch = _seed_yt(db)
db.add_all(
[
ComfortPin(user_id=user.id, video_id="vshort", position=0),
ComfortPin(user_id=user.id, video_id="vmed", position=1),
ComfortPin(user_id=user.id, video_id="vlong", position=2),
]
)
db.commit()
db.execute(delete(ComfortPin).where(ComfortPin.video_id.in_(("vshort", "vmed"))))
db.commit()
db.add(ComfortPin(user_id=user.id, channel_id=ch.id, position=pins.next_position(db, user)))
db.commit()
assert [c["kind"] for c in pins.shelf(db, user)] == ["video", "channel"]
positions = (
db.execute(select(ComfortPin.position).order_by(ComfortPin.position)).scalars().all()
)
assert len(set(positions)) == len(positions) # and no two tiles claim the same slot
def test_a_self_playing_pin_does_not_reference_itself(db):
"""`next` must be a distinct dict even when the pin plays itself. Aliasing the card into its own
`next` is a cycle, and the serializer turns the whole shelf into a 500 over it."""
user, _ = _seed_yt(db)
db.add(ComfortPin(user_id=user.id, video_id="vmed", position=0))
db.commit()
(card,) = pins.shelf(db, user)
assert card["next"] is not card
assert "next" not in card["next"]
def test_the_resume_rail_says_what_would_actually_play(db):
"""A show tile that only opens the show page moves the decision instead of removing it — the
click lands on six seasons to choose between, which is the paralysis this whole surface exists
for. The shelf and the hero already resolved; this rail did not (reported 2026-09-06). The tile
stays the show (its artwork is what the user recognises); `next` is what starts."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
_enable_plex(db)
db.add(
PlexState(
user_id=user.id,
item_id=ep1.id,
status="watched",
watched_at=NOW,
progress_updated_at=NOW,
)
)
db.commit()
rail = rails.continue_watching(db, user)
tile = next(c for c in rail if c["kind"] == "show")
assert tile["id"] == "ps1" # still the show
assert tile["next"]["kind"] == "episode"
assert tile["next"]["id"] == "ps1e2" # ep1 is finished → ep2 is what plays
def test_only_a_show_row_pays_for_a_resolution(db):
"""`next` is the answer to "a show has no Play button". Attaching it to a movie or a video would
both cost a query for nothing and re-open the self-reference cycle the shelf already hit.
The states are what make this test exist: without them every rail query inner-joins its way to
zero rows, the loop body never runs, and the assertion is vacuous — which is exactly how the
epic-end review found it."""
user, _ = _seed_yt(db)
movie, _, ep1 = _seed_plex(db, user)
_enable_plex(db)
db.add_all(
[
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
),
PlexState(
user_id=user.id,
item_id=movie.id,
status="new",
position_seconds=600,
progress_updated_at=NOW,
),
PlexState(
user_id=user.id,
item_id=ep1.id,
status="new",
position_seconds=120,
progress_updated_at=NOW,
),
]
)
db.commit()
rail = rails.continue_watching(db, user)
# A video, a movie and a show — one row of each kind that can appear here, so the assertion below
# actually exercises both sides of the equality.
assert {c["kind"] for c in rail} == {"video", "movie", "show"}
for card in rail:
assert ("next" in card) == (card["kind"] == "show"), card["id"]
def test_next_episode_takes_an_episode_with_no_state_row_at_all(db):
"""The NULL trap in the SQL rewrite: `status NOT IN ('watched','hidden')` is UNKNOWN — not true —
for an episode the user has never touched, so a bare filter drops exactly the common case and the
rail would offer episode 1 of a show already half-watched."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
db.add(PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW))
db.commit() # ep2 has NO PlexState row
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e2"
def test_a_finished_show_starts_again_from_the_top(db):
"""Nothing unwatched left is not "nothing to play" — it is "watch it again"."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
db.add_all(
[
PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW),
PlexState(user_id=user.id, item_id=ep2.id, status="watched", watched_at=NOW),
]
)
db.commit()
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e1"
def test_next_episode_agrees_with_the_show_page_when_two_are_in_progress(db):
"""One show, one answer. `plex.rollup.rollup` — which drives the show page's Resume button —
takes the LAST in-progress episode in broadcast order; taking the FIRST here gave the same show
two different "continue" answers, with the rail ranking it by its most recent touch and then
offering the episode abandoned weeks earlier. Two in-progress episodes are not exotic: a scrub to
the end leaves a stale `status='new', position>0` row until the daily reconcile."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
db.add_all(
[
PlexState(
user_id=user.id,
item_id=ep1.id,
status="new",
position_seconds=300,
progress_updated_at=NOW - timedelta(days=9),
),
PlexState(
user_id=user.id,
item_id=ep2.id,
status="new",
position_seconds=600,
progress_updated_at=NOW,
),
]
)
db.commit()
got = queries.next_episode(db, user.id, show.id)
assert got["id"] == "ps1e2"
# Asserted against the other implementation rather than against a literal, so the two cannot
# drift apart again without this failing.
ordered = [
leaf_card(ep, db.execute(select(PlexState).where(PlexState.item_id == ep.id)).scalar_one())
for ep in (ep1, ep2)
]
assert rollup.rollup(ordered)["resume"]["id"] == got["id"]
def test_next_episode_prefers_the_one_in_progress(db):
"""In progress beats "next unwatched": resuming is cheaper than starting."""
user, _ = _seed_yt(db)
_, show, ep1 = _seed_plex(db, user)
ep2 = db.execute(select(PlexItem).where(PlexItem.rating_key == "ps1e2")).scalar_one()
db.add_all(
[
PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW),
PlexState(user_id=user.id, item_id=ep2.id, status="new", position_seconds=120),
]
)
db.commit()
assert queries.next_episode(db, user.id, show.id)["id"] == "ps1e2"
# ── HTTP surface ──────────────────────────────────────────────────────────────────────────────────
def test_discover_endpoint_answers_in_one_pass(db, as_user):
user, _ = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
)
)
db.commit()
r = as_user(user).get("/api/discover?budget=medium")
assert r.status_code == 200
body = r.json()
assert body["budget"] == "medium"
assert body["hero"]["tier"] == "resume"
assert [c["id"] for c in body["continue_watching"]] == ["vmed"]
assert body["comfort"] == []
def test_an_unknown_budget_is_refused_rather_than_silently_widened(db, as_user):
user, _ = _seed_yt(db)
assert as_user(user).get("/api/discover?budget=whenever").status_code == 422
def test_pinning_twice_is_the_same_shelf(db, as_user):
"""The gesture is "make sure this is on my shelf", so a re-pin is a no-op, not a 409."""
user, _ = _seed_yt(db)
c = as_user(user)
first = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()
second = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()
assert first["created"] is True and second["created"] is False
assert first["id"] == second["id"]
assert len(c.get("/api/discover/pins").json()["items"]) == 1
def test_pinning_a_plex_show_resolves_to_the_episode_that_would_play(db, as_user):
"""Two of the five pinnable kinds are Plex, and the rating_key → internal-id translation is
theirs alone — a wrong column or a wrong model in either arm pins the wrong row, or 404s every
Plex pin. Neither arm had a test before the epic-end review."""
user, _ = _seed_yt(db)
_, _, ep1 = _seed_plex(db, user)
_enable_plex(db)
db.add(PlexState(user_id=user.id, item_id=ep1.id, status="watched", watched_at=NOW))
db.commit()
c = as_user(user)
assert c.post("/api/discover/pins", json={"kind": "plex_show", "ref": "ps1"}).status_code == 200
(tile,) = c.get("/api/discover/pins").json()["items"]
assert (tile["kind"], tile["id"]) == ("show", "ps1")
assert tile["next"]["id"] == "ps1e2" # ep1 is finished
def test_pinning_a_plex_movie_plays_itself_without_a_cycle(db, as_user):
"""The self-reference that once 500'd the whole shelf, on the other leaf kind: `next` has to be a
distinct dict, not the tile aliased into itself. Covered for a video since S1, never for a movie."""
user, _ = _seed_yt(db)
_seed_plex(db, user)
_enable_plex(db)
c = as_user(user)
assert c.post("/api/discover/pins", json={"kind": "plex_item", "ref": "pm1"}).status_code == 200
(tile,) = c.get("/api/discover/pins").json()["items"]
assert tile["id"] == "pm1" and tile["next"]["id"] == "pm1"
def test_the_reroll_endpoint_parses_its_whole_exclusion_list_and_its_press_count(db, as_user):
"""The reroll's behaviour is covered at the domain level; the ROUTE's query-string translation —
the comma split and the `start` passthrough — was not covered at all, so a change there could
hand back the same card on every press with the suite green. Excluding all three candidates is
what proves the whole list parsed: a delimiter change leaves one unsplit string in the set, and a
hero comes back.
NOT the `MAX_EXCLUDE` clamp, which this sends three keys against a cap of 32 and cannot see. The
docstring used to claim it did (round 2). The clamp is deliberately left unpinned: it guards
nothing the route would otherwise do — see the constant's own comment — and a test that sent 33
keys could only assert that the request still answers, which the comma split already proves."""
user, _ = _seed_yt(db)
db.add(
VideoState(
user_id=user.id,
video_id="vmed",
status="new",
position_seconds=300,
progress_updated_at=NOW,
)
)
db.add(ComfortPin(user_id=user.id, video_id="vlong", position=0))
db.commit()
c = as_user(user)
# `start` rotates which tier answers first: 0 leads with resume, 1 with the shelf.
assert c.get("/api/discover/pick?seed=1&start=0").json()["hero"]["tier"] == "resume"
assert c.get("/api/discover/pick?seed=1&start=1").json()["hero"]["tier"] == "comfort"
skip = ",".join(f"youtube:{v}" for v in ("vshort", "vmed", "vlong"))
assert c.get(f"/api/discover/pick?seed=2&exclude={skip}").json()["hero"] is None
assert c.get("/api/discover/pick?budget=whenever").status_code == 422
def test_pinning_something_that_does_not_exist_is_a_404(db, as_user):
user, _ = _seed_yt(db)
r = as_user(user).post("/api/discover/pins", json={"kind": "video", "ref": "nope"})
assert r.status_code == 404
def test_a_pin_whose_target_vanishes_mid_request_is_a_404_not_a_500(db, as_user, monkeypatch):
"""`add_pin`'s `except IntegrityError` arm was reached by no test at all, so the 500→404
narrowing could be reverted with the whole suite green (round 2). The 404 above comes from the
`resolve_ref` PRE-CHECK, a different branch that returns before the INSERT.
The race the arm exists for — a Plex sync dropping the row between `resolve_ref` confirming it
and the INSERT — needs two transactions, but the BRANCH does not: a `resolve_ref` pointing at an
id no row carries lets the pre-check pass and trips the same FK on commit, with the re-query
finding no winner. The sentinel has to FIT `comfort_pins.video_id` (varchar(16)) — a longer one
raises `DataError` on the way in, which this arm does not catch and never meant to. Its sibling arm (a concurrent duplicate, where the re-query DOES find a winner
and answers 200) genuinely needs a second connection and stays uncovered."""
user, _ = _seed_yt(db)
monkeypatch.setattr(pins, "resolve_ref", lambda *a, **k: ("video_id", "gone-mid-req"))
r = as_user(user).post("/api/discover/pins", json={"kind": "video", "ref": "vshort"})
assert r.status_code == 404
def test_another_users_playlist_is_not_pinnable(db, as_user):
"""404, not 403 — a 403 would confirm the id exists to a stranger."""
user, _ = _seed_yt(db)
other = User(email="other-e24@example.com")
db.add(other)
db.flush()
pl = Playlist(user_id=other.id, name="Theirs")
db.add(pl)
db.commit()
r = as_user(user).post("/api/discover/pins", json={"kind": "playlist", "ref": str(pl.id)})
assert r.status_code == 404
def test_reorder_ignores_ids_that_are_not_yours(db, as_user):
"""A stale tab replaying an order that includes a since-deleted pin should reorder what still
exists rather than fail wholesale."""
user, ch = _seed_yt(db)
c = as_user(user)
a = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()["id"]
b = c.post("/api/discover/pins", json={"kind": "channel", "ref": ch.id}).json()["id"]
assert c.post("/api/discover/pins/reorder", json={"ids": [b, 9999, a]}).status_code == 200
assert [p["pin_id"] for p in c.get("/api/discover/pins").json()["items"]] == [b, a]
def test_deleting_a_pin_reports_what_went(db, as_user):
user, _ = _seed_yt(db)
c = as_user(user)
pin_id = c.post("/api/discover/pins", json={"kind": "video", "ref": "vmed"}).json()["id"]
r = c.delete(f"/api/discover/pins/{pin_id}")
assert r.status_code == 200 and r.json() == {"deleted": pin_id}
assert c.get("/api/discover/pins").json()["items"] == []
+257
View File
@@ -0,0 +1,257 @@
"""S6 — per-user, per-host download cookies: host-scope, cache privacy, encryption, write-only API.
The two load-bearing security properties are covered as behaviour, not prose:
* **cache privacy** — a cookie-authed download is a PRIVATE, salted asset; a second user without a
cookie enqueuing the same URL gets the shared (unsalted) asset and can never be cache-served the
authed one, and the worker attaches a cookie ONLY to that private identity.
* **host scope (SEC-12)** — a cookie is resolved ONLY for a URL whose netloc the credential owns; a
look-alike or unrelated host resolves to no host_key and gets no cookie.
Pure-logic parts run with no DB; the CRUD / enqueue-salting / API parts use the `db`/`as_user` lane.
"""
import os
from pathlib import Path
import pytest
from cryptography.fernet import Fernet
from app import security
from app.downloads import credentials, formats, service
from app.models import DownloadCredential, MediaAsset, User
@pytest.fixture(autouse=True)
def _fernet(monkeypatch):
# A throwaway key so encrypt/decrypt work without TOKEN_ENCRYPTION_KEY in the env (mirrors
# test_security). Autouse: every credential write in this module encrypts.
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
COOKIES = "# Netscape HTTP Cookie File\n.facebook.com\tTRUE\t/\tTRUE\t0\tc_user\t123\n"
# --- host scope (pure logic) ------------------------------------------------------------------
class TestHostKeyForUrl:
def test_matches_a_supported_host(self):
assert credentials.host_key_for_url("https://www.facebook.com/watch?v=1") == "facebook"
assert credentials.host_key_for_url("https://fb.watch/abc/") == "facebook"
assert credentials.host_key_for_url("https://x.com/i/status/1") == "twitter"
def test_look_alike_host_never_matches(self):
assert credentials.host_key_for_url("https://myfacebook.com/x") is None
assert credentials.host_key_for_url("https://facebook.com.evil.test/x") is None
def test_unrelated_or_empty_url_has_no_host_key(self):
assert credentials.host_key_for_url("https://www.youtube.com/watch?v=abc") is None
assert credentials.host_key_for_url("") is None
assert credentials.host_key_for_url(None) is None
class TestValidateCookies:
def test_accepts_a_netscape_file(self):
assert credentials.validate_cookies(COOKIES).endswith("\n")
def test_rejects_empty(self):
with pytest.raises(credentials.InvalidCookies):
credentials.validate_cookies(" ")
def test_rejects_non_cookie_paste(self):
# Comments only, no tab-separated data line -> not a usable cookies.txt.
with pytest.raises(credentials.InvalidCookies):
credentials.validate_cookies("just some pasted text\n# a comment")
def test_authed_sig_differs_per_user_and_from_the_shared_sig():
spec = formats.normalize({})
shared = formats.format_sig(spec)
a = credentials.authed_format_sig(spec, 1, "facebook")
b = credentials.authed_format_sig(spec, 2, "facebook")
assert a != shared and b != shared and a != b
def test_write_cookiefile_is_private_and_survives_the_retry_wipe(tmp_path):
staging = tmp_path / "staging"
staging.mkdir()
path = credentials._write_cookiefile(COOKIES, staging)
assert path.read_text() == COOKIES
# 0600 (owner-only) — POSIX perms only (the worker is Linux-only; Windows has no mode bits).
if os.name == "posix":
assert (path.stat().st_mode & 0o777) == 0o600
# It lives in a SUBDIR so the download retry loop (which unlinks files in the staging ROOT) and
# _produced_file (which scans the root) never see it.
assert path.parent == staging / ".auth"
assert not any(p.is_file() for p in staging.iterdir())
# --- CRUD + encryption at rest (DB) -----------------------------------------------------------
def _user(db, email="cred@example.com") -> User:
u = User(email=email)
db.add(u)
db.flush()
return u
class TestCrud:
def test_upsert_stores_ciphertext_not_plaintext(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
row = db.get(DownloadCredential, 1)
assert row.cookies_enc != COOKIES # encrypted at rest
assert COOKIES not in row.cookies_enc
assert security.decrypt(row.cookies_enc) == COOKIES
def test_upsert_replaces_in_place(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
credentials.upsert_credential(db, u.id, "facebook", COOKIES.replace("123", "999"))
db.commit()
rows = db.query(DownloadCredential).filter_by(user_id=u.id, host_key="facebook").all()
assert len(rows) == 1
assert "999" in security.decrypt(rows[0].cookies_enc)
def test_status_list_never_leaks_the_blob(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
status = credentials.status_list(db, u.id)
fb = next(h for h in status if h["host_key"] == "facebook")
assert fb["configured"] is True and fb["updated_at"] is not None
ig = next(h for h in status if h["host_key"] == "instagram")
assert ig["configured"] is False and ig["updated_at"] is None
# No field anywhere carries the cookie value.
assert all(COOKIES not in str(v) for h in status for v in h.values())
def test_delete(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
assert credentials.delete_credential(db, u.id, "facebook") is True
db.commit()
assert credentials.get_credential(db, u.id, "facebook") is None
assert credentials.delete_credential(db, u.id, "facebook") is False
# --- the crux: enqueue salts privately, no cross-user cache serve (DB) -------------------------
FB_URL = "https://www.facebook.com/watch?v=100"
class TestCachePrivacy:
def test_enqueue_with_a_cookie_creates_a_private_salted_asset(self, db):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
job = service.enqueue(db, u.id, "url", FB_URL, {})
asset = db.get(MediaAsset, job.asset_id)
spec = formats.normalize({})
assert asset.format_sig == credentials.authed_format_sig(spec, u.id, "facebook")
assert asset.format_sig != formats.format_sig(spec)
def test_enqueue_without_a_cookie_uses_the_shared_sig(self, db):
u = _user(db)
job = service.enqueue(db, u.id, "url", FB_URL, {})
asset = db.get(MediaAsset, job.asset_id)
assert asset.format_sig == formats.format_sig(formats.normalize({}))
def test_two_users_never_share_the_authed_asset(self, db):
a = _user(db, "a@example.com")
b = _user(db, "b@example.com")
credentials.upsert_credential(db, a.id, "facebook", COOKIES)
db.commit()
ja = service.enqueue(db, a.id, "url", FB_URL, {}) # A: authed, private
jb = service.enqueue(db, b.id, "url", FB_URL, {}) # B: no cookie, shared
assert ja.asset_id != jb.asset_id
# B's asset is the shared identity — A's private group video can never be served to B.
assert db.get(MediaAsset, jb.asset_id).format_sig == formats.format_sig(
formats.normalize({})
)
# --- the worker gate: attach only to the private identity, host-scoped (DB) --------------------
class TestResolveCookiefile:
def _asset_sig(self, sig: str) -> MediaAsset:
return MediaAsset(
source_kind="url", source_ref=FB_URL, format_sig=sig, status="downloading"
)
def test_attaches_to_the_private_salted_asset(self, db, tmp_path):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
path = credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path)
assert path is not None and Path(path).read_text() == COOKIES
def test_never_attaches_to_a_shared_asset(self, db, tmp_path):
# Cookie exists, but the asset is the UNSALTED shared identity -> attaching would poison the
# global cache with a private-group video. Must return None.
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
shared = formats.format_sig(spec)
assert credentials.resolve_cookiefile(db, u.id, FB_URL, shared, spec, tmp_path) is None
def test_non_matching_host_gets_no_cookie(self, db, tmp_path):
u = _user(db)
credentials.upsert_credential(db, u.id, "facebook", COOKIES)
db.commit()
spec = formats.normalize({})
# A YouTube URL has no supported host_key -> no cookie regardless of what's stored.
yt = "https://www.youtube.com/watch?v=abc"
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, yt, salted, spec, tmp_path) is None
def test_no_credential_gets_no_cookie(self, db, tmp_path):
u = _user(db)
spec = formats.normalize({})
salted = credentials.authed_format_sig(spec, u.id, "facebook")
assert credentials.resolve_cookiefile(db, u.id, FB_URL, salted, spec, tmp_path) is None
# --- API: write-only, never echoes the blob (DB) ----------------------------------------------
class TestApi:
def test_put_get_delete_roundtrip_never_returns_the_blob(self, db, as_user):
u = _user(db)
db.commit()
client = as_user(u)
r = client.put("/api/downloads/credentials/facebook", json={"cookies": COOKIES})
assert r.status_code == 200
body = r.json()
assert body["ok"] is True
assert COOKIES not in r.text # the response must never carry the value
fb = next(h for h in body["hosts"] if h["host_key"] == "facebook")
assert fb["configured"] is True
r = client.get("/api/downloads/credentials")
assert r.status_code == 200 and COOKIES not in r.text
r = client.delete("/api/downloads/credentials/facebook")
assert r.status_code == 200
assert credentials.get_credential(db, u.id, "facebook") is None
def test_unknown_host_is_404(self, db, as_user):
u = _user(db)
db.commit()
r = as_user(u).put("/api/downloads/credentials/nope", json={"cookies": COOKIES})
assert r.status_code == 404
def test_bad_cookies_are_rejected(self, db, as_user):
u = _user(db)
db.commit()
r = as_user(u).put("/api/downloads/credentials/facebook", json={"cookies": " "})
assert r.status_code == 400
assert r.json()["detail"]["code"] == "cookies"
+103
View File
@@ -0,0 +1,103 @@
"""Worker-child resource de-prioritization (S4): the encoder thread cap, the `-threads` flag on the
browser-safe re-encode, and the child's startup renice/ionice/OMP handicap. These keep a CPU-heavy
transcode from pegging every core on the box it shares with the API, Postgres, and Plex playback."""
import subprocess
import pytest
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import runner
class TestEncodeThreadCap:
def test_pinned_setting_wins(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 3)
assert editmod.encode_thread_cap() == 3
def test_zero_reserves_one_core(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: 8)
assert editmod.encode_thread_cap() == 7
def test_single_core_box_never_caps_below_one(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: 1)
assert editmod.encode_thread_cap() == 1
def test_unknown_cpu_count_falls_back(self, monkeypatch):
monkeypatch.setattr(settings, "download_ffmpeg_threads", 0)
monkeypatch.setattr(editmod.os, "cpu_count", lambda: None)
assert editmod.encode_thread_cap() == 1 # max(1, 2 - 1)
class TestBrowserSafeThreads:
def test_threads_flag_added_when_capped(self):
cmd = editmod.build_browser_safe_cmd(
"/in.mkv", "/out.mp4", reencode_video=True, reencode_audio=False, threads=2
)
assert "-threads" in cmd
assert cmd[cmd.index("-threads") + 1] == "2"
# The cap precedes the output file, so it binds the encoder (not input decoding).
assert cmd.index("-threads") < cmd.index("/out.mp4")
@pytest.mark.parametrize("threads", [0, None])
def test_no_threads_flag_when_uncapped(self, threads):
cmd = editmod.build_browser_safe_cmd(
"/in.mkv", "/out.mp4", reencode_video=True, reencode_audio=False, threads=threads
)
assert "-threads" not in cmd
class TestDeprioritize:
def test_renices_by_configured_increment(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 12)
monkeypatch.setattr(settings, "download_child_ionice_class", 0) # skip ionice branch
calls = []
monkeypatch.setattr(runner.os, "nice", lambda n: calls.append(n))
monkeypatch.setattr(runner.os, "environ", {})
runner._deprioritize()
assert calls == [12]
def test_nice_zero_does_not_renice(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 0)
calls = []
monkeypatch.setattr(runner.os, "nice", lambda n: calls.append(n))
monkeypatch.setattr(runner.os, "environ", {})
runner._deprioritize()
assert calls == []
def test_sets_omp_budget_to_thread_cap(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 0)
monkeypatch.setattr(settings, "download_ffmpeg_threads", 4)
env: dict = {}
monkeypatch.setattr(runner.os, "environ", env)
runner._deprioritize()
assert env["OMP_NUM_THREADS"] == "4"
def test_ionice_invoked_when_binary_present(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 3)
monkeypatch.setattr(runner.os, "environ", {})
monkeypatch.setattr(runner.shutil, "which", lambda name: "/usr/bin/ionice")
ran = []
monkeypatch.setattr(
runner.subprocess,
"run",
lambda cmd, **kw: ran.append(cmd) or subprocess.CompletedProcess(cmd, 0),
)
runner._deprioritize()
assert ran and ran[0][:3] == ["ionice", "-c", "3"]
def test_ionice_skipped_when_binary_absent(self, monkeypatch):
monkeypatch.setattr(settings, "download_child_nice", 0)
monkeypatch.setattr(settings, "download_child_ionice_class", 3)
monkeypatch.setattr(runner.os, "environ", {})
monkeypatch.setattr(runner.shutil, "which", lambda name: None)
ran = []
monkeypatch.setattr(runner.subprocess, "run", lambda cmd, **kw: ran.append(cmd))
runner._deprioritize()
assert ran == []
+27
View File
@@ -1,5 +1,6 @@
"""Category classification of raw yt-dlp / ffmpeg download-failure strings (S2)."""
from app.downloads import errors as dlerrors
from app.downloads.errors import classify
@@ -14,6 +15,13 @@ def test_real_ytdlp_messages_map_to_categories():
"ERROR: [youtube] abc: Sign in to confirm you're not a bot": "login_required",
"ERROR: [youtube] abc: Join this channel to get access to members-only content": "login_required",
"ERROR: [Reddit] 1vi1x56: Account authentication is required.": "login_required",
# Closed-group / private-post auth walls: yt-dlp either says to sign in or suggests cookies.
"ERROR: [facebook] 123: You must be logged in to see this post": "login_required",
"ERROR: [facebook] 123: Login required. Use --cookies for the authentication.": "login_required",
"ERROR: [instagram] abc: This content is only available for registered users": "login_required",
"ERROR: [youtube] abc: Sign in to confirm... Use --cookies-from-browser or --cookies": "login_required",
# A site yt-dlp has no extractor for at all — distinct from an auth wall, still non-retryable.
"ERROR: Unsupported URL: https://example.com/whatever": "unsupported_host",
"ERROR: [youtube] abc: The uploader has not made this video available in your country": "geo_blocked",
"ERROR: [youtube] abc: Requested format is not available": "format_unavailable",
"ERROR: [youtube] abc: Only images are available for download": "format_unavailable",
@@ -47,3 +55,22 @@ def test_login_beats_generic_unavailable_when_both_present():
"This video is unavailable"
)
assert classify(msg) == "login_required"
def test_our_own_worker_refusals_classify_as_themselves():
# The worker raises these itself (not yt-dlp), and the rule that catches each one matches only a
# fragment of the text. Pinning message and rule together is what stops an edit to the wording
# from silently orphaning the category — the UI would then fall back to "Download failed".
assert classify(dlerrors.QUOTA_FULL_MESSAGE) == "quota_exceeded"
def test_a_url_jobs_network_failure_is_redacted_but_youtube_keeps_its_text():
# SEC-12: the raw transport text for a user-chosen host:port is a port-scanner readout, so it is
# replaced for `url` jobs. Everything else must survive verbatim, or a real diagnosis is lost.
raw = "ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>"
assert dlerrors.user_error_text("url", raw, "network") == dlerrors.NETWORK_REDACTED_MESSAGE
# Same failure, but the destination was never user-chosen — nothing to leak, keep the detail.
assert dlerrors.user_error_text("youtube", raw, "network") == raw
# A user-chosen host, but a category that says something real about the CONTENT, not the network.
login = "ERROR: [facebook] 123: You must be logged in to see this post"
assert dlerrors.user_error_text("url", login, "login_required") == login
+109
View File
@@ -10,6 +10,8 @@ Uses the `db` fixture (real Postgres) — skipped when no database is reachable.
its OWN session, so every test COMMITS its setup before calling it.
"""
from app.config import settings
from app.downloads import pipeline, quota
from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.worker import _claim_job
@@ -169,3 +171,110 @@ def test_list_downloads_is_capped(db, monkeypatch):
monkeypatch.setattr(dl, "_MAX_LIST", 2)
rows = dl.list_downloads(user=u, db=db)
assert len(rows) == 2 # bounded by the cap, not the 3 rows on disk
# --- E12 · S3 · the worker-side max_bytes remainder ------------------------------------------
def _sized_asset(db, size_bytes, ref):
a = MediaAsset(
source_kind="url",
source_ref=ref,
format_sig="s",
status="ready",
ref_count=1,
size_bytes=size_bytes,
)
db.add(a)
db.flush()
return a
def test_remaining_bytes_is_none_for_an_unlimited_user(db):
u = User(email="unlimited@example.com")
db.add(u)
db.flush()
_quota(db, u.id, max_concurrent=2, unlimited=True)
db.flush()
# None, not a number: "no cap" and "no budget left" must never be the same value downstream —
# build_ydl_opts leaves max_filesize unset for both, so only the caller can tell them apart.
assert quota.remaining_bytes(db, u.id) is None
def test_remaining_bytes_is_the_unspent_part_of_the_cap(db):
u = User(email="partial@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 400, "partial-1")
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 600
def test_remaining_bytes_is_zero_once_the_cap_is_met(db):
# THE case this exists for: `check_enqueue` let these jobs in while the user was under the cap,
# and an earlier one has filled it since. Zero is what makes the worker refuse; if this ever
# returned the raw negative or None instead, the user already over quota would be the only one
# downloading with no size ceiling at all.
u = User(email="atcap@example.com")
db.add(u)
db.flush()
db.add(
DownloadQuota(user_id=u.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False)
)
a = _sized_asset(db, 1500, "atcap-1") # overshot: one job finished larger than the remainder
_job(db, u.id, queue_pos=0, status="done", asset_id=a.id)
db.flush()
assert quota.remaining_bytes(db, u.id) == 0
def test_a_refusal_for_one_user_does_not_poison_the_shared_asset(db, monkeypatch, tmp_path):
"""A user at their cap fails their own job, and another user's queued job for the same video
still downloads.
MediaAssets are deduped across users, and the generic failure handler writes the message onto
that shared row — correct for an upstream failure, wrong for "YOUR storage is full", which
`worker._process_job` would then hand verbatim to someone with an empty library.
"""
monkeypatch.setattr(settings, "download_root", str(tmp_path))
over, other = User(email="over@example.com"), User(email="other@example.com")
db.add_all([over, other])
db.flush()
db.add(
DownloadQuota(
user_id=over.id, max_bytes=1000, max_concurrent=2, max_jobs=100, unlimited=False
)
)
full = _sized_asset(db, 1500, "already-held")
_job(db, over.id, queue_pos=0, status="done", asset_id=full.id)
# `downloading` is the state the worker leaves it in — `_claim_asset` compare-and-sets it before
# the child runs — and `_reset_asset_pending` only acts on that status. Seeded as `pending` this
# assertion would pass with the release deleted.
shared = MediaAsset(
source_kind="youtube",
source_ref="x" * 11,
format_sig="s",
status="downloading",
ref_count=2,
)
db.add(shared)
db.flush()
mine = _job(db, over.id, queue_pos=1, status="running", asset_id=shared.id)
theirs = _job(db, other.id, queue_pos=2, asset_id=shared.id)
db.commit()
# The refusal happens before extraction, so nothing needs stubbing — if it ever stops happening
# first, this test starts reaching the network, which is its own signal.
pipeline._download(mine.id, shared.id, "youtube", "x" * 11, {}, over.id)
db.expire_all()
assert db.get(DownloadJob, mine.id).status == "error"
assert db.get(DownloadJob, mine.id).error_code == "quota_exceeded"
# The asset is released, NOT marked error — otherwise the next line's job inherits the refusal.
assert db.get(MediaAsset, shared.id).status == "pending"
assert db.get(MediaAsset, shared.id).error is None
assert db.get(DownloadJob, theirs.id).status == "queued"
+58 -21
View File
@@ -6,7 +6,7 @@ with "PosixPath('.') has an empty name" instead of downloading."""
import pytest
from app.downloads.edit import _concat_quote, parse_probe_streams
from app.downloads.edit import _codecs_from_streams, _concat_quote, run_ffprobe_json
from app.downloads.pipeline import _media_info, _produced_file
from pathlib import Path, PurePosixPath
@@ -105,39 +105,76 @@ class TestMediaInfo:
assert _media_info(info)["id"] == "vid"
class TestParseProbeStreams:
class TestCodecsFromStreams:
# ffprobe emits codec_name BEFORE codec_type, so the old line parser shifted every name onto
# the previous stream and an H.264+AAC mp4 came back as (aac, None) → a needless re-encode.
# _codecs_from_streams reads each stream dict by its own codec_type, immune to that ordering.
def test_reads_each_stream_by_its_own_type(self):
payload = '{"streams":[{"codec_name":"h264","codec_type":"video"},{"codec_name":"aac","codec_type":"audio"}]}'
assert parse_probe_streams(payload) == ("h264", "aac")
streams = [
{"codec_name": "h264", "codec_type": "video"},
{"codec_name": "aac", "codec_type": "audio"},
]
assert _codecs_from_streams(streams) == ("h264", "aac")
def test_audio_first_ordering_is_handled(self):
payload = '{"streams":[{"codec_name":"AAC","codec_type":"audio"},{"codec_name":"H264","codec_type":"video"}]}'
assert parse_probe_streams(payload) == ("h264", "aac")
streams = [
{"codec_name": "AAC", "codec_type": "audio"},
{"codec_name": "H264", "codec_type": "video"},
]
assert _codecs_from_streams(streams) == ("h264", "aac")
def test_extra_streams_do_not_win_over_the_first_of_their_kind(self):
payload = (
'{"streams":[{"codec_name":"h264","codec_type":"video"},'
'{"codec_name":"vp9","codec_type":"video"},'
'{"codec_name":"mov_text","codec_type":"subtitle"},'
'{"codec_name":"opus","codec_type":"audio"}]}'
)
assert parse_probe_streams(payload) == ("h264", "opus")
streams = [
{"codec_name": "h264", "codec_type": "video"},
{"codec_name": "vp9", "codec_type": "video"},
{"codec_name": "mov_text", "codec_type": "subtitle"},
{"codec_name": "opus", "codec_type": "audio"},
]
assert _codecs_from_streams(streams) == ("h264", "opus")
def test_audio_only_and_video_only(self):
assert parse_probe_streams('{"streams":[{"codec_name":"opus","codec_type":"audio"}]}') == (
assert _codecs_from_streams([{"codec_name": "opus", "codec_type": "audio"}]) == (
None,
"opus",
)
assert parse_probe_streams('{"streams":[{"codec_name":"av1","codec_type":"video"}]}') == (
"av1",
None,
)
assert _codecs_from_streams([{"codec_name": "av1", "codec_type": "video"}]) == ("av1", None)
def test_garbage_is_not_a_crash(self):
assert parse_probe_streams("") == (None, None)
assert parse_probe_streams("not json") == (None, None)
def test_empty_stream_list(self):
assert _codecs_from_streams([]) == (None, None)
class TestRunFfprobeJson:
"""The single ffprobe-JSON invocation point (shared by probe_codecs + probe.probe_media). A failed
probe or unparseable output must degrade to None, not raise — the malformed-input safety the old
parse_probe_streams string parser used to own now lives here."""
def test_unparseable_output_returns_none(self, monkeypatch):
import subprocess
class _R:
stdout = "not json"
monkeypatch.setattr(subprocess, "run", lambda *a, **k: _R())
assert run_ffprobe_json(Path("x")) is None
def test_ffprobe_failure_returns_none(self, monkeypatch):
import subprocess
def _boom(*a, **k):
raise subprocess.CalledProcessError(1, "ffprobe")
monkeypatch.setattr(subprocess, "run", _boom)
assert run_ffprobe_json(Path("x")) is None
def test_parses_streams_from_valid_json(self, monkeypatch):
import subprocess
class _R:
stdout = '{"streams":[{"codec_name":"h264","codec_type":"video"}]}'
monkeypatch.setattr(subprocess, "run", lambda *a, **k: _R())
data = run_ffprobe_json(Path("x"), entries="stream=codec_type,codec_name")
assert _codecs_from_streams(data["streams"]) == ("h264", None)
class TestConcatQuote:
+149
View File
@@ -0,0 +1,149 @@
"""An undelivered email leaves a trace in the admins' inbox.
Sending is deliberately fail-soft — a missing SMTP setting or a rejecting relay logs and returns
False so nothing user-facing breaks. The cost of that used to be total silence: an instance whose
SMTP was never filled in swallowed verification links, password resets and access-request notices,
and the operator's first clue was somebody saying nothing arrived.
"""
import smtplib
import pytest
from sqlalchemy import select
from app import email as email_mod
from app.models import Notification
@pytest.fixture(autouse=True)
def _no_smtp_config(monkeypatch):
"""Every test here drives the real `_send`, so make sure it never reaches a socket."""
monkeypatch.setattr(
email_mod,
"_smtp",
lambda: {"host": "", "port": 587, "user": "", "password": "", "from": ""},
)
def _notifications(db, user_id: int) -> list[Notification]:
return list(
db.execute(
select(Notification)
.where(Notification.user_id == user_id, Notification.type == "email_undelivered")
.order_by(Notification.id)
)
.scalars()
.all()
)
def test_a_skipped_email_notifies_every_admin(db, make_user):
admin_a = make_user(email="admin-a@example.com", role="admin")
admin_b = make_user(email="admin-b@example.com", role="admin")
plain = make_user(email="plain@example.com")
assert email_mod.send_test("someone@example.com", "en") is False # fail-soft, as before
db.expire_all()
for admin in (admin_a, admin_b):
(n,) = _notifications(db, admin.id)
assert n.data["reason"] == "not_configured"
assert n.data["subject"] == "Siftlode SMTP test"
assert n.data["recipients"] == ["someone@example.com"]
assert _notifications(db, plain.id) == [] # not an operator's problem to see
def test_the_notice_is_throttled_so_a_dead_relay_cannot_flood_the_inbox(db, make_user):
admin = make_user(email="admin@example.com", role="admin")
for _ in range(5):
email_mod.send_test("someone@example.com", "en")
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_send_failure_is_reported_separately_from_a_missing_config(db, make_user, monkeypatch):
# A configured-but-rejecting relay is a different operator problem from an empty config, and the
# inbox copy differs — so the reason has to survive to the notification.
admin = make_user(email="admin@example.com", role="admin")
monkeypatch.setattr(
email_mod,
"_smtp",
lambda: {
"host": "smtp.example.com",
"port": 587,
"user": "u",
"password": "p",
"from": "Siftlode <u@example.com>",
},
)
def _boom(*a, **kw):
raise smtplib.SMTPException("nope")
monkeypatch.setattr(email_mod.smtplib, "SMTP", _boom)
assert email_mod.send_test("someone@example.com", "en") is False
db.expire_all()
(n,) = _notifications(db, admin.id)
assert n.data["reason"] == "send_failed"
def test_no_admins_means_no_notification_and_no_error(db, make_user):
# A fresh instance can have no admin yet (the wizard has not run). The alert must stay silent
# rather than blowing up inside the fail-soft path it is supposed to make visible.
plain = make_user(email="plain@example.com")
assert email_mod.send_test("someone@example.com", "en") is False
db.expire_all()
assert _notifications(db, plain.id) == []
def test_a_drop_with_nobody_to_tell_does_not_burn_the_hour(db, make_user):
# The throttle window must only be spent on a notice that was actually written. Otherwise a drop
# during setup — before any admin exists — silences the next hour for the admin created a minute
# later, which is exactly the blind spot this alert exists to remove.
email_mod.send_test("someone@example.com", "en") # no admins yet
admin = make_user(email="admin@example.com", role="admin")
assert email_mod.send_test("someone@example.com", "en") is False
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_failed_alert_does_not_burn_the_hour_either(db, make_user, monkeypatch):
admin = make_user(email="admin@example.com", role="admin")
boom = {"on": True}
def _maybe_boom(*a, **kw):
if boom["on"]:
raise RuntimeError("notification store is down")
from app.notifications import create_notification as real
return real(*a, **kw)
monkeypatch.setattr(email_mod, "create_notification", _maybe_boom)
email_mod.send_test("someone@example.com", "en") # the write blows up; nothing recorded
boom["on"] = False
assert email_mod.send_test("someone@example.com", "en") is False
db.expire_all()
assert len(_notifications(db, admin.id)) == 1
def test_a_broken_alert_never_breaks_the_send(db, make_user, monkeypatch):
# The alert is the fallback path; if IT fails, the caller must still get the ordinary fail-soft
# False rather than an exception surfacing from a background task.
make_user(email="admin@example.com", role="admin")
def _boom(*a, **kw):
raise RuntimeError("notification store is down")
monkeypatch.setattr(email_mod, "create_notification", _boom)
assert email_mod.send_test("someone@example.com", "en") is False

Some files were not shown because too many files have changed in this diff Show More