Compare commits

..
144 Commits
Author SHA1 Message Date
peter cc1f196a2c fix(build): drop the unused readMerged export so the publish gate passes
gate / frontend (push) Failing after 1m22s
gate / backend (push) Successful in 2m5s
knip flagged src/lib/storage.ts readMerged as an unused export — its only
caller is readAccountMerged in the same module, so it doesn't need to be
exported. This aborted the 0.72.0 publish gate after the merge; no behaviour
change.
2026-08-23 07:31:24 +02:00
peter e3dac34bb0 release: 0.72.0 (feature/plex-filter-bar-overhaul)
gate / frontend (push) Failing after 41s
gate / backend (push) Successful in 2m6s
2026-08-23 07:27:18 +02:00
peter 938135bb59 chore(release): 0.72.0 — Plex browsing overhaul + editable Collection page
Plex filter-bar overhaul, in-header Discover/Browse + Playlist/Collection
modes, filter-aware Discover rails, poster-collage tiles, Recently-watched
rail, capped unseen Surprise, saved views on Plex, per-page scroll memory, and
a full Collection info page with write-through Plex editing (create/rename/
add/remove/delete). Migration 0073 (saved_views.source). Two deep-review rounds.
2026-08-23 07:26:43 +02:00
peter 7d2bee59a3 fix(plex): round-2 deep-review follow-ups on the review fixes
- collections: the resync-failed-after-successful-write path now returns 422
  (honest "reconcile on next sync" detail), NOT 502. The frontend treats
  502/503/504 as gateway-unreachable and raises a sticky offline banner
  (api/core.ts RETRIABLE_GATEWAY); the app answered, so a 422 action-error
  toast is correct. Removes the false-offline regression the 502 introduced,
  and — since 422 isn't retried — the create double-write vector with it.
- tests: make the mutate-then-resync rollback assertion non-vacuous — the
  resync stub now flushes a partial local change before raising, so the test
  proves db.rollback() actually undoes it; split into a write-failed (422)
  case and a resync-failed (422 + reconcile detail + rollback) case.
- New-collection dup hint is now muted-tone with its own nameMaybeTaken copy
  (per-library uniqueness means the name may be free here) instead of the red
  authoritative nameTaken, which the still-enabled create flow contradicts.
- history: cover the navCounter post-reload collision guard (ensureNavId
  advances the counter past a higher persisted sfScroll id).
2026-08-23 07:22:53 +02:00
peter 6af238c2ea fix(plex): resolve deep-review findings on the Discover/Collections batch
Round of USER-triggered deep-review fixes (11 CONFIRMED + 1 PLAUSIBLE, all
verified) on the 22-commit branch:

- collections: split the mutate-then-resync error surface. A resync READ that
  fails after a successful Plex mutation no longer reports a false 422 "Plex
  write failed" + rollback; it returns 502 "sync pending" (the next full sync
  reconciles the mirror). New _write_or_pending_error helper; delete is exempt
  (no Plex read). Covered by new happy-path / 422-rollback / 502 tests.
- New-collection dup-name guard is now advisory only: existingNames is the
  search-scoped, all-library union, but the backend enforces uniqueness PER
  library — so it no longer blocks a valid same-name collection in another
  library, and the backend 409 is localized to nameTaken.
- member-remove now confirms before the write-through (matches delete-collection).
- add-title / new-collection library search render an error state on request
  failure instead of a misleading "No matches".
- history: seed navCounter past the current entry's persisted sfScroll on load
  so post-reload ids can't collide with back-stack entries and mis-restore scroll.
- PageScroller: LRU-cap the navId-keyed savedScroll map (was unbounded).
- share isIntrinsicReadOnly between the browse tile and the info page so the
  lock/edit affordance can't drift.
2026-08-23 06:57:33 +02:00
peter b71b33654a feat(plex): forbid duplicate collection names + a "New collection" button
Two follow-ups after testing the collection editing against real Plex, which revealed that
Plex CONFLATES same-named collections (two rating_keys resolve to one merged member list) —
a genuine footgun.

- Duplicate-name guard: create and rename now reject (409) a title that already exists in the
  target section (case-insensitive), before any Plex write. create() can also derive the
  library from the seed item, so the caller needn't know the section key. The create dialogs
  surface the 409, and the New-collection panel warns inline before submit.
- A "New collection" button on the Collections browse surface (admin only) opens a panel:
  a name field (with the live duplicate check) + a search to pick the FIRST title — Plex can't
  create an empty collection, it needs a seed. Creating opens the new collection's detail page
  straight away, so you can keep adding titles there. This is the B-route the user wanted,
  vs. the only prior path (a movie's "add to collection").

Verified live: the 409 guard (creating an existing name is blocked, no Plex write), and the
full New-collection flow (name + first title → create → lands on the detail page); test
collections created and deleted, real account left clean.

Gate: tsc/lint/knip/prettier clean, 424 vitest; backend ruff clean, 7 collections tests
(+ a duplicate-name-rejection assertion).
2026-08-23 05:51:48 +02:00
peter 5d99a5f253 feat(plex): Collection info page — hero, play/shuffle, and write-through editing (B)
Clicking a collection tile now opens a dedicated Collection info page (like the Plex-web
one) instead of just applying it as a filter. New GET /collections/{rk} returns the hero
aggregates (year range, most-restrictive member cert, total runtime) + the members ordered
by release year then title; a new PlexCollectionView renders the hero (art/title/meta/
summary+More) and the member grid, with a Play-all and Shuffle over the playable leaves
(reusing the player's existing queue + auto-advance).

For a non-external, admin-enabled collection the page offers rename / delete / add-title
(a search picker) / remove-item, all through the pre-existing admin endpoints that write
THROUGH to Plex (each edit awaits and re-fetches — no optimism, since a Plex write can 422).
An eligible collection shows "Enable editing" first (the DB editable flag). Read-only ones
(smart / imdb / tmdb / … auto-lists) are marked with a lock on both the info page AND the
browse tile, up front.

New `{ kind: "collection" }` sub-view wired through plexSubUrl (bookmarkable ?v=collection:rk),
PlexBrowse (render branch, Backspace back-nav, tile handler), with a round-trip test.

Verified live: tile→page, hero + chronological members (James Bond 1962→2021, PG-13, 55h),
Play/Shuffle, Enable editing → edit controls + remove-X, the add search (marks already-in
titles "Added"), and the read-only lock on external collections. Real Plex mutations wired
but not executed against the live account.

Gate: tsc/lint/knip/prettier clean, 424 vitest; backend ruff clean, 6 collections tests
(+ a detail-hero/ordering assertion).
2026-08-23 04:40:09 +02:00
peter c0d869089d feat(plex): add a "Recently watched" rail to Discover (D)
A new Discover rail listing what the user last FULLY watched, newest first: a movie as its
own card, a show as the specific EPISODE watched (carrying its show title via leaf_card). Sits
above Recently added. Movies honour the sidebar filters like the other rails; episodes carry no
filterable metadata so they don't. Clicking a card plays it (the existing onCard path already
routes episodes to the player). Bilingual heading added.

Verified live: the rail renders watched movies + episodes with Watched badges.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, 13 discover tests (+ a
recently-watched movie+episode assertion).
2026-08-23 04:08:28 +02:00
peter 15f5a5694a feat(plex): Surprise-me shows only unseen titles, capped 3 movies + 3 shows (C)
The Surprise rail drew a random 20+20 sample including already-watched titles. Its point is
to surface something to WATCH, so exclude seen titles (a watched movie, a fully-watched show
— a partially-watched or never-started show still counts as unseen) and cap at 3 movies + 3
shows (6 total), interleaved.

Verified live: /surprise returns 6 items, 3+3, none watched.

Gate: backend ruff clean, 12 discover tests (+ a watched-exclusion/cap assertion).
2026-08-23 04:01:22 +02:00
peter 14a44bff6e fix(plex): show-library collections get montage art too (A3)
The collection browse-tile montage queried only movie members, so show-library
collections (IMDb Popular, TMDb Trending, … in the TV library) rendered as bare colour
tiles. Union the art from PlexShow members as well (a collection lives in one library, so
exactly one side has art); fall back to the collection's own poster, then the colour hash.
Runtime stays movie-only (shows have no single duration).

Verified live: every collection now shows a montage (0 art-less tiles, was ~all the show
collections).

Gate: backend ruff clean, 5 collections tests (+ a show-collection art assertion).
2026-08-23 03:57:20 +02:00
peter 37d11b602f feat(plex): redesign browse tiles — 2-poster montage top, readable label band
The full-bleed 4-poster collage drowned the on-tile text. Split PosterTile in two: the top
half shows up to 2 montage posters, the bottom half is a solid colour-hash band that carries
the label, so the text is always readable. Genre tiles drop the count (just the label now);
collection tiles show the name on one truncated line plus a sub-line of item count · total
runtime. A group with no art falls back to an all-colour tile.

Backend: `/collections` cards gain `total_runtime_s` (a new `runtime_by_key` grouped sum) and
their montage art is capped at 2. Frontend: PosterTile reworked; a `collectionMeta` helper
builds the count·runtime sub-line via the existing formatRuntime.

Verified live: collection and genre tiles render 2 posters over a readable colour band with
the right sub-lines; colour fallback intact for art-less groups.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, collections tests green
(+ a new art/runtime assertion).
2026-08-23 03:52:21 +02:00
peter da3cb5e694 feat(plex): poster collages on the genre + collection browse tiles
The Browse-by-genre tiles and the Collections browse tiles were flat color-hash
gradients. Give each a 2×2 collage of its top films' poster art, with the color hash
kept as the fallback so a group with no matching art (a show-only or empty collection)
never renders blank.

Backend: a shared `top_art_keys` windowed query returns, per key (a genre value or a
collection rating_key), the top few items' rating_keys — best-rated then newest, in one
query. Wired into the Discover genre tiles and the collections union endpoint; the client
renders the posters from the existing /api/plex/image/{key} endpoint, so no new image
plumbing. Frontend: a shared PosterTile component (collage + scrim + label, color-hash
fallback) replaces the two ad-hoc colored-tile implementations.

Verified live on localdev: genre tiles and collection tiles render real poster collages,
with the color fallback correctly kicking in for groups without art.

Gate: tsc/lint/knip/prettier clean, 423 vitest; backend ruff clean, 11 discover tests
(incl. a new genre-art assertion).
2026-08-23 03:10:36 +02:00
peter 1e5232f47a feat(nav): restore scroll position per history entry, open forward views at the top
The app scrolls one persistent <main> (PageScroller); its saved offsets were keyed by a
content `scrollKey` that is the same for a Plex page's Discover and grid views, so opening
a Browse-by-genre tile inherited the scrolled-down Discover offset (landing mid-page) and
Back couldn't tell the two views apart.

Tie scroll memory to the history entry instead: every pushState that opens a main-content
view stamps a fresh `sfScroll` id into history.state (setPage, Plex sub-view open); the
initial/legacy entries get one lazily via replaceState. PageScroller now saves/restores per
`${entryId}:${scrollKey}` — the entry id makes Back restore exactly where you left an entry
and a forward navigation (new id) open at the top, while the scrollKey still resets a new
feed filter to the top within the same entry. A small nav bus lets a Plex sub-view open
(which doesn't re-render the scroller) trigger the restore; `history.scrollRestoration` is
set to manual since the browser can't see our inner scroller.

Verified live: genre tile opens the grid at the top and Back returns to the Discover scroll;
feed→plex→Back restores the feed scroll; a feed filter change still resets to the top.

Gate: tsc/lint/knip/prettier clean, 5 new nav-id unit tests, full vitest green.
2026-08-23 02:55:59 +02:00
peter dddaeb9a38 fix(plex): make the header Discover/Browse toggle clickable
The floating header wrapper is pointer-events-none (so clicks fall through its empty
area to the content below); every interactive child must re-enable pointer events.
PlexViewToggle rendered its SegmentedToggle without the pointer-events-auto wrapper the
sibling PlexScopeToggle has, so the whole Discover/Browse switch was click-dead — the
buttons rendered and were correctly wired, but no real mouse click ever reached them.

Add `className="pointer-events-auto shrink-0"` to the toggle and a regression test that
asserts the root re-enables pointer events (plus the render/onChange/hide-on-deeper-view
contract). Verified live on localdev: real clicks now switch Discover⟷Browse (?v=browse)
in both directions; a header-wide hit-test audit shows no other dead controls.

Gate: tsc/lint/knip/prettier clean, 418 vitest.
2026-08-23 02:23:09 +02:00
peter 70a74f8629 fix(plex): round-3 deep-review follow-ups
Third review round, scoped to the round-2 fix commit. The production ternary was
confirmed correct; two sibling-surface findings:

- The sidebar CollectionPicker's clear-all still wrote collectionTitles:{} while the
  chip-row remove (fixed in round 2) wrote undefined — a divergence that leaves the
  {}-vs-absent cache split on the picker path. Collapse an empty map to undefined there
  too, so both no-collection routes key-match the pristine state.
- The queryKeys lang test was near-tautological (a variadic-spread factory can't drop a
  passed arg), and nothing asserted the CALL SITE keys carry lang. Add a real guard in
  PlexDiscover.dom.test.tsx that inspects the registered query keys for scope+lang, and
  reword the factory test's comment to say it only shape-pins the segment pass-through.

Gate: tsc/lint/knip/prettier clean, 413 vitest.
2026-08-23 02:09:14 +02:00
peter 8f1309fa18 fix(plex): round-2 deep-review follow-ups on the review fixes
Second review round, scoped to the round-1 fix commit (where round-1's own fixes
introduced new code):

- The collection chip-remove pruned collectionTitles to an empty {} on last removal,
  which still hashes differently from the pristine no-collection state (key absent) in
  the React Query keys → a residual cache split. Collapse an empty map to undefined so
  the key-shape matches the pristine one.
- Replace the tautological "order-independent signature" test (both objects spread from
  EMPTY_PLEX_FILTERS inherited the same key order, so it proved nothing) with full
  literals in genuinely different insertion order that would fail an order-sensitive
  JSON.stringify signature.
- Cover the collectionTitles chip-remove prune (usePlexActiveFilters.dom.test.tsx) and
  the plexSurprise/plexDiscover lang-key split (queryKeys.test.ts) with regression tests.

Gate: tsc/lint/knip/prettier clean, 412 vitest.
2026-08-23 01:53:33 +02:00
peter b7a4eaff8d fix(plex): resolve deep-review findings on the filter-bar overhaul
Self-review of the branch diff (feature/plex-filter-bar-overhaul) against main:

- Surprise rail query key omitted `lang` while the backend /surprise now filters
  against the language-specific genre column → a HU/EN switch under an active genre
  filter left the rail stale. Key it by scope+lang+filters, matching plexDiscover.
- Chip-removing a collection left its entry in `collectionTitles` (hashed into the
  plex query keys), splitting the cache from the pristine no-collection state and
  growing the titles map unboundedly. Prune the title on remove, like CollectionPicker.
- The saved-views default-mirror effect gained `cfg` in its deps (re-memoized on every
  live filter edit) → a synchronous localStorage write per keystroke. Depend on the
  module-stable mirror fn instead.
- Collections-browse search input had no accessible name (placeholder only). Add
  aria-label + type="search".
- Extract the tile hue-hash + gradient duplicated between PlexDiscover and
  plexBrowseLists into a shared lib/hueTile so the two surfaces can't drift.
- Document migration 0073 downgrade() as lossy and de-dup non-feed rows before
  recreating the narrow (user_id, name) constraint (verified by the drift test).
- Add unit tests for coercePlexFilters (legacy single-collection migration) and
  plexFiltersSignature (order-independence, title-ignoring).

Gate: tsc/lint/knip/prettier clean, 410 vitest; backend ruff + migration drift +
saved-views tests green.
2026-08-23 01:39:29 +02:00
peter a20e27800c feat(plex): port the saved-views (personal views) UI to Plex (frontend)
Plex now has the same personal-views system as the feed — the rail header icon
opens the quick-pick popover, the rail title opens the manager modal (list /
create / rename / delete / reorder / set-default / apply), and the active view
leads the chip row with a "modified" dot.

One SavedViewsProvider serves both modules: it switches `source` (feed ⟷ plex)
by the active page (both filter providers are global, so it reads whichever the
page owns) and runs the same logic on per-source wiring — the live filters,
apply, a stable signature (feed filtersToParams / plex plexFiltersSignature),
coercion (asFilters / coercePlexFilters), and what to strip on save. Moved below
PlexProvider in the tree so it can read plex state; resets sticky/modal on a
source switch. The shared popover/modal/chip are unchanged; the modal describes
a plex view's captured filters via a pure buildPlexActiveFilters (extracted from
usePlexActiveFilters — no duplication). api.savedViews/createView take a source;
`SavedView.filters` is now an opaque blob. New coercePlexFilters/plexFilters-
Signature (also dedups PlexProvider's collection migration). Empty-state copy
genericised (no longer says "feed"). A Plex view captures the rail filters (not
watch-state/sort/scope).

Verified end-to-end on localdev: create/apply/preview/default/delete on plex,
feed views stay separate + unregressed (402 vitest), no console errors.
2026-08-22 19:16:30 +02:00
peter fa7a09a420 feat(saved-views): scope saved views by module source (backend)
Saved "smart views" were feed-only; the Plex module is getting them too. Add a
`source` discriminator to saved_views ("feed" | "plex"), default "feed" so every
existing row and any source-less client stays feed. Widen the per-user unique
name constraint to (user_id, source, name) so a feed view and a Plex view can
share a name, and scope create's position numbering + the is_default "clear the
others" to (user, source).

Migration 0073 (ADD source NOT NULL server_default 'feed' + swap the unique
constraint; reversible). Endpoints: GET/POST take `source` (default feed).
Tests: source isolation, per-source default, per-source name uniqueness.
2026-08-22 18:56:52 +02:00
peter f983a491c0 feat(plex): add Playlist/Collection browse modes to the header pill
The header library-scope pill gains two entry points — Playlist and Collection —
next to All/Movies/Shows, a second way into each area (the rail filter is the
other). Picking Playlist shows a grid of the user's playlists (a tile opens that
playlist's view); Collection shows a searchable grid of collections (a tile
filters the title grid by that collection). Pure front-end — reuses the existing
plexPlaylists / plexCollections(union) list endpoints.

Two new top-level sub-views (`playlists` / `collections`, bookmarkable as
?v=playlists / ?v=collections) rendered by PlexBrowse via the new plexBrowseLists
surfaces (hue-tiled like the Discover genre tiles). The PlexViewBridge now carries
the view `kind`; PlexScopeToggle is a 5-way that derives its active segment from
it (browse-mode when in one, else scope) and drives the two views through the
bridge — picking a scope from a browse mode returns to the title grid. The
Discover/Browse toggle hides in these modes; Backspace + the search→grid flip
cover them.

Tests: PlexScopeToggle (5-way + browse-mode drive/active), plexSubUrl round-trip
for the two new views. Gate: tsc/lint/knip/prettier + 402 vitest.
2026-08-22 18:19:08 +02:00
peter f85dc12819 feat(plex): move the Discover/Browse toggle into the header
The Discover ⟷ Browse switch now sits in the header next to the library-scope
pill (All / Movies / Shows), instead of on the chip row / atop the Discover
surface.

It's driven through a small one-way PlexViewBridge: PlexBrowse still OWNS the
sub-view (useHistorySubview) and publishes its top-level view + opener to the
bridge; the always-mounted Header reads it to render PlexViewToggle. A global
sub-view provider was deliberately avoided — PlexBrowse's hook remounts on every
plex entry and re-seeds from history (a page switch clears ?v + drops _sub), so
each entry starts at the Discover root; lifting it out would let the view go
stale across navigation. The toggle hides on deeper views (show/info/person/…)
and on non-plex pages (bridge cleared on unmount). sortView is just SortControl
now; the grid toolbar no longer carries the toggle.

Verified: toggle in the header, switches discover⟷grid, hides on info + on Feed,
and Feed→Plex resets to Discover (no stale view); no console errors.
2026-08-22 18:02:55 +02:00
peter 9798e506cf feat(plex): apply the sidebar filters to the Discover rails too
Discover now behaves like the Browse grid — the active metadata filters (genre /
rating / year / collection / studio / …) narrow every rail: Continue watching,
Recently added, Surprise me, the genre tiles, and Because you watched. Same
LibraryFilters / meta_filter_conds the grid uses, applied to each rail's movie +
show query (movie duration included); a rail may come back empty, which the client
already hides. Genre-language matches the grid.

Because Discover honours filters itself, adding a filter no longer force-flips the
Discover landing to the grid (only a header SEARCH does, since Discover has no
search rail) — so you stay on Discover with narrowed rails.

Frontend threads `filters` into api.plexDiscover/plexSurprise + their query keys
(plexSurprise is variadic now) and the watched-toggle setQueryData; PlexDiscover
reads them from usePlex. Tests: two discover filter-narrowing cases; PlexDiscover
dom test updated for the provider default + 3-arg call.
2026-08-22 17:35:37 +02:00
peter a60164edc4 feat(plex): make the Collection filter multi-select with Any/All
A title can belong to several Plex collections, so the collection filter becomes
multi-select combining with Any (OR, default) / All (AND) — the same shape as the
genre/actor filters.

Backend (query.py): `collection` (single) → `collections` CSV + `collection_mode`
on LibraryFilters; meta_filter_conds ORs/ANDs `collection_keys.contains` per id.
Frontend: PlexFilters `collection`/`collectionTitle` → `collections` + optional
`collectionMode` + `collectionTitles` map; CollectionPicker is now a multi-select
Combobox with the any/all match footer; one chip per collection (usePlexActive-
Filters); the info-page "Browse collection" strip unions into the list; applyFilter
now shallow-merges label maps so adding one keeps the others' titles. PlexProvider
migrates a legacy persisted single `collection` into the new shape on read.

Test: test_plex_collections_filter (single / any-union / all-intersection).
2026-08-22 08:18:25 +02:00
peter 6c7038cf1f feat(plex): make the playlists rail widget a compact combobox
The playlists section used to render an always-open list plus a dedicated
create input, growing the rail taller with every playlist. Replace it with a
single Combobox field (reusing the shared component, like the feed's "Your
tags"): the list of playlists only appears when the dropdown is opened, and
typing a name that doesn't exist offers a "Create …" row. Picking a playlist
opens it (the existing playlistOpen hand-off); the create row creates then opens
the new one. `selected` stays empty — this picks an action, not a persisted value.
2026-08-22 07:59:41 +02:00
peter f8e144a658 fix(combobox): stop a single chip wrapping the chevron to a second line
The field's input carried min-w-16 (64px). Flex line-packing sizes a flex item
at its min-width, so in a narrow rail field (~202px) one chip (~119px) + the
64px-reserved input + the chevron overflowed the line, wrapping the chevron
down while the (grown, empty-looking) input filled the first line — the "breaks
to two rows even though it'd fit" glitch seen on the Plex Collection widget and
the feed's channel picker (both are Combobox). Lowering the floor to min-w-8
(32px) keeps chip + input + chevron on one line; flex-1 still grows the input to
fill the remaining width. Verified in-browser: field height 54px→36px (one row).
2026-08-22 07:56:05 +02:00
peter b4984dc38b feat(plex): move Watch-state out of the rail into a toolbar pill
The "Show" (watch-state) filter leaves the left rail and becomes a FilterSelect
pill next to the item count, mirroring the feed's Watch-state control — a
top-level "what am I looking at" choice, always in reach, not buried in the rail.
It still emits a removable chip when non-default (same as the feed).
2026-08-22 07:51:54 +02:00
peter 21c9b34540 feat(plex): port the filter bar to the YT-feed pattern
Bring the Plex module's filter surface in line with the feed's final shape:

- Flatten the rail: PlexSidebar passes `collapsible={false}` to FilterRail so
  the widgets become plain always-open label + control rows (no collapse
  chevrons; reorder/hide in edit mode stay). Drop the rail's own "Clear all" —
  the chip row above the grid already owns it.
- Move Sort + the Discover/Browse toggle into the applied-filter chip row,
  pinned right directly above the grid (a `sortView` fragment, like the feed's),
  and give ActiveFilterChips `flex-1` so it gets the shared responsive overflow
  (wide: fit + "+N" pill; narrow: single "N filters" count pill) — the chips can
  no longer crowd the controls off-screen.

ItemCount leads its own toolbar row, matching the feed. No behaviour change to
Discover, which keeps its standalone view toggle.
2026-08-22 07:23:20 +02:00
peter 041cd526b7 Merge: YT feed filter-bar overhaul + 3 deep-review rounds
gate / frontend (push) Failing after 38s
gate / backend (push) Successful in 1m57s
Flattened, more compact YT feed filter rail and a responsive applied-filter row:
- Rail: removed the in-rail Clear all; Upload date is now a single-select dropdown
  (Any time + more presets + reveal-on-select Custom range); widgets are plain
  label + control rows (non-collapsible), with the "Your tags" title opening the
  tag manager.
- Sort + View moved into the applied-filter row (pinned right, above the grid)
  with responsive overflow: "+N" pill on wide lanes, a single count pill on narrow.
- Earlier branch work also merges here: typeable combobox filters, hidden-channel
  empty->Library hint, Mine/Library scope toggle in the header, per-category
  any/all tag match, and the saved-views redesign (popover + modal + chip).

Three USER-triggered deep-review rounds (trend 14 -> 4 -> 1); round 2 caught a real
coerce-on-seed migration bug. Gate-green: tsc, 397 vitest, lint, prettier, backend
pytest. Not yet released -- prod stays 0.71.0.
2026-08-22 07:09:25 +02:00
peter 9477fea3dc docs(test): correct the feed-views fixture comment on topic tag distribution
chA/chB each carry a different topic tag (T1 vs T2), not both — the old comment
implied they shared T1/T2. Assertions unchanged (equivalence test).
2026-08-22 07:04:00 +02:00
peter e4b19b5320 fix(feed): round-2 deep-review remediations (incl. a real coerce-on-seed bug)
The re-review of the fix round surfaced 4 findings; fixing the coverage one
exposed that the C1 migration was ineffective:
- FeedFiltersProvider.loadAccountFilters: run coerceTagModes on the RAW stored
  blob, not the object merged over DEFAULT_FILTERS. The merge injects
  DEFAULT_TAG_MODES (all "or"), which coerceTagModes prefers over the legacy
  scalar `tagMode` — so the migration never fired. Now a persisted legacy
  tagMode:'and' actually migrates into tagModes.topic (proven by a new test).
- test_feed_views: the tag_mode_other=and case was vacuous (only chC carried
  either "other" tag, so AND == OR). Give chA an "other" tag too so OR-other
  {chA,chC} differs from AND-other {chC} — the conjunctive path now narrows.
- DateFilterSelect: reset the custom reveal in useLayoutEffect (not useEffect)
  so the correction commits before paint — no one-frame "Custom range" flicker
  on the trigger after an external clear.
- SavedViewsProvider: extract the sticky-view tie-break to a pure, tested
  lib/savedViews.ts (preferSticky).

New tests: savedViews (preferSticky), FeedFiltersProvider.dom (legacy tagMode
migration on the localStorage/default-view seed).
2026-08-22 06:55:16 +02:00
peter 7692253636 test(feed): cover the filter-bar gaps deep-review flagged
- test_feed_views: extend _seed_tagged so a channel carries 2+ tags in a
  non-topic category, and add language-AND / other-AND equivalence cases — the
  per-category any/all generalization was only exercised for topic.
- tagModes.test: coerceTagModes migration (legacy global tagMode -> topic),
  per-category preservation, invalid-value coercion, defaults.
- urlState.dom.test: per-category tag-mode URL round-trip + legacy ?tagmode=and
  migration into the topic slot.
- Combobox.dom.test: the any/all match footer (hidden below 2 selections, reports
  a mode change at 2+).
- chipOverflow.test: the all/some/count fit boundaries and the +N reservation.
2026-08-22 06:33:48 +02:00
peter e90603d4db fix(feed): deep-review remediations for the filter-bar overhaul
Correctness / behaviour:
- Move the Personal Views modal from inside the collapsible filter rail to the
  feed page (moduleRegistry FeedPage), so opening it from the always-visible
  toolbar view chip works while the rail is unpinned (its default) instead of
  no-op'ing and latching modalOpen into a later stale pop-open.
- Migrate a persisted legacy global `tagMode` into per-category `tagModes` on the
  localStorage seed paths too (loadAccountFilters), matching the saved-view and
  URL paths — an upgrading user's topic any/all no longer silently reverts to 'or'.
- DateFilterSelect: reset the custom reveal on an external clear while the popover
  is closed (guard on !open, not maxAgeDays), so the trigger reverts to "Any time"
  instead of lingering on "Custom range" with empty fields.
- PersonalViewsModal delete: only advance the detail selection when the delete
  actually happened (remove() now resolves a boolean) — cancelling the confirm no
  longer switches the selected view.
- SavedViewsProvider: prefer the explicitly-applied (sticky) view among several
  that capture identical filters, so apply(B) isn't re-attributed to an earlier
  duplicate A (wrong "modified" chip name).

A11y:
- Give the clickable panel title a descriptive accessible name (titleActionLabel,
  wired to "Personal views") instead of announcing only "Filters".
- Give the "+N" overflow pill an aria-label ("N more filters").
- Use t("views.reorder") for the saved-view drag handle instead of a hardcoded
  English "reorder" literal.

Also extract the chip-fit decision into a pure, testable lib/chipOverflow.ts and
measure the inter-chip gap live (rem-safe) instead of a frozen 6px constant.
2026-08-22 06:33:48 +02:00
peter dd76e9dcf7 feat(feed): move sort + view into the chip row with responsive overflow
Sort and the view switcher are tightly bound to the grid, so pin them to the
right of the applied-filter row (directly above the cards) instead of the filter
toolbar; they render there on both the feed and channel-scoped views.

ActiveFilterChips now shares that row with the controls and trims itself so it
never crowds them, measuring fit off a hidden mirror of the full chip set:
- wide  -> show as many chips as fit, then a "+N" pill whose popover holds the rest;
- narrow -> collapse every chip into one "N filters" pill whose popover holds them all.
Both popovers list the (overflowed / all) chips, each removable, plus Clear all.
2026-08-22 05:42:45 +02:00
peter fca989c098 feat(feed): flatten the filter rail — dropdown upload-date, non-collapsible widgets, title triggers
Rework the feed filter rail into simpler label + control rows:
- Remove the rail's own "Clear all" (the chip row above the cards already
  surfaces one whenever any filter is active).
- Replace the Upload date chip group with a single-select dropdown
  (DateFilterSelect): an "Any time" clear, more presets (2 weeks / 3 months /
  2 years / 5 years) and a "Custom range" that reveals from/to on select; the
  trigger then reads the chosen range. Selection logic is the pure lib/dateFilter
  (applyDateChip -> selectDateChip, no toggle-off; tests updated).
- Widgets are no longer collapsible: PanelGroup gains `collapsible` (feed passes
  false) so groups render as always-open label + control rows; reorder + hide in
  edit mode stay. Other rails (Plex/Playlists) keep the collapsible islands.
- A non-collapsible group's title can act as its own trigger (PanelItem
  onTitleClick): "Your tags" opens the tag manager, replacing the inline Manage
  button.
2026-08-22 05:42:30 +02:00
peter 7ebbf45921 feat(feed): saved views as a header popover + Personal Views modal + a view chip
Retire the standalone 'Saved views' sidebar panel. The saved-views feature now
lives in three lighter surfaces, coordinated by a new SavedViewsProvider (query +
CRUD + the active/sticky view + modal state):

- the filter panel's header ICON opens a quick-pick popover (apply a view, active
  one checked, plus 'Manage views…');
- the header TITLE ('Filters') opens a new Personal Views modal — a master/detail
  manager: left = the reorderable list + a save-current-filters input, right = the
  selected view's name, its captured filters (shown as the same chips the toolbar
  row uses), default toggle, apply, and delete;
- a SelectedViewChip leads the applied-filter row, naming the active view with a
  warn-coloured 'modified' dot once the live filters diverge from it; clicking it
  reopens the picker.

SidePanel/FilterRail gain optional iconExpanded + onTitleClick slots (feed-only,
hidden for the demo account); the collapsed tab keeps its plain pin behaviour.
SavedViewsWidget is deleted and dropped from the panel layout.

Verified live: icon popover lists HH + Manage; applying HH surfaces the chip and
switches scope; removing a filter lights the dirty dot; the title opens the
master/detail modal with the captured-filters summary. 374 vitest, tsc, eslint
(0 err).
2026-08-22 04:03:58 +02:00
peter 829cadf5fb feat(feed): per-category any/all tag match, in each combobox's popover footer
The tag 'Match: Any/All' toggle was a single GLOBAL setting that only ever
affected topics, shown as its own row under the Topic widget. Make it
per-category — language, topic, and 'your tags' each get their own independent
any/all — and move it into each filter combobox's popover footer (like
SortControl's direction segment), reclaiming the standalone Match row.

Backend: the feed query already groups selected tags by category; replace the
single tag_mode with per-category tag_mode_language/topic/other params and apply
the conjunctive (HAVING all) branch per category. Mirrored across filtered_query,
feed_summary.passes(), and the /facets drill-down exclusion (kept in lockstep;
equivalence test updated to tag_mode_topic). No schema change.

Frontend: FeedFilters.tagMode -> tagModes {language,topic,other} (pure leaf
lib/api/tagModes with a coerceTagModes migration for the legacy global value);
serialized to the API and the shareable URL per category (legacy ?tagmode=and
still resolves into the topic slot); saved-view blobs migrate on read. Combobox
gains an optional matchMode/onMatchModeChange footer segment, shown once 2+
values are picked; the three tag comboboxes wire it to their category.

Verified: backend feed tests green (12, incl. the count/facets equivalence net);
live, the footer appears at 2+ selections, toggles aria-pressed, serializes
tagmode_topic=and to the URL, and the request carries tag_mode_topic=and (200).
374 vitest, tsc, ruff, eslint (0 err).
2026-08-22 03:48:23 +02:00
peter 8f08073564 feat(feed): move the Mine/Library scope toggle beside the header search box
The scope toggle used to sit at the top of the filter rail, which is buried when
the rail is collapsed/unpinned. Relocate it to the header, just right of the feed
search box — mirroring Plex's scope toggle placement — so the primary 'what am I
browsing' control is always visible.

New FeedScopeToggle adapter (SegmentedToggle over filters.scope, same as the Plex
one over its scope); rendered in Header on the feed page between the search box and
the sync chip. Removed the toggle from the Sidebar rail's now-unused `top` slot and
dropped the dead `top` prop from FilterRail. Same filters.scope field, so behaviour
is unchanged.

Verified live: the toggle sits beside the feed search and switches scope; the rail
no longer carries it.
2026-08-22 03:29:15 +02:00
peter af08a7850b feat(feed): guide a hidden-channel empty result to the Library
Filtering by a channel under the Mine scope returns nothing when that channel's
subscription is hidden (or you don't subscribe to it) — its videos only live in
the shared Library, since the Mine feed excludes hidden subscriptions by design.
Previously this left a bare 'No videos match these filters', which reads as a
bug when you know you're subscribed.

Add an empty-state hint + a 'View in Library' button (switches scope to all) when
there's a channel filter under Mine and zero results. Bilingual EN/HU.

Verified live: ?channel=<ColdFusion>&show=all under Mine shows the hint; the
button switches to Library and surfaces the 36 videos.
2026-08-22 03:25:49 +02:00
peter d2193ccd1b fix(combobox): merged input ref so the field is typeable by mouse
The <input> carried only floating-ui's reference ref, so the component's own
inputRef stayed null. The field container's onMouseDown then preventDefault()-ed
every press (its `e.target !== inputRef.current` guard was always true) and the
compensating inputRef.current?.focus() was a no-op — so clicking the box opened
the list but never focused the input, and mouse users could not type to filter.
Tab focus still worked, which is why it read as 'click, can't type'.

Feed a merged callback ref that assigns the node to both floating-ui's
setReference and inputRef. Fixes the Channel, Your tags, Topic, and Language
sidebar filters at once (all use Combobox); the channel list's 50-row cap is now
reachable past 50 via search. Single-select commit no longer re-focuses (that now
actually fires and would re-open the just-closed list). Regression test presses
the field chrome and asserts the input receives focus.

Verified live: clicking the Channel field focuses it and typing 'cold' narrows
the list to ColdFusion / wehatethecold.
2026-08-22 03:22:47 +02:00
peter cda70164ba Merge: toolbar/header UI-consistency refactor + a11y fixes
gate / frontend (push) Failing after 1m13s
gate / backend (push) Successful in 1m52s
Collects the toolbar/header/layout redesign and assorted bugfixes on the
bug/channel-back-header-overlap branch: new FilterSelect / ItemCount /
pill-SortControl primitives, removed ModuleName, the --sbw / --content-mw /
--rail-pinned-w chrome-alignment machinery, the centred search box, and the
two deep-review rounds (menu-role a11y, sort aria-pressed, rem rail threshold,
accessible trigger names, e2e helper fix, added coverage).

Not a release: VERSION stays 0.71.0 and this is not deployed to prod.
2026-08-22 02:07:20 +02:00
peter 5e1cc4d981 fix(a11y,test): deep-review round 2 — e2e helper attr, trigger names, observer test
Second gate round over the full branch surfaced 4 verified findings (1 was a
regression the round-1 role change introduced); a 5th (--sbw load jump) was
dropped as refuted (the header only mounts after auth+data, so --sbw is set
before it ever paints).

- e2e (medium): the round-1 switch of FilterSelect rows from role=option to
  menuitemcheckbox moved their state from aria-selected to aria-checked, but
  the Playwright helper expectContentType still asserted aria-selected, so the
  channel specs would fail whenever the e2e lane runs (not part of the fast
  gate, so it stayed green). Assert aria-checked.
- a11y (low): the FilterSelect and SortControl pill triggers set
  aria-label={label}, which overrides the visible "Label: value" text, so a
  screen reader announced only the group name, never the current selection.
  Carry the value (and, for sort, the direction) in the accessible name,
  matching ViewSwitcher. Updated the unit-test trigger name queries to match.
- tests (low): cover the PageShell font-scale MutationObserver recompute path
  (a post-mount root font-size change → --rail-pinned-w re-computes once, no
  loop) — previously only the pre-render and resize paths were exercised.

Verified live: the triggers now announce e.g. "Sort: Date, Descending" /
"Watch state: All". Gate green: tsc (app+node+e2e), 373 vitest, prettier,
eslint (0 err).
2026-08-22 01:06:22 +02:00
peter 4ee452b854 fix(a11y,layout): deep-review fixes — menu roles, sort aria-pressed, rem rail threshold
Epic-end deep-review of the branch surfaced six verified findings; fix them.

a11y (medium): FilterSelect and SortControl opened their AnchoredPopover in
mode:"listbox" but never passed a role, so the surface fell through to
role="menu" while the rows rendered role="option"+aria-selected — options
orphaned outside a listbox, so AT could not announce the selected value.
Switch both to the house menu pattern (as ViewSwitcher already uses): a menu
surface with menuitemradio (single) / menuitemcheckbox (multi) rows carrying
aria-checked, scoped by a role=group. Trigger stays a button+haspopup=menu.

a11y (low): the SortControl asc/desc direction segment signalled the active
direction by colour alone. Add aria-pressed so the current direction is
programmatically exposed, matching SegmentedToggle.

correctness (low): PageShell converted a rem maxWidth to px with a hardcoded
16 to decide the pinned-rail overlay threshold, but index.css scales the root
font by --font-scale (default 1.06 ~ 17px, slider 0.9-1.3), so the threshold
was wrong at every text size on the rem pages (playlists is the only one with
a pinned rail). Use the live root font-size, scale the 36rem slack the same
way, and recompute when --font-scale changes (a guarded MutationObserver — the
slider fires no resize).

tests: cover the two gaps the review flagged — ItemCount (plural, 1M+ overflow,
undefined icon-only, tooltip) and the PageShell rail-pinned-w/content-mw
threshold math incl. the font-scale scaling. Updated the FilterSelect/
SortControl role queries to the new menuitemradio/checkbox roles.

Verified in the running app: menu surfaces carry role=group + menuitemradio/
checkbox with 0 orphaned options, and the sort direction segment exposes
aria-pressed. Dropped two review findings (max-width duplication, pill-class
reuse) as refuted on adversarial verification. Gate green: tsc, 372 vitest,
prettier, eslint (0 err).
2026-08-21 22:41:57 +02:00
peter edbc9dede5 style(header): centre the search-box group over the page content
The floating header band already centres on --content-mw (the width the
page's cards use), but its search box + scope-toggle/sync-chip group was
packed to the left edge (default justify-start), leaving it offset from
the centred card grid on max-width pages (Plex/Channels/Playlists).

Add justify-center to the band so the group centres within it: once the
column is wider than the box's max-w, the box sits centred over the
cards instead of hugging the left edge. Shared header row, so it applies
to every module with a SearchBar. Verified at 1920px on Plex: group
centre == content-band centre (431px margin each side).
2026-08-21 22:08:23 +02:00
peter 53ca4e2849 feat(layout): a pinned filter rail no longer pushes content when there's room for it
On a page that centres its content at a max width, if the window is wide enough for the rail to fit
in the content's LEFT margin without overlapping it, pinning the rail now leaves it OVERLAY — the
content doesn't shift. It reverts to the old in-flow "push" only when there isn't room: a full-width
page (feed) or a window too narrow for the rail to clear the centred content.

PageShell computes --rail-pinned-w from the page's maxWidth vs the viewport (innerWidth >= maxWidth +
~576px → overlay 2.25rem, else in-flow 16.75rem) on a resize listener; the SidePanel slot reads it.

Verified at 1920px: pinning the playlists rail leaves the table at left 487 (moved 0); the rail sits
256px wide ending at 312 — 175px clear of the content. Feed (full width) still pins in-flow.
2026-08-21 21:53:24 +02:00
peter 1801a58f6b feat(layout): centre the header on each page's content width, centre the playlist table
Every page's floating header (search bar + sync chip) and the toolbar band now centre on the SAME
max-width the content column uses, instead of spanning the whole column while a max-w page centres
its grid. So the search bar lines up with the left edge of the cards/table on every page, and sits
centred with the content — not against the full page width.

- PageShell takes a `maxWidth`, published as --content-mw; the header adds max-w-[--content-mw] mx-auto
  inside its left-4 / right-(1rem+scrollbar) span, so it centres on that width (`none` = full width).
- moduleRegistry PAGE_CONTENT carries each page's content max width (plex 1600px, channels 96rem,
  playlists 64rem, downloads 56rem, notifications/users 48rem; feed stays full width) — one source.
- Playlists: its content (header, filter row, table) was left-aligned max-w-5xl → now mx-auto, so the
  table is centred and the header lines up with it.

Verified at 1920px: header left edge == content left edge — feed 340, channels 213, playlists 487.
2026-08-21 21:38:49 +02:00
peter 011acd919e feat(toolbar): lead the Plex toolbar with ItemCount too, unify its gaps
Plex browse now uses the same fixed-width ItemCount pill as the feed (its title count → "N items"),
placed as the leading element, and the toolbar gap tightened 2 → 1.5 to match. One consistent
toolbar shape across the source modules.
2026-08-21 20:37:36 +02:00
peter 45d8b47366 feat(toolbar): fixed-width ItemCount pill, lead the toolbar with it, align chrome to the cards
- New ItemCount: a generic, FIXED-WIDTH count pill (glass, non-interactive) — "192,050 items"
  (neutral noun, dynamic plural), min-w-[9rem] so a changing number never shifts the controls
  beside it; a million or more collapses to "1M+" with the exact figure in the title tooltip.
- Feed toolbar: ItemCount is the fixed leading element now; dropped the mx-1 separators so every
  gap is a uniform 1.5 (Content-types↔Source used to be wider than Watch-state↔Content-types).
- Chrome-to-card alignment (app-wide, via the shared Header + PageShell): the floating header and
  the toolbar band now span exactly the card grid's width — left-4, and a right edge of
  1rem + the scrollbar gutter (measured once into --sbw), so the fixed chrome lines up with the
  scrolled cards instead of running ~15px past them. Wraps cleanly to multiple rows on mobile.

Renames the count testid feed-result-count → item-count (E2Es updated).
2026-08-21 20:33:55 +02:00
peter c3820a0b44 refactor(header): remove the ModuleName module-switcher pill
The user no longer wants the header's "FEED" pill (module label + accent dot + ◀/▶ module-step
arrows). Drop it everywhere — the header now starts with the module's SearchBar; module switching
stays on the nav rail.

Deletes ModuleName.tsx, the now-unused stepModule() in lib/modules, and the Header's
module-order/label/setPage plumbing that only fed it.
2026-08-21 19:58:19 +02:00
peter c7b393901f style(type): correct the trigger text from 16px back to the view-switcher's real 14px
The earlier bump measured the view-switcher's BUTTON (16px, inherited) instead of its visible
label span (text-sm = 14px), and 14px is the size the user actually finds comfortable. Walk the
over-sized triggers back to text-sm: FilterSelect, SortControl, ViewSwitcher, and the Combobox
input (which was 14px before the bump anyway). The secondary text already at text-sm/text-xs
(count, chips, widget headers, section labels) was correct and stays.
2026-08-21 19:53:14 +02:00
peter f1e14c94c5 style(viewswitcher): unify the ViewSwitcher trigger with the FilterSelect pill
The user prefers FilterSelect's rounded-full glass pill over the ViewSwitcher's squarer
rounded-lg trigger. Give the ViewSwitcher trigger the same look — glass-card/glass-hover,
rounded-full, px-3 py-1, open→border-accent, and a 16px label (was text-sm) — so the two read
as one control family. Applies to every ViewSwitcher consumer (feed, channel manager, Plex
playlists).

Native <select>s, the Combobox input box, and playlist/list cards are a different control class
(form fields / variable-height inputs / cards) and keep their own shape.
2026-08-21 19:49:10 +02:00
peter ddb171fca3 style(type): raise the too-small toolbar and filter text toward the view-switcher size
The view-switcher's 16px reads comfortably; the toolbar and filter controls sat at 12px. Bring
the marked controls up to match, and lift the general scale on the shared filter surfaces.

- Toolbar (feed + channel + Plex, via the shared components): FilterSelect and SortControl trigger
  pills text-xs → text-base (16px, the view-switcher size); the result count → text-sm; the
  active-filter chips → text-sm with the "FILTERED BY" label text-2xs → text-xs.
- Filterbar: the Combobox picker input → text-base (matching the toolbar pills) + its chips → text-sm;
  the panel/widget headers (SidePanel collapsed-rail + panel title, PanelGroup widget titles) up one step.

Channel info and the manager tables were already at 14–16px (readable), so they were left as-is.
2026-08-21 19:37:11 +02:00
peter ec3bbc78dc style(layout): tighten the oversized vertical gaps around the page toolbars
The floating-header → toolbar → filter-chips → content stack carried ~2× the whitespace it
needed. Halve it, app-wide where the chrome is shared:
- PageShell: the --hdr-h gap below the floating header 8px → 4px (every page's header).
- Feed: toolbar wrapper pt-4 → pt-2, the toolbar's own pb-3 → pb-1.5, and the grid's top pt-2
  → pt-1. Measured on the feed: header→toolbar 24→12px, toolbar→chips 12→6px, chips→cards 12→8px.
- Plex / Playlists / ChannelDiscovery / DownloadCenter / Notifications / AdminUsers: toolbar
  wrapper pt-3 → pt-2, so every page's top band matches the tighter feed.

PageShell.dom.test updated for the 4px gap.
2026-08-21 19:13:38 +02:00
peter 1be1dc9d06 feat(sort): give SortControl the FilterSelect pill look with a footer direction segment
Rebuilds SortControl's internals (API unchanged, so all five consumers — feed, Plex,
playlists, and the two table/card ColumnSortControl sites — get it for free) as a labeled
"Label: key ↓ ▾" pill on the AnchoredPopover platform, matching FilterSelect. The listbox
lists the sort keys; an asc/desc segment sits in the panel footer (with the trailing slot —
feed reshuffle, playlists dirty-first — beside it).

Two-dimensional key+direction, so it stays its own component rather than reusing FilterSelect:
the trigger is a single plain button (no nested button), picking a key keeps the panel open so
key AND direction can be set in one visit, and picking "no sort" (tables' defaultLabel) closes
it. Directionless keys (shuffle) hide the segment and drop the trigger arrow.

Drops the now-redundant external "Sort" label spans in Feed and PlexBrowse (the trigger carries
the label now). Rewrites SortControl.dom.test for the popover UI.
2026-08-21 18:11:45 +02:00
peter 6109e0c6f1 feat(feed): convert the Source filter to the FilterSelect dropdown
Replaces the native <select> "Source" control with a labeled FilterSelect pill, matching
the Watch state / Content types pills — one consistent toolbar idiom. Single-select over
the three librarySource values (organic / all / search-only).

Sort and View are intentionally left as-is: Sort is a two-dimensional key+direction control
(plus reshuffle and a tables-only "no sort") shared across five surfaces, and View is already
on the same AnchoredPopover platform with an icon-only trigger tuned for this packed toolbar —
FilterSelect's text trigger would regress it. Neither fits FilterSelect's one-dimensional model.
2026-08-21 17:36:33 +02:00
peter 5851ddd329 feat(feed): compress the toolbar chip groups into labeled FilterSelect dropdowns
Adds a reusable FilterSelect: a "Label: value ▾" pill that opens a listbox (a checkmark
row for single-select, a checkbox row for multi), on the AnchoredPopover platform. It
replaces the two feed-toolbar chip walls:
  - Watch state (single-select, default All) — picking a row replaces + closes.
  - Content types (multi-select, default Normal; requireOne keeps at least one on) — rows
    toggle and the panel stays open.
The trigger summarises multi picks as "First +N". The component is source-agnostic, meant
for reuse anywhere a small known option set currently renders as a chip group.

FilterSelect takes a `testId` → the trigger gets it, each option gets `${testId}-${value}`,
so the feed/channel E2Es keep stable selectors. They now open the dropdown and assert the
option's aria-selected instead of the old inline aria-pressed chips (new expectContentType/
toggleContentType helpers).
2026-08-21 17:28:57 +02:00
peter ad0cf5827b feat(feed): use the Combobox picker for Channel/Your tags/Topic/Language filters
Replaces the hand-rolled channel search list and the flat tag chip walls with the
searchable, ARIA-correct Combobox built for the Plex filter bar (E5), so the YT feed
filters get the same input+listbox+chips UX with facet counts.

- ChannelPicker now wraps Combobox (selectedLabels keeps picks named); drops its bespoke
  input + scroll-fade list.
- Sidebar language/topic/tags: each Combobox owns only its slice of the shared
  filters.tags list (setTagSelection replaces that list's ids, leaves the rest). Topic
  keeps its any/all match toggle, Your tags keeps Manage. TagChip removed.
2026-08-21 17:01:41 +02:00
peter f6192aa6bf fix(feed): re-measure header on feed↔channel Back so --hdr-h can't stick at 8px
PageShell is reused (never remounted) across feed↔channel navigation — same
component position in App.tsx — so a one-shot useLayoutEffect kept its
ResizeObserver bound to the FIRST header node. On channel-open that node detached
and fired the observer with a 0 box, rewriting --hdr-h to 0+0+8=8px on the
still-mounted content div. Back restored the pt-(--hdr-h) pad class with that
stale value, sliding the content toolbar up under the header — stuck until reload.

Hold both the header and content nodes in state (callback refs) and key the
measurement effect to the live nodes, so it re-runs every time the header
mounts/unmounts; drop the stale pad when the header is absent (channel page).

Adds a regression test: header removed then restored must re-measure, not 8px.
2026-08-21 16:51:05 +02:00
peter 6b4febcc95 release: 0.71.0 (improvement/e5-s1-plex-platform)
gate / frontend (push) Failing after 1m9s
gate / backend (push) Successful in 1m57s
2026-08-21 08:26:31 +02:00
peter cb5ac09a80 chore(e5): bump to 0.71.0 + release notes for the Plex Experience epic 2026-08-21 08:21:21 +02:00
peter bad2aae753 fix(e5): deep-review round — combobox Enter, parse_person collision, enrich hardening + coverage
Full-epic /deep-review (11 finders + 12 adversarial verifiers) surfaced no HIGH; every
mechanism finding calibrated to LOW on verification. Fix all 13:

Correctness / reliability:
- Combobox: Enter now commits the VISUALLY highlighted row (highlightIndex ?? activeIndex),
  not the bare keyboard activeIndex — so arrow-then-hover-then-Enter no longer toggles a
  different option than the one painted, and hover-with-no-arrow is no longer a silent no-op.
- parse_person: dedup the known_for pool by (id, media_type), not the bare id — TMDB movie
  and tv id spaces are independent, so a same-id movie+show pair is two distinct credits
  (matches the collision-safe key the rest of the module already uses).
- person-enrich: a per-person in-flight + 60s cooldown guard (people._should_schedule_enrich)
  dedups concurrent opens and rate-limits re-triggering while TMDB is degraded (a transient
  TmdbError leaves the watermark NULL), so bursts can't pile 8s tasks onto the shared threadpool.
- PlexInfo compact()/money(): format with i18n.language (keeping currency USD) instead of a
  hardcoded en/en-US, matching the app's Intl convention.
- delete the dead `plex.filter.add` i18n key (both locales) — the +Add button that used it is gone.

Reuse:
- _continue_watching reuses rollup.agg_subquery(include_resumed=True) + query.apply_show_state_filter
  instead of a third hand-inlined copy of the per-show rollup + in_progress predicate, so the
  Discover rail can't drift from the grid's definition.

Test coverage (the gaps a regression would ship through):
- resolve_library_guids scope-exclusion (a credit in a non-scoped/disabled library is dropped)
  + the []-vs-None distinction; parse_person cross-media-id retention; enrich_person transient
  (NULL watermark) and 404 (stamp) branches; _enrich_person_bg body (fresh session close/rollback,
  None-guard) + the cooldown dedup; continue_watching fully-watched-show exclusion; Combobox
  Enter-commits-highlighted + async/controlled-query mode; PopoverItem aria-checked contract;
  RatingsRow missing-IMDb + imdb==audience branches.

Backend lane 413→420, FE vitest 351→357; all gates green. No migration.
2026-08-21 08:11:18 +02:00
peter 8f6714e678 refactor(e5): deferred review-tail (frontend) — Carousel reuse + shared scroll core
The last of the deep-review deferred reuse items:

- extract the scroll-affordance machinery (watch the scroller AND its swappable/
  lazily-growing children, diff two edge booleans) that useScrollFade (vertical) and
  Carousel's useHScroll (horizontal) each hand-duplicated into one shared
  lib/useScrollEdges(axis); both hooks now just map start/end to their own shape.

- migrate PlexInfo's two remaining hand-rolled `flex overflow-x-auto` strips — the
  cast row and the collection strips — to the S1 <Carousel> primitive, so they get the
  edge fade + hover/focus scroll buttons (only on the clipped side) for free, matching
  the Discover rails / CastStrip / RelatedStrip.

Pure refactor, no behaviour change. Gate green (tsc, vitest 351, eslint, prettier,
knip). Browser-verified on a movie info page: cast row + collection strip render as
Carousels with the edge fade and the right-scroll button on the clipped cast row.
2026-08-21 06:16:15 +02:00
peter 13fb2b4e61 refactor(e5): deferred review-tail (backend) — enrich off request path, reuse, tests
Address the deep-review's deferred backend items so E5 is review-complete:

- person-enrich OFF the request path (ABUSE finding): person_detail no longer makes
  the TMDB call inline (even bounded to 8s it sat on the serve thread, un-throttled,
  demo-reachable — it violated the module's "no live TMDB call at serve time"
  invariant). It now schedules a FastAPI BackgroundTask (people._enrich_person_bg,
  fresh session, best-effort) after the response; the first open serves the row as-is
  and the bio lands by the next open. The inline try/rollback recovery is gone with it.

- reuse: a shared tmdb.resolve_library_guids owns the collision-safe guid→library
  bucketing (movie vs tv ids are independent sequences) that resolve_recs and
  people._resolve_known_for each hand-duplicated; a new rollup.item_state_map dedups
  the "PlexState-by-item_id" fetch across the person + discover paths.

- test gaps: enrich_batch transient TmdbError leaves the watermark NULL to retry;
  guid_to_tmdb_id resolves a tvdb source + tv media_type via /find's tv_results;
  _continue_watching includes a NULL-status started movie and excludes a hidden one;
  /api/search/youtube returns 422 (not 5xx) when ingest fails (route-level, B-core-1).

Backend lane 403 → 413, ruff clean. No migration; serve contracts unchanged.
2026-08-21 06:09:17 +02:00
peter 5316f48af2 feat(e5): first-class combobox platform mode + universal Combobox (A11Y-1)
The E5 filter pickers were a hand-rolled floating-ui combobox (MultiSelectChips)
with no combobox ARIA, no arrow-nav, and options in a body-portal — keyboard/SR
users could not operate it, and the bare role=listbox/option wiring actively
misled screen readers (deep-review A11Y-1, CONFIRMED 3/3 medium). Rather than
one-off-hack the a11y, add combobox as a first-class mode of the AnchoredPopover
platform and rebuild the pickers on a reusable component.

Platform (AnchoredPopover):
- new mode "combobox": reuses the whole positioning core (offset/flip/shift/size/
  hide, isPositioned + transform-clobber flash guards, FloatingPortal + fullscreen
  re-parent, topmost-only Escape layer). Differs only in the focus model — no
  useClick (open is field-driven), useRole role="combobox" so Floating UI wires the
  full ARIA tree (input role=combobox + aria-expanded/controls/autocomplete/
  activedescendant, listbox surface, option ids), and useListNavigation virtual:true
  so ↑/↓ move an aria-activedescendant pointer while DOM focus STAYS on the input.
- PopoverSurface: combobox surfaces host the item registry but are NOT focus-trapped.
- PopoverItem: virtual mode carries a stable id (the activedescendant target), never
  takes roving DOM focus, keeps the input focused on click (mousedown preventDefault),
  and its checked/selected truth wins over getItemProps' single-select guess.

Component (Combobox.tsx, new): the universal react-select/Overseerr combi-input —
chips-in-field + typed search + checklist. Supports multi/single select, client OR
server (async) filtering, custom rows, the big-vocabulary render cap + refine footer,
and a selected-label cache so a server-narrowed single-select still names its chip.

Rollout: MultiSelectChips (genre/age-rating/studio) → Combobox; the CollectionPicker
"+Add" popover → a single-select async Combobox (also fixes its arrow-keys-eaten
story), unifying the two filter idioms. PlexSearchBox is a documented future consumer.

Gate green (tsc, vitest 351, eslint, prettier, knip). Browser E2E on localdev:
role=combobox confirmed on both pickers; ↑/↓ moves aria-activedescendant with focus
staying on the input; Enter + click select; multi-select chips accumulate; Backspace
removes the last chip; async collection server-search picks + closes. No backend change.
2026-08-21 05:43:36 +02:00
peter 0c746f5bcd fix(e5): deep-review survivors — TMDB key log leak, person-enrich rollback + timeout
Three verified deep-review findings (the 4th, the filter combobox a11y gap, is
deferred to a dedicated session on the combobox architecture):

- SEC (medium): TmdbClient sent the api_key as a URL query param, so an HTTP
  error's httpx exception string embedded the full URL incl. the key, which
  enrich_batch logged. _get now raises status+path only (never str(e)/the URL)
  and severs the secret-bearing cause with `from None`. Regression test asserts
  neither the status nor the network error path contains the key.
- STATE (low): person_detail's lazy-enrich except now db.rollback()s, so a
  failed enrich_person commit can't poison the session and 500 the subsequent
  SELECTs (SQLAlchemy 2.x deactivates the tx on a commit-time flush failure and
  does not auto-reset it). Test locks that the page renders despite a poisoned
  enrich.
- ABUSE (low): the request-path lazy person enrich now caps its TMDB client at
  an 8s timeout (was the 30s default) so a slow TMDB can't tie up a worker;
  TmdbClient gained a `timeout` param (batch keeps 30s). The fuller fix (move
  person enrichment off the request path) is logged for later.
2026-08-21 04:42:07 +02:00
peter caced8664c fix(e5): stop the filter dropdown flashing at the origin before it positions
The combi-input popover flashed in the top-left corner for a frame, then
jumped into place. Two causes, both confirmed with a per-frame capture in
the browser:

1. The panel painted before floating-ui computed a position — gated its
   visibility on `isPositioned` (hidden until positioned), same as
   PopoverSurface.
2. The bigger one: the `popIn` entrance animation animates `transform`
   (scale/translateY), and it sat on the SAME element floating-ui positions
   via `transform` — so during the 0.16s animation the keyframes OVERRODE
   the translate(x,y) and the panel rendered at the origin. Moved the glass
   styling + popIn onto an inner child, leaving the positioned element's
   transform untouched.

Frame capture after the fix: frame 0 hidden, first visible frame already at
translate(82,474) — never visible at the origin. Also gave the size
middleware a min width for a narrow rail.
2026-08-21 04:00:54 +02:00
peter 29f2c9059e fix(e5): rebuild the filter combi-input on floating-ui primitives (focus, dismiss, z-order)
The AnchoredPopover-based version had three UAT-caught bugs: the field
showed an ugly "Add" placeholder; the first click couldn't type (the
platform's FloatingFocusManager + focusItemOnOpen pulled focus onto the
list, blurring the field until a second click); and an outside click didn't
close the list. A visual pass also caught the dropdown rendering BEHIND the
sidebar panel (z-menu 20 < z-panel 35).

Rebuilt on floating-ui primitives directly — no focus manager, so focus
stays in the field; open is driven by field focus; useDismiss closes on
outside-press/Escape; the list is a z-popover (60) FloatingPortal sized to
the field width. Placeholder is now a plain "Search" (empty once chips
exist). Verified in the browser: type-on-first-click, filter, chip add,
outside-close, and the list painting above the sidebar.
2026-08-21 03:37:14 +02:00
peter b83248c510 fix(e5): Plex detail/UI polish from review (backdrop reset, layout, combobox, labels)
User-reported issues after the E5 build:

- useCrossfadeBackdrop: the static branch (one image / crossfade off /
  reduced motion) painted #root but returned NO cleanup, so the backdrop
  persisted after leaving the page (e.g. switching modules). Every painted
  branch now returns clear() so unmount always resets the default backdrop.
- scheduler: give the tmdb_enrich job a proper label + description (was the
  raw "tmdb_enrich" id) in EN/HU.
- CastStrip: cap crew names to ONE line (truncate) so the row of circular
  photos sits on one baseline instead of going wavy on 2-line names.
- PlexInfo: lay Details / Keywords / Crew out as side-by-side boxes
  (items-start grid) instead of full-width stacked panels with dead space;
  narrower Crew grid for the boxed width; each panel independently hideable
  from the customize menu (new Details/Keywords/Crew toggles). Tighter
  vertical spacing.
- MultiSelectChips: reshape the trigger from a "+Add" button into an
  Overseerr-style combi input — removable chips inside a typeable field that
  filters the checklist, opening on focus (backspace removes the last chip).
- Discover / person rails: reduce the vertical gap between rails.
2026-08-21 03:16:40 +02:00
peter 2b5de89eda fix(e5): address code-review findings (TMDB guid media_type, search ingest guard, discover keying)
Follow-ups from the epic-end /code-review of the E5 branch:

- tmdb.resolve_recs + people._resolve_known_for: match recommendations/
  credits to the library bucketed by media_type. TMDB movie ids and tv ids
  are independent sequences, so a bare `tmdb:<id>` match could deep-link a
  movie rec to an unrelated show (or vice-versa) on a numeric collision.
  Regression test added (test_resolve_recs_matches_by_media_type_not_bare_id).
- routes/search: wrap the _ingest_candidates call so a DB/enrich failure
  degrades to a 422 with a message instead of a bare 500 hidden behind the
  FE's generic "search failed" (extends the B-core-1 rationale); rollback on
  failure. Stamp enriched_at with its own timestamp taken after the enrich,
  not the stub-insert placeholder.
- routes/plex/discover: key serialized cards back by rating_key (globally
  unique in Plex) instead of id(obj); fix the stale rows 3-tuple annotation.
2026-08-21 03:16:24 +02:00
peter ff45db70ae fix: live YouTube search 500 on the scrape source (published_at NOT NULL)
The scrape source exposes no publish date, so its search stub carried
published_at=None. Migration 0068 (PERF-15, E2/0.68.0) made
videos.published_at NOT NULL, so insert_stubs raised IntegrityError —
outside the route's page-fetch try/except — and the whole
/api/search/youtube request 500'd ("YouTube search failed."). Every
scrape search (the default source) had been broken since 0.68.0.

- routes/search._ingest_candidates: stub published_at falls back to
  now() (mirrors stub_from_playlist_item), satisfying the constraint.
- sync.videos.apply_video_details: prefer the enrich snippet's real
  date whenever present, so the now() placeholder is corrected instead
  of becoming the video's permanent date (the guard only filled a NULL,
  which a non-null placeholder no longer is). Never clobbers a real
  date with now().

Regression tests (DB lane): dateless scrape stub ingests and takes the
real enrich date; an enrich miss keeps a non-null placeholder, unkept.
2026-08-21 02:15:09 +02:00
peter 7d40a8780d E5 S8: crossfading art backdrop (detail + person pages)
The faint HTPC art backdrop now slowly cross-dissolves through a pool of images
instead of showing one static art — the last E5 sprint.

- useCrossfadeBackdrop (replaces useArtBackdrop): a double buffer — the SETTLED
  image lives on #root's background (the proven full-viewport layer); the next
  image cross-fades in on a z-index:-1 overlay layer, then is committed and the
  overlay reset. ⭐ #root is given `isolation: isolate` so it's a stacking context
  — without it the z:-1 overlay paints BEHIND #root's own background and is
  invisible (found + fixed live: a red test layer + a real TMDB backdrop both
  only showed once #root was isolated).
- Art pool: the item's own art + its TMDB backdrops (E5 S2, rendered straight
  from the TMDB CDN — the CSP allows it) + its collection siblings' art; the show
  page adds related art; the PERSON page cross-dissolves through the person's
  films' art (the roadmap's signature touch).
- HARD-OFF under prefers-reduced-motion (static, no overlay) + a new usePref
  "Crossfade backdrop" toggle in the detail customize menu (fade 1.6s / 7s cycle).

Wired into PlexInfo, PlexShowView, PlexSeasonView, PlexPersonView. Tests:
plexBackdrop.dom.test.tsx (5: paint+isolate+overlay, reduced-motion off, single
image, animate=false, disable/unmount cleanup). Browser-E2E'd (isolation +
overlay render + a real backdrop crossfade + the toggle). Gate: vitest 347,
backend lane 403 (unchanged — FE only).

▶ E5 · Plex Experience epic COMPLETE (S1–S8).
2026-08-21 01:00:11 +02:00
peter d56a791c04 E5 S7: Plex filter UI — listbox-select pickers + studio filter
Reworks the always-expanded filter chip walls into compact, Seerr-style
"+ Add" → searchable-popover pickers (the user's space-saving ask), and adds
the missing studio filter.

- New MultiSelectChips: selected values as removable chips + a "+ Add" trigger
  opening a searchable checklist (AnchoredPopover platform — menuitemcheckbox,
  arrow/Escape, anchor flip). Options carry facet counts; a large vocabulary
  renders a 50-row cap + "N more — refine with search" footer (studios run to
  ~1500). Replaces the flat Genre + Age-rating chip lists.
- New Studio filter via the same picker. Backend: /facets now aggregates the
  studio vocabulary (distinct studio + count, self-excluded like genres). The
  studio filter itself already existed (only addable from the detail page before).
- Collection converted to the same "+ Add" pattern (CollectionPicker): a
  single-select popover with server-side search + the 50-cap; the selected
  collection still shows as a removable chip. Retires the always-inline search+list.

New i18n plex.filter.studio/add/moreRefine + common.search/remove/noResults
(en+hu). Tests: MultiSelectChips.dom.test.tsx (5, incl. the cap) +
test_plex_facets studio-vocabulary. Browser-E2E'd: Genre→Horror(402),
Studio→A24(14, capped/searched), Collection→James Bond(26)/harry-search.
Gate: vitest 342, backend lane 403.
2026-08-21 00:40:31 +02:00
peter 187b8007f8 E5 S6: enriched person page (TMDB bio + birth facts + Known-for rail)
The person page now leads with a TMDB bio, lazily fetched the first time it's
opened (off the S2-linked `tmdb_person_id`) and then served from the row:

- Migration 0072: ADD-only nullable person columns on plex_people — biography,
  birthday, deathday, place_of_birth, known_for_department, also_known_as,
  known_for ([{tmdb_id, media_type, title}]), tmdb_enriched_at.
- tmdb.py: TmdbClient.fetch_person (/person/{id}+combined_credits), parse_person
  (bio + a popularity-ranked, deduped known_for), apply_person, enrich_person
  (lazy one-shot, key+id guarded, commits, 404 stamps the watermark).
- people.py person_detail: loads the full person row, lazily enriches (best-effort
  — never fails the page), serves the bio fields + resolves known_for to library
  rows by guid (in-library only, popularity order).
- FE PlexPersonView: bio/birth-facts/aka hero + a "Known for" Carousel rail above
  the existing Movies/Shows filmography; PlexPersonDetail type + i18n (en+hu).

Tests: test_plex_person_bio.py (6: parse / lazy-enrich guards / serve) + extended
PlexPersonView.dom.test.tsx + the unknown-name contract in test_plex_person.py.
Browser-E2E'd on Jim Carrey (live TMDB enrich → bio 1937 chars, born 1962-01-17
Newmarket, Known-for resolved). Gate: vitest 337, backend lane 402.
2026-08-21 00:08:19 +02:00
peter f10915c223 E5 S5: enriched Plex detail page (ratings + facts + keywords + crew)
Consumes the TMDB block the S2 engine already persists + serves (movie/show
detail's `tmdb`), turning the movie info page into a Seerr-style enriched detail
(page variant only — the player overlay stays lean):

- Multi-source ratings row: IMDb (links out) · TMDb (with vote count) · Plex
  audience — whichever are present. Only the primary score (IMDb, else audience)
  carries the ratingMin filter, mirroring the backend rating column.
- Facts panel: release · runtime · cert · budget · revenue · studios · countries.
- Keyword chips → tap to filter the library. Backend: a new `keywords` filter on
  LibraryFilters + meta_filter_conds (`keywords @> [kw]`, both models), wired
  through PlexFilters (appendPlexFilters / plexFilterCount / active-chip / clear).
- Full crew (TMDB key-crew) → tap a name to open their person page.

Tests: test_plex_keyword_filter.py (3) + PlexInfo.dom.test.tsx (5, incl. the
overlay-stays-lean guard). New i18n plex.info.facts.* / keywords / crew / audience
(en+hu). Gate: vitest 337, backend lane 396.
2026-08-20 23:50:50 +02:00
peter f9b63745f9 E5 S4: Plex Discover rail-surface (default landing)
A dedicated Plex "Discover" landing — a stack of horizontal Carousel rails
served from ONE consolidated read (serve-from-DB, no live Plex/TMDB call):

- GET /api/plex/discover → continue_watching (in-progress movies + shows, most
  recently progressed) · recently_added · genres (top-N tiles) ·
  because_you_watched (recs of the most-recent watched, TMDB-enriched movie,
  resolved to library rows by guid via serve_block's resolver).
- GET /api/plex/surprise → a fresh random, scope-bounded sample for the ↻ rail.

Frontend: PlexDiscover.tsx (rails reuse the S1 Carousel + S3 PlexPosterCard;
colored genre tiles → tap to filter the grid; ↻ reshuffles the surprise rail
alone). Discover is now the root sub-view; the flat grid becomes the bookmarkable
"Browse all" secondary view (?v=browse) reached by a header search/filter (a
browse-intent flip) or the new Discover⟷Browse toggle.

resolve_recs is promoted from _resolve_recs (now cross-module). Tests:
test_plex_discover.py (rail shape/order, continue-watching, because-you-watched
+ null, surprise bounds, scope) and PlexDiscover.dom.test.tsx; plexSubUrl
round-trip updated for discover-root/grid-browse. Backend lane 393, vitest 332.
2026-08-20 22:15:54 +02:00
peter 5b5e76b73d E5 S3: unified Plex card — scrim-reveal hover + rest polish
The poster card (grid + person page) gets the Seerr-style hover: on hover a
bottom scrim slides up with the title, a "year · runtime · ★rating" meta line,
and a 3-line synopsis — the info you'd otherwise click through for — plus a
less-flat play ring (radial-gradient + ring shadow) and a top-right quick-action
cluster (Info + watched-toggle, revealed on hover / always on touch). Rest-state
badges (season / type tag) yield to the scrim on hover; the playable-cost tint
stays. (§1 A + C from the enrichment study.)

Backend (the data the scrim needs, mirroring S1's `art`): the grid
(browse.unified_library) now emits a lang-coalesced, truncated `summary` (new
query.summary_col) + the Plex `rating`; movie_card/show_card do the same (EN) for
the detail/related/person cards. PlexCard.rating type added (summary already
existed for episodes).

Gate: tsc x3 / eslint / prettier / knip / vitest 326 green; ruff green; backend
lane full suite green. Self-E2E on the live dev server (rebuilt on this branch,
S2-enriched): the grid serves rating+summary (Shawshank 9.3, Godfather 9.2, real
synopses) and the card renders the play ring + scrim (The Expanse: ring, synopsis,
★8.1) — verified via the DOM (the pane wouldn't composite for a screenshot).
2026-08-20 20:30:30 +02:00
peter 91be3d0535 E5 S2: prioritise TMDB crew, and capture the TV showrunner
A live smoke against the real TMDB API showed two crew problems the mocked
tests missed: (1) a TV show's creator/showrunner lives in the payload's
top-level `created_by`, not the crew list, so it was dropped entirely; and
(2) TMDB's crew order is arbitrary — a TV show's per-episode Directors (dozens)
flooded the key-crew list and buried the show-level roles.

Fix: split the accepted crew jobs by media type — movies key on
Director/Writer/…; TV keys on Creator (from created_by) + the show-level
Executive Producer/Writer/Producer/Composer, deliberately excluding the
per-episode Director/Editor/DP noise. Stored crew is sorted by display
priority and capped, so the lead roles come first.

Verified live: Breaking Bad -> [Gilligan (Creator), Johnson (EP), … , Writer];
The Godfather -> [Coppola (Director), Puzo (Screenplay), …]. test_plex_tmdb
green (9).
2026-08-20 09:24:25 +02:00
peter 3bde114b14 E5 S2: TMDB enrichment engine (Plex catalog)
Matches each mirrored Plex movie/show to TMDB by its stored canonical guid,
fetches once (append_to_response), and PERSISTS the enrichment onto the row —
served from the DB, never a live TMDB call at serve time. Additive: the
Plex-sourced imdb_rating/rating are untouched; tmdb_rating is the distinct
TMDb community score (ratings scope = TMDb + Plex's IMDb/audience, no OMDb/RT).

- config.py: tmdb_api_key + api_base/language/interval/batch defaults.
- sysconfig SPECS: tmdb_api_key (secret, Fernet) + tmdb_language, auto-rendered
  in the admin Config page; read like the Plex token.
- migration 0071 (ADD-only nullable, reversible): plex_items + plex_shows gain
  tmdb_id / tmdb_enriched_at (watermark) / tmdb_rating / tmdb_vote_count /
  budget / revenue / keywords / crew / studios / production_countries /
  recommendations / backdrops; plex_people gains tmdb_person_id (the S2
  foundation for S6 person bios). No watermark reset needed.
- app/plex/tmdb.py: TmdbClient (mirrors PlexClient; throttled; 404->None,
  transient->TmdbError), parse_tmdb / apply_tmdb_detail (one-parser rule),
  guid_to_tmdb_id (tmdb: direct, imdb:/tvdb: via /find), enrich_batch (movies
  then shows, transient error leaves the watermark NULL to retry, definitive
  no-match still stamps so it isn't re-fetched), _link_people (upserts
  tmdb_person_id by name), serve_block (recommendations resolved to library
  rows by guid), run_enrich (no-op without a key).
- scheduler: tmdb_enrich nightly job (default 120 min, no YouTube quota),
  runtime-tunable + "run now" from the Scheduler dashboard.
- detail endpoints emit `tmdb` (movies + shows) so it's curl-verifiable now;
  the FE render lands in S5.

Note: _link_people inserts plex_people rows for TMDB cast/crew; unreferenced
crew (composers etc., not in cast_names/directors) get pruned by the orphan
sweep — acceptable, the person page is cast-driven and cast rows survive.

Gate: ruff + py_compile green; backend test lane FULL suite 387 passed
(incl. migration drift downgrade->re-upgrade + test_plex_tmdb 9 tests).
Frontend unchanged since S1. Not merged (epic branch, stacks on S1).
2026-08-20 09:17:43 +02:00
peter fca3fa13ef E5 S1: shared Carousel primitive + Plex card-art serializer
Wave 5a building blocks under the Seerr-inspired Plex enrichment (E5).

Carousel (frontend/src/components/Carousel.tsx): the one horizontal-rail
component, replacing the hand-rolled `flex overflow-x-auto` strips and the
home of the future Plex-Discover rows. Edge-fade mask, hover/focus scroll
buttons that appear only on the clipped side (useHScroll — the X-axis twin
of useScrollFade, same watch-the-children discipline), native scrollbar
hidden behind .no-scrollbar, reduced-motion-aware paging. First consumers:
CastStrip + RelatedStrip now scroll inside it (dedup + free affordances).
New i18n: common.seeAll / scrollLeft / scrollRight (en+hu).

Card-art serializer (backend/app/plex/cards.py): movie_card + show_card now
emit `art` (the wide backdrop, /api/plex/image/{rk}?variant=art) so rails and
the crossfading hero get a landscape image with no per-item fetch — gated on
art_key so an art-less item stays null and falls back to the poster. FE type
PlexCard.art?.

Tests: Carousel.dom.test (children/header/see-all, clipped-edge buttons +
paging), test_plex_cards (art present/absent by art_key). Added a
ResizeObserver stub to PlexPersonView.dom.test (it renders CastStrip, now a
Carousel consumer).

Gate: tsc x3 / eslint / prettier / knip / vitest 324 green; ruff green;
backend test lane (docker-compose.test.yml) green on the plex card tests.
2026-08-20 08:50:47 +02:00
peter df60bc5ff5 merge: E4 test-coverage backfill (deferred deep-review gaps + 2 review rounds)
Closes every ▶ DEFERRED test-coverage item from the E4 deep-review (Modal
focus-trap, useHistorySubview popstate URL-fallback, PLEX_SUB_URL round-trip,
loadInitialPage precedence + channelViewFrom, readSelectedId ?pl guard,
useUrlMirror dep-keying, usePref flush-on-unmount) + the a11y sweep, and the
findings from two /code-review high rounds. Test-only + behaviour-preserving
seams (parseId/navigationInit/playlistsInit/plexSubUrl extractions); no
user-facing change, no version bump. vitest 257→321. Not deployed.
2026-08-20 05:56:16 +02:00
peter 99d63d272e test(e4): lock accessible names on the Plex card buttons
Closes the re-review's test-coverage finding: a plexCards.dom.test asserting the
icon-only card controls (watched-toggle, Info, add-season/episode) each expose
their aria-label as an accessible name via getByRole({name}). A future edit that
drops a label now fails the gate instead of silently reopening the unnamed-button
gap the a11y sweep closed.

4 tests; full FE gate green (tsc/eslint/prettier/knip, 321 vitest).
2026-08-20 05:52:36 +02:00
peter 482dd28006 fix(e4): address code-review findings (a11y sweep + dedup guards)
Resolves the 5 /code-review high findings on this branch:
- a11y sweep: extend aria-label to the remaining icon-only Plex card buttons the
  first pass missed — PlexPosterCard Info, SeasonCard watched-toggle + add-season
  (EpisodeCard already had them). "Fix a principle everywhere, in one sweep."
- new shared lib/parseId.ts `parsePositiveIntId`, used by BOTH readSelectedId
  (now guarding the stored ?pl fallback the same as the URL — was unguarded) and
  plexSubUrl's ?v=playlist decode; removes the hand-duplicated guard.
- PAGE_KEY exported from navigationInit and imported by NavigationProvider (was
  defined in both).
- hoist a `watchedLabel` const per card so title + aria-label can't drift.

+8 tests (parseId unit + guarded-stored ?pl); full FE gate green (tsc/eslint/
prettier/knip, 317 vitest).
2026-08-20 05:40:59 +02:00
peter 0b7180ce8f a11y(e4): aria-label on the icon-only Plex watched-toggle buttons
The hover watched-toggle on Plex movie/episode cards is icon-only (WatchedIcon,
no text), so it carried a `title` but no accessible name for screen readers —
and the empty-Circle unwatched glyph could read like a radio. Mirror the title
into `aria-label` (the Modal close-button idiom) so the control announces its
action. The PlexInfo watched buttons already have visible text labels, so they
need nothing.

FE gate green (tsc/eslint/prettier/knip, 309 vitest).
2026-08-20 05:25:48 +02:00
peter d2f2085f1f test(e4): readSelectedId ?pl guard, useUrlMirror dep-keying, usePref flush
The remaining E4 deep-review deferred gaps, in one sprint:
- readSelectedId split into playlistsInit.ts (react-refresh-clean, testable) —
  ?pl accepted only as a positive integer; empty/0/negative/fractional/non-numeric
  fall back to the stored selection; null when neither is present.
- useUrlMirror: mounts write the param, a same-serialized rerender does NOT
  re-fire (stable dep), a null value clears the key, other params preserved.
- usePref: a pending debounced write is FLUSHED on unmount (not dropped); an
  unmount with nothing pending stays silent.

20 new test cases; full FE gate green (tsc/eslint/prettier/knip, 309 vitest).
2026-08-20 05:24:11 +02:00
peter 062015ba27 test(e4): loadInitialPage precedence + channelViewFrom (S1 fix)
Split the two URL-derived nav seeds out of NavigationProvider into
navigationInit.ts so they're unit-testable and the provider stays
react-refresh-clean (component-only exports). channelViewFrom also dedups the
identical _chan/?chan resolve the channelView seed and the onPop handler each
hand-rolled — locking the S1 fix (a popstate on a URL-only bookmark entry must
restore the channel from ?chan, not close it). Pure code motion.

Tests: the full loadInitialPage precedence matrix (?page > ?v > ?pl >
filters/?chan > stored landing > feed, with the beats-a-lower-param cases) and
channelViewFrom (_chan+name, ?chan fallback, state-over-url, closed → null).

15 tests; full FE gate green (tsc/eslint/prettier/knip).
2026-08-20 05:21:13 +02:00
peter 0401dae689 test(e4): extract PLEX_SUB_URL for round-trip coverage
Move `type Sub` + `PLEX_SUB_URL` out of the 538-line PlexBrowse into a small
plexSubUrl.ts so the encode/decode round-trip and its guards can be unit-tested
without dragging the component (and its lazy hls.js player) into the test graph.
Pure code motion — PlexBrowse imports the two symbols; the `SubviewUrl` import
moves with them.

Tests lock: encode of each bookmarkable kind, grid/player → null, encode→decode
identity, a person name containing ':' round-tripping, and the decode guards
(positive-integer playlist id; season needs both ids; non-empty show/info/person
arg; unknown kind/garbage → null).

12 tests; full FE gate green (tsc/eslint/prettier/knip).
2026-08-20 05:16:06 +02:00
peter e8a1c252a9 test(e4): useHistorySubview seed + popstate URL-fallback (S2 fix)
Backfills the E4 deep-review's deferred coverage for the subview history
primitive. The keeper case is the S2 bookmark fix: a popstate that lands on a
URL-only entry (no history.state._sub) must restore the sub-view from the ?v
param via the SAME fallback the mount seed uses, not collapse to root.

Covers: seed from URL vs state._sub vs root, the no-SubviewUrl state-only
path, open() push+mirror + encode→null key-clear, and popstate restore from
state / from URL (the fix) / collapse-to-root.

9 tests, gate green.
2026-08-20 05:14:12 +02:00
peter fa8bdf83a8 test(e4): Modal focus-trap coverage (FE-9 deferred gap)
Backfills the E4 deep-review's deferred Modal focus-management test. jsdom
returns null for every offsetParent (no layout), so focusablesIn() would see
zero controls; the test stubs HTMLElement.prototype.offsetParent to report
"visible" for connected nodes (what a real browser reports for the shown
dialog), and a second describe documents the raw-jsdom empty-guard path.

Covers: initial focus on open, Tab/Shift+Tab edge-wrap, pull-back on escaped
focus, the active===panel leak guard, no-trap in the interior, Escape closes
the topmost only, and restore-to-trigger on unmount. useBackToClose is mocked
(its history stack is covered by history.dom.test.ts).

10 tests; full FE gate green (tsc/eslint/prettier/knip, 257 vitest).
2026-08-20 05:12:33 +02:00
peter 868ff533ee style(e4): prettier-format two S6-touched files to unblock the 0.70.0 gate
gate / frontend (push) Successful in 1m2s
gate / backend (push) Successful in 2m1s
The reveal-class swap in S6 shortened two className lines (VideoCard action row,
SavedViewsWidget default-star), which prettier wants collapsed to one line. The
local `npm run lint` doesn't run prettier --check, so it slipped past the sprint
gate and only the release `siftlode check` caught it. Formatting only, no behavior
change. Fixes the pre-publish gate for the already-merged 0.70.0 on main.
2026-08-20 03:52:11 +02:00
peter 00bf9a39f7 release: 0.70.0 (improvement/e4-frontend-platform)
gate / frontend (push) Failing after 38s
gate / backend (push) Successful in 2m51s
2026-08-20 03:50:02 +02:00
peter 640a8efeed chore(release): 0.70.0 — E4 Frontend Platform (bookmarkable URLs + touch a11y)
Bump VERSION to 0.70.0 and add the user-facing release-notes entry for the E4
Frontend Platform epic: full bookmarkable/shareable address-bar URLs (retiring the
"Share view" button), touch-visible card action buttons with a shape (not
colour-only) watched marker, and the bookmark-page-closes-on-modal-close fix — over
a large internal front-end refactor (god-component splits, usePref, useUrlMirror,
FilterRail, Modal focus). No DB migration.
2026-08-20 03:49:07 +02:00
peter 23ac884b42 fix(e4): address deep-review findings — bookmark popstate, focus trap, touch a11y
Deep-review (6 dimensions, 3-verifier adversarial pass) surfaced 7 findings, all
verified; fixed here.

- NavigationProvider + history.ts (medium, correctness): the popstate handlers
  restored the open channel / Plex sub-view from history.state ONLY, but a fresh
  ?chan / ?v bookmark lives in the URL with no marker in state — so any overlay's
  programmatic history.back() (closing a modal/player) silently closed the
  bookmarked channel or collapsed the sub-view to the grid. Give onPop the same
  URL fallback the initializers already use (history.ts extracts a shared readView
  so seed and popstate can't diverge again).
- plexCards SeasonCard (medium, a11y): the "In progress" season badge was a
  reveal-idle swap-badge → display:none on touch, and a season has no progress bar,
  so an in-progress season was indistinguishable from unwatched on touch. Moved it
  to the top-left (the toggle owns top-right) and made it always visible.
- plexCards watched toggle (low, a11y / WCAG 1.4.1): watched state was color-only
  on touch (green vs white CheckCircle2). New WatchedIcon distinguishes by SHAPE —
  CheckCircle2 (watched) vs an empty Circle (not) — used by all three card toggles.
- Modal.tsx (low, a11y): the focus trap leaked on Shift+Tab when focus rested on
  the panel container itself (Node.contains(self) is true, so the pull-back branch
  never ran). Guard now also checks `active === panel`.
- index.css (low, a11y): .reveal-idle faded only on .group:hover though .reveal is
  revealed on hover AND focus-within → the badge overlapped the control on keyboard
  focus. Added the focus-within fade.
- PlexBrowse.tsx (low, correctness): the ?v=playlist decoder accepted a non-integer
  id (Number.isFinite); tightened to Number.isInteger(n) && n > 0, matching the ?pl
  guard.

Gate green (tsc, knip, lint 0-err, 247 vitest). Self-E2E (localdev): ?v=show and
?chan bookmarks survive a modal open+close (were collapsing/closing before); Modal
Shift+Tab from the panel stays inside; in-progress season badge visible on touch;
watched toggle shows an empty-circle-vs-check shape difference; 0 console errors.
2026-08-20 03:26:19 +02:00
peter 1579a81395 fix(e4): address code-review findings — usePref flush, ?pl guard, dedup (E4 review)
Applies the /code-review high findings on the E4 branch:

- usePref: flush the pending debounced server write on unmount instead of
  clearing it, so the last change isn't lost (and can't later be overwritten by
  the stale server value via the adopt effect). [correctness]
- PlaylistsProvider.readSelectedId: accept ?pl only when it's a positive integer
  — "?pl=" gave Number("")===0 (finite) and "?pl=1.5" a fractional id, both of
  which selected a non-existent playlist. [correctness]
- usePref: extract mergePrefsCache() so useSavePref and usePref.set share the
  one me-cache merge shape instead of duplicating it. [reuse]
- New lib/useUrlMirror: the shared replaceState(urlWithParams(...)) mirror
  effect, so FeedFiltersProvider and PlaylistsProvider stop hand-rolling the
  identical block (keyed on the serialized delta so it only fires on change).
  [reuse]
- plexCards: swapBadgeReveal() helper for the repeated
  reveal-idle/group-hover:opacity-0 swap-badge ternary. [simplification]

Gate green (tsc, knip, lint 0-err, 247 vitest). Self-E2E (localdev): feed filter
mirrors to ?show and clears on leaving the feed / re-mirrors on return via
useUrlMirror; ?pl= (empty) and ?pl=abc no longer select a bogus playlist (fall
back to the stored selection); Plex reveal-idle badges still display:none on
touch after the helper refactor; 0 console errors.
2026-08-20 02:13:49 +02:00
peter 4fe6f38f1b refactor(rail): extract shared FilterRail shell + name the MediaSource seam (E4 S7)
The three filter rails — feed Sidebar, PlexSidebar, PlaylistsRail — each rendered
the same SidePanel chrome (title bar, customize `editing` state, edit-toggle,
reset-to-layout-default, edit hint) around a PanelGroups body, differing only in
identity + filter content. Extracted that shell into components/FilterRail.tsx:
each rail now supplies only what differs (title/icon, active-filter count +
Clear-all action, its `items`, and optional content above/below the groups —
`top` for the feed's scope toggle, `children` for its tag manager). FilterRail
owns the `editing` state, so all three drop their duplicated useState + SidePanel
wiring.

MediaSource facade (lib/mediaSource.ts): a thin `MediaSourceKind = "feed" | "plex"`
naming the two browsable, filterable sources (a PanelId minus "playlists", which
is a rail but not a source). Already unified in practice — one SortControl, the
source-agnostic ActiveFilter[] chips, PanelId layouts, now the shared shell — so
this is formalization, not new capability: a single named seam for a future third
source or a cross-source feature. The two media-source rails annotate their `kind`
with it; PlaylistsRail passes the plain PanelId. Per the recon, the playlist-dialog
merge was deliberately NOT forced (it proved forced + low value). UX-3 control-vocab
sweep deliberately deferred — the rails already use the canonical primitives.

Behaviour-preserving. Gate green (tsc, knip, lint 0-err, 247 vitest). Self-E2E
(localdev): all three rails render via FilterRail — feed (scope toggle + panel
groups + Customize→editing with reset + 56 reorder affordances, exits clean), Plex
(filter groups + customize), Playlists (sections + customize); 0 JS console errors.
2026-08-20 00:47:09 +02:00
peter 94a608053a feat(a11y): touch-visible card controls + modal focus trap/restore (E4 S6)
UX-7 — hover-only card actions were `opacity-0 group-hover:opacity-100`: fine on
a mouse, but invisible AND still tappable on a touch device (no hover). New
`.reveal` primitive (index.css) gates the hide on `@media (hover: hover)`, so a
touch device keeps the control visible; its partner `.reveal-idle` is the idle
badge a control replaces (watched/in-progress pill under the watched-toggle),
faded on hover and dropped on touch so the two don't collide at the same corner.
Migrated the true controls — feed card action toolbar (VideoCard), Plex poster
card watched-toggle/info/add across all card variants + their swap badges
(plexCards), message pop-out (Messages), edit-interval pencil (Scheduler), and
saved-view star/rename/delete (SavedViewsWidget). Pure play-scrim overlays (the
whole card is the tap target) stay hover-only by design.

FE-9 — Modal.tsx had aria-modal but no focus management. Adds: a Tab/Shift+Tab
focus trap (topmost layer only, wraps at the edges and pulls escaped focus back);
initial focus into the dialog unless an autoFocus child already placed it; focus
restore to the trigger on close (captured in a useState initializer during render,
before commit runs autoFocus and moves activeElement inside); and aria-labelledby
wiring the dialog to its title.

Gate green (tsc, knip, lint 0-err, 247 vitest). Self-E2E (localdev): FE-9 trap
both directions + pull-back + Escape-restore-to-trigger + aria-labelledby verified;
UX-7 feed toolbar opacity 0 on desktop / 1 under mobile (hover:none) emulation,
Plex reveal buttons opacity 1 + reveal-idle badges display:none on touch.
2026-08-19 21:35:52 +02:00
peter aea99cc71a refactor(downloads): split DownloadCenter god-file into row + modals + admin (E4 S5)
Pure code-motion, DownloadCenter 1088->359 L. Group by kind, mirroring the S4
plexCards/plexViews split:

- components/downloadRow.tsx (309 L): the user-list presentation + shared row
  primitives — StatusPill, Thumb, IconBtn, SourceRef, ChannelLine, DownloadRow,
  UsageBar (+ jobTitle/jobChannel/displayUrl helpers, status/phase constants).
- components/downloadModals.tsx (178 L): the per-item user modals — MetaEditModal
  (rename/metadata) and ErrorModal (failure detail + retry).
- components/downloadAdmin.tsx (267 L): the admin "System" tab — AdminSystem plus
  QuotaModal and QuotaUserPicker.

Every extracted piece is prop-driven with no shell-state reach-back, so the shell
just imports them back; the tab logic, mutations and modal state stay in
DownloadCenter. Behaviour-preserving.

Gate green (tsc first try, knip, lint 0-err, 247 vitest). Self-E2E (localdev):
Queue/Library/Shared/System tabs render, UsageBar, admin storage/footprint/job
list, MetaEditModal open/close, 0 console errors.
2026-08-19 21:14:49 +02:00
peter 180ab56379 refactor(plex): split PlexBrowse god-file into cards + views + shared UI (E4 S4)
Pure behaviour-preserving code-motion — PlexBrowse 1751 -> 538 lines. The ~11 inlined
components are self-contained (every view is invoked with shell callbacks as props; no
view reaches back into shell state), so they just move to sibling files.

New files:
- lib/plexWatch.ts: toggleWatchState (shared by the grid + person page)
- components/plexCards.tsx: PlexPosterCard, SeasonCard, CastStrip, RelatedStrip, EpisodeCard
- components/plexViews.tsx: PlexInfoView, PlexPersonView, PlexShowView, PlexSeasonView
- lib/plexDetailUi.tsx (extended): BackBtn, ActionBtn, WatchedToggle, DetailHero, epLabel
  (the file's own remit — "shared detail-page UI")

PlexBrowse keeps the Sub router + PLEX_SUB_URL + PAGE + PlexBrowse() shell. Repointed
the only external importer (PlexPersonView.dom.test) to plexCards/plexViews.

Self-E2E (localdev): Plex grid renders (poster cards + episodes); The Expanse show-drill
renders the hero + action buttons + season cards; ?v=show URL-mirroring intact; 0 console
errors. Gate green: typecheck, knip, eslint (0 err), prettier, 247 vitest.
2026-08-19 06:10:48 +02:00
peter 0b592345e6 fix(player): remove the leaked window "blur" listener in useNativeControlsYield
The native-controls-yield effect registered addEventListener("blur", onBlur) but its
cleanup removed "blur-sm" (a stray Tailwind-class find-replace artifact), so the blur
listener was never removed — leaking one per player open/close. Match the names.
2026-08-19 05:56:20 +02:00
peter 4800df4208 refactor(player): split PlayerModal god-component into hooks + controls (E4 S3)
Mirror the Plex player split (plexPlayerHooks.ts / plexPlayerControls.tsx). The shell
keeps owning the shared refs and passes them into each hook; the once-bound keyboard
handler + layer-stack registration stay in the shell (they orchestrate every cluster).
Behaviour-preserving code-motion — PlayerModal 1332 -> 890 lines.

New lib/playerModalHooks.ts:
- useYtVolume (per-account wheel/arrow volume, on shared playerVolume.ts)
- useNativeControlsYield (iframe-focus overlay yield)
- useYtBigView (F-maximise + our true-fullscreen; YT-specific, not Plex's useFullscreen)
- useYtProgress (player create + resume + 5s checkpoint + auto-watch + auto-advance)

New components/playerModalControls.tsx: PlayerTransportControls (moved as-is) + the
AutoMode/LoopMode types.

Self-E2E (localdev): play a feed video -> plays; transport bar (Auto/Loop); wheel volume
persists ({"volume":100}); F maximises + docked bar; exit big view; Close destroys the
player. Gate green: typecheck, knip, eslint (0 err), prettier, 247 vitest.

Noted a pre-existing bug (kept byte-identical here, flagged separately): the native-yield
effect removes a "blur-sm" listener but adds "blur" — the blur listener leaks.
2026-08-19 05:41:52 +02:00
peter 8bfc01684f feat(nav): full bookmarkable URL model — retire minimalist-URL + Share-view (E4 S2)
The query string is now the source of truth for the current view, so every URL is
bookmarkable/shareable. Reverses the early minimalist-URL rule (user's call).

- urlState: urlWithParams (read-modify-write delta on the live query, each provider
  owns its keys) + clearParams (strip only transient auth params) replace the whole-
  query stripUrlParams; shareUrl deleted; filterParamUpdates helper.
- NavigationProvider: page + ?chan mirrored to the address bar (pushState URL arg;
  history.state mechanics untouched); loadInitialPage resolves deep links
  (?v->plex, ?pl->playlists, filter/?chan->feed, bare->stored landing).
- FeedFiltersProvider: feed filters mirrored to the URL, gated on page===feed.
- useHistorySubview: optional SubviewUrl mapping; PlexBrowse mirrors the sub-view to
  ?v=<kind>:<args> (show/season/info/person/playlist; grid+player stay ephemeral).
- PlaylistsProvider: selected playlist mirrored to ?pl on the Playlists page.
- App: the 5 stripUrlParams calls become targeted clearParams (link/oauth/admin/
  access/verify/reset); the share-link hydrate effect removed.
- Retire the "Share view" feature: Sidebar + SavedViewsWidget buttons + i18n keys.

Browser tab title stays the short module/channel name (document.title untouched) —
the query never leaks into it, per the user's request.

Gate green: typecheck, knip, eslint (0 err), prettier, 247 vitest (urlState.dom.test
+9). No browser-E2E yet (most visual sprint — flagged for the epic-end review).
2026-08-19 04:47:36 +02:00
peter 48aa1a25f5 feat(prefs): usePref — one per-account+server preference hook (E4 S1)
Fold the hand-rolled server-pref plumbing (savePrefs + local mirror + me-cache
merge + adopt-on-login) into a single usePref<T>(serverKey, default, opts) hook,
plus useSavePref for dynamic multi-key patches.

Migrated (6 savePrefs call sites + 2 duplicated adopt effects -> the hook):
- FeedViewProvider: view
- App: filterCollapsed, playlistsCollapsed (+ their adopt block removed)
- PlayerModal: playerAutoAdvance, playerLoop
- plexDetailUi: plexInfoArtBg/Cast/Related (savePref via useSavePref)

Left as-is by design: panel layouts (a dynamically-indexed Record, not one key),
language (i18n store, no React state), PrefsProvider (debounced multi-field
auto-save + baseline echo-suppression - a different model).

Behaviour-equivalent (net -84 LOC at the call sites). usePref.dom.test covers
the three seed sources, persist path, debounce, adopt, parse, and useSavePref.
Frontend gate green: typecheck, knip, eslint (0 err), prettier, 238 vitest.
2026-08-19 03:07:25 +02:00
peter c82cf25bcf release: 0.69.2 (bug/download-poster-account-scope)
gate / frontend (push) Successful in 56s
gate / backend (push) Successful in 1m43s
2026-08-19 02:00:44 +02:00
peter 634e9bb17c chore(release): 0.69.2 — fix multi-account Library thumbnails 2026-08-19 02:00:24 +02:00
peter ce2954cf2b fix(downloads): carry the active account on the Library poster URL
An <img> can't send the X-Siftlode-Account header, so the poster.jpg request
resolved to the session-default account and _accessible_job 404'd it for any
download owned by a different (per-tab) account — every card then fell back to
the source's remote thumbnail, which for Facebook is an expiring fbcdn URL, so
the Library showed blank thumbnails in a multi-account session. The 0.69.1
local-poster-primary change made this the primary image, exposing it broadly.

Append ?account= to the poster URL via a new api.posterImageUrl(), the same
wallet-gated header-less selection downloadFileUrl / storyboardImageUrl already
use (resolved_user_id reads ?account= for exactly these contexts).
2026-08-19 02:00:24 +02:00
peter 2229b01a91 release: 0.69.1 (bug/feed-scroll-thumbnails-about)
gate / frontend (push) Successful in 56s
gate / backend (push) Failing after 38s
2026-08-19 01:42:54 +02:00
peter b17b47f54e chore(release): 0.69.1 — feed read-ahead, download thumbnails, About popover
Three prod bug fixes: local-poster-primary Library thumbnails, viewport-scaled
feed read-ahead (fixes the fast-scroll "Load more" stall), and the channel About
popover fitting/blank-line cleanup.
2026-08-19 01:42:35 +02:00
peter 1e2b3c5d29 docs(sync): scope normalize_description to channels + lock line-ending behavior
Review follow-up. The docstring said "channel/video description" but the helper is
wired only to the channel store sites; clarify it is deliberately channel-only
(video descriptions are user-facing prose where paragraph breaks carry meaning,
unlike a channel's metadata block) so it doesn't read as an asymmetric fix. Also
document that str.splitlines() folds CR/CRLF and Unicode line separators to "\n",
and lock it with a test.
2026-08-19 01:32:26 +02:00
peter d065902404 refactor(feed): cap the read-ahead lookahead and extract it to a tested helper
Review follow-up. The viewport-scaled lookahead was unbounded, so a tall window
prefetched several feed pages on every mount / filter change before any scroll —
wasted work on the app's heaviest query. Clamp it to MAX_LOOKAHEAD_ROWS, and move
the policy into a pure readAhead.ts (readAheadLookahead / shouldPrefetchNext) with
unit tests covering the floor, the scaling, the cap boundary, and the fetch trigger
— the core of the feed-scroll fix now has coverage instead of only manual E2E.
2026-08-19 01:32:26 +02:00
peter a300977ed4 fix(feed): read further ahead so fast scrolling doesn't stall at "Load more"
Keyset pagination is strictly serial (a page needs the previous page's cursor)
and the read-ahead only fired within a fixed 4 rows of the loaded end, pulling
one 60-item page per cycle. A fast free-spin flick (MX Master & co.) out-scrolls
that tiny lookahead, lands on an empty bottom, and waits out a chain of serial
round-trips — the multi-second "Load more" hang. Scale the lookahead to the
rendered row count (PREFETCH_VIEWPORTS screenfuls); since the effect re-runs when
a page settles it chains, keeping a couple of screens buffered ahead of the fold
so the content is usually already there. Bounded, and a no-op for finite lists.
2026-08-19 01:13:57 +02:00
peter ddb7818d38 fix(sync): strip blank lines from channel descriptions before storing
YouTube channel descriptions pad their contact/link blocks with runs of blank
lines, which the About popover renders verbatim (whitespace-pre-wrap) — wasted
space and storage, and noise in the search_vector. Add normalize_description()
and apply it at both channel store sites (full details + initial import) so the
DB holds the clean text. New channels are clean immediately; existing rows heal
on their next metadata re-sync. Unit-tested in test_utils.
2026-08-19 01:13:56 +02:00
peter 48787af11c fix(channels): keep the About popover on-screen at high browser zoom
At elevated zoom the CSS viewport shrinks; a long channel About wanted its fixed
max-h-112 (448px) height, so when floating-ui flipped it above the trigger its
top rendered above the viewport edge, clipped and unreachable (the inner
overflow-y-auto can't scroll past its own top). Enable the popover's fitHeight so
the `size` middleware caps the panel to the available space. Because the surface
splits into an outer positioned element and an inner styled/scroll wrapper, the
middleware now also publishes the available height as a --fit-max-h custom
property the inner scroller consumes via maxHeight: min(28rem, var(--fit-max-h)),
keeping the 28rem design cap on a tall viewport and shrinking to fit on a short one.
2026-08-19 01:13:40 +02:00
peter b733ee84fa fix(downloads): use the local poster as the Library thumbnail primary
The Library card used the source's remote thumbnail_url as the primary <img>
src and only fell back to our self-hosted poster on error. For Facebook
downloads (the bulk of a typical library) that URL is an fbcdn signed link that
expires within days, and is also blocked by ad-blockers / Brave Shields on the
facebook domain — so the thumbnail broke even though the local poster was
present on disk. Flip the priority: serve /api/downloads/{id}/poster.jpg first
(always present, never expires, same-origin) and keep the remote as the fallback
for the rare poster-less asset.
2026-08-19 01:13:40 +02:00
peter 96413c3d5e release: 0.69.0 (improvement/e3-s1-plex-split)
gate / frontend (push) Successful in 59s
gate / backend (push) Successful in 1m51s
2026-08-19 00:14:03 +02:00
peter d9677e0e3a chore(release): 0.69.0 — E3 backend architecture consolidation
Bump VERSION + user-facing release notes for the E3 epic (Plex god-module split,
downloads owner-unification, backend contract & conventions). Observable wins:
Plex catalog prune, download reliability (no false stalls, fair queue, no
file-delete-under-live-download). 20 commits, four clean /code-review high rounds.
2026-08-19 00:13:00 +02:00
peter 2b804300d5 fix(scheduler): address /code-review R3 findings on the maintenance handler (3)
Round-3 review of the S3 config-fold, all in routes/scheduler.py:
- stale docstring: "Persisted to app_state" -> "Persisted via sysconfig" (the
  batch moved stores in B-core-11; the comment didn't follow).
- drop the manual min/max check in the PATCH — sysconfig.set_value already
  validates against the same spec bounds; catch its ValueError -> 400, matching
  the config PATCH pattern (bounds enforced in one place).
- fetch sysconfig.spec(...) once into a local in the GET instead of twice.

No behaviour change beyond the 400 message wording (untested path). 368 pass.
2026-08-18 23:54:32 +02:00
peter 3f047bbad1 refactor(config): config/state mutators flush, callers commit (E3 S3, B-core-10)
Restores CLAUDE.md's canonical transaction rule ("the handler commits; helpers
flush; only quota owns its session") on the config path: sysconfig.set_value /
sysconfig.reset and state.set_sync_paused now flush instead of committing, so a
change lands in ONE transaction with the audit row the route records for it —
previously the helper's internal commit split them, and a crash between left the
mutation persisted with no audit trail.

All callers already commit (the config PATCH/DELETE + audit, the sync pause/resume
+ audit, the scheduler maintenance PATCH); the setup wizard, which relied on the
internal commit, gets an explicit db.commit() after its loop. apply_interval is
deliberately left committing — it performs a live APScheduler reschedule that must
follow the DB persist, so it owns its own unit of work (a legitimate exception,
like the download service transitions). New test locks the flush-not-commit
contract (a rollback before the caller commits discards the change). 368 pass.
2026-08-18 23:46:48 +02:00
peter 8a97c79d37 refactor(config): fold maintenance_revalidate_batch from app_state into sysconfig (E3 S3, B-core-11)
The admin-tunable maintenance re-validation batch lived in its own app_state
column with a hand-rolled override-or-default reader in app.state — a fourth
config store duplicating the sysconfig SPECS pattern. Move it to a sysconfig key
(`maintenance` group, min=100 max=100_000): the reader (sync/maintenance.py) and
the Scheduler dashboard (routes/scheduler.py PATCH + GET) now go through
sysconfig.get_int / set_value / spec, and app.state loses get/set_maintenance_batch
+ the MIN/MAX constants + the settings import. Migration 0070 carries any existing
override into the system_config store before dropping the app_state column
(reversible; restores + copies back on downgrade). Reuses the existing
sysconfig.spec() for the dashboard's min/max, so the bounds have one source.

Full lane 365 pass (migration drift + reversibility exercised), ruff clean.
2026-08-18 23:40:45 +02:00
peter d37b6ef07f refactor(routes): delete 5 dead endpoints the frontend never calls (E3 S3, B-core-8)
None are referenced by the SPA or tests; the Scheduler dashboard's "run now"
covers every manual trigger. Removed: POST /api/sync/{rss,backfill,enrich}
(+ the now-dead _user_channels helper and three sync.runner imports),
POST /api/tags/recompute (+ its admin_user / run_autotag_all imports), and
GET /auth/me (a lesser duplicate of the canonical /api/me in routes/me.py).

This also retires a latent bug: POST /api/sync/backfill reset backfill_done /
backfill_cursor on already-fully-backfilled channels (tipping them back to
"needs full history") for up to 25 channels per curl. Full lane 365 pass.
2026-08-18 23:32:49 +02:00
peter 88c5d5c5cb docs: refresh 7 stale module headers that describe shipped work as "later phase" (E3 S3, B-core-12)
This codebase leans on "why" comments as its architecture doc, so a header that
claims the opposite of the code below it is worse than none. Fix the seven the
review flagged: channels.py (subscribe/unsubscribe shipped), playlists.py +
sync/playlists.py (YouTube two-way sync + write-back shipped), setup.py (drop the
"epic 6c" ref — the wizard steps exist), config.py ("auth._client_ip" ->
"routes.auth._client_ip"), db.py ("routes/plex.py" -> "routes/plex/" after the
package split), sysconfig.py (the "Phase C" note — the watch-sync job that reads
plex_watch_sync_interval_min has landed; clarified it's a boot default, not a knob).
Docs only.
2026-08-18 23:28:25 +02:00
peter cff8bed62b fix(downloads): heartbeat only when no write touched the row this interval (E3 review R2)
Round-2 review of the R1 fixes found that the two-independent-limiter progress
writer issued a redundant heartbeat every interval during an ACTIVE download: a
percent change writes progress (which already refreshes updated_at, the stall
watchdog's liveness source), then the sub-integer calls that follow fire the
separate beat budget — ~2 DB writes/interval instead of the throttle's intended 1.

Replace the second limiter with a shared `last_touch` clock (last DB write OR
beat). Progress writes keep their own `allow_write` budget (so a beat still can't
delay a real write — the R1 concern), but the heartbeat now fires only when
nothing touched the row for a full interval. Active download → ~1 write/interval;
a genuinely stuck re-encode → 1 heartbeat/interval. New regression test:
an unchanged-percent call in the same interval as a write does NOT heartbeat.

ruff clean, full DB-lane 365 pass.
2026-08-18 23:16:48 +02:00
peter bc79720e81 fix(e3): address /code-review high findings (5)
Round-1 review fixes on the E3 branch:

- prune partial-fetch (correctness): `_paginate` now reports whether it reached
  the server's reported `total`; `_prune_stale` prunes only on a COMPLETE
  non-empty listing. A truncated fetch (a mid-sequence empty/short page before
  total — a transient Plex hiccup) no longer deletes the rows it merely failed
  to list. New test: a listing that claims 5 but returns 1 prunes nothing.
- `_prune_stale` (robustness): `stats.get("pruned", 0) + 1` — no KeyError when a
  caller's stats dict omits the key.
- service transitions (conventions): pause/resume/redownload use the JobStatus
  StrEnum instead of loose "queued"/"paused"/"error" literals, matching cancel.
- ffmpeg progress writer (correctness): a stuck-percent heartbeat now runs on its
  OWN 1/interval budget, so it can't consume the progress-write slot and delay a
  real percent change by up to an interval.
- backend/app/README.md (conventions): reword the transaction-ownership rule to
  quote and extend CLAUDE.md's canonical ("handler/job-wrapper commits; helpers
  flush; only quota owns its session") instead of stating a contradicting
  "helpers MAY commit" carve-out.

ruff clean, full DB-lane 364 pass.
2026-08-18 22:47:25 +02:00
peter b02a9e37e8 docs: write the backend conventions + a development guide (E3 S3, B-core-20)
The layering rule ("request-only logic in routes/; scheduler+worker-shared logic
in the plex|downloads|sync packages; cross-cutting primitives as flat core
modules; extract at the SECOND consumer") was real but unwritten — it lived only
in three module docstrings and a one-line README mention. Collect it, plus the
transaction-ownership rule (a mid-operation helper stages, the caller commits),
the atomic-counter / JobStatus-StrEnum conventions, and the one-home-per-config
rule, into backend/app/README.md. Add docs/development.md (run locally + the gate
commands + branching), so a new session/contributor has one place to look instead
of copying whichever pattern they happen to find first.
2026-08-18 22:02:18 +02:00
peter b2013a079a refactor(downloads): split worker.py parent/child into pipeline.py + jobdb.py (E3 S2, B-sub-9)
worker.py (~1090 L) mixed the PARENT supervisor (claim + supervise + the process
loop) with the CHILD pipeline (the yt-dlp/ffmpeg download+edit that runs in the
spawned subprocess). Extract them:

  worker.py  1090 -> 475   parent: claim/supervise/loop
  downloads/pipeline.py  (new 592)  the child pipeline (_download/_process_edit +
                                    hooks/progress/ffmpeg helpers + _Aborted)
  downloads/jobdb.py     (new  73)  the short-lived-session helpers both sides
                                    share (_set_job/_job_status/_touch_job/
                                    _requeue_if_running/_reset_asset_pending +
                                    the staging-dir paths)

The graph is one-directional (runner -> pipeline -> jobdb; worker -> jobdb; no
worker<->pipeline edge) — the parent never calls the child directly, it spawns
runner.py as a subprocess. Consumers rewired: runner import, and two tests follow
the moved symbols (_media_info/_produced_file -> pipeline; the ffmpeg-callback
patches target pipeline where the names are looked up).

Pure code motion, verbatim bodies. ruff clean, full lane 363 pass (unchanged).
2026-08-18 21:58:23 +02:00
peter 0fab0ce779 refactor(auth): typed Pydantic request bodies for the 9 payload:dict handlers (E3 S3, B-core-6)
request-access / demo / register / verify / verify-resend / password-login /
password-reset(request|confirm) / set-password each took an untyped `payload:
dict` and hand-picked fields with `.get(...)`. Replace with five small BaseModels
(EmailIn, EmailPasswordIn, TokenIn, TokenPasswordIn, SetPasswordIn) so FastAPI
validates the body shape and publishes the schema. Fields are `str | None` to
preserve exact behaviour — a missing/null field still coerces via the existing
`(... or "")`, no stricter 422s. Green under the auth-session suite (7), ruff clean.
2026-08-18 21:53:57 +02:00
peter 4ace486618 refactor(downloads): service owns the job state transitions (E3 S2, B-sub-8)
pause/resume/cancel/delete/redownload each hand-rolled their asset bookkeeping
inline in the route (release, failed-asset reset, the 20-line fresh-asset
rebuild), duplicating logic the worker/GC also touch. Move each transition into a
service.* method (service.pause/resume/cancel/delete/redownload) that owns the
mutation + its commit; the routes keep only auth, the HTTP 409 guards, and
serialization. Drops the now-dead _set_status route helper and two stale imports.

Pure behaviour-preserving code motion (routes -56 lines). Green under the E3 S2
transition net + the full downloads suite (363 pass), ruff clean.
2026-08-18 21:46:52 +02:00
peter 3334a8e895 test(downloads): E3 S2 characterization net — job state transitions (B-sub-8)
Pins the observable HTTP behaviour of pause/resume/cancel/delete/redownload
(status changes, asset release, failed-asset reset, fresh-asset rebuild) before
the inline route logic moves into service.* methods. Green on the current routes.
2026-08-18 21:43:57 +02:00
peter e2362e1ed1 fix(downloads): heartbeat a slow re-encode so the stall watchdog can't false-kill it (E3 S2, B-sub-4)
The supervisor kills a job that writes nothing for download_stall_timeout_seconds
(150s) and labels it a "network" stall. But the ffmpeg progress writer only wrote
on an integer-percent CHANGE, so a re-encode that sits on one percent for minutes
(AV1/VP9 -> H.264 or a crop on the CPU-only prod host) produced no row write and
was killed mid-encode as "stalled" — and the retry stalled the same way.

Fix: the writer now heartbeats (`_touch_job`, bumping updated_at without touching
the progress bar) once per interval even when the whole percent is unchanged, so
a live-but-quiet re-encode keeps a fresh updated_at. Reorders the rate-limit gate
first so a burst still costs one DB op — either a progress write or a heartbeat.

Pure-logic test with an injected clock: three stuck-at-10% intervals write no new
progress but heartbeat each time. Full lane green, ruff clean.
2026-08-18 21:34:36 +02:00
peter 7d89bc8b8e fix(downloads): exclude in-flight assets from the claim, drop the busy-wait (E3 S2, B-sub-3)
Two jobs on the same shared asset (same video+format queued twice, or by two
users) starved the worker pool: worker A downloads the asset; worker B claims the
sibling job, finds the asset `downloading`, requeues it, sleep(2)s, and re-claims
it on the next tick — burning a whole worker slot behind one in-flight asset for
the entire download (at concurrency=2, the queue stalls).

Fix: the claim SQL now excludes any job whose asset is already `downloading`
(`NOT EXISTS … media_assets.status='downloading'`), so a competing sibling simply
waits until the asset settles and the workers pick up unrelated jobs meanwhile.
The `_process_job` busy/lost-race branches drop their `time.sleep(2)` — the
exclusion keeps a requeued job out of the claim window, so there's no spin.

TDD: the new test failed on the old claim (claimed the sibling), passes now.
Full lane green, ruff clean.
2026-08-18 21:31:18 +02:00
peter f0a0214d97 fix(downloads): atomic ref_count increment/decrement (E3 S2, B-sub-5)
MediaAsset.ref_count (how many jobs hold a shared cache asset) was a Python
read-modify-write at four sites (enqueue, enqueue_edit, release_asset, the
redownload route). Two concurrent enqueues of the same (source,format) both read
the same value and write the same +1 → the count ends up one short → the first
release drives it to zero and deletes the file out from under a live holder.

Replace all four with atomic DB ops: incr_ref_count (`ref_count = ref_count + 1`)
and decr_ref_count (`greatest(ref_count - 1, 0)` RETURNING the new value, so the
release decides whether to free the asset from the post-decrement count while
holding the row's write lock — Postgres serialises concurrent releases on that
lock, closing the double-free). Pure DB-atomicity change, no semantic shift.

Green under the E3 S2 ref_count net (5) + full lane (356 pass), ruff clean.
2026-08-18 21:27:39 +02:00
peter e824fe0ae9 test(downloads): E3 S2 characterization net — ref_count enqueue/release semantics (B-sub-5)
Locks the observable single-threaded ref_count accounting before the atomic-SQL
conversion: two enqueues of one (source,format) share a single asset and stack
ref_count to 2; release keeps the asset until the last holder then frees the
ready row; a non-holding (canceled) job's release is a no-op; the decrement
clamps at 0; an errored asset is kept at zero (only ready is freed). Green on the
current RMW code (5 pass) — the fix must keep every assertion.
2026-08-18 21:24:46 +02:00
peter 49011940d6 feat(plex): catalog prune — delete titles that disappeared from Plex (E3 S1, B-sub-6)
A sync was upsert-only: a movie/show/season/episode removed from Plex stayed
browsable forever and 404'd on playback. Each section now diffs the rating_keys
it just listed against the stored ones and deletes the stale rows (watch-state /
playlist links cascade via the existing FKs), mirroring the collections prune
that already shipped.

Safety: the prune is GUARDED on a non-empty listing (_prune_stale returns early
when `seen` is empty), so a transient Plex glitch that yields an empty page can
never mass-delete a library — the metadata is cheap to re-sync, an accidental
wipe is the only real risk. Shows prune leaf-first (episodes -> seasons -> shows)
so a stale row is deleted directly before its parent's cascade reaches it.

Out of scope (separate follow-up): library-level reconcile of a whole removed
section + retiring the dead `PlexLibrary.enabled` column.

Tests: movie prune + watch-state cascade, the empty-listing guard (mutation-
verified — removing the guard wipes everything), show/season/episode prune.
351 DB-lane pass, ruff clean.
2026-08-18 21:13:18 +02:00
peter 52b6535dc6 refactor(plex): split browse.py god-module into per-concern routers (E3 S1)
routes/plex/browse.py was a 1693-line god-module (libraries + facets +
collections + playlists + show/item detail + people). Split it into focused
sibling routers under the same /api/plex prefix — pure code motion, no SQL /
route-path / response-shape changes:

  browse.py  1693 -> 469  (list_libraries, unified_library, facets)
  collections.py  (new 253)  browse + admin collection editor; owns _collection_source
  playlists.py    (new 405)  the user's orderable playlists
  detail.py       (new 438)  show/item detail + enrichment; _lib_key lives here
  people.py       (new 139)  cast/crew chips + person filmography

Cross-module helpers _norm_lang + _content_libs move to _shared.py (their one
home); detail imports _collection_source from collections (one-way, no cycle).
__init__.py includes the four new routers. Three DB-lane tests follow the moved
symbols (import path for _content_libs; monkeypatch targets for PlexClient /
_show_enrichment now on detail).

Behaviour-equivalent: ruff clean, 348 DB-lane pass (unchanged), parity net green.
2026-08-18 21:06:04 +02:00
peter 9bf7da2cbe refactor(plex): dedup movie/show watch-state predicates into query.py (E3 S1)
The `show` scope (all|watched|in_progress|unwatched) filter and the status CASE
were hand-duplicated between unified_library (the grid) and facets.wsq — the
exact drift risk the review flagged (a hidden movie kept out of the grid must
not inflate facets). Extract them into four shared query.py helpers
(movie_status_case, show_status_case, apply_movie_state_filter,
apply_show_state_filter), model-agnostic (callers pass the joined state
expressions), and point all three call sites at them.

Pure behaviour-equivalence refactor: browse.py -67 lines, no query changes.
Green under the E3 S1 parity net + query_conversion/facets/field-scoped/lang
tests (348 DB-lane pass, ruff clean).
2026-08-18 20:52:39 +02:00
peter 3301976375 test(plex): E3 S1 characterization net — grid⟷facets watch-state parity
Coverage-first safety net for the E3 S1 Plex god-module split. The movie/show
watch-state predicates are currently hand-duplicated in unified_library (the
grid) and facets.wsq (the facet aggregation); this pins the invariant the
facets docstring promises — with no metadata filter active, a facet's genre /
content-rating counts equal the same counts over the grid's visible items — for
every (scope × show) combination, deriving the expected counts from the grid's
own response. Locks the two branch-sets together before they merge into one
shared helper, so the merge is provably behaviour-equivalent.

Mutation-verified: dropping the facets hidden-exclusion branch makes both tests
fail (Drama 3≠2, R 2≠1). Green on the current code (348 DB-lane pass).
2026-08-18 20:48:43 +02:00
peter fa176a3e6a release: 0.68.0 (improvement/e2-s6-job-index-hygiene)
gate / frontend (push) Successful in 52s
gate / backend (push) Successful in 1m37s
2026-08-18 19:20:43 +02:00
peter e8771f4d6b chore(release): 0.68.0 — E2 Feed & Search performance epic
Bump VERSION 0.67.0 -> 0.68.0 and add the user-facing release-notes entry for
the E2 epic (feed sort index, keyset/NOT NULL, search GIN + state inner-joins,
single /feed/summary pass, no card-action refetch storm, job partial indexes).
2026-08-18 19:20:18 +02:00
peter 94dbe4b228 fix(jobs): match the boolean partial predicates to the IS TRUE/IS FALSE queries (E2 S6 review)
The via_search / shorts_probed partials used `WHERE via_search` / `WHERE NOT
shorts_probed`, but the queries filter with `.is_(True)`/`.is_(False)` (SQL
`IS TRUE`/`IS FALSE`), which Postgres's partial-index prover does NOT match to a
bare-boolean predicate — so both fell back to a full seq scan (the shorts-probe
job kept full-scanning for its work queue, the exact thing PERF-3 meant to fix).

Align the predicates to the query form: `WHERE via_search IS TRUE` and
`WHERE shorts_probed IS FALSE`. Verified on the prod-sized localdev: both now
Index Only Scan (via_search 44,683 -> 3 buffers; shorts_probed 44,683 -> 1).
Gate green (346 pytest).
2026-08-18 18:31:38 +02:00
peter 0ed4ac6958 perf(jobs): partial indexes for the enrich/maintenance scanners + boolean cleanup (E2 S6, PERF-17/18/3)
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17 enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes):
  44,680 buffers even with 0 pending -> a partial ix_videos_enrich_pending makes
  it a 1-buffer index scan (empty when nothing pending).
- PERF-18 maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY
  last_checked_at NULLS FIRST LIMIT N`, ~300 pages/run): 44,754 buffers + top-N
  sort -> a partial ix_videos_recheck_order in that exact order = a 42-buffer
  index scan. The order_by is rewritten to `asc().nulls_first()` (equivalent to
  the old `(IS NULL) DESC, ASC`) so it rides the index; the unused plain
  ix_videos_last_checked_at is dropped.

PERF-3: drop the never-picked boolean btrees (is_short, via_explore) and slim the
useful ones to partials over the queried subset (via_search=true, NOT
shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. Model: those 6 columns lose their inline index=True
(the partials are migration-defined, like the search_vector GIN); live_status
left alone.

Measured on the prod-sized localdev reseed: enrich 44,680 -> 1 buffer (25.9 ->
0.02 ms), maintenance 44,754 -> 42 buffers (35.5 -> 0.17 ms); all 11 CONCURRENTLY
index ops on 257k rows in 0.75s. Gate green (346 pytest incl. drift + downgrade
round-trip).
2026-08-18 17:01:13 +02:00
peter 8b8b0ca5ab perf(feed): skip the empty IN() query in tags_by_category (E2 review round 2)
The tags_by_category extraction dropped feed_summary's `if selected:` guard, so
a /feed/summary request with no tag filter (the common case) ran a pointless
`SELECT ... WHERE id IN ()` round-trip. Early-return {} for empty tag_ids —
restores the no-query hot path and covers filtered_query + future callers too.
Gate green (346 pytest).
2026-08-18 16:49:29 +02:00
peter 6116ad4bf4 refactor(feed): address /code-review high findings on the E2 stack
Review round on the E2 stack (S1-S5). No correctness bugs; the surviving
low/medium findings addressed:

- altitude (#1): extract the shared `tags_by_category` helper used by BOTH
  filtered_query (SQL tag WHERE) and feed_summary (Python facet/count), removing
  the category-grouping duplication, and add bidirectional "keep in lockstep —
  equivalence-tested by test_feed_summary_matches_count_plus_facets" comments so
  the SQL<->Python tag rule can't silently drift.
- test-coverage (#2): the /feed/summary equivalence test seeded only system tags,
  leaving the `ChannelTag.user_id == user.id` visibility branch untested. Add a
  user-owned Tag + link + two combos; the suite is now 346 (was 344) and the new
  cases pass (feed_summary agrees with /feed/count + /facets on user-visible tags).
- dead-code (#3): annotate get_feed_count, get_facets, and the FE api.feedCount/
  facets as "SUPERSEDED by /feed/summary — kept only as the equivalence oracle;
  retire together" so the intentionally-retained surface reads as deliberate.

Not changed (confirmed no-op): the search channel-name resolve (#4) is
once-per-request on a 633-row table and a deliberate pre-resolve for PERF-12's
BitmapOr plan; the facet loop (#5) already computes passes() once per
(channel, category) with a distinct exclude each, so there is no recomputation.

Gate green: ruff + 346 pytest; frontend tsc/knip/prettier (FE change is a comment).
2026-08-18 16:39:04 +02:00
peter 943143ef45 perf(feed): stop the card-action + player-close feed refetch storm (E2 S5, PERF-2)
Marking a card watched/hidden/reset and closing the player each fired a full
`invalidateQueries(qk.feed())`, refetching EVERY loaded infinite-query page — 10
pages x a few quick marks = ~50 feed refetches, each a full scan (x PERF-1). The
optimistic `overrides` layer + the matchesView filter already render the correct
state (a hidden/watched card drops out instantly), so the refetch was pure waste.

Switch these paths to `refetchType: "none"`: the queries are marked stale but not
refetched now; the next filter change / remount pulls fresh server data. Applies
to the two Feed card-action paths (setState/clearState) and the PlayerModal
close (feed + playlists — progress is already persisted server-side, only the
on-screen resume bar waits for the next natural refetch).

Left as immediate refetch on purpose: channel/tag/playlist mutations (one-off, on
pages where the main feed is unmounted anyway) and the onboarding / notification-
undo paths (which must reveal the changed content right away). Gate green:
frontend tsc/eslint/knip/prettier/222 vitest (backend unchanged, 344 pytest).
2026-08-18 15:49:49 +02:00
peter 21a3aae700 perf(feed): collapse count + facets into one /feed/summary pass (E2 S4, PERF-13)
The FE fired /feed/count (1 pass) + /facets (3 passes) on every filter change =
4 full scans of the subscribed set. New /api/feed/summary does ONE grouped query
(the base filter with tags stripped, GROUP BY channel_id -> per-channel video
counts) and derives the total count + every tag's distinct-channel facet count in
Python on the ~few-hundred-row channel map, faithfully mirroring filtered_query's
channel-based tag logic + the /facets drill-down (own category dropped, unless a
conjunctive topic-AND). Returns {count, facets} — the same numbers the two old
endpoints returned.

FE: a single shared query (Feed reads .count, Sidebar reads .facets); same key +
filters, so React Query dedupes to ONE request. Invalidations repointed to
qk.feedSummary. The old /feed/count + /facets endpoints (and their now-unused FE
api/qk) are kept as the equivalence-test oracle + safety net; a later cleanup can
drop them.

New equivalence test (test_feed_views, 8 filter combos incl. cross-category,
disjunctive/conjunctive topic modes, search): /feed/summary count == /feed/count
and facets == /facets, byte-identical. Structural win: 2 requests / 4-6 queries
-> 1 request / 2-3 queries (the review measured 530 -> 144 ms). Gate green:
344 pytest, frontend tsc/eslint/knip/prettier/222 vitest.
2026-08-18 05:01:27 +02:00
peter 6a35e83ded perf(feed): un-block the search GIN + inner-join the state views (E2 S3, PERF-12/16)
PERF-12: the search ORed `unaccent(channels.title) ILIKE '%q%'` straight into
the per-video predicate, which forced a full scan + detoast of every
search_vector and a per-candidate channel fetch. Pre-resolve the matching
channel ids (a few-hundred-row scan) and match `Video.channel_id.in_(ids)`
instead, so the planner keeps the search_vector GIN as a BitmapOr branch
(alongside ix_videos_channel_pub_desc). Empty match -> false() collapses the OR.

PERF-16: show=watched|hidden|in_progress used an outer join + coalesce(status,
'new'), so the planner couldn't start from the user's small video_states set.
Those views only return rows that HAVE a state, so inner-join video_states and
filter the raw column (status is NOT NULL on a present row, so it's identical).
'unwatched'/'all' keep the outer join. Dropped the now-dead position_expr.

New regression tests (test_feed_views): search returns both text- and channel-
name matches; the four watch-state views return the exact expected sets.

Measured on the prod-sized localdev reseed (prod planner, 0067/0068 applied):
- search /feed/count q=magyar: 663,463 -> 19,206 buffers (1,406 -> 31 ms; OLD
  seq-scan+detoast -> NEW BitmapOr GIN+channel index)
- search feed page: 2,908 -> 1,479 buffers (32.6 -> 1.4 ms)
- show=watched (user with 64 watched): 40,661 -> 264 buffers (~154x)
Gate green (336 pytest, incl. the 2 new equivalence tests).
2026-08-18 04:41:16 +02:00
peter 8c08ea3133 perf(feed): published_at NOT NULL + range-anchored keyset for deep pages (E2 S2, PERF-15)
A nullable published_at forced the feed keyset into a non-sargable
`... OR published_at IS NULL` disjunction: with the 0067 index the planner still
applied it as a FILTER and scanned from the top of the index, so a deep `newest`
page discarded every earlier row (O(depth)).

- Migration 0068: backfill any NULLs (0 in prod) -> SET NOT NULL + DEFAULT now().
- Model: Video.published_at is now non-Optional, nullable=False, server_default now().
- feedquery: the `pub` key is nullable=False, and keyset_predicate is rebuilt
  recursively so a non-nullable leading key gets a non-strict range bound
  (`published_at <= v`) as an AND conjunct — an index range Cond instead of a
  filter. Nullable keys keep the NULL-aware OR form (a `<= v` bound would drop
  their NULL group). Equivalence proven by test_feed_keyset (all 13 sorts,
  cursor-walk == unpaged).
- Ingest paths (stub_from_playlist_item, apply_video_details, playlists sync)
  fall back to now() so they never write an explicit NULL.

Measured on the prod-sized localdev reseed at a ~50k-deep cursor (0067 index +
prod planner): OLD `Rows Removed by Filter: 50001`, 25,687 buffers / 9.16 ms ->
NEW `Index Cond: published_at <= v`, 1 row filtered, 47 buffers / 0.037 ms. SET
NOT NULL on 257k rows: 0.32s. Gate green (334 pytest).
2026-08-18 04:30:07 +02:00
peter 3af3799b25 perf(feed): index the default published_at DESC NULLS LAST sort (E2 S1, PERF-1)
The feed's default `newest` sort is `published_at DESC NULLS LAST, id ASC`, but
the only published_at index was a plain ASC btree — read backwards it satisfies
DESC NULLS FIRST, not NULLS LAST — so every feed page fell back to a full
`videos` seq scan + top-N sort. Migration 0067 adds the two matching indexes
(global + per-channel) CONCURRENTLY and drops the now-redundant single-column
ix_videos_channel_id (the channel composite leads with channel_id, a strict
superset).

Measured on the prod-sized localdev reseed (257,312 videos, prod planner):
- global newest:  44,747 buffers / 154 ms  ->  60 buffers / 0.32 ms
- per-channel(12.6k): 7,604 buffers / 96 ms -> 54 buffers / 0.14 ms
CONCURRENTLY build of both on 257k rows: 1.7s total (~0 downtime). This is the
likely root cause of the "prod feed is slow" symptom.

Model: drop the inline index=True on Video.channel_id (the composite supersedes
it); the composites are migration-defined like the search_vector GIN. Gate green
(334 pytest incl. the migration drift + downgrade/re-upgrade round-trip).
2026-08-18 04:16:08 +02:00
223 changed files with 18752 additions and 6656 deletions
+1 -1
View File
@@ -1 +1 @@
0.67.0 0.72.0
@@ -0,0 +1,55 @@
"""feed sort indexes: published_at DESC NULLS LAST composite(s) for the default feed (PERF-1)
The feed's default `newest` sort is `published_at DESC NULLS LAST, id ASC` (feedquery.order_by), but the
only published_at index was a plain ASC btree (`ix_videos_published_at`). A btree read backwards satisfies
`DESC NULLS FIRST`, not the `DESC NULLS LAST` the query asks for, so the planner could not use it for
ordering — every feed page fell back to a full `videos` scan + top-N sort. Measured on a prod-sized reseed
(257k videos): the default page went from ~44.8k shared buffers (dev planner) / ~169k (prod planner, the
per-channel nested loop that is the likely "prod feed is slow" root cause) down to ~624 buffers with the
matching index.
This adds two indexes in the exact feed order:
- ix_videos_published_at_desc (published_at DESC NULLS LAST, id ASC) — the global feed / Library sort.
- ix_videos_channel_pub_desc (channel_id, published_at DESC NULLS LAST, id ASC) — the per-channel feed
(channel page, channel_ids filter). It leads with channel_id, so it is a strict superset of the old
single-column `ix_videos_channel_id` (FK cascade, sync upsert, autotag, channel-detail all still ride
it) — that redundant index is dropped.
The plain ASC `ix_videos_published_at` stays: it already serves the `oldest` sort (ASC NULLS LAST = btree
default) as an Index Scan.
CONCURRENTLY (in alembic's autocommit_block, since it can't run inside a transaction) so a live prod builds
these without blocking reads/writes; downtime ~0. IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade round-trip idempotent.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0067_feed_sort_indexes"
down_revision: Union[str, None] = "0066_notif_created_at_notnull"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.get_context().autocommit_block():
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_published_at_desc "
"ON videos (published_at DESC NULLS LAST, id ASC)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_pub_desc "
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)"
)
# Redundant now — ix_videos_channel_pub_desc leads with channel_id (a strict superset).
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_id")
def downgrade() -> None:
with op.get_context().autocommit_block():
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_channel_id ON videos (channel_id)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_channel_pub_desc")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_published_at_desc")
@@ -0,0 +1,36 @@
"""videos.published_at NOT NULL DEFAULT now() (PERF-15)
Every video in practice carries a publish date (0 NULLs in the 257k-row prod-sized reseed), but the
column was nullable, which forced the feed's keyset predicate into a non-sargable
`... OR published_at IS NULL` disjunction — blocking the ix_videos_published_at_desc range so deep
`newest` pages scanned from the top of the index and filtered (51.8k buffers). Making it NOT NULL lets
the keyset drop that disjunction and take a `published_at <=` range bound (deep page -> ~0.5k buffers).
server_default now() is the safety net for the rare date-less ingest path (the enrich overwrites it with
the real date); the app-side ingest paths also fall back to now() so they never write an explicit NULL.
Note: `SET NOT NULL` validates with a brief ACCESS EXCLUSIVE lock + one table scan (it cannot run
CONCURRENTLY). On the prod row count (~235k) that is sub-second; still, it is a short write-blocking lock
rather than the ~0 of 0067's CONCURRENTLY index build.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0068_published_at_not_null"
down_revision: Union[str, None] = "0067_feed_sort_indexes"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Backfill any stray NULLs first so SET NOT NULL can't fail (0 in prod, but be safe).
op.execute("UPDATE videos SET published_at = now() WHERE published_at IS NULL")
op.alter_column(
"videos", "published_at", nullable=False, server_default=sa.text("now()")
)
def downgrade() -> None:
op.alter_column("videos", "published_at", nullable=True, server_default=None)
@@ -0,0 +1,93 @@
"""scheduler-job partial indexes + boolean-index cleanup (PERF-17/18/3)
Two scheduler jobs full-seq-scanned the 257k-row videos table every run:
- PERF-17: enrich_pending (`WHERE enriched_at IS NULL LIMIT N`, every few minutes) — 44.6k buffers /
run even when nothing is pending. A partial index over just the pending rows makes it a 1-buffer
(empty) index scan.
- PERF-18: maintenance revalidate (`WHERE unavailable_since IS NULL ORDER BY last_checked_at NULLS
FIRST LIMIT N`, ~300 pages/run) — 44.7k buffers/page; ix_videos_last_checked_at had idx_scan=0. A
partial index in that exact order serves it as an index scan (~36 buffers/page) and replaces the
unused plain one.
PERF-3: the videos table also carried plain btrees on low-cardinality boolean columns the planner never
picks (a `col IS false` majority predicate can't ride a btree). Drop the never-useful ones (is_short,
via_explore) and slim the rest to partial indexes over the small/shrinking subset the job queries target
(via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — less write amplification on the
biggest table, same lookups. live_status is left alone (a separate finding).
All CONCURRENTLY (autocommit_block) — downtime ~0; IF (NOT) EXISTS keeps the drift test's
downgrade->re-upgrade idempotent.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0069_job_index_hygiene"
down_revision: Union[str, None] = "0068_published_at_not_null"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.get_context().autocommit_block():
# PERF-17: enrich-pending partial (empty when nothing is pending -> 1-buffer scan).
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_enrich_pending "
"ON videos (id) WHERE enriched_at IS NULL"
)
# PERF-18: maintenance revalidate order, as a partial in the exact WHERE + ORDER BY shape.
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_recheck_order "
"ON videos (last_checked_at ASC NULLS FIRST) WHERE unavailable_since IS NULL"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_last_checked_at")
# PERF-3: drop the never-picked boolean btrees; slim the rest to partials over the target subset.
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_is_short")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_explore")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
# Predicate MUST match the query's `.is_(True)`/`.is_(False)` form (SQL `IS TRUE`/`IS FALSE`):
# Postgres's partial-index prover does NOT match a `col IS TRUE` clause to a bare `col` predicate,
# so `WHERE via_search` / `WHERE NOT shorts_probed` would leave the queries seq-scanning.
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search "
"ON videos (id) WHERE via_search IS TRUE"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed "
"ON videos (id) WHERE shorts_probed IS FALSE"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
"ON videos (unavailable_since) WHERE unavailable_since IS NOT NULL"
)
def downgrade() -> None:
with op.get_context().autocommit_block():
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_enrich_pending")
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_recheck_order")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_last_checked_at "
"ON videos (last_checked_at)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_is_short ON videos (is_short)"
)
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_explore ON videos (via_explore)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_via_search")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_via_search ON videos (via_search)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shorts_probed")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_shorts_probed ON videos (shorts_probed)"
)
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_unavailable_since")
op.execute(
"CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_videos_unavailable_since "
"ON videos (unavailable_since)"
)
@@ -0,0 +1,47 @@
"""Fold the maintenance re-validation batch from app_state into sysconfig (B-core-11)
The admin-tunable `maintenance_revalidate_batch` lived in its own `app_state` column with a
hand-rolled override-or-default reader in `app.state`, duplicating the sysconfig SPECS pattern. It's
now a sysconfig key (`system_config` store), so every admin knob lives in one place. This migration
carries any existing override into the sysconfig store, then drops the dead column. Reversible.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0070_fold_maintenance_batch"
down_revision: Union[str, None] = "0069_job_index_hygiene"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Preserve a customized override: copy it into the sysconfig store before the column goes.
op.execute(
"""
INSERT INTO system_config (key, value, encrypted)
SELECT 'maintenance_revalidate_batch', maintenance_revalidate_batch::text, false
FROM app_state
WHERE id = 1 AND maintenance_revalidate_batch IS NOT NULL
ON CONFLICT (key) DO NOTHING
"""
)
op.drop_column("app_state", "maintenance_revalidate_batch")
def downgrade() -> None:
op.add_column(
"app_state",
sa.Column("maintenance_revalidate_batch", sa.Integer(), nullable=True),
)
# Carry the value back onto the row, then drop the sysconfig override (symmetric restore).
op.execute(
"""
UPDATE app_state SET maintenance_revalidate_batch =
(SELECT value::int FROM system_config WHERE key = 'maintenance_revalidate_batch')
WHERE id = 1
AND EXISTS (SELECT 1 FROM system_config WHERE key = 'maintenance_revalidate_batch')
"""
)
op.execute("DELETE FROM system_config WHERE key = 'maintenance_revalidate_batch'")
@@ -0,0 +1,52 @@
"""plex TMDB enrichment columns (E5 S2)
Additive TMDB-sourced fields on plex_items + plex_shows — keywords, full crew, budget/revenue, TMDb
community score, production companies/countries, recommendations, and a backdrop-path pool — plus a
`tmdb_id` cache and a `tmdb_enriched_at` watermark. Filled by the new TMDB enrichment job (matched by
the stored canonical `guid`), SERVED FROM THE ROW (never a live TMDB call). Distinct from the existing
Plex-sourced `imdb_rating`/`rating`, which are left untouched. Plus `plex_people.tmdb_person_id` (the S2
foundation for S6's person bios). ADD-only, all nullable → no table rewrite, no ACCESS EXCLUSIVE lock.
No watermark reset needed: `tmdb_enriched_at` starts NULL, so every eligible row is picked up once.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0071_plex_tmdb_fields"
down_revision: Union[str, None] = "0070_fold_maintenance_batch"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_TABLES = ("plex_items", "plex_shows")
# (name, type) — same set on both movie/leaf rows and show rows. budget/revenue stay NULL for shows.
_COLS: tuple[tuple[str, sa.types.TypeEngine], ...] = (
("tmdb_id", sa.Integer()),
("tmdb_enriched_at", sa.DateTime(timezone=True)),
("tmdb_rating", sa.Float()),
("tmdb_vote_count", sa.Integer()),
("budget", sa.BigInteger()),
("revenue", sa.BigInteger()),
("keywords", postgresql.JSONB()),
("crew", postgresql.JSONB()),
("studios", postgresql.JSONB()),
("production_countries", postgresql.JSONB()),
("recommendations", postgresql.JSONB()),
("backdrops", postgresql.JSONB()),
)
def upgrade() -> None:
for t in _TABLES:
for name, type_ in _COLS:
op.add_column(t, sa.Column(name, type_, nullable=True))
op.add_column("plex_people", sa.Column("tmdb_person_id", sa.Integer(), nullable=True))
def downgrade() -> None:
op.drop_column("plex_people", "tmdb_person_id")
for t in _TABLES:
for name, _ in reversed(_COLS):
op.drop_column(t, name)
@@ -0,0 +1,40 @@
"""plex person TMDB bio columns (E5 S6)
Additive TMDB-sourced person fields on plex_people — biography, birth/death dates, place of birth,
known-for department, also-known-as, and a `known_for` credit pool ([{tmdb_id, media_type, title}]
resolved to library rows by guid at serve time) — plus a `tmdb_enriched_at` watermark. Filled the
first time a person page is opened (from /person/{tmdb_person_id}), then SERVED FROM THE ROW. The
`tmdb_person_id` link itself already exists (0071). ADD-only, all nullable → no table rewrite, no
ACCESS EXCLUSIVE lock. `tmdb_enriched_at` starts NULL, so every person is enriched once on first view.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0072_plex_person_bio"
down_revision: Union[str, None] = "0071_plex_tmdb_fields"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_COLS: tuple[tuple[str, sa.types.TypeEngine], ...] = (
("biography", sa.Text()),
("birthday", sa.String(length=10)),
("deathday", sa.String(length=10)),
("place_of_birth", sa.Text()),
("known_for_department", sa.String(length=64)),
("also_known_as", postgresql.JSONB()),
("known_for", postgresql.JSONB()),
("tmdb_enriched_at", sa.DateTime(timezone=True)),
)
def upgrade() -> None:
for name, type_ in _COLS:
op.add_column("plex_people", sa.Column(name, type_, nullable=True))
def downgrade() -> None:
for name, _ in reversed(_COLS):
op.drop_column("plex_people", name)
@@ -0,0 +1,39 @@
"""saved_views.source (ask 4: per-module saved views)
Saved "smart views" were feed-only; the Plex module now has them too. Add a `source` discriminator
(default "feed" so every existing row stays a feed view) and widen the per-user unique name constraint
to (user_id, source, name) so a feed view and a Plex view can share a name. `source` is NOT NULL with a
server_default, so the ADD is a metadata-only rewrite-free change; the constraint swap is quick on a
small per-user table.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0073_saved_view_source"
down_revision: Union[str, None] = "0072_plex_person_bio"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"saved_views",
sa.Column("source", sa.String(length=16), nullable=False, server_default="feed"),
)
op.drop_constraint("uq_user_saved_view", "saved_views", type_="unique")
op.create_unique_constraint(
"uq_user_source_saved_view", "saved_views", ["user_id", "source", "name"]
)
def downgrade() -> None:
# LOSSY / effectively forward-only: upgrade lets a user hold a feed view and a plex view of the same
# name, so re-creating the narrow (user_id, name) constraint can raise IntegrityError on such a pair.
# De-dup by dropping the non-feed rows first (they can't exist in the pre-0073 schema anyway) so the
# narrow constraint re-applies cleanly for the drift test's downgrade→re-upgrade cycle.
op.execute("DELETE FROM saved_views WHERE source <> 'feed'")
op.drop_constraint("uq_user_source_saved_view", "saved_views", type_="unique")
op.create_unique_constraint("uq_user_saved_view", "saved_views", ["user_id", "name"])
op.drop_column("saved_views", "source")
+75
View File
@@ -0,0 +1,75 @@
# Backend conventions
The rules below are already followed across `backend/app` — they lived only in a few module
docstrings (`demo.py`, `userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
## Module layering
Three layers, by *who* runs the code — not by feature:
1. **`routes/…` — request-only.** A route module holds HTTP concerns: auth dependencies, request/
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
2. **`plex/`, `downloads/`, `sync/` — domain logic shared by more than one caller.** Anything the
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
route, the worker, and the scheduler share one definition. Examples from this codebase:
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
layers import. They depend on nothing above them (a core module must never import a `routes/…`
module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't
pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
## Shared counters and status: do it atomically at the DB
- **Never read-modify-write a shared counter in Python** (`x.count = x.count + 1`). Concurrent sessions
lose an update. Use a DB-atomic statement — `col = col + 1`, or `greatest(col - 1, 0)` with
`RETURNING` so the caller branches on the post-value under the row lock (`service.incr_ref_count` /
`decr_ref_count`).
- **Download job status is a `JobStatus` StrEnum** (`downloads/status.py`), not bare strings. The
`HOLDING` set defines which states hold a `MediaAsset` (so release accounting stays correct);
`STOPPED_BY_ROUTE` is the route-initiated stop set. A worker child's terminal write goes through
`service.finalize_if_running` (an atomic `UPDATE … WHERE status='running'`) so a finishing child can't
resurrect a job a route just stopped.
## Config: one home per kind
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
## The gate
Everything below must be green before a change is done (run via `siftlode check`):
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
+15 -1
View File
@@ -57,7 +57,7 @@ class Settings(BaseSettings):
# VPS Caddy front door. A request from any other peer (someone hitting the app port directly) # VPS Caddy front door. A request from any other peer (someone hitting the app port directly)
# is NOT trusted and its raw socket IP is used, so it can't spoof its rate-limit identity via # is NOT trusted and its raw socket IP is used, so it can't spoof its rate-limit identity via
# XFF. Empty = trust no proxy (use the direct peer) — correct for a directly-exposed / local-dev # XFF. Empty = trust no proxy (use the direct peer) — correct for a directly-exposed / local-dev
# deploy. Comma-separated. See auth._client_ip + the deploy docs. # deploy. Comma-separated. See routes.auth._client_ip + the deploy docs.
trusted_proxy_ips: str = "" trusted_proxy_ips: str = ""
# --- Outbound email (onboarding: access-request + approval notices) --- # --- Outbound email (onboarding: access-request + approval notices) ---
@@ -245,6 +245,20 @@ class Settings(BaseSettings):
# local disk, so the default is fine there. # local disk, so the default is fine there.
plex_hls_dir: str = "" plex_hls_dir: str = ""
# --- TMDB enrichment (E5 S2; optional — enriches the Plex catalog from TMDB) ---
# TMDB API key (v3 api_key, 32 hex chars). Server-side only; stored encrypted when set via the
# admin Config page. Empty → the TMDB enrichment job no-ops. Feeds keywords / crew / budget /
# revenue / TMDb score / recommendations / backdrops onto the mirrored Plex rows, matched by the
# stored canonical guid, and SERVED FROM THE DB — never a live TMDB call at serve time.
tmdb_api_key: str = ""
tmdb_api_base: str = "https://api.themoviedb.org/3"
# Metadata language for TMDB fetches (keyword/overview locale). Stored EN by default.
tmdb_language: str = "en-US"
# How often the TMDB enrichment job fetches a batch of not-yet-enriched Plex titles, in minutes.
tmdb_enrich_interval_min: int = 120
# Titles fetched per TMDB enrichment run (TMDB is fast + generously rate-limited).
tmdb_enrich_batch: int = 100
@property @property
def allowed_email_set(self) -> set[str]: def allowed_email_set(self) -> set[str]:
return {e.strip().lower() for e in self.allowed_emails.split(",") if e.strip()} return {e.strip().lower() for e in self.allowed_emails.split(",") if e.strip()}
+1 -1
View File
@@ -12,7 +12,7 @@ class Base(DeclarativeBase):
# Pool sized above the sync-endpoint threadpool (~40) so a burst of short DB sessions can't starve on # Pool sized above the sync-endpoint threadpool (~40) so a burst of short DB sessions can't starve on
# checkout. The image proxies deliberately DON'T hold a connection during their slow external fetch # checkout. The image proxies deliberately DON'T hold a connection during their slow external fetch
# (see routes/plex.py), so this headroom is only for the many brief sessions, not long ones. # (see routes/plex/), so this headroom is only for the many brief sessions, not long ones.
engine = create_engine( engine = create_engine(
settings.database_url, settings.database_url,
pool_pre_ping=True, pool_pre_ping=True,
+73
View File
@@ -0,0 +1,73 @@
"""Short-lived-session DB helpers + staging paths shared by the worker parent and the child pipeline."""
from pathlib import Path
from sqlalchemy import text
from app.config import settings
from app.db import SessionLocal
from app.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
def _set_job(job_id: int, **fields) -> None:
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
if job is None:
return
for k, v in fields.items():
setattr(job, k, v)
db.commit()
def _job_status(job_id: int) -> str | None:
with SessionLocal() as db:
return db.execute(
text("SELECT status FROM download_jobs WHERE id = :id"), {"id": job_id}
).scalar()
def _touch_job(job_id: int) -> None:
"""Bump a running job's `updated_at` without changing any value — a stall-watchdog heartbeat for
phases that legitimately write no progress. Extraction writes nothing, and a multi-client-set
fallback re-writes identical values (progress 0 / phase 'downloading'), so ORM `onupdate` would
NOT fire and two slow extraction attempts could otherwise cross the stall timeout and be killed
mid-extraction though they'd have succeeded."""
with SessionLocal() as db:
db.execute(
text("UPDATE download_jobs SET updated_at=now() WHERE id=:id AND status='running'"),
{"id": job_id},
)
db.commit()
def _requeue_if_running(job_id: int, phase: str) -> None:
"""Put the job back to 'queued' for a later retry, but ONLY while it is still 'running'. The
route flips status to paused/cancelled to stop a job; a blind `status='queued'` here would
clobber that and silently revive a job the user just cancelled. The conditional WHERE makes the
requeue a no-op unless we still own a running job."""
with SessionLocal() as db:
db.execute(
text(
"UPDATE download_jobs SET status='queued', phase=:phase, updated_at=now() "
"WHERE id=:id AND status='running'"
),
{"id": job_id, "phase": phase},
)
db.commit()
def _reset_asset_pending(asset_id: int) -> None:
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset and asset.status == "downloading":
asset.status = "pending"
db.commit()
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
+601
View File
@@ -0,0 +1,601 @@
"""Child-side download/edit pipeline — the yt-dlp download + ffmpeg edit that runs in the spawned subprocess."""
import logging
import re
import shutil
import time
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
import yt_dlp
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, formats, service, storage
from app.downloads.jobdb import (
_edit_staging_dir,
_job_status,
_reset_asset_pending,
_set_job,
_staging_dir,
_touch_job,
)
from app.downloads.status import JobStatus
from app.models import DownloadJob, MediaAsset
from app.notifications import create_notification
from app.titles import normalize_title
log = logging.getLogger("siftlode.worker")
class _Aborted(Exception):
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
# opens a session and SELECTs; `_set_job` opens one and UPDATE+COMMITs), so unthrottled they cost
# ~10^5 queries on a long job. ONE interval, ONE implementation, used by both paths — three
# hand-rolled copies is how the ffmpeg side ended up unthrottled while the download side was fine.
_DB_WRITE_INTERVAL_S = 1.0
_CANCEL_POLL_INTERVAL_S = 1.0
_THUMB_SUFFIXES = (".jpg", ".jpeg", ".png", ".webp")
# yt-dlp's working files in the staging dir. None of them is ever the finished media, but several
# CAN be the biggest file there (a video-only per-format stream, an abandoned fragment run), so the
# staging fallback must exclude them by name — picking one would store e.g. a silent video-only
# stream as a ready asset.
_YTDLP_TEMP_RE = re.compile(
r"\.part(-Frag\d+)?$" # in-flight download / fragment
r"|\.ytdl$" # resume state
r"|\.temp\.[^.]+$" # merge/postprocess temp
r"|\.f\d+\.[^.]+$", # per-format stream kept before the merge
re.IGNORECASE,
)
def _notify_download_done(
db, user_id: int | None, title: str | None, job_id, *, kind: str
) -> None:
"""Post a best-effort 'ready' notice to the owner's inbox — downloads/edits used to finish
silently while scheduler jobs and playlist-push already notified (R11 feedback symmetry). A
notification failure must never mask the completed job, so it is isolated + rolled back (mirrors
scheduler._notify_done). `kind` ("download"/"edit") lets the inbox pick accurate copy; the
English title here is only a fallback for an unknown type. `user_id`/`title` are captured by the
caller BEFORE its commit, so reading them here needs no post-commit refresh of an expired row."""
if user_id is None:
return
try:
create_notification(
db,
user_id=user_id,
type="download_done",
title=f"Ready to watch: {title}" if title else "Ready to watch",
data={"title": title, "kind": kind},
)
except Exception:
db.rollback()
log.exception("failed to write download-done notification for job %s", job_id)
def _parse_upload_date(raw) -> date | None:
if not raw:
return None
try:
return datetime.strptime(str(raw), "%Y%m%d").date()
except ValueError:
return None
def _fill_asset_meta_early(asset_id: int, info: dict) -> None:
"""Populate an asset's display metadata from yt-dlp's info_dict the first time it's seen
(only if not already filled at enqueue from our catalog) — so an ad-hoc URL's row shows a
real title + thumbnail while it downloads, not just the URL. Zero extra network cost."""
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset is None or asset.title:
return
# `or info.get("id")` avoids a blank card while downloading when the cleaned title is empty
# (e.g. a title that was only a social engagement prefix); the completion path later sets
# the final title with its own source_ref fallback.
asset.title = normalize_title(info.get("title")) or info.get("id")
asset.uploader = info.get("uploader") or info.get("channel")
asset.uploader_url = adapters.resolve_adapter(None, None, info).uploader_url(info)
asset.thumbnail_url = info.get("thumbnail")
asset.duration_s = int(info["duration"]) if info.get("duration") else None
asset.upload_date = _parse_upload_date(info.get("upload_date"))
asset.source_webpage_url = info.get("webpage_url")
db.commit()
def _make_progress_hook(job_id: int, asset_id: int):
state = {"meta_done": False}
cancelled = _cancel_poll(job_id) # same policy as the ffmpeg path
allow_write = _rate_limiter(_DB_WRITE_INTERVAL_S)
def hook(d: dict) -> None:
# Cooperative pause/cancel: if the job is no longer 'running', abort the download.
if cancelled():
raise _Aborted
if not state["meta_done"] and d.get("info_dict"):
state["meta_done"] = True
try:
_fill_asset_meta_early(asset_id, d["info_dict"])
except Exception: # noqa: BLE001 — metadata is best-effort, never fail a download
pass
if d.get("status") != "downloading":
return
if not allow_write():
return
# yt-dlp downloads the video and audio streams SEPARATELY (each 0→100%), then merges.
# Label the phase so the user understands the bar resetting between streams instead of
# seeing a mysterious 95%→5% jump.
info = d.get("info_dict") or {}
vcodec = info.get("vcodec") or "none"
acodec = info.get("acodec") or "none"
if vcodec != "none" and acodec == "none":
phase = "video"
elif acodec != "none" and vcodec == "none":
phase = "audio"
else:
phase = "media"
total = d.get("total_bytes") or d.get("total_bytes_estimate") or 0
done = d.get("downloaded_bytes") or 0
pct = int(done * 100 / total) if total else 0
_set_job(
job_id,
progress=min(pct, 99),
speed_bps=int(d["speed"]) if d.get("speed") else None,
eta_s=int(d["eta"]) if d.get("eta") else None,
phase=phase,
)
return hook
def _rate_limiter(interval_s: float, *, now=time.monotonic):
"""`should_run()` → True at most once per `interval_s` (the first call always passes).
`now` is injectable so tests drive a clock instead of patching the global one."""
# `None`, not 0.0: "never ran" has to be a VALUE, not a falsy timestamp. A truthiness test here
# reads a legitimate `now() == 0.0` as "never ran" and lets a second call straight through —
# unreachable with real `time.monotonic()`, but `now` exists to be injected, and a test clock
# starting at 0 is the obvious choice. That would let a throttle test encode the wrong
# behaviour and still pass.
state: dict[str, float | None] = {"at": None}
def should_run() -> bool:
t = now()
last = state["at"]
if last is not None and t - last < interval_s:
return False
state["at"] = t
return True
return should_run
def _cancel_poll(job_id: int, interval_s: float = _CANCEL_POLL_INTERVAL_S, *, now=time.monotonic):
"""Cooperative "has this job been paused/cancelled?" poll, rate-limited and sticky: once it
says yes it never queries again. Shared by the yt-dlp hook and run_ffmpeg so cancel latency is
one policy, not one per call site."""
allow = _rate_limiter(interval_s, now=now)
state = {"cancelled": False}
def cancelled() -> bool:
if not state["cancelled"] and allow():
state["cancelled"] = _job_status(job_id) not in ("running", None)
return state["cancelled"]
return cancelled
def _ffmpeg_progress_writer(job_id: int, phase: str, *, now=time.monotonic):
"""`on_progress` for run_ffmpeg: persist a progress row write at most once per interval and only
when the whole-percent value changed (the bar shows integers), plus a separate once-per-interval
heartbeat while the percent is stuck (so a slow re-encode isn't mistaken for a stall)."""
# Progress writes have their OWN 1/interval budget so a heartbeat can never delay a real write (the
# round-1 concern). The heartbeat, in turn, is gated on `last_touch` — the last DB write OR beat —
# so a progress write (which already refreshes updated_at) suppresses a redundant heartbeat that
# same interval, keeping active downloads at ~1 write/interval, not two.
allow_write = _rate_limiter(_DB_WRITE_INTERVAL_S, now=now)
state = {"pct": -1}
last_touch: dict = {"t": None} # monotonic time of the last DB write/heartbeat
def write(pct: float) -> None:
whole = int(pct)
if whole != state["pct"]:
if allow_write():
state["pct"] = whole
_set_job(job_id, progress=whole, phase=phase)
last_touch["t"] = now()
return
# Same integer percent, but ffmpeg IS still working — a slow re-encode (AV1/VP9 -> H.264 or a
# crop on the CPU-only host) can sit on one percent for minutes. Heartbeat updated_at so the
# stall watchdog doesn't kill a live job (B-sub-4) — but only if no write/beat already touched
# the row this interval, so a progressing download doesn't pay a redundant write.
t = now()
if last_touch["t"] is None or t - last_touch["t"] >= _DB_WRITE_INTERVAL_S:
_touch_job(job_id)
last_touch["t"] = t
return write
def _pp_phase(pp: str) -> str:
"""Map a yt-dlp postprocessor class name to a user-facing phase label. ffmpeg post-steps
aren't byte-progress operations (no %), so the UI shows the step name + an indeterminate
pulse — this makes the long merge/finalize legible instead of a silent 'Processing'."""
if "Merg" in pp:
return "merging"
if "ExtractAudio" in pp:
return "audio_extract"
if "Thumbnail" in pp:
return "thumbnail"
if "SponsorBlock" in pp or "ModifyChapters" in pp:
return "sponsorblock"
if "Metadata" in pp:
return "metadata"
return "processing"
def _make_pp_hook(job_id: int):
"""Postprocessor hook: surface the current ffmpeg step so the bar shows a concrete label
(Merging / Embedding thumbnail / …) instead of freezing at 100% after the streams finish."""
def hook(d: dict) -> None:
if d.get("status") in ("started", "processing"):
_set_job(job_id, phase=_pp_phase(d.get("postprocessor") or ""), speed_bps=None, eta_s=None)
return hook
def _find_thumbnail(staging: Path, media: Path) -> Path | None:
for p in staging.iterdir():
if p != media and p.suffix.lower() in _THUMB_SUFFIXES:
return p
return None
def _produced_file(info: dict, staging: Path) -> Path:
"""The media file yt-dlp actually wrote.
`requested_downloads[0].filepath` is the authoritative answer on YouTube, but an arbitrary-URL
extractor may not fill it in at all — and `Path("")` is `Path(".")`, which *passes* `.exists()`,
so a missing value used to sail past the fallback and blow up two steps later (`with_name` on an
empty name: the BBC-news failure). Anything that isn't a real file counts as absent; then fall
back to the largest file left in staging that is neither a thumbnail nor one of yt-dlp's own
working files (see `_YTDLP_TEMP_RE`)."""
req = (info.get("requested_downloads") or [{}])[0]
fp = req.get("filepath") or req.get("_filename") or info.get("filepath") or info.get("_filename")
if fp:
produced = Path(str(fp))
if produced.is_file():
return produced
cands = [
p
for p in staging.iterdir()
if p.is_file()
and p.suffix.lower() not in _THUMB_SUFFIXES
and not _YTDLP_TEMP_RE.search(p.name)
]
if not cands:
raise RuntimeError("yt-dlp produced no output file")
return max(cands, key=lambda p: p.stat().st_size)
def _media_info(info: dict) -> dict:
"""Unwrap a playlist/multi_video extraction to the entry that carries the download.
A single non-YouTube page (a news article's video, a multi-rendition embed) can extract as a
wrapper whose own dict has no `requested_downloads` — the media lives on an entry. Pick the
FIRST entry that actually downloaded something: on a page with a skipped teaser in front of the
real video, entries[0] carries no file, and taking it blindly would label the produced file
with the wrong id/title/duration."""
entries = [e for e in (info.get("entries") or []) if e]
if not entries or info.get("requested_downloads"):
return info
chosen = next((e for e in entries if e.get("requested_downloads")), entries[0])
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
for i, clients in enumerate(client_sets):
opts = formats.build_ydl_opts(
spec, outtmpl, _make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
# multi-client-set fallback would otherwise cross the stall timeout with a stale updated_at.
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
try:
with yt_dlp.YoutubeDL(opts) as ydl:
return ydl.extract_info(url, download=True)
except yt_dlp.utils.DownloadError as exc:
last_exc = exc
if i < len(client_sets) - 1 and adapter.retriable(str(exc)):
log.info("job %s: clients %s failed (%s); retrying with fallback", job_id, clients, str(exc)[:70])
for p in list(staging.iterdir()): # drop partials before the retry
try:
p.unlink()
except OSError:
pass
_set_job(job_id, progress=0, phase="downloading")
continue
raise
raise last_exc # unreachable (loop either returns or raises)
def _download_settings() -> tuple[str, int]:
"""Layout + retention days from the DB config (admin-overridable on the Configuration page),
falling back to the env defaults. Read here, not from `settings`, so an admin edit takes effect."""
with SessionLocal() as db:
return (
sysconfig.get_str(db, "download_layout"),
sysconfig.get_int(db, "download_retention_days"),
)
def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict) -> Path:
"""Make an mp4-compat-profile download play in every browser: H.264 + AAC/MP3 in mp4. If the
download (whatever the source shipped) isn't already that, re-encode the offending stream(s) —
or just remux when the codecs are fine but the container isn't. **Scope:** only the mp4 compat
profile is normalized; an explicit `vcodec` or a non-mp4 custom profile is the user opting out
of the guarantee and is left as-is (so a `webm` profile can still ship VP9/Opus). Returns the
file to store (converted, or the original) and corrects `info`'s codec fields to match disk."""
if spec.get("container") != "mp4" or spec.get("vcodec"):
return produced
vcodec, acodec = editmod.probe_codecs(produced)
is_mp4 = produced.suffix.lower() == ".mp4"
needs, reenc_v, reenc_a = editmod.browser_transcode_flags(vcodec, acodec, is_mp4)
if not needs:
return produced
dest = produced.with_name(produced.stem + ".browser.mp4")
action = "re-encode" if (reenc_v or reenc_a) else "remux"
log.info("job %s: %s (v=%s a=%s mp4=%s) -> browser-safe mp4", job_id, action, vcodec, acodec, is_mp4)
_set_job(job_id, phase="processing", progress=0, speed_bps=None, eta_s=None)
# Prefer the info duration; probe the file when the extractor didn't provide one, so a long
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(produced, dest, reenc_v, reenc_a),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
should_cancel=_cancel_poll(job_id),
)
produced.unlink(missing_ok=True)
# Keep the stored asset metadata truthful about what's now on disk.
if reenc_v:
info["vcodec"] = "h264"
if reenc_a:
info["acodec"] = "aac"
return dest
def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict) -> None:
staging = _staging_dir(asset_id)
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
info = _media_info(_extract_with_fallback(job_id, asset_id, url, spec, staging, adapter))
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
produced = _produced_file(info, staging)
produced = _ensure_browser_playable(job_id, produced, spec, info)
ext = produced.suffix.lstrip(".").lower()
meta = storage.MediaMeta(
video_id=info.get("id") or source_ref,
title=normalize_title(info.get("title")) or source_ref,
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
upload_date=_parse_upload_date(info.get("upload_date")),
uniqueid_type=adapter.nfo_uniqueid_type(),
duration_s=int(info["duration"]) if info.get("duration") else None,
description=info.get("description"),
thumbnail_url=info.get("thumbnail"),
)
layout, retention_days = _download_settings()
rel = storage.rel_path(meta, ext, layout, asset_id=asset_id)
thumb = _find_thumbnail(staging, produced)
# Convert the raw thumbnail (webp/png/jpeg) to a real JPEG ourselves — robustly — for the
# poster + og:image sidecar, now that yt-dlp's fatal thumbnail convertor is gone. On failure
# this is None and write_sidecars writes no poster, so ensure_poster cuts a frame instead.
poster_jpg = editmod.thumbnail_to_jpg(thumb, staging / "poster.jpg")
storage.place_file(produced, settings.download_root, rel)
nfo_ok = storage.write_sidecars(settings.download_root, rel, meta, poster_jpg)
final = storage.abs_path(settings.download_root, rel)
size = final.stat().st_size if final.exists() else None
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = size
asset.duration_s = meta.duration_s
asset.width = int(info["width"]) if info.get("width") else None
asset.height = int(info["height"]) if info.get("height") else None
asset.container = ext
asset.vcodec = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
asset.acodec = info.get("acodec") if info.get("acodec") not in (None, "none") else None
asset.title = meta.title
asset.uploader = meta.uploader
asset.uploader_url = adapter.uploader_url(info)
asset.upload_date = meta.upload_date
asset.thumbnail_url = meta.thumbnail_url
# Record a self-hosted poster for every download: it's the reliable link-preview
# (og:image) source — a remote thumbnail, esp. Facebook's signed CDN URL, expires
# and can't be relied on cross-origin. write_sidecars already wrote <base>.jpg from
# the thumbnail; ensure_poster returns that, or cuts a frame if there was none.
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.source_webpage_url = info.get("webpage_url")
asset.nfo_written = nfo_ok
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture the owner before commit expires the row (avoids a refresh SELECT just to notify).
owner_id = job.user_id if job else None
fields = {"status": JobStatus.DONE, "progress": 100, "phase": None, "speed_bps": None,
"eta_s": None}
if job and not job.display_name and meta.title:
# display_name is VARCHAR(255) — clip a long title (see _fill_asset_meta_early).
fields["display_name"] = meta.title[:255]
# Atomic terminal write: finalizes ONLY while the row is still running, so a child that
# finishes just after a route cancel/pause can't resurrect it (see finalize_if_running).
finalized = service.finalize_if_running(db, job_id, **fields)
db.commit()
if finalized:
_notify_download_done(db, owner_id, meta.title, job_id, kind="download")
log.info("job %s done: %s", job_id, rel)
except (_Aborted, editmod.EditAborted):
# Paused/canceled mid-download or mid-transcode: free the asset for a later retry; keep the
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
_reset_asset_pending(asset_id)
log.info("job %s aborted (pause/cancel)", job_id)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "error"
asset.error = msg
# Atomic fail: only while still running, so a route-set cancel/pause is never overwritten
# with `error` (error also holds the asset → would re-open the double-release class).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
)
db.commit()
log.warning("job %s failed: %s", job_id, msg)
finally:
shutil.rmtree(staging, ignore_errors=True)
def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "error"
asset.error = msg
# Atomic fail: only while still running (see the download error path).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
)
db.commit()
log.warning("edit job %s failed: %s", job_id, msg)
def _process_edit(job_id: int, asset_id: int) -> None:
"""Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
src = db.get(MediaAsset, job.source_asset_id) if job and job.source_asset_id else None
edit_spec = dict(job.edit_spec or {}) if job else {}
user_id = job.user_id if job else None
src_ready = src is not None and src.status == "ready" and bool(src.rel_path)
src_rel = src.rel_path if src else None
src_w, src_h, src_dur = (src.width, src.height, src.duration_s) if src else (None, None, None)
if not src_ready or not src_rel:
_fail_edit(job_id, asset_id, "The source download is no longer available.")
return
src_path = storage.abs_path(settings.download_root, src_rel)
if not src_path.exists():
_fail_edit(job_id, asset_id, "The source file is missing.")
return
src_ext = src_path.suffix.lstrip(".").lower() or "mp4"
crop = editmod.has_crop(edit_spec)
out_ext = "mp4" if editmod.needs_reencode(edit_spec) else src_ext
total_s = editmod.clip_duration(edit_spec, src_dur) or src_dur or 0
staging = _edit_staging_dir(asset_id)
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
dest_staging = staging / f"out.{out_ext}"
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
if edit_spec.get("segments"):
cmd, prep = editmod.build_concat_plan(src_path, dest_staging, edit_spec, out_ext, staging)
for p, content in prep.items():
p.write_text(content, encoding="utf-8")
else:
cmd = editmod.build_edit_cmd(src_path, dest_staging, edit_spec, out_ext)
editmod.run_ffmpeg(
cmd,
total_s,
on_progress=_ffmpeg_progress_writer(job_id, "editing"),
should_cancel=_cancel_poll(job_id),
)
if not dest_staging.exists():
raise RuntimeError("ffmpeg produced no output file")
rel = storage.edit_rel_path(user_id, asset_id, out_ext)
storage.place_file(dest_staging, settings.download_root, rel)
final = storage.abs_path(settings.download_root, rel)
size = final.stat().st_size if final.exists() else None
_, retention_days = _download_settings()
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = size
asset.container = out_ext
asset.duration_s = total_s or asset.duration_s
if crop:
asset.width = edit_spec["crop"]["w"]
asset.height = edit_spec["crop"]["h"]
else:
asset.width, asset.height = src_w, src_h
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture owner + title before commit expires the rows (avoids a refresh SELECT).
owner_id = job.user_id if job else None
dl_title = asset.title if asset else None
# Atomic terminal write (see the download path): don't resurrect a route-stopped job.
finalized = service.finalize_if_running(
db, job_id, status=JobStatus.DONE, progress=100, phase=None
)
db.commit()
if finalized:
_notify_download_done(db, owner_id, dl_title, job_id, kind="edit")
log.info("edit job %s done: %s", job_id, rel)
except editmod.EditAborted:
# Paused/canceled mid-encode: free the asset for a later retry (ref_count is the route's job).
_reset_asset_pending(asset_id)
log.info("edit job %s aborted (pause/cancel)", job_id)
except Exception as exc: # noqa: BLE001 — surface the failure, keep the worker alive
_fail_edit(job_id, asset_id, str(exc)[:500])
finally:
shutil.rmtree(staging, ignore_errors=True)
+2 -2
View File
@@ -38,8 +38,8 @@ def main() -> None:
return return
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has # Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
# real work, and to keep the import graph obvious: runner is a thin entry over worker's helpers. # real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
from app.worker import _download, _process_edit from app.downloads.pipeline import _download, _process_edit
if job_kind == "edit": if job_kind == "edit":
_process_edit(job_id, asset_id) _process_edit(job_id, asset_id)
+107 -4
View File
@@ -112,6 +112,34 @@ def next_queue_pos(db: Session) -> int:
return (db.execute(select(func.coalesce(func.max(DownloadJob.queue_pos), 0))).scalar() or 0) + 1 return (db.execute(select(func.coalesce(func.max(DownloadJob.queue_pos), 0))).scalar() or 0) + 1
def incr_ref_count(db: Session, asset_id: int) -> None:
"""Atomically +1 an asset's hold count AT THE DB. A Python read-modify-write
(`asset.ref_count = asset.ref_count + 1`) loses an update when two enqueues of the same
(source,format) run in parallel — both read the same value and write the same +1 — leaving the
count one short, so the first release can free a file another job still holds (B-sub-5)."""
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id)
.values(ref_count=MediaAsset.ref_count + 1)
.execution_options(synchronize_session=False)
)
def decr_ref_count(db: Session, asset_id: int) -> int:
"""Atomically -1 an asset's hold count (clamped at 0) and RETURN the new value. The caller decides
whether to free the asset from the returned count while still holding the row's write lock, so two
concurrent releases can't both observe the same pre-decrement value and double-free — Postgres
serialises the two UPDATEs on the row lock (B-sub-5). synchronize_session=False: the ORM copy is
intentionally left stale; callers read the returned value, not `asset.ref_count`."""
return db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id)
.values(ref_count=func.greatest(MediaAsset.ref_count - 1, 0))
.returning(MediaAsset.ref_count)
.execution_options(synchronize_session=False)
).scalar_one()
def enqueue( def enqueue(
db: Session, db: Session,
user_id: int, user_id: int,
@@ -143,7 +171,7 @@ def enqueue(
queue_pos=next_queue_pos(db), queue_pos=next_queue_pos(db),
) )
db.add(job) db.add(job)
asset.ref_count = (asset.ref_count or 0) + 1 incr_ref_count(db, asset.id)
# Cache hit: the file already exists — no worker round-trip needed. # Cache hit: the file already exists — no worker round-trip needed.
if asset.status == "ready": if asset.status == "ready":
@@ -212,7 +240,7 @@ def enqueue_edit(
queue_pos=next_queue_pos(db), queue_pos=next_queue_pos(db),
) )
db.add(job) db.add(job)
asset.ref_count = (asset.ref_count or 0) + 1 incr_ref_count(db, asset.id)
if asset.status == "ready": # identical clip already produced for this user → instant if asset.status == "ready": # identical clip already produced for this user → instant
job.status = "done" job.status = "done"
@@ -273,8 +301,83 @@ def release_asset(db: Session, job: DownloadJob) -> None:
asset = db.get(MediaAsset, job.asset_id) asset = db.get(MediaAsset, job.asset_id)
if asset is None: if asset is None:
return return
asset.ref_count = max(0, (asset.ref_count or 0) - 1) new_count = decr_ref_count(db, asset.id)
if asset.ref_count == 0 and asset.status == "ready": if new_count == 0 and asset.status == "ready":
if asset.rel_path: if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path) storage.delete_asset_files(settings.download_root, asset.rel_path)
db.delete(asset) db.delete(asset)
# --- Job state transitions ---------------------------------------------------------------------
# The single home for the pause/resume/cancel/delete/redownload transitions + their asset
# bookkeeping (release, failed-asset reset, fresh-asset rebuild), so the routes are thin callers and
# the worker/GC could share the same definitions (B-sub-8). Each commits its own transition; the
# route keeps only auth + HTTP mapping + serialization.
def pause(db: Session, job: DownloadJob) -> None:
"""Pause a queued/running job (a running worker aborts cooperatively via the cancel poll); a no-op
in any other state."""
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING):
job.status = JobStatus.PAUSED
db.commit()
def resume(db: Session, job: DownloadJob) -> None:
"""Requeue a paused/errored job for a fresh attempt, clearing its error. If the SHARED asset itself
failed, reset it to pending so the worker actually re-downloads — otherwise the requeued job would
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset)."""
if job.status in (JobStatus.PAUSED, JobStatus.ERROR):
job.progress = 0
job.error = None
job.error_code = None
if job.asset_id:
asset = db.get(MediaAsset, job.asset_id)
if asset and asset.status == "error": # MediaAsset.status (no enum), not a JobStatus
asset.status = "pending"
asset.error = None
job.status = JobStatus.QUEUED
db.commit()
def cancel(db: Session, job: DownloadJob) -> None:
"""Cancel a job unless it's already done/canceled, releasing its asset hold WHILE it still counts
as holding — `canceled` ∉ HOLDING, so a later delete won't double-release."""
if job.status not in (JobStatus.DONE, JobStatus.CANCELED):
release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
def delete(db: Session, job: DownloadJob) -> None:
"""Delete a job, releasing its asset hold first (which frees the file iff this was the last holder)."""
release_asset(db, job)
db.delete(job)
db.commit()
def redownload(db: Session, job: DownloadJob) -> None:
"""Rebuild a finished/failed job's file into a FRESH, private asset (the cache sig is salted with
the job id + current asset id, so it never reuses the dedup pool) and requeue the SAME job row —
so any public share link (DownloadLink -> job_id) keeps resolving to the newly-downloaded file.
The old hold is dropped first (freeing the old file iff this job was its last holder). The caller
has already rejected edit clips + still-in-progress jobs (HTTP 409)."""
spec = formats.normalize(job.profile_snapshot)
sig = formats.format_sig(spec, salt=f"{job.id}:{job.asset_id}")
asset = get_or_create_asset(db, job.source_kind, job.source_ref, sig)
populate_from_catalog(db, asset)
# Drop the old hold BEFORE re-pointing the job — release_asset keys off the job's current asset.
release_asset(db, job)
incr_ref_count(db, asset.id)
job.asset_id = asset.id
job.status = JobStatus.QUEUED
job.progress = 0
job.phase = None
job.error = None
job.speed_bps = None
job.eta_s = None
# Re-derive the display name from the freshly cleaned title on completion (else an older, un-cleaned
# name would survive the rebuild).
job.display_name = None
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
db.commit()
+146 -34
View File
@@ -100,12 +100,24 @@ def serialize(row) -> dict:
} }
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
"""Group tag ids by their category (language/topic/other). The single source for the category split
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
two can't disagree on which selected tag belongs to which category."""
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
for tid, cat in db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids))).all():
out.setdefault(cat, []).append(tid)
return out
def filtered_query( def filtered_query(
db: Session, db: Session,
user: User, user: User,
*, *,
tags: list[int], tags: list[int],
tag_mode: str, tag_modes: dict[str, str],
channel_ids: list[str], channel_ids: list[str],
q: str | None, q: str | None,
min_duration: int | None, min_duration: int | None,
@@ -131,7 +143,6 @@ def filtered_query(
either mode via the VideoState outer join.""" either mode via the VideoState outer join."""
state = aliased(VideoState) state = aliased(VideoState)
status_expr = func.coalesce(state.status, "new") status_expr = func.coalesce(state.status, "new")
position_expr = func.coalesce(state.position_seconds, 0)
query = select(*feed_columns(user, state)).join( query = select(*feed_columns(user, state)).join(
Channel, Channel.id == Video.channel_id Channel, Channel.id == Video.channel_id
@@ -163,7 +174,14 @@ def filtered_query(
and_(SearchFind.video_id == Video.id, SearchFind.user_id == user.id), and_(SearchFind.video_id == Video.id, SearchFind.user_id == user.id),
) )
query = query.outerjoin( # The state-gated views (watched/hidden/in_progress) only ever return videos that HAVE a
# video_states row, so an INNER join lets the planner drive from the user's own small video_states
# set (~100 rows) instead of scanning all videos and outer-joining (PERF-16). status is NOT NULL on
# a present row (server_default 'new'), so the raw-column filters below match the coalesce form
# exactly. 'unwatched'/'all' include row-less 'new' videos, so they keep the outer join.
state_required = show in ("watched", "hidden", "in_progress")
join_state = query.join if state_required else query.outerjoin
query = join_state(
state, and_(state.video_id == Video.id, state.user_id == user.id) state, and_(state.video_id == Video.id, state.user_id == user.id)
) )
@@ -246,15 +264,25 @@ def filtered_query(
# channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort. # channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort.
rank_expr = None rank_expr = None
if q: if q:
# Resolve the channels whose NAME matches up front (there are only a few hundred channels, so
# this is a tiny standalone scan) and match videos by channel_id. ORing an unindexed
# `unaccent(channels.title) ILIKE '%q%'` straight into the per-video predicate defeated the
# search_vector GIN — the planner detoasted every non-matching tsvector to test the `@@`
# (~2.6s/search on the prod-sized set); channel_id.in_(...) keeps the GIN as a BitmapOr branch
# (PERF-12). Empty match -> false() so the OR collapses to the text match alone.
like = func.unaccent(f"%{q}%")
matching_channel_ids = (
db.execute(select(Channel.id).where(func.unaccent(Channel.title).ilike(like)))
.scalars()
.all()
)
channel_match = (
Video.channel_id.in_(matching_channel_ids) if matching_channel_ids else false()
)
ts_str = to_tsquery_str(q) ts_str = to_tsquery_str(q)
if ts_str: if ts_str:
tsq = func.to_tsquery("public.unaccent_simple", ts_str) tsq = func.to_tsquery("public.unaccent_simple", ts_str)
query = query.where( query = query.where(or_(Video.search_vector.op("@@")(tsq), channel_match))
or_(
Video.search_vector.op("@@")(tsq),
func.unaccent(Channel.title).ilike(func.unaccent(f"%{q}%")),
)
)
# Scale ts_rank (a float4) to an integer score so the keyset cursor compares it # Scale ts_rank (a float4) to an integer score so the keyset cursor compares it
# EXACTLY — a raw float4 vs the float8 cursor value mismatches on round-trip and # EXACTLY — a raw float4 vs the float8 cursor value mismatches on round-trip and
# breaks paging (the same page repeats). 1e6 granularity is ample; ties break on # breaks paging (the same page repeats). 1e6 granularity is ample; ties break on
@@ -262,21 +290,15 @@ def filtered_query(
rank_expr = cast(func.ts_rank(Video.search_vector, tsq) * 1000000, BigInteger) rank_expr = cast(func.ts_rank(Video.search_vector, tsq) * 1000000, BigInteger)
else: else:
# No usable tokens (e.g. only punctuation): fall back to a plain substring match. # No usable tokens (e.g. only punctuation): fall back to a plain substring match.
like = func.unaccent(f"%{q}%") query = query.where(or_(func.unaccent(Video.title).ilike(like), channel_match))
query = query.where(
or_(
func.unaccent(Video.title).ilike(like),
func.unaccent(Channel.title).ilike(like),
)
)
if tags: if tags:
# AND across tag categories (e.g. language AND topic narrows), OR within a # AND across tag categories (e.g. language AND topic narrows), OR within a category — unless that
# category; the any/all toggle controls multiple topic tags. # category's mode is "and" (its per-category any/all toggle), which requires a channel to carry
cat_rows = db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tags))).all() # ALL its selected tags. ⚠️ feed_summary.passes() reimplements this exact rule in Python for the
by_category: dict[str, list[int]] = {} # sidebar count/facets — keep the two in lockstep (equivalence-tested by
for tag_id, category in cat_rows: # test_feed_summary_matches_count_plus_facets).
by_category.setdefault(category, []).append(tag_id) by_category = tags_by_category(db, tags)
# Facet counting drops the category being counted so its own chips don't zero each # Facet counting drops the category being counted so its own chips don't zero each
# other out (standard drill-down faceting: a category's count ignores its own # other out (standard drill-down faceting: a category's count ignores its own
# selections but still honours the other categories' filters). # selections but still honours the other categories' filters).
@@ -284,7 +306,7 @@ def filtered_query(
by_category.pop(exclude_tag_category, None) by_category.pop(exclude_tag_category, None)
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id) visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
for category, ids in by_category.items(): for category, ids in by_category.items():
if category == "topic" and tag_mode == "and" and len(ids) > 1: if tag_modes.get(category, "or") == "and" and len(ids) > 1:
sub = ( sub = (
select(ChannelTag.channel_id) select(ChannelTag.channel_id)
.where(ChannelTag.tag_id.in_(ids), visible) .where(ChannelTag.tag_id.in_(ids), visible)
@@ -315,20 +337,96 @@ def filtered_query(
if show == "unwatched": if show == "unwatched":
query = query.where(status_expr.notin_(("watched", "hidden"))) query = query.where(status_expr.notin_(("watched", "hidden")))
elif show == "in_progress": elif show == "in_progress":
# Started but not finished: a stored resume position, not yet watched/hidden. # Started but not finished: a stored resume position, not yet watched/hidden. INNER-joined
# (state_required) so the raw state columns are safe and the planner can use video_states.
query = query.where( query = query.where(
and_(position_expr > 0, status_expr.notin_(("watched", "hidden"))) and_(state.position_seconds > 0, state.status.notin_(("watched", "hidden")))
) )
elif show == "watched": elif show == "watched":
query = query.where(status_expr == "watched") query = query.where(state.status == "watched")
elif show == "hidden": elif show == "hidden":
query = query.where(status_expr == "hidden") query = query.where(state.status == "hidden")
else: # all else: # all
query = query.where(status_expr != "hidden") query = query.where(status_expr != "hidden")
return query, rank_expr return query, rank_expr
def feed_summary(db: Session, user: User, **params) -> dict:
"""Single-pass replacement for /feed/count + 3x /facets (PERF-13): the sidebar's total video count
and every tag's distinct-channel facet count, derived in Python from ONE grouped query instead of
four full passes over the subscribed set on every filter change.
The one query is the base filter with the TAG predicate removed, grouped by channel: it yields every
channel that has a matching video, with its video count. From that ~few-hundred-row map we compute:
- count = sum of the video counts of the channels that pass the SELECTED tag filter;
- facets = per tag, the number of distinct channels that carry it AND pass the drill-down filter
(the tag's own category dropped, unless it's a conjunctive topic-AND — mirroring the
old /facets exclude_tag_category logic exactly).
Tag matching is channel-based, identical to filtered_query's tag block (AND across categories, OR
within, a category in "and" mode requires ALL its selected tags)."""
tag_modes = params.get("tag_modes", {})
# ONE pass: per-channel video counts for the base filter, tags stripped so every candidate channel
# is present to facet over.
base, _rank = filtered_query(db, user, **{**params, "tags": []})
ch_col = base.with_only_columns(Video.channel_id).subquery()
channel_counts: dict[str, int] = dict(
db.execute(select(ch_col.c.channel_id, func.count()).group_by(ch_col.c.channel_id)).all()
)
if not channel_counts:
return {"count": 0, "facets": {}}
# channel -> its visible tag ids, plus each tag's category (for grouping the facet output).
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
channel_tags: dict[str, set[int]] = {}
tag_category: dict[int, str] = {}
for cid, tid, cat in db.execute(
select(ChannelTag.channel_id, ChannelTag.tag_id, Tag.category)
.join(Tag, Tag.id == ChannelTag.tag_id)
.where(visible, ChannelTag.channel_id.in_(channel_counts.keys()))
).all():
channel_tags.setdefault(cid, set()).add(tid)
tag_category[tid] = cat
# Selected tags grouped by category (read from Tag via the shared helper, so a selected tag that no
# base channel carries still filters everything out for its category — matching the SQL `IN (empty)`
# behaviour).
selected_by_cat: dict[str, set[int]] = {
cat: set(ids) for cat, ids in tags_by_category(db, params.get("tags") or []).items()
}
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
# The Python mirror of filtered_query's SQL tag block (AND across categories, OR within,
# topic-AND requires all selected) — keep in lockstep; see the ⚠️ note there.
for cat, sel in selected_by_cat.items():
if cat == exclude_cat:
continue
if tag_modes.get(cat, "or") == "and" and len(sel) > 1:
if not sel <= ch_tags: # conjunctive: channel must carry ALL this category's selected tags
return False
elif not (sel & ch_tags): # disjunctive: channel must carry ANY selected tag
return False
return True
count = sum(
n for cid, n in channel_counts.items() if passes(channel_tags.get(cid, set()), None)
)
facets: dict[int, int] = {}
for category in ("language", "topic", "other"):
# Drop this category's own selections so its chips keep independent counts; a conjunctive
# topic-AND keeps them (each remaining chip narrows to channels that ALSO have the selected).
exclude = None if tag_modes.get(category, "or") == "and" else category
for cid, ch_tags in channel_tags.items():
if not passes(ch_tags, exclude):
continue
for tid in ch_tags:
if tag_category.get(tid) == category:
facets[tid] = facets.get(tid, 0) + 1
return {"count": count, "facets": facets}
# --- Keyset (cursor) pagination --------------------------------------------- # --- Keyset (cursor) pagination ---------------------------------------------
# #
# Each sort is described as an ordered list of key columns (expression + direction # Each sort is described as an ordered list of key columns (expression + direction
@@ -339,7 +437,7 @@ def filtered_query(
# and the page boundary is stable even as the scheduler ingests new videos mid-scroll # and the page boundary is stable even as the scheduler ingests new videos mid-scroll
# (OFFSET would shift and duplicate/skip rows). # (OFFSET would shift and duplicate/skip rows).
def sort_keys(sort: str, seed: int, rank_expr=None) -> list[dict]: def sort_keys(sort: str, seed: int, rank_expr=None) -> list[dict]:
pub = {"expr": Video.published_at, "asc": False, "nullable": True, "kind": "dt"} pub = {"expr": Video.published_at, "asc": False, "nullable": False, "kind": "dt"}
table: dict[str, list[dict]] = { table: dict[str, list[dict]] = {
"newest": [pub], "newest": [pub],
"oldest": [{**pub, "asc": True}], "oldest": [{**pub, "asc": True}],
@@ -383,7 +481,15 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
Lexicographic over the key columns then id, NULL-aware for `nulls_last`: a NULL Lexicographic over the key columns then id, NULL-aware for `nulls_last`: a NULL
sorts after every non-NULL value, so a cursor sitting on a non-NULL value must sorts after every non-NULL value, so a cursor sitting on a non-NULL value must
also pull in the NULL group, and equality treats NULL == NULL.""" also pull in the NULL group, and equality treats NULL == NULL.
Built recursively so that for a NON-nullable leading key we can prepend a non-strict
range bound (`col <= v` for desc / `col >= v` for asc) as an AND conjunct — logically
a no-op over the OR of strict-after branches, but it lets the planner bound the index
scan instead of scanning from the top and filtering every earlier row (PERF-15: deep
`newest` pages went 51.8k -> ~0.5k buffers). A NULLABLE key can't take that bound (the
NULL group sorts after every value and `col <= v` would drop it), so it keeps the pure
NULL-aware OR form."""
def eq(k): def eq(k):
return k["expr"].is_(None) if k["value"] is None else k["expr"] == k["value"] return k["expr"].is_(None) if k["value"] is None else k["expr"] == k["value"]
@@ -393,11 +499,17 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
cmp = k["expr"] > k["value"] if k["asc"] else k["expr"] < k["value"] cmp = k["expr"] > k["value"] if k["asc"] else k["expr"] < k["value"]
return or_(cmp, k["expr"].is_(None)) if k["nullable"] else cmp return or_(cmp, k["expr"].is_(None)) if k["nullable"] else cmp
branches = [] def rec(i: int):
for j in range(len(keys)): if i == len(keys):
branches.append(and_(*[eq(keys[i]) for i in range(j)], after(keys[j]))) return Video.id > cursor_id # all keys tied -> break by the unique id
branches.append(and_(*[eq(k) for k in keys], Video.id > cursor_id)) k = keys[i]
return or_(*branches) tail = or_(after(k), and_(eq(k), rec(i + 1)))
if not k["nullable"] and k["value"] is not None:
bound = k["expr"] >= k["value"] if k["asc"] else k["expr"] <= k["value"]
return and_(bound, tail)
return tail
return rec(0)
def encode_cursor(keys: list[dict], row) -> str: def encode_cursor(keys: list[dict], row) -> str:
+81 -17
View File
@@ -251,14 +251,23 @@ class Video(Base, TimestampMixin):
__tablename__ = "videos" __tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_desc`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0067 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) AND the per-channel feed
# sort. Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column( channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True ForeignKey("channels.id", ondelete="CASCADE")
) )
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles) title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
description: Mapped[str | None] = mapped_column(Text) description: Mapped[str | None] = mapped_column(Text)
published_at: Mapped[datetime | None] = mapped_column( # NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
DateTime(timezone=True), index=True # forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
published_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
) )
thumbnail_url: Mapped[str | None] = mapped_column(String(1024)) thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
duration_seconds: Mapped[int | None] = mapped_column(Integer) duration_seconds: Mapped[int | None] = mapped_column(Integer)
@@ -276,18 +285,22 @@ class Video(Base, TimestampMixin):
# folds YouTube's own relevance judgement into the local search: a video YouTube returned # folds YouTube's own relevance judgement into the local search: a video YouTube returned
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it. # for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
search_terms: Mapped[str | None] = mapped_column(Text) search_terms: Mapped[str | None] = mapped_column(Text)
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
# boolean btrees and replaced the useful ones with partial indexes over the target subset
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
# search_vector GIN.
is_short: Mapped[bool] = mapped_column( is_short: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True Boolean, default=False, server_default="false"
) )
# Whether the youtube.com/shorts/<id> probe has run for this video. # Whether the youtube.com/shorts/<id> probe has run for this video.
shorts_probed: Mapped[bool] = mapped_column( shorts_probed: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True Boolean, default=False, server_default="false"
) )
# True when this video first entered the catalog via a live YouTube search (not an # True when this video first entered the catalog via a live YouTube search (not an
# organic channel/playlist sync). Used to hide search-discovered "noise" from the # organic channel/playlist sync). Used to hide search-discovered "noise" from the
# Library by default; left False by the organic ingest paths. # Library by default; left False by the organic ingest paths.
via_search: Mapped[bool] = mapped_column( via_search: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True Boolean, default=False, server_default="false"
) )
# True when this video entered the catalog because a user EXPLORED its (un-subscribed) # True when this video entered the catalog because a user EXPLORED its (un-subscribed)
# channel. Like via_search it's hidden from the shared Library by default, but it's # channel. Like via_search it's hidden from the shared Library by default, but it's
@@ -295,7 +308,7 @@ class Video(Base, TimestampMixin):
# explored or subscribe to its channel (a gate in the feed query), so one user's # explored or subscribe to its channel (a gate in the feed query), so one user's
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths. # curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
via_explore: Mapped[bool] = mapped_column( via_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True Boolean, default=False, server_default="false"
) )
# none | live | upcoming | premiere | was_live # none | live | upcoming | premiere | was_live
live_status: Mapped[str] = mapped_column( live_status: Mapped[str] = mapped_column(
@@ -311,11 +324,14 @@ class Video(Base, TimestampMixin):
# (least-recently-checked first); unavailable_since marks a video as currently # (least-recently-checked first); unavailable_since marks a video as currently
# unplayable (hidden from the feed immediately, hard-deleted after a grace period); # unplayable (hidden from the feed immediately, hard-deleted after a grace period);
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live). # unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
last_checked_at: Mapped[datetime | None] = mapped_column( last_checked_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True DateTime(timezone=True)
) )
unavailable_since: Mapped[datetime | None] = mapped_column( unavailable_since: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True DateTime(timezone=True)
) )
unavailable_reason: Mapped[str | None] = mapped_column(String(24)) unavailable_reason: Mapped[str | None] = mapped_column(String(24))
@@ -531,9 +547,6 @@ class AppState(Base):
sync_paused: Mapped[bool] = mapped_column( sync_paused: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false" Boolean, default=False, server_default="false"
) )
# Admin override for the maintenance job's rolling re-validation batch (videos re-checked
# per run). NULL = use the config/env default (settings.maintenance_revalidate_batch).
maintenance_revalidate_batch: Mapped[int | None] = mapped_column(Integer)
# First-run install wizard. `configured` is False on a fresh instance (the app runs in # First-run install wizard. `configured` is False on a fresh instance (the app runs in
# setup mode: only the wizard works) and flipped True when the wizard finishes. While # setup mode: only the wizard works) and flipped True when the wizard finishes. While
# unconfigured, a one-time setup token (only its SHA-256 hash stored) gates the wizard; # unconfigured, a one-time setup token (only its SHA-256 hash stored) gates the wizard;
@@ -633,19 +646,23 @@ class PlaylistItem(Base):
class SavedView(Base, TimestampMixin, UpdatedAtMixin): class SavedView(Base, TimestampMixin, UpdatedAtMixin):
"""A per-user named snapshot of the feed's filter/sort state (a "smart view"). """A per-user named snapshot of a module's filter/sort state (a "smart view").
`filters` is the serialized FeedFilters JSON the frontend applies with one click. `source` names the module the view belongs to ("feed" | "plex"); `filters` is that module's
`position` orders them in the sidebar (drag-reorder); `is_default` marks the one view serialized filter JSON the frontend applies with one click. `position` orders them in the sidebar
applied automatically on load — at most one per user, enforced by the API on set.""" (drag-reorder, per source); `is_default` marks the one view applied automatically on load — at most
one per (user, source), enforced by the API on set. The name is unique per (user, source)."""
__tablename__ = "saved_views" __tablename__ = "saved_views"
__table_args__ = (UniqueConstraint("user_id", "name", name="uq_user_saved_view"),) __table_args__ = (
UniqueConstraint("user_id", "source", "name", name="uq_user_source_saved_view"),
)
id: Mapped[int] = mapped_column(primary_key=True) id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column( user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE") ForeignKey("users.id", ondelete="CASCADE")
) )
source: Mapped[str] = mapped_column(String(16), default="feed", server_default="feed")
name: Mapped[str] = mapped_column(String(80)) name: Mapped[str] = mapped_column(String(80))
filters: Mapped[dict] = mapped_column(JSON) filters: Mapped[dict] = mapped_column(JSON)
position: Mapped[int] = mapped_column(Integer, default=0, server_default="0") position: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
@@ -1054,6 +1071,20 @@ class PlexShow(Base, TimestampMixin, UpdatedAtMixin):
imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link
tagline: Mapped[str | None] = mapped_column(Text) tagline: Mapped[str | None] = mapped_column(Text)
full_cast: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # ordered [{name, role, thumb}] full_cast: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # ordered [{name, role, thumb}]
# --- TMDB enrichment (E5 S2): see the identical block on plex_items for the full rationale. Additive;
# served from the row; NULL until enriched. (`budget`/`revenue` stay NULL for shows — TV has none.) ---
tmdb_id: Mapped[int | None] = mapped_column(Integer)
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
tmdb_rating: Mapped[float | None] = mapped_column(Float) # TMDb vote_average (0–10)
tmdb_vote_count: Mapped[int | None] = mapped_column(Integer)
budget: Mapped[int | None] = mapped_column(BigInteger)
revenue: Mapped[int | None] = mapped_column(BigInteger)
keywords: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
crew: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [{name, job, tmdb_id, thumb}]
studios: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # TMDB production companies [str]
production_countries: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
# Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched, # Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched,
# so serving falls back to the EN columns. `people_text` is language-neutral, reused for both vectors. # so serving falls back to the EN columns. `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512)) title_hu: Mapped[str | None] = mapped_column(String(512))
@@ -1171,6 +1202,24 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link imdb_id: Mapped[str | None] = mapped_column(String(16)) # tt… → the clickable IMDb link
tagline: Mapped[str | None] = mapped_column(Text) tagline: Mapped[str | None] = mapped_column(Text)
full_cast: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # ordered [{name, role, thumb}] full_cast: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # ordered [{name, role, thumb}]
# --- TMDB enrichment (E5 S2): matched by the canonical `guid` ("tmdb:12345"), filled by the TMDB
# backfill job, SERVED FROM THIS ROW (never a live TMDB call). Additive — imdb_rating/rating
# above stay Plex-sourced; `tmdb_rating` is the distinct TMDb community score. NULL until enriched;
# `tmdb_enriched_at` is the "tried" watermark (stamped on a definitive no-match too, so an
# unresolvable title isn't re-fetched every run; left NULL only on a transient fetch error). ---
tmdb_id: Mapped[int | None] = mapped_column(Integer)
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
tmdb_rating: Mapped[float | None] = mapped_column(Float) # TMDb vote_average (0–10)
tmdb_vote_count: Mapped[int | None] = mapped_column(Integer)
budget: Mapped[int | None] = mapped_column(BigInteger) # USD (movies; TV has none)
revenue: Mapped[int | None] = mapped_column(BigInteger)
keywords: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
crew: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [{name, job, tmdb_id, thumb}]
studios: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # TMDB production companies [str]
production_countries: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
# [{tmdb_id, title, poster, media_type}] — resolved to library rows (by guid) at serve time.
recommendations: Mapped[list | None] = mapped_column(JSONB(none_as_null=True))
backdrops: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # TMDB backdrop file_paths [str] (crossfade pool, S8)
# Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched # Hungarian-metadata twin fields, written by the overlay sync (E5 S5). NULL = no HU twin matched
# (serving falls back to the EN columns). `people_text` is language-neutral, reused for both vectors. # (serving falls back to the EN columns). `people_text` is language-neutral, reused for both vectors.
title_hu: Mapped[str | None] = mapped_column(String(512)) title_hu: Mapped[str | None] = mapped_column(String(512))
@@ -1211,6 +1260,21 @@ class PlexPerson(Base, TimestampMixin, UpdatedAtMixin):
id: Mapped[int] = mapped_column(primary_key=True) id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String(255), unique=True, index=True) name: Mapped[str] = mapped_column(String(255), unique=True, index=True)
photo_thumb: Mapped[str | None] = mapped_column(Text) # proxied /person-image URL, or None photo_thumb: Mapped[str | None] = mapped_column(Text) # proxied /person-image URL, or None
# TMDB person id, linked by name from a title's TMDB credits (E5 S2 lays the foundation; the /person
# biography fetch + person-page render land in S6). NULL until a TMDB-enriched title names this person.
tmdb_person_id: Mapped[int | None] = mapped_column(Integer)
# --- TMDB person enrichment (E5 S6): fetched from /person/{tmdb_person_id} the first time the person
# page is opened, then SERVED FROM THE ROW. `tmdb_enriched_at` is the watermark (set even on a
# no-op fetch so it isn't retried every open). `known_for` = top combined-credits [{tmdb_id,
# media_type, title}], resolved to library rows by guid at serve time (like recommendations). ---
biography: Mapped[str | None] = mapped_column(Text)
birthday: Mapped[str | None] = mapped_column(String(10)) # "YYYY-MM-DD", served as-is
deathday: Mapped[str | None] = mapped_column(String(10))
place_of_birth: Mapped[str | None] = mapped_column(Text)
known_for_department: Mapped[str | None] = mapped_column(String(64))
also_known_as: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [str]
known_for: Mapped[list | None] = mapped_column(JSONB(none_as_null=True)) # [{tmdb_id, media_type, title}]
tmdb_enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
class PlexCollection(Base, TimestampMixin, UpdatedAtMixin): class PlexCollection(Base, TimestampMixin, UpdatedAtMixin):
+21
View File
@@ -6,14 +6,32 @@ router. Pure functions — no DB, no request state."""
from app.models import PlexItem, PlexShow, PlexState from app.models import PlexItem, PlexShow, PlexState
# A card's wide backdrop, for a rail's landscape tile or a crossfading hero — but ONLY when the item
# actually has art. `thumb` is always emitted (its endpoint placeholders a missing poster), but an
# `art` URL for an item with no `art_key` would resolve to a broken/blank image on every card that
# tried to use it, so gate it on the stored key and let the client fall back to the poster.
def _art_url(rating_key: str, art_key: str | None) -> str | None:
return f"/api/plex/image/{rating_key}?variant=art" if art_key else None
# Card hover-overview: a short synopsis for the reveal panel, trimmed so a strip/related row of cards
# stays a few hundred bytes each rather than carrying full synopses. (These non-grid card paths serve
# the EN summary/rating; the main grid coalesces the HU twin in browse.unified_library.)
def _card_summary(summary: str | None) -> str | None:
return (summary or "")[:300] or None
def movie_card(it: PlexItem, st: PlexState | None) -> dict: def movie_card(it: PlexItem, st: PlexState | None) -> dict:
return { return {
"id": it.rating_key, "id": it.rating_key,
"type": "movie", "type": "movie",
"title": it.title, "title": it.title,
"year": it.year, "year": it.year,
"rating": it.rating,
"summary": _card_summary(it.summary),
"duration_seconds": it.duration_s, "duration_seconds": it.duration_s,
"thumb": f"/api/plex/image/{it.rating_key}", "thumb": f"/api/plex/image/{it.rating_key}",
"art": _art_url(it.rating_key, it.art_key),
"playable": it.playable, "playable": it.playable,
"status": st.status if st else "new", "status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0, "position_seconds": st.position_seconds if st else 0,
@@ -26,7 +44,10 @@ def show_card(sh: PlexShow, status: str = "new") -> dict:
"type": "show", "type": "show",
"title": sh.title, "title": sh.title,
"year": sh.year, "year": sh.year,
"rating": sh.rating,
"summary": _card_summary(sh.summary),
"thumb": f"/api/plex/image/{sh.rating_key}", "thumb": f"/api/plex/image/{sh.rating_key}",
"art": _art_url(sh.rating_key, sh.art_key),
"season_count": sh.child_count, "season_count": sh.child_count,
# Aggregate watch-state across the show's episodes (new | in_progress | watched) → grid badge. # Aggregate watch-state across the show's episodes (new | in_progress | watched) → grid badge.
"status": status, "status": status,
+129 -5
View File
@@ -6,7 +6,7 @@ Pure query fragments — every function returns SQLAlchemy expressions or a para
from dataclasses import dataclass from dataclasses import dataclass
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from sqlalchemy import and_, func, or_, type_coerce from sqlalchemy import and_, case, func, or_, select, type_coerce
from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.dialects.postgresql import JSONB
TS_CONFIG = "public.unaccent_simple" TS_CONFIG = "public.unaccent_simple"
@@ -29,6 +29,11 @@ def title_col(model, lang: str):
return func.coalesce(model.title_hu, model.title) if lang == "hu" else model.title return func.coalesce(model.title_hu, model.title) if lang == "hu" else model.title
def summary_col(model, lang: str):
"""DISPLAY summary, HU-coalesced (EN fallback) — the card hover-overview + info page read this."""
return func.coalesce(model.summary_hu, model.summary) if lang == "hu" else model.summary
def genres_col(model, lang: str): def genres_col(model, lang: str):
# type_coerce so the JSONB comparator (`.contains` / `@>`, GIN-backed) survives the coalesce. # type_coerce so the JSONB comparator (`.contains` / `@>`, GIN-backed) survives the coalesce.
if lang == "hu": if lang == "hu":
@@ -36,6 +41,58 @@ def genres_col(model, lang: str):
return model.genres return model.genres
def top_art_keys(
db, model, lib_ids: list[int], key_col, keys: list[str], *, per: int = 4, extra_conds=()
) -> dict[str, list[str]]:
"""For each value in `keys` (a member of the JSONB array `key_col` — e.g. a genre value or a
collection rating_key), the rating_keys of its top `per` items (best-rated, then most-recent) as
{key: [rating_key, …]}. ONE windowed query: unnest the array, rank within each key, take the top.
The caller renders posters from /api/plex/image/{rating_key}, so only the keys are returned."""
if not lib_ids or not keys:
return {}
k = func.jsonb_array_elements_text(key_col).label("k")
expanded = (
select(model.rating_key.label("rk"), model.rating.label("r"), model.added_at.label("a"), k)
.where(model.library_id.in_(lib_ids), func.jsonb_typeof(key_col) == "array", *extra_conds)
.subquery()
)
ranked = select(
expanded.c.rk,
expanded.c.k,
func.row_number()
.over(
partition_by=expanded.c.k,
order_by=(expanded.c.r.desc().nullslast(), expanded.c.a.desc().nullslast()),
)
.label("rn"),
).subquery()
out: dict[str, list[str]] = {}
for rk, key in db.execute(
select(ranked.c.rk, ranked.c.k)
.where(ranked.c.k.in_(keys), ranked.c.rn <= per)
.order_by(ranked.c.k, ranked.c.rn)
).all():
out.setdefault(key, []).append(rk)
return out
def runtime_by_key(db, model, lib_ids: list[int], key_col, keys: list[str]) -> dict[str, int]:
"""Total runtime (sum of duration_s, seconds) of the items under each key in `keys` (a member of the
JSONB array `key_col`), as {key: seconds}. ONE grouped query. Used for a collection tile's runtime."""
if not lib_ids or not keys:
return {}
k = func.jsonb_array_elements_text(key_col).label("k")
exp = (
select(func.coalesce(model.duration_s, 0).label("d"), k)
.where(model.library_id.in_(lib_ids), func.jsonb_typeof(key_col) == "array")
.subquery()
)
rows = db.execute(
select(exp.c.k, func.sum(exp.c.d)).where(exp.c.k.in_(keys)).group_by(exp.c.k)
).all()
return {key: int(total or 0) for key, total in rows}
def vector_col(model, lang: str): def vector_col(model, lang: str):
# RANKING vector. In HU we rank by the Hungarian vector; a row matched only via its English vector # RANKING vector. In HU we rank by the Hungarian vector; a row matched only via its English vector
# (see `vector_match`) gets rank 0 and sorts last, which is the intended HU-first ordering. # (see `vector_match`) gets rank 0 and sorts last, which is the intended HU-first ordering.
@@ -135,8 +192,12 @@ def meta_filter_conds(model, p: dict, lang: str = "en") -> list:
stds = split_csv(p.get("studios")) stds = split_csv(p.get("studios"))
if stds: # OR if stds: # OR
conds.append(model.studio.in_(stds)) conds.append(model.studio.in_(stds))
if p.get("collection"): for kw in split_csv(p.get("keywords")): # AND — TMDB keyword chips from the detail page (E5 S5)
conds.append(model.collection_keys.contains([p["collection"]])) conds.append(model.keywords.contains([kw]))
csel = split_csv(p.get("collections"))
if csel: # OR (any) or AND (all) — a title can belong to several collections, mirroring genre_mode
cc = [model.collection_keys.contains([c]) for c in csel]
conds.append(and_(*cc) if p.get("collection_mode") == "all" else or_(*cc))
return conds return conds
@@ -166,7 +227,9 @@ class LibraryFilters:
"any" # how multiple actors combine: any (OR, default) | all (AND) "any" # how multiple actors combine: any (OR, default) | all (AND)
) )
studios: str | None = None studios: str | None = None
collection: str | None = None keywords: str | None = None # TMDB keyword chips (E5 S5), CSV; language-neutral
collections: str | None = None # CSV of collection rating_keys
collection_mode: str = "any" # how multiple collections combine: any (OR, default) | all (AND)
def as_dict(self) -> dict: def as_dict(self) -> dict:
return { return {
@@ -183,9 +246,70 @@ class LibraryFilters:
"actors": self.actors, "actors": self.actors,
"actor_mode": self.actor_mode, "actor_mode": self.actor_mode,
"studios": self.studios, "studios": self.studios,
"collection": self.collection, "keywords": self.keywords,
"collections": self.collections,
"collection_mode": self.collection_mode,
} }
def like_escape(s: str) -> str: def like_escape(s: str) -> str:
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
# --- Watch-state scope (shared by the /library grid AND the /facets aggregation) ---------------
#
# The `show` scope (all | watched | in_progress | unwatched) filters the browse result by per-user
# watch state. The SAME predicate has to be applied in two places — `unified_library` (the grid) and
# `facets.wsq` (so the facet counts reflect exactly the visible set) — and they MUST stay identical:
# a hidden movie kept out of the grid must not inflate the facets either. These helpers are the one
# home for that predicate so the two call sites can't drift (E3 S1; pinned by test_plex_facets_grid_
# parity). Callers pass the already-joined state expressions, so this module stays model-agnostic:
# * movies — `st` is an aliased PlexState outer-joined on (item_id, user_id).
# * shows — `tot`/`wat`/`inp` are the coalesced totals from `rollup.agg_subquery`.
def movie_status_case(st):
"""The per-movie status label (new | in_progress | watched) from its PlexState alias `st`."""
return case(
(st.status == "watched", "watched"),
(
and_(st.position_seconds > 0, or_(st.status.is_(None), st.status != "watched")),
"in_progress",
),
else_="new",
)
def show_status_case(tot, wat, inp):
"""The per-show status label rolled up from its episodes' totals (total/watched/in-progress)."""
return case(
(and_(tot > 0, wat >= tot), "watched"),
(or_(wat > 0, inp > 0), "in_progress"),
else_="new",
)
def apply_movie_state_filter(query, st, show: str):
"""Restrict a movie query to the `show` scope via its PlexState alias `st`. The default (`all`)
excludes hidden movies — the exclusion that must match between grid and facets."""
if show == "watched":
return query.where(st.status == "watched")
if show == "in_progress":
return query.where(
st.position_seconds > 0, or_(st.status.is_(None), st.status != "watched")
)
if show == "unwatched":
return query.where(or_(st.status.is_(None), st.status.notin_(["watched", "hidden"])))
return query.where(or_(st.status.is_(None), st.status != "hidden"))
def apply_show_state_filter(query, tot, wat, inp, show: str):
"""Restrict a show query to the `show` scope via its rollup totals. `all` is a no-op (shows have
no hidden concept); the caller only joins the rollup for the three non-`all` scopes."""
if show == "watched":
return query.where(tot > 0, wat >= tot)
if show == "in_progress":
return query.where(or_(wat > 0, inp > 0), wat < tot)
if show == "unwatched":
return query.where(wat == 0, inp == 0)
return query
+41 -18
View File
@@ -42,27 +42,32 @@ def rollup(cards: list[dict]) -> dict:
} }
def agg_subquery(db: Session, user_id: int): def agg_subquery(db: Session, user_id: int, include_resumed: bool = False):
"""Per-show episode counts for a user: total, watched, in-progress. Reused to (a) filter the show """Per-show episode counts for a user: total, watched, in-progress. Reused to (a) filter the show
grid by aggregate watch-state and (b) compute each card's badge.""" grid by aggregate watch-state, (b) compute each card's badge, and (c) drive the Discover
continue-watching rail. `include_resumed` adds `resumed_at` = max(progress_updated_at) across the
show's episodes, so the rail can order shows by how recently they were last progressed."""
ps = aliased(PlexState) ps = aliased(PlexState)
return ( cols = [
select( PlexItem.show_id.label("show_id"),
PlexItem.show_id.label("show_id"), func.count(PlexItem.id).label("total"),
func.count(PlexItem.id).label("total"), func.count(case((ps.status == "watched", 1))).label("watched"),
func.count(case((ps.status == "watched", 1))).label("watched"), func.count(
func.count( case(
case( (
( and_(
and_( ps.position_seconds > 0,
ps.position_seconds > 0, or_(ps.status.is_(None), ps.status != "watched"),
or_(ps.status.is_(None), ps.status != "watched"), ),
), 1,
1,
)
) )
).label("inprog"), )
) ).label("inprog"),
]
if include_resumed:
cols.append(func.max(ps.progress_updated_at).label("resumed_at"))
return (
select(*cols)
.outerjoin(ps, and_(ps.item_id == PlexItem.id, ps.user_id == user_id)) .outerjoin(ps, and_(ps.item_id == PlexItem.id, ps.user_id == user_id))
.where(PlexItem.kind == "episode", PlexItem.show_id.isnot(None)) .where(PlexItem.kind == "episode", PlexItem.show_id.isnot(None))
.group_by(PlexItem.show_id) .group_by(PlexItem.show_id)
@@ -76,3 +81,21 @@ def state_map(db: Session, user_id: int, show_ids: list[int]) -> dict[int, str]:
agg_subquery(db, user_id).where(PlexItem.show_id.in_(show_ids)) agg_subquery(db, user_id).where(PlexItem.show_id.in_(show_ids))
).all() ).all()
return {r.show_id: show_status(r.total, r.watched, r.inprog) for r in rows} return {r.show_id: show_status(r.total, r.watched, r.inprog) for r in rows}
def item_state_map(
db: Session, user_id: int, item_ids: list[int]
) -> dict[int, PlexState]:
"""A user's PlexState rows for a set of leaf items (movies or episodes), keyed by item_id — the
leaf analogue of `state_map` (which rolls a show's episodes up to a status string). One batched
fetch, so the caller serializes cards without a per-card query. Empty in → empty dict (no `IN ()`)."""
if not item_ids:
return {}
return {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user_id, PlexState.item_id.in_(item_ids)
)
).scalars()
}
+62 -9
View File
@@ -5,9 +5,14 @@ Playback is from the local physical file, so this only mirrors METADATA + the ph
(no ffprobe needed) to classify each item's browser playability. Movies + episodes become playable (no ffprobe needed) to classify each item's browser playability. Movies + episodes become playable
LEAF rows (plex_items); shows/seasons are mirrored for the drill-down + episode ordering. LEAF rows (plex_items); shows/seasons are mirrored for the drill-down + episode ordering.
The sync is a full reconcile of the enabled sections (upsert by rating_key). Pruning of items that The sync is a full reconcile of the enabled sections (upsert by rating_key) AND prunes rows for
disappeared from Plex is intentionally left for later (a follow-up can diff rating_keys per library). titles that disappeared from Plex: each section diffs the rating_keys it just listed against the ones
Cast + intro/credit markers are fetched lazily on the item-detail/player path (P2), not here. stored, and deletes the stale ones (their watch-state / playlist links cascade). The prune is guarded
on a NON-empty listing — a section that returns zero items is left untouched, so a transient Plex
glitch that yields an empty page can't wipe a library (the metadata is cheap to re-sync; an accidental
mass-delete is the only real risk). Library-level reconcile (dropping a whole removed section, the
dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are fetched lazily on the
item-detail/player path (P2), not here.
""" """
import logging import logging
from datetime import datetime, timezone from datetime import datetime, timezone
@@ -150,7 +155,7 @@ def sync(db: Session) -> dict:
return {"skipped": "disabled"} return {"skipped": "disabled"}
stats = { stats = {
"libraries": 0, "movies": 0, "shows": 0, "seasons": 0, "episodes": 0, "libraries": 0, "movies": 0, "shows": 0, "seasons": 0, "episodes": 0,
"collections": 0, "overlaid": 0, "collections": 0, "overlaid": 0, "pruned": 0,
} }
wanted = _enabled_section_keys(db) wanted = _enabled_section_keys(db)
overlay_map = _overlay_map(db) # {huKey: enKey} overlay_map = _overlay_map(db) # {huKey: enKey}
@@ -279,8 +284,14 @@ def _upsert_library(db: Session, key: str, title: str, kind: str) -> PlexLibrary
return lib return lib
def _paginate(plex: PlexClient, key: str, item_type: int): def _paginate(plex: PlexClient, key: str, item_type: int, progress: dict | None = None):
"""Yield a section's items page by page. If `progress` is given, set `progress["complete"]` True
ONLY when the walk reached the server's reported `total` — a truncated fetch (an empty/short page
mid-sequence, e.g. a transient Plex hiccup, before `total` is reached) leaves it False so the
caller can skip pruning. Pruning on a partial listing would delete rows the listing merely failed
to mention (the metadata is re-syncable, but that still cascades watch-state)."""
start = 0 start = 0
total = None
while True: while True:
items, total = plex.section_items(key, item_type, start, _PAGE) items, total = plex.section_items(key, item_type, start, _PAGE)
if not items: if not items:
@@ -292,6 +303,10 @@ def _paginate(plex: PlexClient, key: str, item_type: int):
break break
if len(items) < _PAGE: if len(items) < _PAGE:
break break
if progress is not None:
# A truthy `total` we reached = a trustworthy complete listing. `total` unknown/zero-with-rows
# (a Plex that doesn't report totalSize) is treated as NOT complete, so prune stays conservative.
progress["complete"] = bool(total) and start >= total
# The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this # The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this
@@ -351,6 +366,24 @@ def _apply_item(row: PlexItem, lib_id: int, kind: str, meta: dict) -> None:
row.playable = classify_playable(mf.get("codec_video"), mf.get("codec_audio"), mf.get("container")) row.playable = classify_playable(mf.get("codec_video"), mf.get("codec_audio"), mf.get("container"))
def _prune_stale(
db: Session, existing: dict, seen: set[str], stats: dict, *, complete: bool
) -> None:
"""Delete stored rows whose rating_key is no longer in the freshly-listed `seen` set — the titles
that disappeared from Plex (their watch-state / playlist links cascade via the FKs). GUARDED on a
COMPLETE, non-empty listing: a truncated fetch (`complete` False — a mid-sequence empty/short page
before the server's total) or a zero-item listing prunes NOTHING, so a transient Plex hiccup can
never wipe a library over rows it merely failed to list. Callers pass the rating_key -> row map
they already built for the upsert."""
if not complete or not seen:
return
for rk, row in existing.items():
if rk not in seen:
db.delete(row)
stats["pruned"] = stats.get("pruned", 0) + 1
db.flush()
def _sync_movies(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None: def _sync_movies(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None:
existing = { existing = {
r.rating_key: r r.rating_key: r
@@ -358,7 +391,9 @@ def _sync_movies(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -
select(PlexItem).where(PlexItem.library_id == lib.id, PlexItem.kind == "movie") select(PlexItem).where(PlexItem.library_id == lib.id, PlexItem.kind == "movie")
).scalars() ).scalars()
} }
for meta in _paginate(plex, lib.plex_key, _TYPE_MOVIE): seen: set[str] = set()
progress: dict = {}
for meta in _paginate(plex, lib.plex_key, _TYPE_MOVIE, progress):
rk = str(meta.get("ratingKey") or "") rk = str(meta.get("ratingKey") or "")
if not rk: if not rk:
continue continue
@@ -366,8 +401,10 @@ def _sync_movies(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -
if row.id is None: if row.id is None:
db.add(row) db.add(row)
_apply_item(row, lib.id, "movie", meta) _apply_item(row, lib.id, "movie", meta)
seen.add(rk)
stats["movies"] += 1 stats["movies"] += 1
db.flush() db.flush()
_prune_stale(db, existing, seen, stats, complete=progress.get("complete", False))
def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None: def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None:
@@ -378,10 +415,13 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
select(PlexShow).where(PlexShow.library_id == lib.id) select(PlexShow).where(PlexShow.library_id == lib.id)
).scalars() ).scalars()
} }
for meta in _paginate(plex, lib.plex_key, _TYPE_SHOW): shows_seen: set[str] = set()
shows_progress: dict = {}
for meta in _paginate(plex, lib.plex_key, _TYPE_SHOW, shows_progress):
rk = str(meta.get("ratingKey") or "") rk = str(meta.get("ratingKey") or "")
if not rk: if not rk:
continue continue
shows_seen.add(rk)
sh = shows.get(rk) or PlexShow(rating_key=rk) sh = shows.get(rk) or PlexShow(rating_key=rk)
if sh.id is None: if sh.id is None:
db.add(sh) db.add(sh)
@@ -407,11 +447,14 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
select(PlexSeason).join(PlexShow).where(PlexShow.library_id == lib.id) select(PlexSeason).join(PlexShow).where(PlexShow.library_id == lib.id)
).scalars() ).scalars()
} }
for meta in _paginate(plex, lib.plex_key, _TYPE_SEASON): seasons_seen: set[str] = set()
seasons_progress: dict = {}
for meta in _paginate(plex, lib.plex_key, _TYPE_SEASON, seasons_progress):
rk = str(meta.get("ratingKey") or "") rk = str(meta.get("ratingKey") or "")
sid = show_id.get(str(meta.get("parentRatingKey") or "")) sid = show_id.get(str(meta.get("parentRatingKey") or ""))
if not rk or sid is None: if not rk or sid is None:
continue continue
seasons_seen.add(rk)
se = seasons.get(rk) or PlexSeason(rating_key=rk) se = seasons.get(rk) or PlexSeason(rating_key=rk)
if se.id is None: if se.id is None:
db.add(se) db.add(se)
@@ -431,10 +474,13 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
select(PlexItem).where(PlexItem.library_id == lib.id, PlexItem.kind == "episode") select(PlexItem).where(PlexItem.library_id == lib.id, PlexItem.kind == "episode")
).scalars() ).scalars()
} }
for meta in _paginate(plex, lib.plex_key, _TYPE_EPISODE): episodes_seen: set[str] = set()
episodes_progress: dict = {}
for meta in _paginate(plex, lib.plex_key, _TYPE_EPISODE, episodes_progress):
rk = str(meta.get("ratingKey") or "") rk = str(meta.get("ratingKey") or "")
if not rk: if not rk:
continue continue
episodes_seen.add(rk)
row = episodes.get(rk) or PlexItem(rating_key=rk) row = episodes.get(rk) or PlexItem(rating_key=rk)
if row.id is None: if row.id is None:
db.add(row) db.add(row)
@@ -446,6 +492,13 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
stats["episodes"] += 1 stats["episodes"] += 1
db.flush() db.flush()
# Prune leaf-first (episodes -> seasons -> shows) so a stale row is deleted directly, before its
# parent's cascade would reach it — avoids deleting the same ORM object twice. Each level is
# independently guarded on a non-empty listing by _prune_stale.
_prune_stale(db, episodes, episodes_seen, stats, complete=episodes_progress.get("complete", False))
_prune_stale(db, seasons, seasons_seen, stats, complete=seasons_progress.get("complete", False))
_prune_stale(db, shows, shows_seen, stats, complete=shows_progress.get("complete", False))
def _sync_overlay( def _sync_overlay(
db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict
+492
View File
@@ -0,0 +1,492 @@
"""TMDB enrichment for the mirrored Plex catalog (E5 S2).
A thin synchronous TMDB v3 client plus the parse/persist of the enrichment fields onto plex_items /
plex_shows, matched by the stored canonical `guid` ("tmdb:12345", or imdb/tvdb resolved via /find).
Server-side ONLY (the api_key from sysconfig). Everything fetched is PERSISTED and served from the DB
row — there is never a live TMDB call at serve time (the user's explicit rule). Additive: the existing
Plex-sourced `imdb_rating` / `rating` are left untouched; `tmdb_rating` is the distinct TMDb score.
One-parser rule (mirrors app/plex/detail.py): `parse_tmdb` is the pure parser, `apply_tmdb_detail` writes
the row. The enrich batch + scheduler entry live here too so the whole TMDB integration is one module.
"""
import logging
import time
from datetime import datetime, timezone
import httpx
from sqlalchemy import func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import sysconfig
from app.config import settings
from app.models import PlexItem, PlexPerson, PlexShow
log = logging.getLogger("siftlode.tmdb")
# Politeness throttle: TMDB tolerates ~50 req/s; ~33/s leaves comfortable headroom for a shared host.
_MIN_INTERVAL = 0.03
# The crew jobs worth surfacing on an info page, in DISPLAY-priority order (the stored crew is sorted by
# this — TMDB's own crew order is arbitrary and buries the lead roles). MOVIE and TV differ on purpose:
# on a TV show `Director`/`Editor`/`Director of Photography` are PER-EPISODE credits (dozens of names
# that would flood the list and bury the showrunner), so TV keys on `Creator` (from the payload's
# top-level `created_by`, not the crew list) + the show-level producers/writers instead.
_MOVIE_CREW_JOBS = (
"Director",
"Writer",
"Screenplay",
"Story",
"Producer",
"Executive Producer",
"Editor",
"Director of Photography",
"Original Music Composer",
)
_TV_CREW_JOBS = ("Creator", "Executive Producer", "Writer", "Producer", "Original Music Composer")
_MOVIE_PRIORITY = {job: i for i, job in enumerate(_MOVIE_CREW_JOBS)}
_TV_PRIORITY = {job: i for i, job in enumerate(_TV_CREW_JOBS)}
# append_to_response bundles the sub-resources into ONE request per title (keywords + credits +
# recommendations + images + external ids). TV uses aggregate_credits (per-person aggregated jobs).
_MOVIE_APPEND = "keywords,credits,recommendations,images,external_ids"
_TV_APPEND = "keywords,aggregate_credits,recommendations,images,external_ids"
class TmdbError(Exception):
"""Any transient failure talking to TMDB (network, non-404 HTTP, bad payload). The caller leaves
the row's `tmdb_enriched_at` NULL so the next run retries."""
class TmdbNotConfigured(TmdbError):
"""No TMDB API key is set — the module can't do anything."""
class TmdbClient:
"""Bound to the configured api_key + language. Use as a context manager so the httpx client closes."""
def __init__(self, db: Session, timeout: float = 30.0):
self.key = sysconfig.get_str(db, "tmdb_api_key")
if not self.key:
raise TmdbNotConfigured("TMDB API key is not configured")
self.base = settings.tmdb_api_base.rstrip("/")
self.language = sysconfig.get_str(db, "tmdb_language") or settings.tmdb_language
# `timeout` is caller-tunable so the request-path lazy person enrich can cap how long it may
# block a worker thread (the batch job keeps the generous default).
self._http = httpx.Client(timeout=timeout)
self._last = 0.0
def __enter__(self) -> "TmdbClient":
return self
def __exit__(self, *exc) -> None:
self._http.close()
def close(self) -> None:
self._http.close()
def _throttle(self) -> None:
wait = self._last + _MIN_INTERVAL - time.monotonic()
if wait > 0:
time.sleep(wait)
self._last = time.monotonic()
def _get(self, path: str, params: dict | None = None) -> dict | None:
"""GET a TMDB path. Returns the JSON dict, None on a 404 (a definitive "no such id"), and raises
TmdbError on any transient failure (network / other HTTP status / non-JSON)."""
self._throttle()
p = {"api_key": self.key, "language": self.language, **(params or {})}
try:
r = self._http.get(f"{self.base}{path}", params=p)
except httpx.HTTPError as e:
# NEVER put str(e)/the URL in the message: the request URL carries `?api_key=<secret>` in
# its query string, and this message is logged (enrich_batch). Report the type + path only.
raise TmdbError(f"TMDB request error on {path}: {type(e).__name__}") from None
if r.status_code == 404:
return None
try:
r.raise_for_status()
except httpx.HTTPStatusError as e:
# Same reason: the HTTPStatusError string embeds the full URL incl. the api_key. Sever the
# cause (`from None`) so no traceback can resurface it, and log only status + path.
raise TmdbError(f"TMDB {path} returned {e.response.status_code}") from None
try:
return r.json()
except ValueError as e:
raise TmdbError("TMDB returned a non-JSON response") from e
def fetch(self, tmdb_id: int, media_type: str) -> dict | None:
"""Full enrichment payload for a movie or show (one request via append_to_response)."""
if media_type == "tv":
return self._get(f"/tv/{tmdb_id}", {"append_to_response": _TV_APPEND})
return self._get(f"/movie/{tmdb_id}", {"append_to_response": _MOVIE_APPEND})
def find(self, external_id: str, source: str) -> dict | None:
"""Resolve an external id (imdb_id / tvdb_id) to TMDB items via /find."""
return self._get(f"/find/{external_id}", {"external_source": source})
def fetch_person(self, person_id: int) -> dict | None:
"""One person's bio + combined credits (E5 S6), in a single append_to_response request."""
return self._get(f"/person/{person_id}", {"append_to_response": "combined_credits"})
def guid_to_tmdb_id(guid: str | None, media_type: str, tmdb: TmdbClient) -> int | None:
"""Map a stored canonical guid to a TMDB id. `tmdb:` is direct; `imdb:`/`tvdb:` go through /find
(picking the movie_results or tv_results list per `media_type`). Returns None for an unmatchable
guid — the caller then stamps the watermark so it isn't retried every run."""
if not guid or ":" not in guid:
return None
provider, _, ext = guid.partition(":")
if provider == "tmdb":
try:
return int(ext)
except ValueError:
return None
source = {"imdb": "imdb_id", "tvdb": "tvdb_id"}.get(provider)
if not source:
return None
res = tmdb.find(ext, source)
if not res:
return None
items = res.get("tv_results" if media_type == "tv" else "movie_results") or []
return items[0].get("id") if items else None
def _dedup_crew(crew: list[dict]) -> list[dict]:
seen: set[tuple] = set()
out: list[dict] = []
for c in crew:
key = (c["name"], c["job"])
if key in seen:
continue
seen.add(key)
out.append(c)
return out
def parse_tmdb(payload: dict, media_type: str) -> dict:
"""Pure parser: pull the enrichment fields out of a TMDB movie/tv payload (append_to_response form).
Best-effort — any missing sub-resource is just omitted. Returns the row fields plus a `people` list
([{name, tmdb_id}] over cast+crew) for the plex_people id-linking. TMDB image fields are stored as
bare paths ("/abc.jpg"); the client builds the image.tmdb.org URL."""
is_tv = media_type == "tv"
votes = payload.get("vote_count") or 0
kw_container = payload.get("keywords") or {}
kw_list = kw_container.get("results") if is_tv else kw_container.get("keywords")
keywords = [k.get("name") for k in (kw_list or []) if k.get("name")][:20]
priority = _TV_PRIORITY if is_tv else _MOVIE_PRIORITY
credits = payload.get("aggregate_credits" if is_tv else "credits") or {}
crew: list[dict] = []
# A TV show's creators live at the top level (created_by), not in the crew list — surface them.
if is_tv:
for c in payload.get("created_by") or []:
if c.get("name"):
crew.append(
{"name": c["name"], "job": "Creator", "tmdb_id": c.get("id"), "profile_path": c.get("profile_path")}
)
for c in credits.get("crew") or []:
# Movie crew entries carry `job`; TV aggregate_credits carry a `jobs` array per person.
job = (c.get("jobs") or [{}])[0].get("job") if is_tv else c.get("job")
if job in priority and c.get("name"):
crew.append(
{"name": c["name"], "job": job, "tmdb_id": c.get("id"), "profile_path": c.get("profile_path")}
)
# Dedup by (name, job) — a person can hold two distinct key roles (Coppola: Director + Writer) — then
# order by display priority so the lead roles come first, and cap.
crew = _dedup_crew(crew)
crew.sort(key=lambda c: priority.get(c["job"], 99))
crew = crew[:12]
# name -> tmdb person id, for the plex_people foundation (S6 fetches bios off these ids).
people: list[dict] = []
for c in (credits.get("cast") or [])[:30]:
if c.get("name") and c.get("id"):
people.append({"name": c["name"], "tmdb_id": c["id"]})
for c in crew:
if c.get("tmdb_id"):
people.append({"name": c["name"], "tmdb_id": c["tmdb_id"]})
recs: list[dict] = []
for r in ((payload.get("recommendations") or {}).get("results") or [])[:20]:
recs.append(
{
"tmdb_id": r.get("id"),
"title": r.get("title") or r.get("name"),
"poster_path": r.get("poster_path"),
"media_type": r.get("media_type") or media_type,
}
)
backdrops = [
b.get("file_path")
for b in ((payload.get("images") or {}).get("backdrops") or [])
if b.get("file_path")
][:10]
return {
"tmdb_id": payload.get("id"),
# vote_average is meaningless with no votes (TMDB returns 0.0) → treat as unknown.
"tmdb_rating": payload.get("vote_average") if votes else None,
"tmdb_vote_count": votes or None,
# 0 is TMDB's "unknown" for money fields; TV has neither.
"budget": (payload.get("budget") or None) if not is_tv else None,
"revenue": (payload.get("revenue") or None) if not is_tv else None,
"keywords": keywords or None,
"crew": crew or None,
"studios": [c.get("name") for c in (payload.get("production_companies") or []) if c.get("name")][:8]
or None,
"production_countries": [
c.get("name") for c in (payload.get("production_countries") or []) if c.get("name")
]
or None,
"recommendations": recs or None,
"backdrops": backdrops or None,
"people": people,
}
def apply_tmdb_detail(row, parsed: dict) -> None:
"""Persist the parsed TMDB fields onto a movie/show row. Does NOT touch full_cast / imdb_rating /
rating (Plex-owned). The caller stamps `tmdb_enriched_at` and links people separately."""
row.tmdb_id = parsed["tmdb_id"]
row.tmdb_rating = parsed["tmdb_rating"]
row.tmdb_vote_count = parsed["tmdb_vote_count"]
row.budget = parsed["budget"]
row.revenue = parsed["revenue"]
row.keywords = parsed["keywords"]
row.crew = parsed["crew"]
row.studios = parsed["studios"]
row.production_countries = parsed["production_countries"]
row.recommendations = parsed["recommendations"]
row.backdrops = parsed["backdrops"]
def _link_people(db: Session, links: dict[str, int]) -> None:
"""Upsert plex_people by name, stamping `tmdb_person_id` — keeping any id already set (first wins),
filling it where NULL, inserting a bare row where the person isn't cached yet. Deduped by name so a
single statement can't hit the conflict target twice (the same rule as people.enrich_row_people)."""
values = [{"name": nm, "tmdb_person_id": pid} for nm, pid in links.items() if nm and pid]
if not values:
return
stmt = pg_insert(PlexPerson).values(values)
db.execute(
stmt.on_conflict_do_update(
index_elements=[PlexPerson.name],
set_={"tmdb_person_id": func.coalesce(PlexPerson.tmdb_person_id, stmt.excluded.tmdb_person_id)},
)
)
def enrich_batch(db: Session, tmdb: TmdbClient, limit: int) -> int:
"""Enrich up to `limit` not-yet-TMDB-enriched titles (movies first, shows fill the remainder —
mirrors _enrich_people_batch). A transient TmdbError leaves the watermark NULL for retry; a
definitive no-match (unresolvable guid / TMDB 404) still stamps the watermark so it isn't re-fetched
every run. One commit at the end. Returns the number processed (matched OR definitively unmatched)."""
movies = (
db.execute(
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.guid.is_not(None),
PlexItem.tmdb_enriched_at.is_(None),
)
.order_by(PlexItem.id)
.limit(limit)
)
.scalars()
.all()
)
shows: list[PlexShow] = []
if len(movies) < limit:
shows = (
db.execute(
select(PlexShow)
.where(PlexShow.guid.is_not(None), PlexShow.tmdb_enriched_at.is_(None))
.order_by(PlexShow.id)
.limit(limit - len(movies))
)
.scalars()
.all()
)
processed = 0
links: dict[str, int] = {}
for row, media_type in [(m, "movie") for m in movies] + [(s, "tv") for s in shows]:
try:
tmdb_id = guid_to_tmdb_id(row.guid, media_type, tmdb)
payload = tmdb.fetch(tmdb_id, media_type) if tmdb_id else None
except TmdbError as e:
log.warning("tmdb enrich %s failed (will retry): %s", row.rating_key, e)
continue # transient — leave tmdb_enriched_at NULL for the next run
if payload:
parsed = parse_tmdb(payload, media_type)
apply_tmdb_detail(row, parsed)
for p in parsed["people"]:
links.setdefault(p["name"], p["tmdb_id"])
# Stamp even on a definitive no-match so an unresolvable title isn't re-fetched every run.
row.tmdb_enriched_at = datetime.now(timezone.utc)
processed += 1
if links:
_link_people(db, links)
db.commit()
return processed
def run_enrich(db: Session) -> dict:
"""Scheduler entry point. No-op (never an error) when the TMDB key isn't configured."""
if not sysconfig.get_str(db, "tmdb_api_key"):
return {"enriched": 0, "skipped": "tmdb_api_key not set"}
with TmdbClient(db) as tmdb:
n = enrich_batch(db, tmdb, settings.tmdb_enrich_batch)
return {"enriched": n}
def resolve_library_guids(
db: Session,
entries: list[dict] | None,
*,
movie_lib_ids: list[int] | None = None,
show_lib_ids: list[int] | None = None,
) -> tuple[dict[str, PlexItem], dict[str, PlexShow]]:
"""Match a list of TMDB credits/recs (each `{tmdb_id, media_type, …}`) to our library rows by the
canonical guid `tmdb:<id>`, returning `(movies_by_guid, shows_by_guid)` of the ORM rows.
⭐ TMDB movie ids and tv ids are INDEPENDENT sequences (id 603 can be BOTH a movie and a show), so
each entry matches ONLY its own media_type's table — a movie credit against a library movie, a tv
credit against a library show; a blind numeric match would resolve to an unrelated same-id title.
Pass `movie_lib_ids`/`show_lib_ids` to scope to the enabled libraries (the person page); leave them
None to match across all libraries (the detail-page recs). An explicit empty list means "that side
has no enabled library" → its bucket is skipped entirely."""
if not entries:
return {}, {}
movie_guids = [
f"tmdb:{e['tmdb_id']}" for e in entries if e.get("tmdb_id") and e.get("media_type") != "tv"
]
tv_guids = [
f"tmdb:{e['tmdb_id']}" for e in entries if e.get("tmdb_id") and e.get("media_type") == "tv"
]
movies: dict[str, PlexItem] = {}
if movie_guids and movie_lib_ids != []:
conds = [PlexItem.kind == "movie", PlexItem.guid.in_(movie_guids)]
if movie_lib_ids is not None:
conds.append(PlexItem.library_id.in_(movie_lib_ids))
for m in db.execute(select(PlexItem).where(*conds)).scalars():
movies.setdefault(m.guid, m)
shows: dict[str, PlexShow] = {}
if tv_guids and show_lib_ids != []:
conds = [PlexShow.guid.in_(tv_guids)]
if show_lib_ids is not None:
conds.append(PlexShow.library_id.in_(show_lib_ids))
for s in db.execute(select(PlexShow).where(*conds)).scalars():
shows.setdefault(s.guid, s)
return movies, shows
def resolve_recs(db: Session, recs: list[dict] | None) -> list[dict]:
"""Tag each recommendation with whether it's in OUR library (matched by guid `tmdb:<id>`, bucketed
by media_type — see resolve_library_guids) and, if so, its rating_key — so the client can deep-link
the ones we have and merely display the rest."""
if not recs:
return []
movies, shows = resolve_library_guids(db, recs)
out: list[dict] = []
for r in recs:
g = f"tmdb:{r.get('tmdb_id')}"
row = shows.get(g) if r.get("media_type") == "tv" else movies.get(g)
out.append(
{**r, "in_library": row is not None, "rating_key": row.rating_key if row else None}
)
return out
def serve_block(db: Session, row) -> dict | None:
"""The TMDB extras for a movie/show detail response, served from the row (no live TMDB call).
None until the row has been TMDB-enriched (the client then shows nothing extra)."""
if row.tmdb_enriched_at is None:
return None
return {
"tmdb_id": row.tmdb_id,
"rating": row.tmdb_rating,
"vote_count": row.tmdb_vote_count,
"budget": row.budget,
"revenue": row.revenue,
"keywords": row.keywords or [],
"crew": row.crew or [],
"studios": row.studios or [],
"production_countries": row.production_countries or [],
"recommendations": resolve_recs(db, row.recommendations),
"backdrops": row.backdrops or [],
}
# --- Person enrichment (E5 S6) -------------------------------------------------------------------
# A person's bio + birth facts + a "Known for" credit pool, fetched once (the first person-page open)
# and served from the plex_people row. `known_for` stores the top combined-credits ids; the person
# endpoint resolves them to library rows by guid at serve time (like recommendations).
_KNOWN_FOR_MAX = 15
def parse_person(payload: dict) -> dict:
"""TMDB /person/{id} (+combined_credits) → the persisted bio fields. `known_for` = the person's
most-popular movie/tv credits (deduped by id), top `_KNOWN_FOR_MAX`, resolved to library later."""
credits = (payload.get("combined_credits") or {}).get("cast") or []
# Dedup on (id, media_type), not the bare id: TMDB movie ids and tv ids are independent sequences
# (id 603 can be both a movie and a show — see resolve_library_guids), so a same-id movie+show pair
# is two distinct credits and both must survive.
seen: set[tuple[int, str]] = set()
known: list[dict] = []
for c in sorted(credits, key=lambda c: c.get("popularity") or 0, reverse=True):
tid, mt = c.get("id"), c.get("media_type")
if not tid or mt not in ("movie", "tv") or (tid, mt) in seen:
continue
seen.add((tid, mt))
known.append({"tmdb_id": tid, "media_type": mt, "title": c.get("title") or c.get("name")})
if len(known) >= _KNOWN_FOR_MAX:
break
return {
"biography": (payload.get("biography") or "").strip() or None,
"birthday": payload.get("birthday") or None,
"deathday": payload.get("deathday") or None,
"place_of_birth": payload.get("place_of_birth") or None,
"known_for_department": payload.get("known_for_department") or None,
"also_known_as": payload.get("also_known_as") or None,
"known_for": known or None,
}
def apply_person(row: PlexPerson, parsed: dict) -> None:
row.biography = parsed["biography"]
row.birthday = parsed["birthday"]
row.deathday = parsed["deathday"]
row.place_of_birth = parsed["place_of_birth"]
row.known_for_department = parsed["known_for_department"]
row.also_known_as = parsed["also_known_as"]
row.known_for = parsed["known_for"]
row.tmdb_enriched_at = datetime.now(timezone.utc)
def enrich_person(db: Session, person: PlexPerson) -> bool:
"""Lazily fetch + persist one person's TMDB bio (first person-page open), committing the write so it
survives regardless of the read path. No-op (False) without a key or a `tmdb_person_id`; a transient
error leaves `tmdb_enriched_at` NULL so the next open retries. A 404 (deleted person) stamps the
watermark with empty fields so we don't refetch it every open."""
if not person.tmdb_person_id or not sysconfig.get_str(db, "tmdb_api_key"):
return False
try:
# Called from a background task (off the request path — see people._enrich_person_bg), so it no
# longer blocks a serve; still cap the outbound TMDB call with a short timeout so a hanging TMDB
# doesn't tie up a threadpool worker for the default 30s.
with TmdbClient(db, timeout=8.0) as tmdb:
payload = tmdb.fetch_person(person.tmdb_person_id)
except TmdbError:
return False
if payload is None:
person.tmdb_enriched_at = datetime.now(timezone.utc)
else:
apply_person(person, parse_person(payload))
db.commit()
return True
+53 -33
View File
@@ -10,6 +10,7 @@ from datetime import datetime, timedelta, timezone
from authlib.integrations.starlette_client import OAuth, OAuthError from authlib.integrations.starlette_client import OAuth, OAuthError
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
from fastapi.responses import JSONResponse, RedirectResponse from fastapi.responses import JSONResponse, RedirectResponse
from pydantic import BaseModel
from sqlalchemy import select, update from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -187,6 +188,36 @@ def _oauth_redirect_uri(request: Request) -> str:
return settings.oauth_redirect_url return settings.oauth_redirect_url
# --- Request bodies (typed contracts; B-core-6) ---------------------------------------------
# Fields are `str | None` (not required) so the handlers keep their exact prior behaviour — a
# missing/null field reads as None and the existing `(... or "")` coercions still apply — while
# FastAPI now validates the body is a JSON object and publishes the schema.
class EmailIn(BaseModel):
email: str | None = None
class EmailPasswordIn(BaseModel):
email: str | None = None
password: str | None = None
class TokenIn(BaseModel):
token: str | None = None
class TokenPasswordIn(BaseModel):
token: str | None = None
password: str | None = None
class SetPasswordIn(BaseModel):
password: str | None = None
current_password: str | None = None
@router.get("/login") @router.get("/login")
async def login(request: Request, db: Session = Depends(get_db)): async def login(request: Request, db: Session = Depends(get_db)):
# access_type=offline ensures a refresh_token on first authorization. We avoid # access_type=offline ensures a refresh_token on first authorization. We avoid
@@ -448,7 +479,7 @@ async def logout(request: Request):
@router.post("/request-access") @router.post("/request-access")
async def request_access( async def request_access(
payload: dict, body: EmailIn,
request: Request, request: Request,
background: BackgroundTasks, background: BackgroundTasks,
db: Session = Depends(get_db), db: Session = Depends(get_db),
@@ -462,7 +493,7 @@ async def request_access(
is careful to avoid, cf. demo_login); an already-allowed user can still just sign in with Google. is careful to avoid, cf. demo_login); an already-allowed user can still just sign in with Google.
Rate-limited per client IP like register/login/reset, and a block returns "pending" too so the Rate-limited per client IP like register/login/reset, and a block returns "pending" too so the
endpoint can't be probed for throttling either.""" endpoint can't be probed for throttling either."""
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
if not valid_email(email): if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.") raise HTTPException(status_code=400, detail="Enter a valid email address.")
if not _request_access_limiter.allow(_client_ip(request)): if not _request_access_limiter.allow(_client_ip(request)):
@@ -477,7 +508,7 @@ async def request_access(
@router.post("/demo") @router.post("/demo")
def demo_login(payload: dict, request: Request, db: Session = Depends(get_db)) -> dict: def demo_login(body: EmailIn, request: Request, db: Session = Depends(get_db)) -> dict:
"""Hidden demo entry: a whitelisted email logs straight into the shared demo account, """Hidden demo entry: a whitelisted email logs straight into the shared demo account,
no Google OAuth. The login page fires this quietly (debounced) as the user types/pastes, no Google OAuth. The login page fires this quietly (debounced) as the user types/pastes,
so there is no visible 'demo login' button. Rate-limited per IP; when blocked it answers so there is no visible 'demo login' button. Rate-limited per IP; when blocked it answers
@@ -485,7 +516,7 @@ def demo_login(payload: dict, request: Request, db: Session = Depends(get_db)) -
enumeration oracle. On a match it sets the session and the client reloads into the app.""" enumeration oracle. On a match it sets the session and the client reloads into the app."""
if not _demo_limiter.allow(_client_ip(request)): if not _demo_limiter.allow(_client_ip(request)):
return {"authenticated": False} return {"authenticated": False}
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
if valid_email(email) and is_demo_allowed(db, email): if valid_email(email) and is_demo_allowed(db, email):
demo = get_or_create_demo_user(db) demo = get_or_create_demo_user(db)
# Isolate the demo: drop any real-account wallet on this tab first, so the demo session # Isolate the demo: drop any real-account wallet on this tab first, so the demo session
@@ -556,7 +587,7 @@ def _consume_token(db: Session, raw: str | None, kind: str) -> User | None:
@router.post("/register") @router.post("/register")
def register( def register(
payload: dict, body: EmailPasswordIn,
request: Request, request: Request,
background: BackgroundTasks, background: BackgroundTasks,
db: Session = Depends(get_db), db: Session = Depends(get_db),
@@ -565,8 +596,8 @@ def register(
verification link, and records an access request for admin approval. Two gates before verification link, and records an access request for admin approval. Two gates before
sign-in: verified email AND admin approval. Responses are uniform once input is valid, so sign-in: verified email AND admin approval. Responses are uniform once input is valid, so
the endpoint can't be used to discover which emails are registered.""" the endpoint can't be used to discover which emails are registered."""
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
password = payload.get("password") or "" password = body.password or ""
# Input validation is safe to reveal (independent of whether the email exists). # Input validation is safe to reveal (independent of whether the email exists).
if not valid_email(email): if not valid_email(email):
raise HTTPException(status_code=400, detail="Enter a valid email address.") raise HTTPException(status_code=400, detail="Enter a valid email address.")
@@ -628,10 +659,10 @@ def _register_account(email: str, password: str) -> None:
@router.post("/verify") @router.post("/verify")
def verify_email_confirm(payload: dict, db: Session = Depends(get_db)) -> dict: def verify_email_confirm(body: TokenIn, db: Session = Depends(get_db)) -> dict:
"""Confirm an email-verification token the SPA read from the URL fragment (SB3). Uniform 400 """Confirm an email-verification token the SPA read from the URL fragment (SB3). Uniform 400
on any invalid/expired/used token.""" on any invalid/expired/used token."""
user = _consume_token(db, payload.get("token"), "verify") user = _consume_token(db, body.token, "verify")
if user is None: if user is None:
raise HTTPException( raise HTTPException(
status_code=400, detail="This verification link is invalid or has expired." status_code=400, detail="This verification link is invalid or has expired."
@@ -643,7 +674,7 @@ def verify_email_confirm(payload: dict, db: Session = Depends(get_db)) -> dict:
@router.post("/verify/resend") @router.post("/verify/resend")
def verify_email_resend( def verify_email_resend(
payload: dict, body: EmailIn,
request: Request, request: Request,
background: BackgroundTasks, background: BackgroundTasks,
) -> dict: ) -> dict:
@@ -653,7 +684,7 @@ def verify_email_resend(
account (the lookup runs off the response path), so it can't probe for registered emails.""" account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _verify_resend_limiter.allow(_client_ip(request)): if not _verify_resend_limiter.allow(_client_ip(request)):
return {"status": "ok"} # silently throttle; uniform response return {"status": "ok"} # silently throttle; uniform response
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
if valid_email(email): if valid_email(email):
background.add_task(_resend_verify_if_eligible, email) background.add_task(_resend_verify_if_eligible, email)
return {"status": "ok"} return {"status": "ok"}
@@ -688,7 +719,7 @@ def verify_email(token: str, db: Session = Depends(get_db)):
@router.post("/password-login") @router.post("/password-login")
def password_login( def password_login(
payload: dict, body: EmailPasswordIn,
request: Request, request: Request,
background: BackgroundTasks, background: BackgroundTasks,
db: Session = Depends(get_db), db: Session = Depends(get_db),
@@ -700,8 +731,8 @@ def password_login(
raise HTTPException( raise HTTPException(
status_code=429, detail="Too many attempts. Try again shortly." status_code=429, detail="Too many attempts. Try again shortly."
) )
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
password = payload.get("password") or "" password = body.password or ""
user = db.execute(select(User).where(User.email == email)).scalar_one_or_none() user = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
# Always run the KDF (against a decoy hash for an unknown / passwordless / demo account) so the # Always run the KDF (against a decoy hash for an unknown / passwordless / demo account) so the
# response time is the same whether or not the email exists — no timing-based enumeration. # response time is the same whether or not the email exists — no timing-based enumeration.
@@ -731,7 +762,7 @@ def password_login(
@router.post("/password-reset/request") @router.post("/password-reset/request")
def password_reset_request( def password_reset_request(
payload: dict, body: EmailIn,
request: Request, request: Request,
background: BackgroundTasks, background: BackgroundTasks,
) -> dict: ) -> dict:
@@ -739,7 +770,7 @@ def password_reset_request(
password account (the lookup runs off the response path), so it can't probe for registered emails.""" password account (the lookup runs off the response path), so it can't probe for registered emails."""
if not _reset_limiter.allow(_client_ip(request)): if not _reset_limiter.allow(_client_ip(request)):
return {"status": "ok"} return {"status": "ok"}
email = (payload.get("email") or "").strip().lower() email = (body.email or "").strip().lower()
# Do the account lookup + token issue + email OFF the response path (a background task with its # Do the account lookup + token issue + email OFF the response path (a background task with its
# own session), so the endpoint takes the same time for any valid email whether or not it has a # own session), so the endpoint takes the same time for any valid email whether or not it has a
# password account — no timing-based enumeration. Any valid email schedules the same task. # password account — no timing-based enumeration. Any valid email schedules the same task.
@@ -761,10 +792,10 @@ def _send_reset_if_eligible(email: str) -> None:
@router.post("/password-reset/confirm") @router.post("/password-reset/confirm")
def password_reset_confirm(payload: dict, db: Session = Depends(get_db)) -> dict: def password_reset_confirm(body: TokenPasswordIn, db: Session = Depends(get_db)) -> dict:
"""Set a new password from a valid reset token, then invalidate any other reset tokens.""" """Set a new password from a valid reset token, then invalidate any other reset tokens."""
token = payload.get("token") token = body.token
new_password = payload.get("password") or "" new_password = body.password or ""
if len(new_password) < PASSWORD_MIN_LEN: if len(new_password) < PASSWORD_MIN_LEN:
raise HTTPException( raise HTTPException(
status_code=400, status_code=400,
@@ -872,7 +903,7 @@ async def link_google(
@router.post("/set-password") @router.post("/set-password")
def set_password( def set_password(
payload: dict, body: SetPasswordIn,
request: Request, request: Request,
user: User = Depends(current_user), user: User = Depends(current_user),
db: Session = Depends(get_db), db: Session = Depends(get_db),
@@ -884,7 +915,7 @@ def set_password(
raise HTTPException( raise HTTPException(
status_code=403, detail="Not available in the demo account." status_code=403, detail="Not available in the demo account."
) )
new_password = payload.get("password") or "" new_password = body.password or ""
if len(new_password) < PASSWORD_MIN_LEN: if len(new_password) < PASSWORD_MIN_LEN:
raise HTTPException( raise HTTPException(
status_code=400, status_code=400,
@@ -892,7 +923,7 @@ def set_password(
) )
if user.password_hash: if user.password_hash:
if not verify_password( if not verify_password(
payload.get("current_password") or "", user.password_hash body.current_password or "", user.password_hash
): ):
raise HTTPException( raise HTTPException(
status_code=403, detail="Current password is incorrect." status_code=403, detail="Current password is incorrect."
@@ -941,17 +972,6 @@ async def upgrade(
) )
@router.get("/me")
async def me(user: User = Depends(current_user)) -> dict:
return {
"id": user.id,
"email": user.email,
"display_name": user.display_name,
"avatar_url": user.avatar_url,
"role": user.role,
}
@router.get("/config") @router.get("/config")
def auth_config(db: Session = Depends(get_db)) -> dict: def auth_config(db: Session = Depends(get_db)) -> dict:
"""Public: which sign-in options this instance offers, so the login page can hide what's off """Public: which sign-in options this instance offers, so the login page can hide what's off
+2 -2
View File
@@ -1,6 +1,6 @@
"""Channel manager: the user's subscriptions with their personal overrides (priority, """Channel manager: the user's subscriptions with their personal overrides (priority,
hidden, tags) and per-channel sync state. Read + local-write only — YouTube-side writes hidden, tags) and per-channel sync state. Local reads/writes plus YouTube-side writes
(unsubscribe) live behind the optional write scope and are added in a later phase.""" (subscribe / unsubscribe) that require the optional write scope."""
import logging import logging
from datetime import datetime, timezone from datetime import datetime, timezone
+6 -56
View File
@@ -21,8 +21,7 @@ from app.config import settings
from app.db import get_db from app.db import get_db
from app.downloads import edit as editmod from app.downloads import edit as editmod
from app.downloads import links as linksmod from app.downloads import links as linksmod
from app.downloads import formats, quota, service, storage from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.models import ( from app.models import (
DownloadJob, DownloadJob,
DownloadLink, DownloadLink,
@@ -454,18 +453,12 @@ def update_download_meta(
return _serialize_job(db, job) return _serialize_job(db, job)
def _set_status(db: Session, job: DownloadJob, status: str) -> None:
job.status = status
db.commit()
@router.post("/{job_id}/pause") @router.post("/{job_id}/pause")
def pause_download( def pause_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db) job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict: ) -> dict:
job = _own_job(db, user, job_id) job = _own_job(db, user, job_id)
if job.status in ("queued", "running"): service.pause(db, job)
_set_status(db, job, "paused") # a running worker aborts cooperatively via the hook
return _serialize_job(db, job) return _serialize_job(db, job)
@@ -474,19 +467,7 @@ def resume_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db) job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict: ) -> dict:
job = _own_job(db, user, job_id) job = _own_job(db, user, job_id)
if job.status in ("paused", "error"): service.resume(db, job)
job.progress = 0
job.error = None
job.error_code = None
# If the shared asset itself failed, reset it to pending so the worker actually
# re-downloads — otherwise the requeued job would instantly re-inherit the asset's
# stale error (the worker short-circuits a job whose asset is already errored).
if job.asset_id:
asset = db.get(MediaAsset, job.asset_id)
if asset and asset.status == "error":
asset.status = "pending"
asset.error = None
_set_status(db, job, "queued")
return _serialize_job(db, job) return _serialize_job(db, job)
@@ -510,33 +491,7 @@ def redownload_download(
raise HTTPException(status_code=409, detail="Editor clips can't be re-downloaded.") raise HTTPException(status_code=409, detail="Editor clips can't be re-downloaded.")
if job.status in ("queued", "running", "paused"): if job.status in ("queued", "running", "paused"):
raise HTTPException(status_code=409, detail="This download is still in progress.") raise HTTPException(status_code=409, detail="This download is still in progress.")
service.redownload(db, job)
# Resolve the fresh private asset FIRST (before touching any file). Salting with the current
# asset id makes it distinct from both the dedup pool and this job's current asset, so the
# insert never collides — no IntegrityError/rollback can strand us after a file was deleted.
spec = formats.normalize(job.profile_snapshot)
sig = formats.format_sig(spec, salt=f"{job.id}:{job.asset_id}")
asset = service.get_or_create_asset(db, job.source_kind, job.source_ref, sig)
service.populate_from_catalog(db, asset)
# Now drop the old hold — `service.release_asset` deletes the old file + row only if this job was its
# last holder (a co-held asset is left intact for its other holders).
service.release_asset(db, job)
asset.ref_count = (asset.ref_count or 0) + 1
job.asset_id = asset.id
job.status = "queued"
job.progress = 0
job.phase = None
job.error = None
job.speed_bps = None
job.eta_s = None
# Re-derive the display name (card title + Content-Disposition filename) from the freshly
# cleaned title — otherwise a name auto-set from an older, un-cleaned title (e.g. still carrying
# a "1.6M views · … |" social prefix) would survive the rebuild. The worker fills it on completion.
job.display_name = None
job.queue_pos = service.next_queue_pos(db) # tail of the queue, not its stale original slot
db.commit()
return _serialize_job(db, job) return _serialize_job(db, job)
@@ -545,10 +500,7 @@ def cancel_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db) job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict: ) -> dict:
job = _own_job(db, user, job_id) job = _own_job(db, user, job_id)
if job.status not in (JobStatus.DONE, JobStatus.CANCELED): service.cancel(db, job)
service.release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
return _serialize_job(db, job) return _serialize_job(db, job)
@@ -557,9 +509,7 @@ def delete_download(
job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db) job_id: int, user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict: ) -> dict:
job = _own_job(db, user, job_id) job = _own_job(db, user, job_id)
service.release_asset(db, job) service.delete(db, job)
db.delete(job)
db.commit()
return {"deleted": job_id} return {"deleted": job_id}
+33 -4
View File
@@ -13,6 +13,7 @@ from app.db import get_db
from app.feedquery import ( from app.feedquery import (
decode_cursor, decode_cursor,
encode_cursor, encode_cursor,
feed_summary,
filtered_query, filtered_query,
keyset_predicate, keyset_predicate,
order_by, order_by,
@@ -47,7 +48,11 @@ FINISH_MARGIN_SECONDS = 10
# Shared query parameters for /feed and /feed/count. # Shared query parameters for /feed and /feed/count.
def _feed_params( def _feed_params(
tags: list[int] = Query(default=[]), tags: list[int] = Query(default=[]),
tag_mode: str = "or", # Per-category tag match: each of language / topic / other independently ORs (any) or ANDs (all)
# its own selected tags. Defaults to "or". (Superseded the single global `tag_mode`.)
tag_mode_language: str = "or",
tag_mode_topic: str = "or",
tag_mode_other: str = "or",
channel_ids: list[str] = Query(default=[]), channel_ids: list[str] = Query(default=[]),
q: str | None = None, q: str | None = None,
min_duration: int | None = None, min_duration: int | None = None,
@@ -64,7 +69,11 @@ def _feed_params(
) -> dict: ) -> dict:
return { return {
"tags": tags, "tags": tags,
"tag_mode": tag_mode, "tag_modes": {
"language": tag_mode_language,
"topic": tag_mode_topic,
"other": tag_mode_other,
},
"channel_ids": channel_ids, "channel_ids": channel_ids,
"q": q, "q": q,
"min_duration": min_duration, "min_duration": min_duration,
@@ -127,6 +136,9 @@ def get_feed_count(
user: User = Depends(current_user), user: User = Depends(current_user),
db: Session = Depends(get_db), db: Session = Depends(get_db),
) -> dict: ) -> dict:
# SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
# oracle that test_feed_summary_matches_count_plus_facets asserts /feed/summary's `count` against;
# remove both together with that test if this endpoint is ever retired.
query, _rank = filtered_query(db, user, **params) query, _rank = filtered_query(db, user, **params)
total = db.scalar(select(func.count()).select_from(query.subquery())) total = db.scalar(select(func.count()).select_from(query.subquery()))
return {"count": total or 0} return {"count": total or 0}
@@ -146,7 +158,11 @@ def get_facets(
The count is the number of distinct channels that have at least one video in the current The count is the number of distinct channels that have at least one video in the current
view, matching the existing channel-count chip semantics. JSON object keys are strings view, matching the existing channel-count chip semantics. JSON object keys are strings
(tag ids).""" (tag ids).
SUPERSEDED by /feed/summary (PERF-13) — the app no longer calls this. Kept as the reference
oracle for test_feed_summary_matches_count_plus_facets (it asserts /feed/summary's `facets`
against these `counts`); retire the two together."""
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id) visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
counts: dict[int, int] = {} counts: dict[int, int] = {}
for category in ("language", "topic", "other"): for category in ("language", "topic", "other"):
@@ -154,7 +170,7 @@ def get_facets(
# independent counts (you can OR more of them in). Conjunctive (AND, topics only) # independent counts (you can OR more of them in). Conjunctive (AND, topics only)
# keeps them applied, so each remaining chip narrows to channels that ALSO have all # keeps them applied, so each remaining chip narrows to channels that ALSO have all
# already-selected topics — and tags that can't co-occur drop to zero (hidden). # already-selected topics — and tags that can't co-occur drop to zero (hidden).
conjunctive = category == "topic" and params.get("tag_mode") == "and" conjunctive = params.get("tag_modes", {}).get(category) == "and"
base, _rank = filtered_query( base, _rank = filtered_query(
db, db,
user, user,
@@ -174,6 +190,19 @@ def get_facets(
return {"counts": counts} return {"counts": counts}
@router.get("/feed/summary")
def get_feed_summary(
params: dict = Depends(_feed_params),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The sidebar's total video count + per-tag channel facet counts in ONE request and ONE grouped
query, replacing the separate /feed/count (1 pass) + /facets (3 passes) the FE fired on every filter
change (PERF-13). Response: {"count": <videos>, "facets": {<tag_id>: <channel_count>}} — the same
numbers /feed/count's `count` and /facets' `counts` return, so the two are interchangeable."""
return feed_summary(db, user, **params)
class VideoStateIn(BaseModel): class VideoStateIn(BaseModel):
status: str status: str
+4 -3
View File
@@ -1,6 +1,7 @@
"""Local playlists: per-user named, ordered collections of videos from the shared """Playlists: per-user named, ordered collections of videos from the shared catalog. Local
catalog. Foundation phase = local only (create / rename / delete, add / remove / reorder CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-way sync —
items). YouTube two-way sync (mirror existing playlists, push edits) lands in later phases.""" mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
from fastapi import APIRouter, Depends, HTTPException from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel from pydantic import BaseModel
from sqlalchemy import func, select, and_ from sqlalchemy import func, select, and_
+15 -1
View File
@@ -10,9 +10,23 @@ wiring expects (`plex_routes.router`), so main.py is unchanged.
from fastapi import APIRouter from fastapi import APIRouter
from app.routes.plex import admin, browse, playback from app.routes.plex import (
admin,
browse,
collections,
detail,
discover,
people,
playback,
playlists,
)
router = APIRouter() router = APIRouter()
router.include_router(admin.router) router.include_router(admin.router)
router.include_router(browse.router) router.include_router(browse.router)
router.include_router(collections.router)
router.include_router(detail.router)
router.include_router(discover.router)
router.include_router(people.router)
router.include_router(playback.router) router.include_router(playback.router)
router.include_router(playlists.router)
+21
View File
@@ -3,10 +3,12 @@
import logging import logging
from fastapi import HTTPException from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.models import ( from app.models import (
PlexItem, PlexItem,
PlexLibrary,
) )
from app.plex.lookups import by_rating_key from app.plex.lookups import by_rating_key
@@ -18,3 +20,22 @@ def item_or_404(db: Session, rating_key: str) -> PlexItem:
if it is None: if it is None:
raise HTTPException(status_code=404, detail="Unknown Plex item") raise HTTPException(status_code=404, detail="Unknown Plex item")
return it return it
def _norm_lang(lang: str | None) -> str:
"""The overlay has an EN base + an optional HU twin; anything other than `hu` serves EN."""
return "hu" if lang == "hu" else "en"
def _content_libs(db: Session) -> tuple[list[int], list[int]]:
"""The BROWSABLE content libraries, partitioned into (movie_lib_ids, show_lib_ids) — the single
source of the browse scope. Excludes HU overlay libraries (`overlay_of` set): those are never
browsed on their own, they only lend their Hungarian metadata to the EN twin's rows (E5 S5)."""
libs = db.execute(
select(PlexLibrary).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
).scalars().all()
movie_lib_ids = [lb.id for lb in libs if lb.kind == "movie"]
show_lib_ids = [lb.id for lb in libs if lb.kind == "show"]
return movie_lib_ids, show_lib_ids
File diff suppressed because it is too large Load Diff
+390
View File
@@ -0,0 +1,390 @@
"""Plex collections: browse mirrored collections + the admin manual-collection editor."""
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.auth import admin_user, current_user
from app.db import get_db
from app.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexShow,
User,
)
from app.plex import rollup
from app.plex import sync as plex_sync
from app.plex.cards import movie_card, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.plex.query import like_escape, runtime_by_key, top_art_keys
from app.routes.plex._shared import _content_libs
# The most-restrictive → least ordering used to pick a collection's effective content rating from its
# members (Plex shows one cert on the collection; we don't store it, so derive it).
_CERT_RANK = {
"G": 1, "TV-Y": 1, "TV-Y7": 1, "TV-G": 1,
"PG": 2, "TV-PG": 2,
"PG-13": 3, "TV-14": 3,
"R": 4, "TV-MA": 4,
"NC-17": 5,
}
def _collection_cert(certs: list[str | None]) -> str | None:
ranked = [(c, _CERT_RANK.get(c, 0)) for c in certs if c]
return max(ranked, key=lambda x: x[1])[0] if ranked else None
router = APIRouter(prefix="/api/plex", tags=["plex"])
_STRIP_PROVIDERS = ("imdb", "tmdb", "tvdb", "trakt")
def _collection_source(col: PlexCollection) -> str:
first = (col.title or "").strip().split(" ", 1)[0].lower()
if first in _STRIP_PROVIDERS:
return first
if col.smart:
return "smart"
return "collection"
def _collection_editable(c: PlexCollection) -> bool:
"""Effective editability: the admin marked it editable AND it's a plain manual collection — never
smart, never an external auto-list (IMDb/TMDb/…), which are managed outside Plex."""
return bool(c.editable) and not c.smart and _collection_source(c) == "collection"
def _collection_card(c: PlexCollection) -> dict:
return {
"id": c.rating_key,
"title": c.title,
"summary": c.summary,
"thumb": f"/api/plex/image/{c.rating_key}" if c.thumb_key else None,
"child_count": c.child_count,
"smart": c.smart,
"source": _collection_source(c),
"editable": c.editable,
"can_edit": _collection_editable(
c
), # whether the edit controls should be offered (admin UI)
}
@router.get("/collections")
def collections(
library: str | None = None,
q: str | None = None,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Mirrored collections as cards (most-populated first), optionally name-filtered by `q`. With a
`library` plex_key → that one library's collections (used by the collection editor). WITHOUT one →
the UNION across all enabled libraries (the unified-scope sidebar picker, which has no single
library selected)."""
if library is not None:
lib = db.execute(
select(PlexLibrary).where(PlexLibrary.plex_key == str(library))
).scalars().first()
if lib is None:
return {"collections": []}
query = select(PlexCollection).where(PlexCollection.library_id == lib.id)
else:
enabled_ids = list(
db.execute(
select(PlexLibrary.id).where(
PlexLibrary.enabled.is_(True), PlexLibrary.overlay_of.is_(None)
)
).scalars()
)
if not enabled_ids:
return {"collections": []}
query = select(PlexCollection).where(
PlexCollection.library_id.in_(enabled_ids)
)
term = (q or "").strip()
if term:
query = query.where(PlexCollection.title.ilike(f"%{like_escape(term)}%"))
rows = db.execute(
query.order_by(
PlexCollection.child_count.desc().nullslast(),
func.lower(PlexCollection.title),
)
).scalars().all()
# A montage of the top titles per tile + the collection's total runtime. Art unions MOVIE and SHOW
# members (a collection lives in one library, so exactly one side has art) — a show collection would
# otherwise render blank; the collection's own poster is the next fallback, then the colour hash.
# Runtime is movie-only (shows have no single duration).
movie_ids, show_ids = _content_libs(db)
keys = [c.rating_key for c in rows]
art_m = top_art_keys(db, PlexItem, movie_ids, PlexItem.collection_keys, keys, per=2)
art_s = top_art_keys(db, PlexShow, show_ids, PlexShow.collection_keys, keys, per=2)
runtime = runtime_by_key(db, PlexItem, movie_ids, PlexItem.collection_keys, keys)
def _art(c: PlexCollection) -> list[str]:
member = art_m.get(c.rating_key) or art_s.get(c.rating_key)
return member or ([c.rating_key] if c.thumb_key else [])
return {
"collections": [
{
**_collection_card(c),
"art": _art(c),
"total_runtime_s": runtime.get(c.rating_key, 0),
}
for c in rows
]
}
@router.get("/collections/{rating_key}")
def collection_detail(
rating_key: str,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""One collection's hero data + its members (movies + shows), ordered by release year then title —
the Collection info page. Read-only here; edits go through the admin mutation endpoints below."""
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
movies = db.execute(
select(PlexItem)
.where(PlexItem.kind == "movie", PlexItem.collection_keys.contains([rating_key]))
.order_by(PlexItem.year.asc().nullslast(), func.lower(PlexItem.title))
).scalars().all()
shows = db.execute(
select(PlexShow)
.where(PlexShow.collection_keys.contains([rating_key]))
.order_by(PlexShow.year.asc().nullslast(), func.lower(PlexShow.title))
).scalars().all()
mstate = rollup.item_state_map(db, user.id, [m.id for m in movies])
sstate = rollup.state_map(db, user.id, [s.id for s in shows])
# Merge the two type-homogeneous lists into ONE member list ordered by (year, title).
tagged = [(m.year, (m.title or "").lower(), "movie", m) for m in movies] + [
(s.year, (s.title or "").lower(), "show", s) for s in shows
]
tagged.sort(key=lambda t: (t[0] is None, t[0] or 0, t[1]))
items = [
movie_card(o, mstate.get(o.id)) if kind == "movie" else show_card(o, sstate.get(o.id, "new"))
for _, _, kind, o in tagged
]
years = [y for y, _, _, _ in tagged if y is not None]
certs = [m.content_rating for m in movies] + [s.content_rating for s in shows]
return {
**_collection_card(col),
"content_rating": _collection_cert(certs),
"year_min": min(years) if years else None,
"year_max": max(years) if years else None,
"total_runtime_s": sum(m.duration_s or 0 for m in movies),
"items": items,
}
def _editable_collection_or_403(db: Session, rating_key: str) -> PlexCollection:
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
if not _collection_editable(col):
raise HTTPException(status_code=403, detail="This collection is read-only")
return col
def _reject_duplicate_name(
db: Session, library_id: int, title: str, exclude_rk: str | None = None
) -> None:
"""No two collections in a section may share a name: Plex CONFLATES same-named collections'
membership (both rating_keys resolve to one merged member list — a real footgun), so forbid it up
front on create + rename rather than mirror the mess."""
dup = db.execute(
select(PlexCollection).where(
PlexCollection.library_id == library_id,
func.lower(PlexCollection.title) == title.lower(),
)
).scalars().first()
if dup is not None and dup.rating_key != exclude_rk:
raise HTTPException(status_code=409, detail="A collection with this name already exists")
def _write_or_pending_error(wrote: bool, e: Exception) -> HTTPException:
"""Map a PlexError raised inside a mutate-then-resync block to the honest message. If the mutating
call itself failed (``wrote`` False) the write never landed → "Plex write failed". If the write
succeeded but the follow-up resync READ (Plex metadata/children) failed, the change IS on the Plex
server → don't claim a write failure; say the local copy will reconcile on the next sync.
Both use 422, NOT a 5xx: the frontend treats 502/503/504 as "gateway unreachable" and raises a
sticky offline banner (see api/core.ts RETRIABLE_GATEWAY). A 422 surfaces the detail as an ordinary
action-error toast, which is what both of these are — the app answered, it didn't go unreachable."""
if wrote:
detail = (
"Collection updated in Plex, but refreshing the local copy failed; "
f"it will reconcile on the next sync ({e})"
)
else:
detail = f"Plex write failed: {e}"
return HTTPException(status_code=422, detail=detail)
class CreateCollectionIn(BaseModel):
library: str = ""
title: str = ""
item_rating_key: str = ""
@router.post("/collections")
def create_collection(
body: CreateCollectionIn,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Create a manual collection (seeded with one item — Plex needs at least one). Editable by default."""
library = body.library
title = body.title.strip()
seed = body.item_rating_key
if not title or not seed:
raise HTTPException(
status_code=400, detail="title and item_rating_key are required"
)
if library:
lib = db.execute(
select(PlexLibrary).where(PlexLibrary.plex_key == library)
).scalars().first()
else:
# The "New collection" flow only knows the seed title → derive its library.
seed_item = by_rating_key(db, PlexItem, seed) or by_rating_key(db, PlexShow, seed)
lib = db.get(PlexLibrary, seed_item.library_id) if seed_item else None
if lib is None:
raise HTTPException(status_code=404, detail="Unknown library")
_reject_duplicate_name(db, lib.id, title)
wrote = False
try:
with PlexClient(db) as plex:
new = plex.create_collection(lib.plex_key, title, seed)
wrote = True
col = plex_sync.resync_collection(db, plex, lib, str(new.get("ratingKey")))
col.editable = True
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
return _collection_card(col)
@router.post("/collections/{rating_key}/items/{item_rating_key}")
def collection_add_item(
rating_key: str,
item_rating_key: str,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
plex.add_collection_item(rating_key, str(item_rating_key))
wrote = True
col = plex_sync.resync_collection(db, plex, lib, rating_key)
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
return _collection_card(col)
@router.delete("/collections/{rating_key}/items/{item_rating_key}")
def collection_remove_item(
rating_key: str,
item_rating_key: str,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
plex.remove_collection_item(rating_key, str(item_rating_key))
wrote = True
col = plex_sync.resync_collection(db, plex, lib, rating_key)
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
return _collection_card(col)
class CollectionRenameIn(BaseModel):
title: str = ""
@router.patch("/collections/{rating_key}")
def collection_rename(
rating_key: str,
body: CollectionRenameIn,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
col = _editable_collection_or_403(db, rating_key)
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
_reject_duplicate_name(db, col.library_id, title, exclude_rk=rating_key)
lib = db.get(PlexLibrary, col.library_id)
wrote = False
try:
with PlexClient(db) as plex:
plex.rename_collection(lib.plex_key, rating_key, title)
wrote = True
col = plex_sync.resync_collection(db, plex, lib, rating_key)
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
return _collection_card(col)
@router.delete("/collections/{rating_key}")
def collection_delete(
rating_key: str, _: User = Depends(admin_user), db: Session = Depends(get_db)
) -> dict:
col = _editable_collection_or_403(db, rating_key)
lib = db.get(PlexLibrary, col.library_id)
try:
with PlexClient(db) as plex:
plex.delete_collection(rating_key)
plex_sync.delete_collection_local(db, lib, rating_key)
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise HTTPException(status_code=422, detail=f"Plex write failed: {e}")
return {"deleted": rating_key}
class CollectionEditableIn(BaseModel):
editable: bool = False
@router.post("/collections/{rating_key}/editable")
def collection_set_editable(
rating_key: str,
body: CollectionEditableIn,
_: User = Depends(admin_user),
db: Session = Depends(get_db),
) -> dict:
"""Mark an existing Plex collection as 'mine to edit' (or unmark). Smart / auto-list collections
(IMDb/TMDb/…) can never be made editable — they're regenerated outside Plex."""
col = by_rating_key(db, PlexCollection, rating_key)
if col is None:
raise HTTPException(status_code=404, detail="Unknown collection")
want = body.editable
if want and (col.smart or _collection_source(col) != "collection"):
raise HTTPException(
status_code=400,
detail="Smart / auto-list collections can't be made editable",
)
col.editable = want
db.commit()
return _collection_card(col)
+443
View File
@@ -0,0 +1,443 @@
"""Plex show/item detail: the show drill-down + the per-item player payload (markers, cast, nav)."""
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import get_db
from app.models import (
PlexCollection,
PlexItem,
PlexLibrary,
PlexSeason,
PlexShow,
PlexState,
User,
)
from app.plex import detail as plex_detail
from app.plex import people as plex_people
from app.plex import rollup, subtitles
from app.plex import tmdb as plex_tmdb
from app.plex.cards import episode_card, movie_card, show_card
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.lookups import by_rating_key
from app.routes.plex._shared import _norm_lang, item_or_404
from app.routes.plex.collections import _collection_source
router = APIRouter(prefix="/api/plex", tags=["plex"])
def _lib_key(db: Session, library_id: int | None) -> str | None:
"""The Plex section key for a library id — the info/show page passes it to the admin collection
editor (in the unified cross-library view there's no single library prop otherwise)."""
if library_id is None:
return None
lib = db.get(PlexLibrary, library_id)
return lib.plex_key if lib else None
_SHOW_ENRICH_TTL_S = 300
_show_enrich_cache: dict[str, tuple[float, dict, list]] = {}
_show_enrich_lock = threading.Lock()
def _show_enrichment(db: Session, rating_key: str) -> tuple[dict, list]:
"""(meta, related_raw) for a show's live Plex enrichment — served from the per-show TTL cache when
warm, otherwise fetched with the two Plex calls (metadata + related) run IN PARALLEL. Returns
({}, []) when Plex is unavailable/unconfigured (not cached, so it retries on the next open)."""
now = time.monotonic()
with _show_enrich_lock:
hit = _show_enrich_cache.get(rating_key)
if hit is not None and hit[0] > now:
return hit[1], hit[2]
try:
with PlexClient(db) as plex:
with ThreadPoolExecutor(max_workers=2) as ex:
meta_f = ex.submit(plex.metadata, rating_key)
related_f = ex.submit(
plex.related, rating_key
) # swallows its own errors → []
meta = meta_f.result() or {}
related_raw = related_f.result()
except (PlexError, PlexNotConfigured):
return {}, []
# Don't cache an EMPTY related list: plex.related() swallows a transient failure as [] (same as a
# genuinely no-related show), so caching it would serve an empty "related" section for the whole TTL
# even after Plex recovers. Skip the cache in that case (retry next open, as before); shows that DO
# have related items — the payload worth caching — still get cached.
if related_raw:
with _show_enrich_lock:
_show_enrich_cache[rating_key] = (
now + _SHOW_ENRICH_TTL_S,
meta,
related_raw,
)
return meta, related_raw
@router.get("/show/{rating_key}")
def show_detail(
rating_key: str,
lang: str = "en",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""A show's seasons + episodes (the drill-down view) with per-user watch state."""
hu = _norm_lang(lang) == "hu"
sh = by_rating_key(db, PlexShow, rating_key)
if sh is None:
raise HTTPException(status_code=404, detail="Unknown Plex show")
seasons = db.execute(
select(PlexSeason)
.where(PlexSeason.show_id == sh.id)
.order_by(PlexSeason.season_number.asc().nullsfirst())
).scalars().all()
eps = db.execute(
select(PlexItem)
.where(PlexItem.show_id == sh.id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
).scalars().all()
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user.id,
PlexState.item_id.in_([e.id for e in eps] or [0]),
)
).scalars()
}
by_season: dict[int | None, list] = {}
all_cards: list[
dict
] = [] # every episode, in season/episode order → show-level rollup
for e in eps:
card = episode_card(e, states.get(e.id))
by_season.setdefault(e.season_id, []).append(card)
all_cards.append(card)
show_roll = rollup.rollup(all_cards)
# Cast + IMDb + tagline are served from the DB row (persisted by the cast backfill, E5 S5b). The one
# remaining live call is `related` — a Plex recommendation hub we don't mirror — which also carries
# the metadata used to lazily fill a not-yet-enriched show. Any live failure just drops `related` and
# the lazy fill; the row-served detail below still renders (outage-resilient).
related: list[dict] = []
try:
meta, related_raw = _show_enrichment(db, sh.rating_key)
related = _relatedshow_cards(db, user.id, related_raw, exclude=sh.id)
if sh.people_enriched_at is None and meta:
plex_people.enrich_row_people(db, sh, plex_people.cast_from_meta(meta))
plex_detail.apply_rich_detail(sh, meta) # imdb/tagline/full_cast for future views
sh.people_enriched_at = datetime.now(timezone.utc)
try:
db.commit()
except Exception:
db.rollback()
except Exception:
# Best-effort only — the local episode list is already assembled, so any live-Plex failure
# (not configured, HTTP/JSON shape) must degrade, not 500.
pass
imdb_id = sh.imdb_id
return {
"show": {
"id": sh.rating_key,
"title": (sh.title_hu or sh.title) if hu else sh.title,
"summary": (sh.summary_hu or sh.summary) if hu else sh.summary,
"year": sh.year,
"thumb": f"/api/plex/image/{sh.rating_key}",
"art": f"/api/plex/image/{sh.rating_key}?variant=art",
"library": _lib_key(db, sh.library_id),
"content_rating": sh.content_rating,
"rating": sh.rating,
"genres": (sh.genres_hu or sh.genres if hu else sh.genres) or [],
"studio": sh.studio,
"season_count": sh.child_count,
"imdb_rating": sh.imdb_rating,
"imdb_id": imdb_id,
"imdb_url": f"https://www.imdb.com/title/{imdb_id}/" if imdb_id else None,
"cast": sh.full_cast or [],
# Show-level rollup for the hero action buttons.
"status": show_roll["status"],
"resume": show_roll["resume"],
"first": show_roll["first"],
"episode_count": show_roll["episode_count"],
"collection_keys": sh.collection_keys or [],
# TMDB enrichment (E5 S2), served from the row.
"tmdb": plex_tmdb.serve_block(db, sh),
},
"seasons": [_season_block(se, by_season.get(se.id, [])) for se in seasons],
"related": related,
}
def _season_block(se: PlexSeason, cards: list[dict]) -> dict:
"""A season's card for the show page (with its aggregate rollup) plus its episodes (used by the
season subpage, read from the same cached show-detail payload)."""
roll = rollup.rollup(cards)
return {
"id": se.rating_key,
"season_number": se.season_number,
"title": se.title
or (f"Season {se.season_number}" if se.season_number else "Season"),
"thumb": f"/api/plex/image/{se.rating_key}",
"episode_count": roll["episode_count"],
"status": roll["status"],
"resume": roll["resume"],
"first": roll["first"],
"episodes": cards,
}
def _relatedshow_cards(
db: Session, user_id: int, related: list[dict], exclude: int
) -> list[dict]:
"""Map Plex 'related' metadata to show cards for the shows we actually mirror (so they're openable),
excluding the current show. Order preserved; capped."""
rks = [str(m.get("ratingKey")) for m in related if m.get("ratingKey")]
if not rks:
return []
shows = {
s.rating_key: s
for s in db.execute(
select(PlexShow).where(
PlexShow.rating_key.in_(rks[:60]), PlexShow.id != exclude
)
).scalars()
}
statuses = rollup.state_map(db, user_id, [s.id for s in shows.values()])
out: list[dict] = []
seen: set[str] = set()
for rk in rks:
sh = shows.get(rk)
if sh is None or rk in seen:
continue
seen.add(rk)
out.append(show_card(sh, statuses.get(sh.id, "new")))
if len(out) >= 20:
break
return out
def _collection_strips(db: Session, it: PlexItem, user_id: int) -> list[dict]:
"""The collections this movie belongs to, each with its member titles (playable cards) — the
'Avatar Collection' strip on the info page. Pure local lookup (no Plex call)."""
keys = it.collection_keys or []
if not keys or it.kind != "movie":
return []
cols = {
c.rating_key: c
for c in db.execute(
select(PlexCollection).where(PlexCollection.rating_key.in_(keys))
).scalars()
}
# Smallest (most specific — e.g. a franchise) first; the big auto-lists (IMDb/TMDb Popular) last.
ordered = sorted(
(c for c in cols.values()), key=lambda c: (c.child_count or 0, c.title)
)
out = []
for col in ordered:
rk = col.rating_key
members = db.execute(
select(PlexItem)
.where(
PlexItem.library_id == it.library_id,
PlexItem.kind == "movie",
PlexItem.collection_keys.contains([rk]),
)
.order_by(PlexItem.year.asc().nullslast(), func.lower(PlexItem.title))
.limit(60)
).scalars().all()
ids = [m.id for m in members]
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user_id, PlexState.item_id.in_(ids or [0])
)
).scalars()
}
out.append(
{
"id": rk,
"title": col.title,
"source": _collection_source(col),
"items": [movie_card(m, states.get(m.id)) for m in members],
}
)
return out
def _episode_nav(db: Session, it: PlexItem) -> tuple[str | None, str | None]:
if it.kind != "episode" or it.show_id is None:
return None, None
keys = list(
db.execute(
select(PlexItem.rating_key)
.where(PlexItem.show_id == it.show_id, PlexItem.kind == "episode")
.order_by(
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
).scalars()
)
try:
i = keys.index(it.rating_key)
except ValueError:
return None, None
return (keys[i - 1] if i > 0 else None), (
keys[i + 1] if i < len(keys) - 1 else None
)
@router.get("/item/{rating_key}")
def item_detail(
rating_key: str,
lang: str = "en",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Everything the player needs: metadata, cast, intro/credit markers (from Plex), episode
prev/next, and the per-user resume position + watch status."""
lang = _norm_lang(lang)
it = item_or_404(db, rating_key)
st = db.execute(
select(PlexState).where(
PlexState.user_id == user.id, PlexState.item_id == it.id
)
).scalars().first()
# Markers + audio/subtitle streams are file-specific, so they stay LIVE; the rich detail (cast/IMDb/
# tagline/genres) is served from the DB row for movies (persisted below / by the backfill) and only
# parsed from this live meta for episodes (never enriched — not in the grid, no HU overlay).
meta: dict = {}
markers: list[dict] = []
audio_streams: list[dict] = []
subtitle_streams: list[dict] = []
try:
with PlexClient(db) as plex:
meta = plex.metadata(it.rating_key, markers=True) or {}
for m in meta.get("Marker") or []:
if m.get("type") in ("intro", "credits"):
markers.append(
{
"type": m.get("type"),
"start_s": round((m.get("startTimeOffset") or 0) / 1000, 1),
"end_s": round((m.get("endTimeOffset") or 0) / 1000, 1),
}
)
# Audio + subtitle tracks (ordinals among their stream type — used to -map the selection).
part = ((meta.get("Media") or [{}])[0].get("Part") or [{}])[0]
for s in part.get("Stream") or []:
if s.get("streamType") == 2:
audio_streams.append(
{
"ord": len(audio_streams),
"label": s.get("displayTitle")
or s.get("language")
or f"Audio {len(audio_streams) + 1}",
"language": s.get("languageTag") or s.get("language"),
"default": bool(s.get("selected") or s.get("default")),
}
)
elif s.get("streamType") == 3:
codec = (s.get("codec") or s.get("format") or "").lower()
subtitle_streams.append(
{
"ord": len(subtitle_streams),
"label": s.get("displayTitle")
or s.get("language")
or f"Subtitle {len(subtitle_streams) + 1}",
"language": s.get("languageTag") or s.get("language"),
"codec": codec,
# Text subs (srt/ass/…) can be served as WebVTT tracks; image subs (PGS/VobSub)
# can't be turned into text, so the player hides them (burn-in is a later phase).
"text": codec not in subtitles.BITMAP_SUBS,
}
)
except Exception:
pass # live extras (markers/streams) are best-effort; any live-Plex failure must degrade, not 500 — playback works without them
# Lazy enrichment (movies only): the live fetch above carries the FULL cast + rich detail (the sync
# only stored the listing's top ~3), so fill a not-yet-enriched movie — its cast becomes filterable
# and future views serve from the row. Episodes aren't enriched; a show's cast is filled from /show.
if it.kind == "movie" and it.people_enriched_at is None and meta:
cast = plex_people.cast_from_meta(meta)
if cast:
plex_people.enrich_row_people(db, it, cast)
plex_detail.apply_rich_detail(it, meta)
it.people_enriched_at = datetime.now(timezone.utc)
try:
db.commit()
except Exception:
db.rollback()
hu = lang == "hu"
# Movies serve rich detail from the DB row; episodes from the live meta just fetched.
if it.kind == "movie":
_imdb_id = it.imdb_id
rich = {
"cast": it.full_cast or [],
"imdb_rating": it.imdb_rating,
"imdb_id": _imdb_id,
"imdb_url": f"https://www.imdb.com/title/{_imdb_id}/" if _imdb_id else None,
"content_rating": it.content_rating,
"genres": (it.genres_hu or it.genres if hu else it.genres) or [],
"directors": it.directors or [],
"studio": it.studio,
"tagline": it.tagline,
}
else:
rich = plex_detail.rich_detail(meta)
prev_id, next_id = _episode_nav(db, it)
show = db.get(PlexShow, it.show_id) if it.kind == "episode" and it.show_id else None
return {
"id": it.rating_key,
"kind": it.kind,
"title": (it.title_hu or it.title) if hu else it.title,
"summary": (it.summary_hu or it.summary) if hu else it.summary,
"year": it.year,
"duration_seconds": it.duration_s,
"playable": it.playable,
"thumb": f"/api/plex/image/{it.rating_key}",
"art": f"/api/plex/image/{it.rating_key}?variant=art",
"library": _lib_key(db, it.library_id),
"cast": rich.get("cast", []),
"imdb_rating": rich.get("imdb_rating"),
"imdb_id": rich.get("imdb_id"),
"imdb_url": rich.get("imdb_url"),
# Audience score (Plex audienceRating) served alongside the IMDb score — the UI shows the
# IMDb badge when imdb_rating is present, else this. Episodes carry it on the row too.
"rating": it.rating,
"content_rating": rich.get("content_rating"),
"genres": rich.get("genres", []),
"directors": rich.get("directors", []),
"studio": rich.get("studio"),
"tagline": rich.get("tagline"),
"markers": markers,
"audio_streams": audio_streams,
"subtitle_streams": subtitle_streams,
"status": st.status if st else "new",
"position_seconds": st.position_seconds if st else 0,
"show_title": (
(show.title_hu or show.title if hu else show.title) if show else None
),
"season_number": it.season_number,
"episode_number": it.episode_number,
"prev_id": prev_id,
"next_id": next_id,
"collections": _collection_strips(db, it, user.id),
# TMDB enrichment (E5 S2), served from the row — movies only (episodes aren't enriched).
"tmdb": plex_tmdb.serve_block(db, it) if it.kind == "movie" else None,
}
+421
View File
@@ -0,0 +1,421 @@
"""Plex Discover: the rail-surface read-model (E5 S4).
A consolidated payload for the dedicated Discover landing — a stack of horizontal rails:
Continue watching · Recently added · Browse-by-genre tiles · Because you watched X — plus a
separate, reshuffling Surprise-me sample. Serve-from-DB only: the recommendation rail reads the
recs the S2 TMDB engine already persisted (`PlexItem.recommendations`), never a live TMDB call, and
every card comes from the same `movie_card`/`show_card` serializers the grid uses (so rails inherit
S1's `art` + S3's rating/summary for free). The rails carry EN metadata (like the other non-grid
card paths); the HU overlay coalesces only in the main grid.
The sidebar metadata filters (genre / rating / year / collection / …) narrow EVERY rail too, so the
Discover surface behaves like the Browse grid — the same `LibraryFilters` / `meta_filter_conds` the
grid uses, applied to each rail's movie + show query (a rail may legitimately come back empty, which
the client hides). Genre-language matches the grid; duration is movie-only.
"""
from fastapi import APIRouter, Depends, Query
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import get_db
from app.models import PlexItem, PlexShow, PlexState, User
from app.plex import rollup
from app.plex.cards import leaf_card, movie_card, show_card
from app.plex.query import (
LibraryFilters,
apply_meta_filters,
apply_show_state_filter,
genres_col,
meta_filter_conds,
top_art_keys,
)
from app.plex.tmdb import resolve_recs
from app.routes.plex._shared import _content_libs, _norm_lang
router = APIRouter(prefix="/api/plex", tags=["plex"])
RAIL_LIMIT = 20
GENRE_TILES = 12
SURPRISE_PER_KIND = 3 # Surprise-me: at most this many movies + this many shows (6 total)
def _scope_libs(db: Session, scope: str) -> tuple[list[int], list[int]]:
"""The movie/show library ids visible for this scope (empty list = out of scope)."""
movie_ids, show_ids = _content_libs(db)
return (
movie_ids if scope in ("movie", "both") else [],
show_ids if scope in ("show", "both") else [],
)
def _movie_meta(q, p: dict, lang: str, f: LibraryFilters):
"""Apply the shared metadata filters + movie-only duration to a movie query (same as the grid)."""
q = apply_meta_filters(q, PlexItem, p, lang)
if f.duration_min is not None:
q = q.where(PlexItem.duration_s >= f.duration_min)
if f.duration_max is not None:
q = q.where(PlexItem.duration_s <= f.duration_max)
return q
def _movie_cards(db: Session, uid: int, items: list[PlexItem]) -> list[dict]:
sts = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, sts.get(it.id)) for it in items]
def _show_cards(db: Session, uid: int, shows: list[PlexShow]) -> list[dict]:
statuses = rollup.state_map(db, uid, [s.id for s in shows])
return [show_card(s, statuses.get(s.id, "new")) for s in shows]
def _recently_added(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], p: dict, lang: str, f: LibraryFilters
) -> list[dict]:
"""Newest movies + shows, interleaved by `added_at` desc into one rail."""
movies = (
db.execute(
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie", PlexItem.library_id.in_(movie_ids)
),
p, lang, f,
)
.order_by(PlexItem.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
).scalars().all()
if movie_ids
else []
)
shows = (
db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.library_id.in_(show_ids)), PlexShow, p, lang
)
.order_by(PlexShow.added_at.desc().nullslast())
.limit(RAIL_LIMIT)
).scalars().all()
if show_ids
else []
)
# Merge the two type-homogeneous lists by added_at, then serialize each side in one batch so the
# per-card state lookups stay batched (no N+1). The `(is not None, value)` key sorts a missing
# timestamp last under reverse.
tagged = [(m.added_at, "movie", m) for m in movies] + [
(s.added_at, "show", s) for s in shows
]
tagged.sort(key=lambda t: (t[0] is not None, t[0]), reverse=True)
top = tagged[:RAIL_LIMIT]
# Key the serialized cards back by rating_key (globally unique in Plex — a movie and a show never
# share one), so the merge doesn't depend on object identity staying alive.
top_movies = [o for _, kind, o in top if kind == "movie"]
top_shows = [o for _, kind, o in top if kind == "show"]
mcards = {m.rating_key: c for m, c in zip(top_movies, _movie_cards(db, uid, top_movies))}
scards = {s.rating_key: c for s, c in zip(top_shows, _show_cards(db, uid, top_shows))}
return [
mcards[o.rating_key] if kind == "movie" else scards[o.rating_key]
for _, kind, o in top
]
def _recently_watched(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], p: dict, lang: str, f: LibraryFilters
) -> list[dict]:
"""Recently watched rail: the titles the user last FULLY watched, newest first. A movie is its own
card; a show contributes the specific EPISODE watched (carrying its show title). Movies honour the
sidebar filters (like the other rails); episodes carry no filterable metadata, so they don't."""
rows: list[tuple[object, dict]] = [] # (watched_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
)
)
for it, st in db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all():
rows.append((st.watched_at, leaf_card(it, st)))
if show_ids:
eq = (
select(PlexItem, PlexState, PlexShow.title, PlexShow.rating_key)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.join(PlexShow, PlexShow.id == PlexItem.show_id)
.where(
PlexItem.kind == "episode",
PlexItem.library_id.in_(show_ids),
PlexState.status == "watched",
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(RAIL_LIMIT)
)
for ep, st, sh_title, sh_rk in db.execute(eq).all():
rows.append((st.watched_at, leaf_card(ep, st, show_title=sh_title, show_rk=sh_rk)))
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _continue_watching(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], p: dict, lang: str, f: LibraryFilters
) -> list[dict]:
"""Resume rail: in-progress movies + partially-watched shows, most-recently-progressed first.
A movie's resume time is its own `progress_updated_at`; a show's is the max across its episodes."""
rows: list[tuple[object, dict]] = [] # (resumed_at, card)
if movie_ids:
mq = (
select(PlexItem, PlexState)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.position_seconds > 0,
# A NULL status is still "started" (mirrors apply_movie_state_filter's in_progress) —
# `notin_` alone would drop it, since `NULL NOT IN (...)` is NULL, not TRUE.
or_(
PlexState.status.is_(None),
PlexState.status.notin_(["watched", "hidden"]),
),
)
)
mrows = db.execute(
_movie_meta(mq, p, lang, f)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for it, st in mrows:
rows.append((st.progress_updated_at, movie_card(it, st)))
if show_ids:
# Reuse the shared per-show episode rollup (total/watched/inprog) rather than re-inlining it, plus
# its `resumed_at` = max(progress_updated_at); the in-progress predicate comes from the same shared
# `apply_show_state_filter` the grid uses, so the rail can't drift from the grid's definition.
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at).join(prog, prog.c.show_id == PlexShow.id),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_ids))
srows = db.execute(
apply_meta_filters(sq, PlexShow, p, lang)
.order_by(prog.c.resumed_at.desc().nullslast())
.limit(RAIL_LIMIT)
).all()
for sh, resumed_at in srows:
rows.append((resumed_at, show_card(sh, "in_progress")))
rows.sort(key=lambda r: (r[0] is not None, r[0]), reverse=True)
return [card for _, card in rows[:RAIL_LIMIT]]
def _genre_tiles(
db: Session, movie_ids: list[int], show_ids: list[int], p: dict, lang: str
) -> list[dict]:
"""The most-common genres across the scope, as {value, count} tiles → tap to filter the grid.
The genre vocabulary is language-specific (coalesce(genres_hu, genres) in HU), matching the grid.
The active metadata filters narrow the counts too, so the tiles agree with the filtered rails."""
counts: dict[str, int] = {}
for model, ids in ((PlexItem, movie_ids), (PlexShow, show_ids)):
if not ids:
continue
gcol = genres_col(model, lang)
base = select(func.jsonb_array_elements_text(gcol).label("g")).where(
model.library_id.in_(ids),
func.jsonb_typeof(gcol) == "array",
*meta_filter_conds(model, p, lang),
)
if model is PlexItem:
base = base.where(model.kind == "movie")
expanded = base.subquery()
for g, c in db.execute(
select(expanded.c.g, func.count()).group_by(expanded.c.g)
).all():
counts[g] = counts.get(g, 0) + c
top = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:GENRE_TILES]
# A 2×2 poster collage per tile: the top movies in that genre (color hash is the client fallback).
art = top_art_keys(
db,
PlexItem,
movie_ids,
genres_col(PlexItem, lang),
[g for g, _ in top],
extra_conds=(PlexItem.kind == "movie", *meta_filter_conds(PlexItem, p, lang)),
)
return [{"value": g, "count": c, "art": art.get(g, [])} for g, c in top]
def _because_you_watched(
db: Session, uid: int, movie_ids: list[int], show_ids: list[int], p: dict, lang: str, f: LibraryFilters
) -> dict | None:
"""Recommendations anchored on the user's most-recently-watched, TMDB-enriched movie. Only the
recs that resolve to OUR library (by guid) become cards — the rest aren't actionable here. Returns
None (rail hidden) when there's no enriched watch history or none of its recs are in the library.
The metadata filters narrow the resolved recs (not the anchor pick) so the rail matches the grid."""
if not movie_ids:
return None
anchor = db.execute(
select(PlexItem)
.join(
PlexState,
and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid),
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
PlexState.status == "watched",
PlexItem.recommendations.isnot(None),
)
.order_by(PlexState.watched_at.desc().nullslast())
.limit(1)
).scalars().first()
if anchor is None:
return None
resolved = resolve_recs(db, anchor.recommendations)
# Preserve TMDB's recommendation order, keep only in-library hits, and drop the anchor itself.
rks = [
r["rating_key"]
for r in resolved
if r.get("in_library") and r.get("rating_key") != anchor.rating_key
]
if not rks:
return None
movies = {
m.rating_key: m
for m in db.execute(
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie", PlexItem.rating_key.in_(rks)
),
p, lang, f,
)
).scalars().all()
}
shows = {
s.rating_key: s
for s in db.execute(
apply_meta_filters(
select(PlexShow).where(PlexShow.rating_key.in_(rks)), PlexShow, p, lang
)
).scalars().all()
}
ordered_movies = [movies[rk] for rk in rks if rk in movies]
ordered_shows = [shows[rk] for rk in rks if rk in shows]
# rating_key is globally unique in Plex, so it keys the serialized cards back safely (no id()).
mcards = {m.rating_key: c for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies))}
scards = {s.rating_key: c for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows))}
items: list[dict] = []
for rk in rks:
if rk in mcards:
items.append(mcards[rk])
elif rk in scards:
items.append(scards[rk])
if not items:
return None
return {"anchor": {"id": anchor.rating_key, "title": anchor.title}, "items": items}
@router.get("/discover")
def discover(
scope: str = "both",
lang: str = "en",
f: LibraryFilters = Depends(),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The Discover landing payload — every rail in one pass (serve-from-DB), narrowed by the active
sidebar filters. Empty rails come back as empty lists (the client hides them); `because_you_watched`
is null when there's no anchor."""
lang = _norm_lang(lang)
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
return {
"scope": scope,
"continue_watching": _continue_watching(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_watched": _recently_watched(db, user.id, movie_ids, show_ids, p, lang, f),
"recently_added": _recently_added(db, user.id, movie_ids, show_ids, p, lang, f),
"genres": _genre_tiles(db, movie_ids, show_ids, p, lang),
"because_you_watched": _because_you_watched(db, user.id, movie_ids, show_ids, p, lang, f),
}
@router.get("/surprise")
def surprise(
scope: str = "both",
lang: str = "en",
limit: int = Query(default=2 * SURPRISE_PER_KIND, ge=1, le=40),
f: LibraryFilters = Depends(),
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""A fresh random sample for the reshuffling Surprise-me rail (a random draw each call), narrowed by
the active filters. Only titles the user has NOT seen — a watched movie / a fully-watched show is
excluded (the point is to surface something to watch). Capped at SURPRISE_PER_KIND movies +
SURPRISE_PER_KIND shows, drawn separately then interleaved so a lopsided library doesn't crowd out
one type."""
lang = _norm_lang(lang)
p = f.as_dict()
movie_ids, show_ids = _scope_libs(db, scope)
per = min(limit, SURPRISE_PER_KIND)
movies = []
if movie_ids:
# Exclude movies this user has watched.
watched_movie = (
select(PlexState.id)
.where(
PlexState.item_id == PlexItem.id,
PlexState.user_id == user.id,
PlexState.status == "watched",
)
.exists()
)
movies = db.execute(
_movie_meta(
select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_ids),
~watched_movie,
),
p, lang, f,
)
.order_by(func.random())
.limit(per)
).scalars().all()
shows = []
if show_ids:
# Exclude fully-watched shows (LEFT JOIN so a never-started show — no rollup row — still counts
# as unseen). `total IS NULL / == 0 / watched < total` == "not fully watched".
prog = rollup.agg_subquery(db, user.id).subquery()
not_fully_watched = or_(
prog.c.total.is_(None), prog.c.total == 0, prog.c.watched < prog.c.total
)
shows = db.execute(
apply_meta_filters(
select(PlexShow)
.outerjoin(prog, prog.c.show_id == PlexShow.id)
.where(PlexShow.library_id.in_(show_ids), not_fully_watched),
PlexShow, p, lang,
)
.order_by(func.random())
.limit(per)
).scalars().all()
mcards = _movie_cards(db, user.id, movies)
scards = _show_cards(db, user.id, shows)
# Interleave movie/show cards, then trim to `limit`.
items: list[dict] = []
for i in range(max(len(mcards), len(scards))):
if i < len(mcards):
items.append(mcards[i])
if i < len(scards):
items.append(scards[i])
return {"scope": scope, "items": items[:limit]}
+242
View File
@@ -0,0 +1,242 @@
"""Plex people: cast/crew filter-chip cards + a single person's filmography page."""
import logging
import threading
import time
from fastapi import APIRouter, BackgroundTasks, Depends
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import SessionLocal, get_db
from app.models import (
PlexItem,
PlexPerson,
PlexShow,
User,
)
from app.plex import rollup
from app.plex import tmdb as plex_tmdb
from app.plex.cards import movie_card, show_card
from app.plex.query import split_csv
from app.routes.plex._shared import _content_libs
router = APIRouter(prefix="/api/plex", tags=["plex"])
log = logging.getLogger("siftlode.plex")
def _resolve_known_for(
db: Session,
known_for: list | None,
uid: int,
movie_lib_ids: list[int],
show_lib_ids: list[int],
) -> list[dict]:
"""The person's TMDB "Known for" credits (top by popularity) that are IN our library, as cards, in
popularity order. Matched by the canonical guid `tmdb:<id>`; out-of-library credits are dropped."""
if not known_for:
return []
# Match each credit to a library row by guid, bucketed by media_type (movie ids and tv ids collide
# numerically) and scoped to the enabled libraries — the shared helper owns that collision-safe
# logic (the detail-page recs use the same one).
movies, shows = plex_tmdb.resolve_library_guids(
db, known_for, movie_lib_ids=movie_lib_ids, show_lib_ids=show_lib_ids
)
m_states = rollup.item_state_map(db, uid, [m.id for m in movies.values()])
show_statuses = rollup.state_map(db, uid, [s.id for s in shows.values()])
out: list[dict] = []
for k in known_for:
g = f"tmdb:{k.get('tmdb_id')}"
# A credit resolves against its OWN media_type's bucket only.
if k.get("media_type") == "tv":
if g in shows:
out.append(show_card(shows[g], show_statuses.get(shows[g].id, "new")))
elif g in movies:
out.append(movie_card(movies[g], m_states.get(movies[g].id)))
return out
@router.get("/people/cards")
def people_cards(
names: str,
scope: str = "both",
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Cards for the actor filter-chips currently applied: each name + its cached headshot + how many
movies/shows (in the given scope) feature them. Photos come from the plex_people cache (populated
as items are rich-fetched — info-page view + the rich re-sync); counts are live against cast_names.
A name with no cached photo yet still returns a card (the UI shows a placeholder)."""
wanted = split_csv(names)[:16]
if not wanted:
return {"people": []}
movie_lib_ids, show_lib_ids = _content_libs(db)
want_movies = scope in ("movie", "both") and bool(movie_lib_ids)
want_shows = scope in ("show", "both") and bool(show_lib_ids)
photos = {
p.name: p.photo_thumb
for p in db.execute(
select(PlexPerson).where(PlexPerson.name.in_(wanted))
).scalars()
}
out = []
for nm in wanted:
count = 0
if want_movies:
count += (
db.scalar(
select(func.count(PlexItem.id)).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_lib_ids),
PlexItem.cast_names.contains([nm]),
)
)
or 0
)
if want_shows:
count += (
db.scalar(
select(func.count(PlexShow.id)).where(
PlexShow.library_id.in_(show_lib_ids),
PlexShow.cast_names.contains([nm]),
)
)
or 0
)
out.append({"name": nm, "photo": photos.get(nm), "count": count})
return {"people": out}
# Cap each filmography section — a prolific person shouldn't return an unbounded list the client then
# mounts a card per row for (the grid paginates at PAGE; this page renders all at once). Ample for a
# personal library; the most-recent titles win via the year-desc order.
_PERSON_SECTION_MAX = 200
# A person's bio is enriched at most once per this window: it dedups concurrent opens of the same person
# (each would otherwise schedule its own 8s TMDB task) AND rate-limits re-triggering when TMDB is
# down/slow (a transient failure leaves tmdb_enriched_at NULL, so without this every reload would spawn a
# fresh task and pile up on the shared threadpool). In-process + best-effort: a restart just clears it, at
# worst costing a few extra fetches. A successful enrich stamps the watermark, so the reservation only
# ever expires harmlessly.
_ENRICH_COOLDOWN_S = 60.0
_enrich_inflight: dict[int, float] = {}
_enrich_lock = threading.Lock()
def _should_schedule_enrich(person_id: int) -> bool:
"""True (and reserves a cooldown) if this person isn't already being enriched / recently attempted."""
now = time.monotonic()
with _enrich_lock:
deadline = _enrich_inflight.get(person_id)
if deadline is not None and now < deadline:
return False
_enrich_inflight[person_id] = now + _ENRICH_COOLDOWN_S
# Bound the map: drop expired reservations once it grows (a large library has many people).
if len(_enrich_inflight) > 1024:
for pid in [p for p, d in _enrich_inflight.items() if d <= now]:
del _enrich_inflight[pid]
return True
def _enrich_person_bg(person_id: int) -> None:
"""Fetch + persist one person's TMDB bio OFF the request path — a FastAPI background task that runs
AFTER the response is sent, so the outbound TMDB call never sits on the serve latency (the module's
"no live TMDB call at serve time" invariant). The request's session is already closed by then, so
open a fresh one. Best-effort: a TMDB failure just leaves `tmdb_enriched_at` NULL to retry on the
next open, and the bio lands from the row by then."""
db = SessionLocal()
try:
person = db.get(PlexPerson, person_id)
if person is not None:
plex_tmdb.enrich_person(db, person)
except Exception: # noqa: BLE001 — background best-effort; never surfaces to a user
db.rollback()
log.warning("background person bio enrich failed for id=%s", person_id, exc_info=True)
finally:
db.close()
@router.get("/person/{name}")
def person_detail(
name: str,
background_tasks: BackgroundTasks,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""One cast/crew member's filmography: their cached headshot + the movies and shows (in the enabled
libraries) whose cast OR directors include this name — the inverse of the cast-name → library-filter
link. The name string is the identity key (Plex cast carries no stable per-person id), so distinct
people who share a name share a page. Each section is capped at _PERSON_SECTION_MAX. An unknown name
simply returns empty sections.
E5 S6: the first time a page is opened, the person's TMDB bio + "Known for" credits are lazily
fetched (from the S2-linked `tmdb_person_id`) and persisted, then served from the row on every open.
The fetch runs OFF the request path (a background task); the first open serves whatever's in the row
(bio possibly still null) and the bio lands from the row by the next open.
"""
person = db.execute(select(PlexPerson).where(PlexPerson.name == name)).scalars().first()
# Lazy one-time bio enrich, scheduled AFTER the response so the TMDB call is never on the serve path.
# Gated by a per-person cooldown so concurrent/rapid opens (or repeated opens while TMDB is degraded)
# can't pile background tasks onto the shared threadpool.
if (
person is not None
and person.tmdb_person_id
and person.tmdb_enriched_at is None
and _should_schedule_enrich(person.id)
):
background_tasks.add_task(_enrich_person_bg, person.id)
movie_lib_ids, show_lib_ids = _content_libs(db)
movies: list[dict] = []
if movie_lib_ids:
items = db.execute(
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_lib_ids),
or_(
PlexItem.cast_names.contains([name]),
PlexItem.directors.contains([name]),
),
)
.order_by(PlexItem.year.desc().nullslast(), func.lower(PlexItem.title))
.limit(_PERSON_SECTION_MAX)
).scalars().all()
states = rollup.item_state_map(db, user.id, [it.id for it in items])
movies = [movie_card(it, states.get(it.id)) for it in items]
shows: list[dict] = []
if show_lib_ids:
show_rows = db.execute(
select(PlexShow)
.where(
PlexShow.library_id.in_(show_lib_ids),
or_(
PlexShow.cast_names.contains([name]),
PlexShow.directors.contains([name]),
),
)
.order_by(PlexShow.year.desc().nullslast(), func.lower(PlexShow.title))
.limit(_PERSON_SECTION_MAX)
).scalars().all()
statuses = rollup.state_map(db, user.id, [s.id for s in show_rows])
shows = [show_card(sh, statuses.get(sh.id, "new")) for sh in show_rows]
known_for = _resolve_known_for(
db, person.known_for if person else None, user.id, movie_lib_ids, show_lib_ids
)
return {
"name": name,
"photo": person.photo_thumb if person else None,
"biography": person.biography if person else None,
"birthday": person.birthday if person else None,
"deathday": person.deathday if person else None,
"place_of_birth": person.place_of_birth if person else None,
"known_for_department": person.known_for_department if person else None,
"also_known_as": (person.also_known_as if person else None) or [],
"known_for": known_for,
"movies": movies,
"shows": shows,
}
+405
View File
@@ -0,0 +1,405 @@
"""Plex playlists: the current user's personal, orderable playlists of playable items."""
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import delete, func, select, update
from sqlalchemy.orm import Session
from app.auth import current_user
from app.db import get_db
from app.models import (
PlexItem,
PlexPlaylist,
PlexPlaylistItem,
PlexShow,
PlexState,
User,
)
from app.plex.cards import leaf_card
from app.plex.lookups import by_rating_key
from app.routes._common import owned_or_404
router = APIRouter(prefix="/api/plex", tags=["plex"])
def _own_playlist_or_404(db: Session, pid: int, user: User) -> PlexPlaylist:
return owned_or_404(db, PlexPlaylist, pid, user, detail="Playlist not found")
def _playlist_card(pl: PlexPlaylist, count: int, thumb_rk: str | None) -> dict:
return {
"id": pl.id,
"title": pl.title,
"item_count": count,
"thumb": f"/api/plex/image/{thumb_rk}" if thumb_rk else None,
}
def _playlist_item_query(pid: int):
return (
select(PlexPlaylistItem, PlexItem)
.join(PlexItem, PlexItem.id == PlexPlaylistItem.item_id)
.where(PlexPlaylistItem.playlist_id == pid)
.order_by(PlexPlaylistItem.position, PlexPlaylistItem.id)
)
@router.get("/playlists")
def playlists(
contains: str | None = None,
contains_group: str | None = None,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The current user's playlists (newest-touched first), each with an item count + a cover thumb.
`contains`=an item rating_key adds `has_item` per playlist (for the single 'add to playlist' dialog).
`contains_group`=a comma-separated list of rating_keys adds `group_in` per playlist (how many of the
group it already holds) + a top-level `group_size`, for the 'add a whole season/show' dialog."""
contains_id = None
if contains:
contains_id = db.scalar(
select(PlexItem.id).where(PlexItem.rating_key == str(contains))
)
group_ids: set[int] | None = None
if contains_group is not None:
rks = [s for s in (contains_group or "").split(",") if s]
group_ids = set(
db.execute(
select(PlexItem.id).where(PlexItem.rating_key.in_(rks or [""]))
).scalars()
)
out = []
for pl in db.execute(
select(PlexPlaylist)
.where(PlexPlaylist.user_id == user.id)
.order_by(PlexPlaylist.updated_at.desc())
).scalars():
count = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pl.id
)
)
or 0
)
first = db.execute(_playlist_item_query(pl.id)).first()
card = _playlist_card(pl, count, first[1].rating_key if first else None)
if contains_id is not None:
card["has_item"] = (
db.execute(
select(PlexPlaylistItem.id).where(
PlexPlaylistItem.playlist_id == pl.id,
PlexPlaylistItem.item_id == contains_id,
)
).first()
is not None
)
if group_ids is not None:
card["group_in"] = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pl.id,
PlexPlaylistItem.item_id.in_(group_ids or [0]),
)
)
or 0
)
out.append(card)
resp: dict = {"playlists": out}
if group_ids is not None:
resp["group_size"] = len(group_ids)
return resp
class CreatePlexPlaylistIn(BaseModel):
title: str = ""
item_rating_key: str = ""
@router.post("/playlists")
def create_playlist(
body: CreatePlexPlaylistIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
pl = PlexPlaylist(user_id=user.id, title=title)
db.add(pl)
db.flush()
seed = body.item_rating_key
count = 0
thumb_rk = None
if seed:
it = by_rating_key(db, PlexItem, seed)
if it is not None:
db.add(PlexPlaylistItem(playlist_id=pl.id, item_id=it.id, position=0))
count, thumb_rk = 1, it.rating_key
db.commit()
return _playlist_card(pl, count, thumb_rk)
@router.get("/playlists/{pid}")
def playlist_detail(
pid: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
rows = db.execute(_playlist_item_query(pid)).all()
items = [it for (_pi, it) in rows]
states = {
s.item_id: s
for s in db.execute(
select(PlexState).where(
PlexState.user_id == user.id,
PlexState.item_id.in_([it.id for it in items] or [0]),
)
).scalars()
}
shows = {
sh.id: sh
for sh in db.execute(
select(PlexShow).where(
PlexShow.id.in_([it.show_id for it in items if it.show_id] or [0])
)
).scalars()
}
cards = [
leaf_card(
it,
states.get(it.id),
shows[it.show_id].title if it.show_id in shows else None,
shows[it.show_id].rating_key if it.show_id in shows else None,
)
for it in items
]
return {"id": pl.id, "title": pl.title, "items": cards}
class RenamePlexPlaylistIn(BaseModel):
title: str = ""
@router.patch("/playlists/{pid}")
def rename_playlist(
pid: int,
body: RenamePlexPlaylistIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
title = body.title.strip()
if not title:
raise HTTPException(status_code=400, detail="title is required")
pl.title = title
db.commit()
count = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pl.id
)
)
or 0
)
first = db.execute(_playlist_item_query(pl.id)).first()
return _playlist_card(pl, count, first[1].rating_key if first else None)
@router.delete("/playlists/{pid}")
def delete_playlist(
pid: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
db.delete(pl)
db.commit()
return {"deleted": pid}
class PlexItemRefIn(BaseModel):
item_rating_key: str = ""
@router.post("/playlists/{pid}/items")
def playlist_add_item(
pid: int,
body: PlexItemRefIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
it = by_rating_key(db, PlexItem, body.item_rating_key)
if it is None:
raise HTTPException(status_code=404, detail="Unknown item")
exists = db.execute(
select(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id == it.id
)
).scalars().first()
if exists is None:
maxpos = db.scalar(
select(func.coalesce(func.max(PlexPlaylistItem.position), -1)).where(
PlexPlaylistItem.playlist_id == pid
)
)
db.add(
PlexPlaylistItem(playlist_id=pid, item_id=it.id, position=int(maxpos) + 1)
)
pl.updated_at = datetime.now(timezone.utc)
db.commit()
return {"ok": True}
@router.delete("/playlists/{pid}/items/{item_rating_key}")
def playlist_remove_item(
pid: int,
item_rating_key: str,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist_or_404(db, pid, user)
it = by_rating_key(db, PlexItem, item_rating_key)
if it is not None:
db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id == it.id
)
)
pl.updated_at = datetime.now(timezone.utc)
db.commit()
return {"ok": True}
class PlexItemKeysIn(BaseModel):
item_rating_keys: list[str] = []
def _items_by_rks(db: Session, rks: list) -> list[PlexItem]:
"""Resolve rating_keys to PlexItems, preserving the input order (drops unknown keys)."""
order = [str(k) for k in rks if str(k)]
if not order:
return []
by_rk = {
it.rating_key: it
for it in db.execute(
select(PlexItem).where(PlexItem.rating_key.in_(order))
).scalars()
}
seen: set[str] = set()
out: list[PlexItem] = []
for rk in order:
it = by_rk.get(rk)
if it is not None and rk not in seen:
seen.add(rk)
out.append(it)
return out
@router.post("/playlists/{pid}/items/bulk")
def playlist_add_bulk(
pid: int,
body: PlexItemKeysIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Append many items at once (a whole season/show). Already-present items are skipped; the input
order is preserved for the newly-added ones. Returns how many were added + the new total."""
pl = _own_playlist_or_404(db, pid, user)
items = _items_by_rks(db, body.item_rating_keys)
existing = set(
db.execute(
select(PlexPlaylistItem.item_id).where(PlexPlaylistItem.playlist_id == pid)
).scalars()
)
pos = int(
db.scalar(
select(func.coalesce(func.max(PlexPlaylistItem.position), -1)).where(
PlexPlaylistItem.playlist_id == pid
)
)
)
added = 0
for it in items:
if it.id in existing:
continue
pos += 1
db.add(PlexPlaylistItem(playlist_id=pid, item_id=it.id, position=pos))
existing.add(it.id)
added += 1
if added:
pl.updated_at = datetime.now(timezone.utc)
db.commit()
total = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pid
)
)
or 0
)
return {"added": added, "item_count": total}
@router.post("/playlists/{pid}/items/remove-bulk")
def playlist_remove_bulk(
pid: int,
body: PlexItemKeysIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Remove many items at once (a whole season/show, or a multi-select). Returns how many were
removed + the new total."""
pl = _own_playlist_or_404(db, pid, user)
ids = [it.id for it in _items_by_rks(db, body.item_rating_keys)]
removed = 0
if ids:
removed = db.execute(
delete(PlexPlaylistItem).where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id.in_(ids)
),
execution_options={"synchronize_session": False},
).rowcount
if removed:
pl.updated_at = datetime.now(timezone.utc)
db.commit()
total = (
db.scalar(
select(func.count(PlexPlaylistItem.id)).where(
PlexPlaylistItem.playlist_id == pid
)
)
or 0
)
return {"removed": removed, "item_count": total}
@router.put("/playlists/{pid}/order")
def reorder_playlist(
pid: int,
body: PlexItemKeysIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Set the playlist order from an explicit list of item rating_keys (0-based positions)."""
pl = _own_playlist_or_404(db, pid, user)
order = body.item_rating_keys
id_by_rk = {
it.rating_key: it.id
for it in db.execute(
select(PlexItem).where(PlexItem.rating_key.in_(order or [""]))
).scalars()
}
for pos, rk in enumerate(order):
iid = id_by_rk.get(rk)
if iid is not None:
db.execute(
update(PlexPlaylistItem)
.where(
PlexPlaylistItem.playlist_id == pid, PlexPlaylistItem.item_id == iid
)
.values(position=pos)
)
pl.updated_at = datetime.now(timezone.utc)
db.commit()
return {"ok": True}
+31 -9
View File
@@ -9,11 +9,19 @@ from app.db import get_db
from app.models import SavedView, User from app.models import SavedView, User
from app.routes._common import owned_or_404 from app.routes._common import owned_or_404
# Saved "smart views": per-user named snapshots of the feed's FeedFilters. Gated behind # Saved "smart views": per-user named snapshots of a module's filters (feed or plex, keyed by
# require_human so the shared demo account can't pollute a communal list (mirrors how other # `source`). Gated behind require_human so the shared demo account can't pollute a communal list
# personal-state writes are scoped). # (mirrors how other personal-state writes are scoped).
router = APIRouter(prefix="/api/saved-views", tags=["saved-views"]) router = APIRouter(prefix="/api/saved-views", tags=["saved-views"])
# The modules that own saved views. A view is scoped to exactly one; the default is "feed" so a caller
# (or an older client) that omits `source` keeps the original feed behaviour.
_SOURCES = {"feed", "plex"}
def _norm_source(source: str) -> str:
return source if source in _SOURCES else "feed"
def _serialize(v: SavedView) -> dict: def _serialize(v: SavedView) -> dict:
return { return {
@@ -27,12 +35,15 @@ def _serialize(v: SavedView) -> dict:
@router.get("") @router.get("")
def list_views( def list_views(
user: User = Depends(require_human), db: Session = Depends(get_db) source: str = "feed",
user: User = Depends(require_human),
db: Session = Depends(get_db),
) -> list[dict]: ) -> list[dict]:
source = _norm_source(source)
rows = ( rows = (
db.execute( db.execute(
select(SavedView) select(SavedView)
.where(SavedView.user_id == user.id) .where(SavedView.user_id == user.id, SavedView.source == source)
.order_by(SavedView.position, SavedView.id) .order_by(SavedView.position, SavedView.id)
) )
.scalars() .scalars()
@@ -44,6 +55,7 @@ def list_views(
class CreateViewIn(BaseModel): class CreateViewIn(BaseModel):
name: str name: str
filters: dict filters: dict
source: str = "feed"
@router.post("") @router.post("")
@@ -55,13 +67,19 @@ def create_view(
name = body.name.strip() name = body.name.strip()
if not name: if not name:
raise HTTPException(status_code=400, detail="name is required") raise HTTPException(status_code=400, detail="name is required")
source = _norm_source(body.source)
# Position is per (user, source), so each module's views number 0..N independently.
maxpos = db.execute( maxpos = db.execute(
select(func.coalesce(func.max(SavedView.position), -1)).where( select(func.coalesce(func.max(SavedView.position), -1)).where(
SavedView.user_id == user.id SavedView.user_id == user.id, SavedView.source == source
) )
).scalar_one() ).scalar_one()
view = SavedView( view = SavedView(
user_id=user.id, name=name[:80], filters=body.filters, position=maxpos + 1 user_id=user.id,
source=source,
name=name[:80],
filters=body.filters,
position=maxpos + 1,
) )
db.add(view) db.add(view)
try: try:
@@ -107,10 +125,14 @@ def update_view(
if "is_default" in sent: if "is_default" in sent:
make_default = bool(body.is_default) make_default = bool(body.is_default)
if make_default: if make_default:
# At most one default per user: clear the flag on the others first. # At most one default per (user, source): clear the flag on this module's others first.
db.execute( db.execute(
update(SavedView) update(SavedView)
.where(SavedView.user_id == user.id, SavedView.id != view.id) .where(
SavedView.user_id == user.id,
SavedView.source == view.source,
SavedView.id != view.id,
)
.values(is_default=False) .values(is_default=False)
) )
view.is_default = make_default view.is_default = make_default
+11 -14
View File
@@ -90,17 +90,13 @@ def set_maintenance_settings(
db: Session = Depends(get_db), db: Session = Depends(get_db),
) -> dict: ) -> dict:
"""Admin-tune the maintenance job's rolling re-validation batch (videos re-checked per """Admin-tune the maintenance job's rolling re-validation batch (videos re-checked per
run). Persisted to app_state; the job reads it each run.""" run). Persisted via sysconfig; the job reads it each run."""
value = body.revalidate_batch # set_value validates against the spec's min/max (one source of bounds), mirroring the config PATCH.
if not (state.MAINTENANCE_BATCH_MIN <= value <= state.MAINTENANCE_BATCH_MAX): try:
raise HTTPException( sysconfig.set_value(db, "maintenance_revalidate_batch", body.revalidate_batch)
status_code=400, except (TypeError, ValueError) as exc:
detail=( raise HTTPException(status_code=400, detail=str(exc) or "Invalid value")
f"revalidate_batch must be between {state.MAINTENANCE_BATCH_MIN} and " new = sysconfig.get_int(db, "maintenance_revalidate_batch")
f"{state.MAINTENANCE_BATCH_MAX}"
),
)
new = state.set_maintenance_batch(db, value)
audit.record(db, admin.id, AuditAction.MAINTENANCE_TUNE, summary=f"revalidate batch = {new}", after={"revalidate_batch": new}) audit.record(db, admin.id, AuditAction.MAINTENANCE_TUNE, summary=f"revalidate batch = {new}", after={"revalidate_batch": new})
db.commit() db.commit()
return {"revalidate_batch": new} return {"revalidate_batch": new}
@@ -167,6 +163,7 @@ def get_scheduler(
"daily_budget": sysconfig.get_int(db, "quota_daily_budget"), "daily_budget": sysconfig.get_int(db, "quota_daily_budget"),
"backfill_reserve": sysconfig.get_int(db, "backfill_quota_reserve"), "backfill_reserve": sysconfig.get_int(db, "backfill_quota_reserve"),
} }
maint = sysconfig.spec("maintenance_revalidate_batch")
return { return {
**snap, **snap,
@@ -174,9 +171,9 @@ def get_scheduler(
"queue": queue, "queue": queue,
"quota": quota_info, "quota": quota_info,
"maintenance": { "maintenance": {
"revalidate_batch": state.get_maintenance_batch(db), "revalidate_batch": sysconfig.get_int(db, "maintenance_revalidate_batch"),
"revalidate_batch_default": settings.maintenance_revalidate_batch, "revalidate_batch_default": settings.maintenance_revalidate_batch,
"min": state.MAINTENANCE_BATCH_MIN, "min": maint.min,
"max": state.MAINTENANCE_BATCH_MAX, "max": maint.max,
}, },
} }
+23 -4
View File
@@ -12,6 +12,7 @@ Ingested results are flagged (videos.via_search / channels.from_search) so the L
this search-discovered "noise" by default. Shorts and currently-live/upcoming videos are never this search-discovered "noise" by default. Shorts and currently-live/upcoming videos are never
ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are ingested this way (Shorts are confirmed by the youtube.com/shorts probe; live/upcoming are
dropped from the search response up front).""" dropped from the search response up front)."""
import logging
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone from datetime import datetime, timezone
@@ -43,6 +44,8 @@ from app.youtube import search_scrape
from app.youtube.client import YouTubeClient, YouTubeError from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short from app.youtube.shorts import make_client, probe_is_short
log = logging.getLogger("siftlode.search")
router = APIRouter(prefix="/api/search", tags=["search"]) router = APIRouter(prefix="/api/search", tags=["search"])
# search.list costs this many quota units per page. # search.list costs this many quota units per page.
@@ -136,6 +139,10 @@ def _ingest_candidates(
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO # 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
# NOTHING), so an organically-synced video stays organic. # NOTHING), so an organically-synced video stays organic.
# published_at is NOT NULL (migration 0068); the scrape source exposes no publish date
# (only a relative "3 years ago"), so fall back to now() to satisfy the constraint — the
# enrich in step 3 overwrites it with the video's real snippet date.
now = datetime.now(timezone.utc)
insert_stubs( insert_stubs(
db, db,
[ [
@@ -143,7 +150,7 @@ def _ingest_candidates(
"id": it["id"], "id": it["id"],
"channel_id": it["channel_id"], "channel_id": it["channel_id"],
"title": it["title"], "title": it["title"],
"published_at": parse_dt(it["published_at"]), "published_at": parse_dt(it["published_at"]) or now,
"thumbnail_url": it["thumbnail_url"], "thumbnail_url": it["thumbnail_url"],
"via_search": True, "via_search": True,
} }
@@ -158,13 +165,13 @@ def _ingest_candidates(
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)} detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError: except YouTubeError:
detail_by_id = {} detail_by_id = {}
now = datetime.now(timezone.utc) enriched_at = datetime.now(timezone.utc) # when THIS enrich ran (distinct from the stub placeholder)
for v in videos: for v in videos:
item = detail_by_id.get(v.id) item = detail_by_id.get(v.id)
if item is None: if item is None:
continue continue
apply_video_details(v, item) apply_video_details(v, item)
v.enriched_at = now v.enriched_at = enriched_at
db.commit() db.commit()
# 4) Confirm/deny Shorts (no quota) so none enter via search. # 4) Confirm/deny Shorts (no quota) so none enter via search.
@@ -282,7 +289,19 @@ def search_youtube(
and it["channel_id"] and it["channel_id"]
and it["channel_id"] not in blocked and it["channel_id"] not in blocked
] ]
collected.extend(_ingest_candidates(db, yt, user, candidates)) # Ingest (DB writes + enrich) runs outside the page-fetch try above; guard it too so a
# write/shape failure degrades to a 422 with a message on this JSON endpoint instead of a
# bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
try:
collected.extend(_ingest_candidates(db, yt, user, candidates))
except Exception as exc: # noqa: BLE001 — any ingest failure, not just YouTubeError
db.rollback()
log.exception("live search ingest failed for %r", term)
if collected:
break # keep what earlier pages already gathered
raise HTTPException(
status_code=422, detail=f"YouTube search failed: {exc}"
) from exc
next_cursor = page["next_page_token"] next_cursor = page["next_page_token"]
# The API source pages cost 100 units each — never auto-page; one page per request. # The API source pages cost 100 units each — never auto-page; one page per request.
if source == "api" or len(collected) >= limit or not next_cursor: if source == "api" or len(collected) >= limit or not next_cursor:
+2 -1
View File
@@ -3,7 +3,7 @@
`GET /api/setup/status` is public (the SPA uses it to decide whether to show the wizard). Every `GET /api/setup/status` is public (the SPA uses it to decide whether to show the wizard). Every
mutating step goes through `require_setup`: it rejects once the instance is configured AND requires mutating step goes through `require_setup`: it rejects once the instance is configured AND requires
the one-time setup token (printed to the container logs at first boot, carried by the wizard as the the one-time setup token (printed to the container logs at first boot, carried by the wizard as the
X-Setup-Token header). The wizard steps themselves are added in epic 6c. X-Setup-Token header).
""" """
from typing import Any from typing import Any
@@ -96,6 +96,7 @@ def setup_config(settings_map: dict[str, Any], db: Session = Depends(require_set
except (ValueError, RuntimeError) as exc: except (ValueError, RuntimeError) as exc:
raise HTTPException(status_code=400, detail=str(exc)) raise HTTPException(status_code=400, detail=str(exc))
saved.append(key) saved.append(key)
db.commit() # sysconfig.set_value only flushes now — persist the wizard's config in one commit
return {"saved": saved} return {"saved": saved}
+1 -50
View File
@@ -7,30 +7,13 @@ from app.audit import AuditAction
from app.auth import admin_user, current_user, has_read_scope, require_human from app.auth import admin_user, current_user, has_read_scope, require_human
from app.db import get_db from app.db import get_db
from app.models import Channel, Subscription, User, Video from app.models import Channel, Subscription, User, Video
from app.sync.runner import ( from app.sync.runner import estimate_deep_backfill
estimate_deep_backfill,
run_enrich,
run_recent_backfill,
run_rss_poll,
)
from app.sync.subscriptions import import_subscriptions from app.sync.subscriptions import import_subscriptions
from app.scheduler import running_job_ids from app.scheduler import running_job_ids
router = APIRouter(prefix="/api/sync", tags=["sync"]) router = APIRouter(prefix="/api/sync", tags=["sync"])
def _user_channels(db: Session, user: User) -> list[Channel]:
return (
db.execute(
select(Channel)
.join(Subscription, Subscription.channel_id == Channel.id)
.where(Subscription.user_id == user.id)
)
.scalars()
.all()
)
@router.post("/subscriptions") @router.post("/subscriptions")
def sync_subscriptions( def sync_subscriptions(
user: User = Depends(require_human), db: Session = Depends(get_db) user: User = Depends(require_human), db: Session = Depends(get_db)
@@ -48,38 +31,6 @@ def sync_subscriptions(
return result return result
@router.post("/rss")
def sync_rss(
user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
# A partial failure comes back in `failed_feeds` so a manual poll reports it instead of an
# innocent-looking "0 new"; a TOTAL outage raises out of run_rss_poll and surfaces as a 500,
# which is the honest answer for a poll that reached nothing.
result = run_rss_poll(db, _user_channels(db, user))
return {"new_videos": result["new"], "failed_feeds": result["failed"]}
@router.post("/backfill")
def sync_backfill(
user: User = Depends(require_human),
db: Session = Depends(get_db),
max_channels: int = 25,
) -> dict:
with quota.measured(db, user.id, quota.QuotaAction.VIDEOS_BACKFILL_RECENT) as m:
result = run_recent_backfill(db, _user_channels(db, user), max_channels=max_channels)
result["quota_used_estimate"] = m.used
return result
@router.post("/enrich")
def sync_enrich(
user: User = Depends(require_human), db: Session = Depends(get_db)
) -> dict:
with quota.measured(db, user.id, quota.QuotaAction.VIDEOS_ENRICH) as m:
enriched = run_enrich(db)
return {"enriched": enriched, "quota_used_estimate": m.used}
@router.get("/status") @router.get("/status")
def sync_status( def sync_status(
user: User = Depends(current_user), db: Session = Depends(get_db) user: User = Depends(current_user), db: Session = Depends(get_db)
+1 -11
View File
@@ -4,11 +4,10 @@ from sqlalchemy import func, or_, select
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.auth import admin_user, current_user from app.auth import current_user
from app.db import get_db from app.db import get_db
from app.models import ChannelTag, Tag, User from app.models import ChannelTag, Tag, User
from app.routes._common import owned_or_404 from app.routes._common import owned_or_404
from app.sync.autotag import run_autotag_all
router = APIRouter(prefix="/api/tags", tags=["tags"]) router = APIRouter(prefix="/api/tags", tags=["tags"])
@@ -113,12 +112,3 @@ def delete_tag(
db.delete(tag) # ChannelTag rows cascade on tag delete. db.delete(tag) # ChannelTag rows cascade on tag delete.
db.commit() db.commit()
return {"deleted": tag_id} return {"deleted": tag_id}
@router.post("/recompute")
def recompute(
_: User = Depends(admin_user),
db: Session = Depends(get_db),
only_missing: bool = False,
) -> dict:
return run_autotag_all(db, only_missing=only_missing)
+10
View File
@@ -17,6 +17,7 @@ from app.demo import reset_demo_state
from app.downloads.gc import run_download_gc from app.downloads.gc import run_download_gc
from app.models import SchedulerSetting, User from app.models import SchedulerSetting, User
from app.plex.sync import sync as run_plex_sync from app.plex.sync import sync as run_plex_sync
from app.plex.tmdb import run_enrich as run_tmdb_enrich
from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync from app.plex.watch_sync import run_plex_watch_reconcile, run_plex_watch_sync
from app.notifications import create_notification from app.notifications import create_notification
from app.state import is_sync_paused from app.state import is_sync_paused
@@ -68,6 +69,7 @@ JOB_INTERVALS: dict[str, int] = {
"plex_sync": settings.plex_sync_interval_min, "plex_sync": settings.plex_sync_interval_min,
"plex_watch_sync": settings.plex_watch_sync_interval_min, "plex_watch_sync": settings.plex_watch_sync_interval_min,
"plex_watch_reconcile": settings.plex_watch_reconcile_interval_min, "plex_watch_reconcile": settings.plex_watch_reconcile_interval_min,
"tmdb_enrich": settings.tmdb_enrich_interval_min,
"audit_gc": settings.audit_gc_minutes, "audit_gc": settings.audit_gc_minutes,
} }
@@ -351,6 +353,13 @@ def _plex_watch_reconcile_job() -> None:
_job("plex_watch_reconcile", run_plex_watch_reconcile) _job("plex_watch_reconcile", run_plex_watch_reconcile)
def _tmdb_enrich_job() -> None:
# Enrich a batch of mirrored Plex titles from TMDB (keywords/crew/budget/revenue/TMDb score/
# recommendations/backdrops), matched by guid. External metadata (no YouTube quota); a no-op when
# the TMDB key is unset or the Plex catalog is empty.
_job("tmdb_enrich", run_tmdb_enrich)
def _audit_gc_job() -> None: def _audit_gc_job() -> None:
# Prune audit-log rows older than the admin-set retention (audit_retention_days; 0 = keep # Prune audit-log rows older than the admin-set retention (audit_retention_days; 0 = keep
# forever). Pure DB, no quota. # forever). Pure DB, no quota.
@@ -378,6 +387,7 @@ JOB_FUNCS: dict[str, Callable[[], None]] = {
"plex_sync": _plex_sync_job, "plex_sync": _plex_sync_job,
"plex_watch_sync": _plex_watch_sync_job, "plex_watch_sync": _plex_watch_sync_job,
"plex_watch_reconcile": _plex_watch_reconcile_job, "plex_watch_reconcile": _plex_watch_reconcile_job,
"tmdb_enrich": _tmdb_enrich_job,
"audit_gc": _audit_gc_job, "audit_gc": _audit_gc_job,
} }
+3 -23
View File
@@ -4,16 +4,11 @@ import secrets
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.config import settings
from app.models import AppState from app.models import AppState
from app.security import hash_token from app.security import hash_token
log = logging.getLogger("siftlode.state") log = logging.getLogger("siftlode.state")
# Bounds for the admin-set maintenance re-validation batch (videos re-checked per run).
MAINTENANCE_BATCH_MIN = 100
MAINTENANCE_BATCH_MAX = 100_000
# Once configuration completes in this process it never reverts, so cache it to skip the per-request # Once configuration completes in this process it never reverts, so cache it to skip the per-request
# DB hit in the setup-mode gate (see main.setup_gate). Starts False; refreshed from the DB. # DB hit in the setup-mode gate (see main.setup_gate). Starts False; refreshed from the DB.
_configured_cache = False _configured_cache = False
@@ -33,10 +28,12 @@ def is_sync_paused(db: Session) -> bool:
def set_sync_paused(db: Session, paused: bool) -> None: def set_sync_paused(db: Session, paused: bool) -> None:
# Flushes, doesn't commit — the caller owns the transaction so the change lands with its audit
# row in one transaction (B-core-10).
row = _row(db) row = _row(db)
row.sync_paused = paused row.sync_paused = paused
db.add(row) db.add(row)
db.commit() db.flush()
log.info("Background sync %s", "paused" if paused else "resumed") log.info("Background sync %s", "paused" if paused else "resumed")
@@ -90,20 +87,3 @@ def check_setup_token(db: Session, raw: str | None) -> bool:
return False return False
stored = _row(db).setup_token_hash stored = _row(db).setup_token_hash
return stored is not None and secrets.compare_digest(stored, hash_token(raw)) return stored is not None and secrets.compare_digest(stored, hash_token(raw))
def get_maintenance_batch(db: Session) -> int:
"""Effective maintenance re-validation batch: the admin override if set, else the
env/config default."""
return _row(db).maintenance_revalidate_batch or settings.maintenance_revalidate_batch
def set_maintenance_batch(db: Session, value: int) -> int:
"""Clamp and persist the maintenance batch override. Returns the applied value."""
value = max(MAINTENANCE_BATCH_MIN, min(MAINTENANCE_BATCH_MAX, int(value)))
row = _row(db)
row.maintenance_revalidate_batch = value
db.add(row)
db.commit()
log.info("Maintenance re-validation batch set to %s", value)
return value
+6 -7
View File
@@ -25,11 +25,10 @@ from datetime import timedelta
from sqlalchemy import select from sqlalchemy import select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app import progress, quota from app import progress, quota, sysconfig
from app.config import settings from app.config import settings
from app.models import Playlist, PlaylistItem, Video from app.models import Playlist, PlaylistItem, Video
from app.notifications import create_notification from app.notifications import create_notification
from app.state import get_maintenance_batch
from app.sync.runner import get_service_user from app.sync.runner import get_service_user
from app.sync.videos import apply_video_details, parse_dt from app.sync.videos import apply_video_details, parse_dt
from app.utils import now_utc as _now from app.utils import now_utc as _now
@@ -182,7 +181,7 @@ def _recheck_flagged(db: Session, yt: YouTubeClient) -> dict:
def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict: def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
"""Phase 2: re-check the least-recently-checked currently-available videos and flag any """Phase 2: re-check the least-recently-checked currently-available videos and flag any
that have become unplayable (hidden immediately; deleted later if they don't recover).""" that have become unplayable (hidden immediately; deleted later if they don't recover)."""
batch = get_maintenance_batch(db) # admin override, else config default batch = sysconfig.get_int(db, "maintenance_revalidate_batch") # admin override, else config default
flagged_new = 0 flagged_new = 0
checked = 0 checked = 0
now = _now() now = _now()
@@ -194,10 +193,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
db.execute( db.execute(
select(Video) select(Video)
.where(Video.unavailable_since.is_(None)) .where(Video.unavailable_since.is_(None))
.order_by( # NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
Video.last_checked_at.is_(None).desc(), # never-checked first # `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
Video.last_checked_at.asc(), # partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
) .order_by(Video.last_checked_at.asc().nulls_first())
.limit(min(50, batch - checked)) .limit(min(50, batch - checked))
) )
.scalars() .scalars()
+6 -4
View File
@@ -1,6 +1,6 @@
"""Read-direction YouTube playlist sync: mirror each user's own YouTube playlists into """YouTube playlist sync, both directions. Read: mirror each user's own YouTube playlists
local `source='youtube'` playlists (kept fresh by the scheduler). One-way (YT -> local); into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: push local
the write-back direction (local edits -> YouTube) is a later phase. edits out to YouTube (`plan_push` / `push_playlist`), behind the optional write scope.
YouTube's special Watch Later / History playlists are not exposed by the Data API and are YouTube's special Watch Later / History playlists are not exposed by the Data API and are
never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and
@@ -90,7 +90,9 @@ def _ensure_videos(db: Session, yt: YouTubeClient, video_ids: list[str]) -> None
if not vid or not cid or vid in seen: if not vid or not cid or vid in seen:
continue continue
seen.add(vid) seen.add(vid)
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt"))) # published_at is NOT NULL (0068); apply_video_details re-fills a missing date, but seed a now()
# fallback here too so the constructor never carries an explicit NULL.
v = Video(id=vid, channel_id=cid, published_at=parse_dt(sn.get("publishedAt")) or now)
apply_video_details(v, it) apply_video_details(v, it)
v.enriched_at = now v.enriched_at = now
db.add(v) db.add(v)
+3 -2
View File
@@ -6,6 +6,7 @@ from sqlalchemy import select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.models import Channel, Subscription, User from app.models import Channel, Subscription, User
from app.utils import normalize_description
from app.youtube.client import YouTubeClient, best_thumbnail from app.youtube.client import YouTubeClient, best_thumbnail
log = logging.getLogger("siftlode.sync") log = logging.getLogger("siftlode.sync")
@@ -55,7 +56,7 @@ def apply_channel_details(db: Session, items: list[dict]) -> None:
channel.title = snippet.get("title") or channel.title channel.title = snippet.get("title") or channel.title
channel.handle = snippet.get("customUrl") channel.handle = snippet.get("customUrl")
channel.description = snippet.get("description") channel.description = normalize_description(snippet.get("description"))
channel.thumbnail_url = ( channel.thumbnail_url = (
best_thumbnail(snippet.get("thumbnails")) or channel.thumbnail_url best_thumbnail(snippet.get("thumbnails")) or channel.thumbnail_url
) )
@@ -104,7 +105,7 @@ def import_subscriptions(db: Session, user: User) -> dict:
channel = Channel( channel = Channel(
id=cid, id=cid,
title=sub.get("title"), title=sub.get("title"),
description=sub.get("description"), description=normalize_description(sub.get("description")),
thumbnail_url=sub.get("thumbnail_url"), thumbnail_url=sub.get("thumbnail_url"),
) )
db.add(channel) db.add(channel)
+13 -3
View File
@@ -90,7 +90,10 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
video_id = content.get("videoId") video_id = content.get("videoId")
if not video_id: if not video_id:
return None return None
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt")) # published_at is NOT NULL (migration 0068); fall back to now() for the rare item that carries no
# date at all, so the bulk stub insert never writes a NULL. The enrich pass replaces it if the
# video's own snippet later yields a real publish date.
published = parse_dt(content.get("videoPublishedAt") or snippet.get("publishedAt")) or _now()
return { return {
"id": video_id, "id": video_id,
"channel_id": channel_id, "channel_id": channel_id,
@@ -242,8 +245,15 @@ def apply_video_details(video: Video, item: dict) -> None:
video.original_title = raw_title video.original_title = raw_title
video.title = normalize_title(raw_title) video.title = normalize_title(raw_title)
video.description = snippet.get("description") video.description = snippet.get("description")
if not video.published_at: # published_at is NOT NULL (0068). Prefer the enrich snippet's real date whenever it has one —
video.published_at = parse_dt(snippet.get("publishedAt")) # this also corrects a now() placeholder written at stub time (e.g. a scrape-search stub, which
# carries no publish date). Only fall back to now() when the row still has nothing at all; never
# clobber an existing real date with now() (a re-enrich whose snippet lacks a date).
snippet_published = parse_dt(snippet.get("publishedAt"))
if snippet_published:
video.published_at = snippet_published
elif not video.published_at:
video.published_at = _now()
video.thumbnail_url = best_thumbnail(snippet.get("thumbnails")) or video.thumbnail_url video.thumbnail_url = best_thumbnail(snippet.get("thumbnails")) or video.thumbnail_url
video.duration_seconds = parse_iso8601_duration(content.get("duration")) video.duration_seconds = parse_iso8601_duration(content.get("duration"))
video.view_count = int(stats["viewCount"]) if stats.get("viewCount") else None video.view_count = int(stats["viewCount"]) if stats.get("viewCount") else None
+18 -6
View File
@@ -59,6 +59,10 @@ SPECS: tuple[ConfigSpec, ...] = (
# --- Batch sizes --- # --- Batch sizes ---
ConfigSpec("enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50), ConfigSpec("enrich_batch_size", "int", "batch", "enrich_batch_size", min=1, max=50),
ConfigSpec("autotag_title_sample", "int", "batch", "autotag_title_sample", min=1, max=1_000), ConfigSpec("autotag_title_sample", "int", "batch", "autotag_title_sample", min=1, max=1_000),
# --- Maintenance (rolling re-validation of already-known videos) ---
# Videos re-checked per maintenance run. Folded here from app_state (B-core-11) so every admin
# knob lives in one store; read by sync/maintenance.py, tuned from the Scheduler dashboard.
ConfigSpec("maintenance_revalidate_batch", "int", "maintenance", "maintenance_revalidate_batch", min=100, max=100_000),
# --- Discovery cleanup (search + explore ephemeral content) --- # --- Discovery cleanup (search + explore ephemeral content) ---
# Days an explored-but-unsubscribed channel is kept before the cleanup job purges it. # Days an explored-but-unsubscribed channel is kept before the cleanup job purges it.
ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650), ConfigSpec("explore_grace_days", "int", "explore", "explore_grace_days", min=0, max=3_650),
@@ -96,8 +100,13 @@ SPECS: tuple[ConfigSpec, ...] = (
ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"), ConfigSpec("plex_overlay_map", "str", "plex", "plex_overlay_map"),
# (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.) # (plex_sync_interval_min is tuned from the Scheduler dashboard, not here — it's a scheduler job.)
ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16), ConfigSpec("plex_max_transcodes", "int", "plex", "plex_max_transcodes", min=0, max=16),
# NOTE: plex_watch_sync_interval_min (config.py default) is surfaced here in Phase C, when the # NOTE: plex_watch_sync_interval_min is a config.py default read by the watch-sync scheduler job
# incremental watch-sync scheduler job that reads it lands. # (scheduler.py); it is intentionally NOT a runtime admin knob, so it stays out of SPECS.
# --- TMDB enrichment (E5 S2; optional) — the API key is the only secret ---
ConfigSpec("tmdb_api_key", "str", "tmdb", "tmdb_api_key", secret=True),
ConfigSpec("tmdb_language", "str", "tmdb", "tmdb_language"),
# (tmdb_enrich_interval_min is a scheduler job, tuned from the Scheduler dashboard; tmdb_enrich_batch
# is a config.py default — both stay out of SPECS, like plex_sync_interval_min.)
) )
_BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS} _BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS}
@@ -182,7 +191,9 @@ def get_bool(db: Session, key: str) -> bool:
def set_value(db: Session, key: str, value) -> None: def set_value(db: Session, key: str, value) -> None:
"""Validate, coerce, and persist a DB override for `key`. Secrets are encrypted.""" """Validate, coerce, and stage a DB override for `key` (secrets are encrypted). Flushes but does
NOT commit — the caller owns the transaction, so the change lands together with its audit row
(B-core-10). The staged value is visible to a same-session `get()` before the commit."""
s = _BY_KEY[key] s = _BY_KEY[key]
if s.type == "int": if s.type == "int":
v = int(value) v = int(value)
@@ -208,15 +219,16 @@ def set_value(db: Session, key: str, value) -> None:
db.add(row) db.add(row)
row.value = stored row.value = stored
row.encrypted = encrypted row.encrypted = encrypted
db.commit() db.flush()
def reset(db: Session, key: str) -> None: def reset(db: Session, key: str) -> None:
"""Remove the DB override for `key` (fall back to the env/config default).""" """Remove the DB override for `key` (fall back to the env/config default). Flushes but does NOT
commit — the caller owns the transaction (B-core-10)."""
row = db.get(SystemConfig, key) row = db.get(SystemConfig, key)
if row is not None: if row is not None:
db.delete(row) db.delete(row)
db.commit() db.flush()
def describe(db: Session) -> dict: def describe(db: Session) -> dict:
+15
View File
@@ -36,3 +36,18 @@ def to_tsquery_str(q: str) -> str | None:
return None return None
tokens[-1] = tokens[-1] + ":*" tokens[-1] = tokens[-1] + ":*"
return " & ".join(tokens) return " & ".join(tokens)
def normalize_description(text: str | None) -> str | None:
"""Strip whitespace-only lines from a CHANNEL description before storing. YouTube channel
descriptions pad their contact/link blocks with runs of blank lines; keeping them wastes storage,
bloats the About popover, and adds nothing to the search document. Drops purely-empty lines and
right-strips each kept line; `str.splitlines()` also folds CR / CRLF and Unicode line separators
to "\\n" (a benign line-ending normalization). None stays None; all-blank collapses to "".
Deliberately CHANNEL-only — video descriptions are left with their original formatting: they are
user-facing prose where paragraph breaks carry meaning, unlike a channel's metadata block."""
if text is None:
return None
kept = [line.rstrip() for line in text.splitlines() if line.strip()]
return "\n".join(kept)
+25 -635
View File
@@ -17,125 +17,40 @@ Design:
""" """
import logging import logging
import os import os
import re
import shutil import shutil
import signal import signal
import subprocess import subprocess
import sys import sys
import threading import threading
import time import time
from datetime import date, datetime, timedelta, timezone from datetime import datetime, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import text from sqlalchemy import text
from app import sysconfig from app import sysconfig
from app.config import settings from app.config import settings
from app.db import SessionLocal from app.db import SessionLocal
from app.downloads import edit as editmod from app.downloads import edit as editmod
from app.downloads import errors as dlerrors from app.downloads import quota, service
from app.downloads import adapters, formats, quota, service, storage from app.downloads.jobdb import (
_edit_staging_dir,
_requeue_if_running,
_reset_asset_pending,
_set_job,
_staging_dir,
)
from app.downloads.status import STOPPED_BY_ROUTE, JobStatus from app.downloads.status import STOPPED_BY_ROUTE, JobStatus
from app.models import DownloadJob, MediaAsset from app.models import DownloadJob, MediaAsset
from app.notifications import create_notification
from app.titles import normalize_title
log = logging.getLogger("siftlode.worker") log = logging.getLogger("siftlode.worker")
def _notify_download_done(
db, user_id: int | None, title: str | None, job_id, *, kind: str
) -> None:
"""Post a best-effort 'ready' notice to the owner's inbox — downloads/edits used to finish
silently while scheduler jobs and playlist-push already notified (R11 feedback symmetry). A
notification failure must never mask the completed job, so it is isolated + rolled back (mirrors
scheduler._notify_done). `kind` ("download"/"edit") lets the inbox pick accurate copy; the
English title here is only a fallback for an unknown type. `user_id`/`title` are captured by the
caller BEFORE its commit, so reading them here needs no post-commit refresh of an expired row."""
if user_id is None:
return
try:
create_notification(
db,
user_id=user_id,
type="download_done",
title=f"Ready to watch: {title}" if title else "Ready to watch",
data={"title": title, "kind": kind},
)
except Exception:
db.rollback()
log.exception("failed to write download-done notification for job %s", job_id)
_stop = threading.Event() _stop = threading.Event()
class _Aborted(Exception):
"""Raised inside the progress hook when the job was paused/canceled mid-download."""
def _handle_signal(signum, frame): def _handle_signal(signum, frame):
_stop.set() _stop.set()
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
def _set_job(job_id: int, **fields) -> None:
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
if job is None:
return
for k, v in fields.items():
setattr(job, k, v)
db.commit()
def _job_status(job_id: int) -> str | None:
with SessionLocal() as db:
return db.execute(
text("SELECT status FROM download_jobs WHERE id = :id"), {"id": job_id}
).scalar()
def _touch_job(job_id: int) -> None:
"""Bump a running job's `updated_at` without changing any value — a stall-watchdog heartbeat for
phases that legitimately write no progress. Extraction writes nothing, and a multi-client-set
fallback re-writes identical values (progress 0 / phase 'downloading'), so ORM `onupdate` would
NOT fire and two slow extraction attempts could otherwise cross the stall timeout and be killed
mid-extraction though they'd have succeeded."""
with SessionLocal() as db:
db.execute(
text("UPDATE download_jobs SET updated_at=now() WHERE id=:id AND status='running'"),
{"id": job_id},
)
db.commit()
def _requeue_if_running(job_id: int, phase: str) -> None:
"""Put the job back to 'queued' for a later retry, but ONLY while it is still 'running'. The
route flips status to paused/cancelled to stop a job; a blind `status='queued'` here would
clobber that and silently revive a job the user just cancelled. The conditional WHERE makes the
requeue a no-op unless we still own a running job."""
with SessionLocal() as db:
db.execute(
text(
"UPDATE download_jobs SET status='queued', phase=:phase, updated_at=now() "
"WHERE id=:id AND status='running'"
),
{"id": job_id, "phase": phase},
)
db.commit()
def _parse_upload_date(raw) -> date | None:
if not raw:
return None
try:
return datetime.strptime(str(raw), "%Y%m%d").date()
except ValueError:
return None
# --- claim + recovery ----------------------------------------------------------------------- # --- claim + recovery -----------------------------------------------------------------------
def _wait_for_schema(timeout: float = 300.0) -> bool: def _wait_for_schema(timeout: float = 300.0) -> bool:
@@ -176,7 +91,14 @@ def _recover_orphans() -> None:
def _claim_job() -> int | None: def _claim_job() -> int | None:
"""Claim the oldest queued job whose user is under their per-user concurrency limit. """Claim the oldest queued job whose user is under their per-user concurrency limit AND whose
shared asset isn't already being downloaded by another worker.
The asset exclusion (`NOT EXISTS … media_assets.status='downloading'`) stops the same-asset ping-
pong (B-sub-3): without it, a second worker claims a queued job that shares an in-flight asset,
finds it busy, requeues it, and immediately re-claims it — burning a worker slot behind one
in-flight asset and starving everyone else. Excluded here, that job simply waits until its asset
settles, so `_process_job` no longer needs to sleep on the busy/lost-race paths.
Users already AT their concurrency limit are excluded IN THE SQL, so one user flooding the Users already AT their concurrency limit are excluded IN THE SQL, so one user flooding the
queue can't monopolize the fetch window and starve everyone behind them. (The old fixed queue can't monopolize the fetch window and starve everyone behind them. (The old fixed
@@ -193,6 +115,10 @@ def _claim_job() -> int | None:
""" """
SELECT id, user_id FROM download_jobs SELECT id, user_id FROM download_jobs
WHERE status='queued' WHERE status='queued'
AND NOT EXISTS (
SELECT 1 FROM media_assets a
WHERE a.id = download_jobs.asset_id AND a.status = 'downloading'
)
AND user_id NOT IN ( AND user_id NOT IN (
SELECT r.user_id SELECT r.user_id
FROM download_jobs r FROM download_jobs r
@@ -463,15 +389,16 @@ def _process_job(job_id: int) -> None:
_set_job(job_id, status="error", error=asset_error or "Download failed", phase=None) _set_job(job_id, status="error", error=asset_error or "Download failed", phase=None)
return return
if asset_status == "downloading": if asset_status == "downloading":
# Another worker owns this asset; wait and let the job be reclaimed later. # Another worker won this asset between our claim and here (a narrow race — the claim SQL
# excludes downloading-asset jobs). Requeue and let it be reclaimed once the asset settles;
# no sleep — the claim exclusion keeps it out of the window until then, so no busy-loop.
_requeue_if_running(job_id, "waiting") _requeue_if_running(job_id, "waiting")
time.sleep(2)
return return
# asset_status == "pending": try to win the download / edit. # asset_status == "pending": try to win the download / edit.
if not _claim_asset(asset_id): if not _claim_asset(asset_id):
# Lost the pending->downloading race; requeue (same reasoning — the exclusion prevents a spin).
_requeue_if_running(job_id, "waiting") _requeue_if_running(job_id, "waiting")
time.sleep(2)
return return
# The asset is now ours ('downloading'): run the heavy work in a supervised child process. # The asset is now ours ('downloading'): run the heavy work in a supervised child process.
@@ -494,543 +421,6 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
db.commit() db.commit()
def _fill_asset_meta_early(asset_id: int, info: dict) -> None:
"""Populate an asset's display metadata from yt-dlp's info_dict the first time it's seen
(only if not already filled at enqueue from our catalog) — so an ad-hoc URL's row shows a
real title + thumbnail while it downloads, not just the URL. Zero extra network cost."""
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset is None or asset.title:
return
# `or info.get("id")` avoids a blank card while downloading when the cleaned title is empty
# (e.g. a title that was only a social engagement prefix); the completion path later sets
# the final title with its own source_ref fallback.
asset.title = normalize_title(info.get("title")) or info.get("id")
asset.uploader = info.get("uploader") or info.get("channel")
asset.uploader_url = adapters.resolve_adapter(None, None, info).uploader_url(info)
asset.thumbnail_url = info.get("thumbnail")
asset.duration_s = int(info["duration"]) if info.get("duration") else None
asset.upload_date = _parse_upload_date(info.get("upload_date"))
asset.source_webpage_url = info.get("webpage_url")
db.commit()
def _make_progress_hook(job_id: int, asset_id: int):
state = {"meta_done": False}
cancelled = _cancel_poll(job_id) # same policy as the ffmpeg path
allow_write = _rate_limiter(_DB_WRITE_INTERVAL_S)
def hook(d: dict) -> None:
# Cooperative pause/cancel: if the job is no longer 'running', abort the download.
if cancelled():
raise _Aborted
if not state["meta_done"] and d.get("info_dict"):
state["meta_done"] = True
try:
_fill_asset_meta_early(asset_id, d["info_dict"])
except Exception: # noqa: BLE001 — metadata is best-effort, never fail a download
pass
if d.get("status") != "downloading":
return
if not allow_write():
return
# yt-dlp downloads the video and audio streams SEPARATELY (each 0→100%), then merges.
# Label the phase so the user understands the bar resetting between streams instead of
# seeing a mysterious 95%→5% jump.
info = d.get("info_dict") or {}
vcodec = info.get("vcodec") or "none"
acodec = info.get("acodec") or "none"
if vcodec != "none" and acodec == "none":
phase = "video"
elif acodec != "none" and vcodec == "none":
phase = "audio"
else:
phase = "media"
total = d.get("total_bytes") or d.get("total_bytes_estimate") or 0
done = d.get("downloaded_bytes") or 0
pct = int(done * 100 / total) if total else 0
_set_job(
job_id,
progress=min(pct, 99),
speed_bps=int(d["speed"]) if d.get("speed") else None,
eta_s=int(d["eta"]) if d.get("eta") else None,
phase=phase,
)
return hook
# Both progress sources call back FAR faster than the database should be touched: yt-dlp fires its
# hook per chunk, and ffmpeg's `-progress pipe:1` emits a block of ~12 lines every ~0.5s while
# run_ffmpeg calls its callbacks once per LINE. Every callback here is a round-trip (`_job_status`
# opens a session and SELECTs; `_set_job` opens one and UPDATE+COMMITs), so unthrottled they cost
# ~10^5 queries on a long job. ONE interval, ONE implementation, used by both paths — three
# hand-rolled copies is how the ffmpeg side ended up unthrottled while the download side was fine.
_DB_WRITE_INTERVAL_S = 1.0
_CANCEL_POLL_INTERVAL_S = 1.0
def _rate_limiter(interval_s: float, *, now=time.monotonic):
"""`should_run()` → True at most once per `interval_s` (the first call always passes).
`now` is injectable so tests drive a clock instead of patching the global one."""
# `None`, not 0.0: "never ran" has to be a VALUE, not a falsy timestamp. A truthiness test here
# reads a legitimate `now() == 0.0` as "never ran" and lets a second call straight through —
# unreachable with real `time.monotonic()`, but `now` exists to be injected, and a test clock
# starting at 0 is the obvious choice. That would let a throttle test encode the wrong
# behaviour and still pass.
state: dict[str, float | None] = {"at": None}
def should_run() -> bool:
t = now()
last = state["at"]
if last is not None and t - last < interval_s:
return False
state["at"] = t
return True
return should_run
def _cancel_poll(job_id: int, interval_s: float = _CANCEL_POLL_INTERVAL_S, *, now=time.monotonic):
"""Cooperative "has this job been paused/cancelled?" poll, rate-limited and sticky: once it
says yes it never queries again. Shared by the yt-dlp hook and run_ffmpeg so cancel latency is
one policy, not one per call site."""
allow = _rate_limiter(interval_s, now=now)
state = {"cancelled": False}
def cancelled() -> bool:
if not state["cancelled"] and allow():
state["cancelled"] = _job_status(job_id) not in ("running", None)
return state["cancelled"]
return cancelled
def _ffmpeg_progress_writer(job_id: int, phase: str, *, now=time.monotonic):
"""`on_progress` for run_ffmpeg: persist at most one row write per interval, and only when the
whole-percent value actually changed (the bar shows integers)."""
allow = _rate_limiter(_DB_WRITE_INTERVAL_S, now=now)
state = {"pct": -1}
def write(pct: float) -> None:
whole = int(pct)
if whole == state["pct"] or not allow():
return
state["pct"] = whole
_set_job(job_id, progress=whole, phase=phase)
return write
def _pp_phase(pp: str) -> str:
"""Map a yt-dlp postprocessor class name to a user-facing phase label. ffmpeg post-steps
aren't byte-progress operations (no %), so the UI shows the step name + an indeterminate
pulse — this makes the long merge/finalize legible instead of a silent 'Processing'."""
if "Merg" in pp:
return "merging"
if "ExtractAudio" in pp:
return "audio_extract"
if "Thumbnail" in pp:
return "thumbnail"
if "SponsorBlock" in pp or "ModifyChapters" in pp:
return "sponsorblock"
if "Metadata" in pp:
return "metadata"
return "processing"
def _make_pp_hook(job_id: int):
"""Postprocessor hook: surface the current ffmpeg step so the bar shows a concrete label
(Merging / Embedding thumbnail / …) instead of freezing at 100% after the streams finish."""
def hook(d: dict) -> None:
if d.get("status") in ("started", "processing"):
_set_job(job_id, phase=_pp_phase(d.get("postprocessor") or ""), speed_bps=None, eta_s=None)
return hook
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
_THUMB_SUFFIXES = (".jpg", ".jpeg", ".png", ".webp")
# yt-dlp's working files in the staging dir. None of them is ever the finished media, but several
# CAN be the biggest file there (a video-only per-format stream, an abandoned fragment run), so the
# staging fallback must exclude them by name — picking one would store e.g. a silent video-only
# stream as a ready asset.
_YTDLP_TEMP_RE = re.compile(
r"\.part(-Frag\d+)?$" # in-flight download / fragment
r"|\.ytdl$" # resume state
r"|\.temp\.[^.]+$" # merge/postprocess temp
r"|\.f\d+\.[^.]+$", # per-format stream kept before the merge
re.IGNORECASE,
)
def _find_thumbnail(staging: Path, media: Path) -> Path | None:
for p in staging.iterdir():
if p != media and p.suffix.lower() in _THUMB_SUFFIXES:
return p
return None
def _produced_file(info: dict, staging: Path) -> Path:
"""The media file yt-dlp actually wrote.
`requested_downloads[0].filepath` is the authoritative answer on YouTube, but an arbitrary-URL
extractor may not fill it in at all — and `Path("")` is `Path(".")`, which *passes* `.exists()`,
so a missing value used to sail past the fallback and blow up two steps later (`with_name` on an
empty name: the BBC-news failure). Anything that isn't a real file counts as absent; then fall
back to the largest file left in staging that is neither a thumbnail nor one of yt-dlp's own
working files (see `_YTDLP_TEMP_RE`)."""
req = (info.get("requested_downloads") or [{}])[0]
fp = req.get("filepath") or req.get("_filename") or info.get("filepath") or info.get("_filename")
if fp:
produced = Path(str(fp))
if produced.is_file():
return produced
cands = [
p
for p in staging.iterdir()
if p.is_file()
and p.suffix.lower() not in _THUMB_SUFFIXES
and not _YTDLP_TEMP_RE.search(p.name)
]
if not cands:
raise RuntimeError("yt-dlp produced no output file")
return max(cands, key=lambda p: p.stat().st_size)
def _media_info(info: dict) -> dict:
"""Unwrap a playlist/multi_video extraction to the entry that carries the download.
A single non-YouTube page (a news article's video, a multi-rendition embed) can extract as a
wrapper whose own dict has no `requested_downloads` — the media lives on an entry. Pick the
FIRST entry that actually downloaded something: on a page with a skipped teaser in front of the
real video, entries[0] carries no file, and taking it blindly would label the produced file
with the wrong id/title/duration."""
entries = [e for e in (info.get("entries") or []) if e]
if not entries or info.get("requested_downloads"):
return info
chosen = next((e for e in entries if e.get("requested_downloads")), entries[0])
return {**info, **chosen}
def _extract_with_fallback(job_id, asset_id, url, spec, staging, adapter):
"""Download via yt-dlp using the source `adapter`'s client sets + options. For YouTube the adapter
supplies the primary player-client set then a POT-backed fallback (retried on a bot/DRM/format
error); other sources make a single attempt. Returns the info dict of the successful attempt."""
client_sets = adapter.client_sets()
outtmpl = str(staging / "%(id)s.%(ext)s")
last_exc: Exception | None = None
for i, clients in enumerate(client_sets):
opts = formats.build_ydl_opts(
spec, outtmpl, _make_progress_hook(job_id, asset_id),
extra=adapter.ydl_opts_extra(clients),
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
_touch_job(job_id) # stall-watchdog heartbeat: extraction writes no progress, so a slow /
# multi-client-set fallback would otherwise cross the stall timeout with a stale updated_at.
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
try:
with yt_dlp.YoutubeDL(opts) as ydl:
return ydl.extract_info(url, download=True)
except yt_dlp.utils.DownloadError as exc:
last_exc = exc
if i < len(client_sets) - 1 and adapter.retriable(str(exc)):
log.info("job %s: clients %s failed (%s); retrying with fallback", job_id, clients, str(exc)[:70])
for p in list(staging.iterdir()): # drop partials before the retry
try:
p.unlink()
except OSError:
pass
_set_job(job_id, progress=0, phase="downloading")
continue
raise
raise last_exc # unreachable (loop either returns or raises)
def _download_settings() -> tuple[str, int]:
"""Layout + retention days from the DB config (admin-overridable on the Configuration page),
falling back to the env defaults. Read here, not from `settings`, so an admin edit takes effect."""
with SessionLocal() as db:
return (
sysconfig.get_str(db, "download_layout"),
sysconfig.get_int(db, "download_retention_days"),
)
def _ensure_browser_playable(job_id: int, produced: Path, spec: dict, info: dict) -> Path:
"""Make an mp4-compat-profile download play in every browser: H.264 + AAC/MP3 in mp4. If the
download (whatever the source shipped) isn't already that, re-encode the offending stream(s) —
or just remux when the codecs are fine but the container isn't. **Scope:** only the mp4 compat
profile is normalized; an explicit `vcodec` or a non-mp4 custom profile is the user opting out
of the guarantee and is left as-is (so a `webm` profile can still ship VP9/Opus). Returns the
file to store (converted, or the original) and corrects `info`'s codec fields to match disk."""
if spec.get("container") != "mp4" or spec.get("vcodec"):
return produced
vcodec, acodec = editmod.probe_codecs(produced)
is_mp4 = produced.suffix.lower() == ".mp4"
needs, reenc_v, reenc_a = editmod.browser_transcode_flags(vcodec, acodec, is_mp4)
if not needs:
return produced
dest = produced.with_name(produced.stem + ".browser.mp4")
action = "re-encode" if (reenc_v or reenc_a) else "remux"
log.info("job %s: %s (v=%s a=%s mp4=%s) -> browser-safe mp4", job_id, action, vcodec, acodec, is_mp4)
_set_job(job_id, phase="processing", progress=0, speed_bps=None, eta_s=None)
# Prefer the info duration; probe the file when the extractor didn't provide one, so a long
# re-encode still shows real progress instead of a frozen 0%.
total_s = int(info["duration"]) if info.get("duration") else editmod.probe_duration(produced)
editmod.run_ffmpeg(
editmod.build_browser_safe_cmd(produced, dest, reenc_v, reenc_a),
total_s=total_s,
on_progress=_ffmpeg_progress_writer(job_id, "processing"),
should_cancel=_cancel_poll(job_id),
)
produced.unlink(missing_ok=True)
# Keep the stored asset metadata truthful about what's now on disk.
if reenc_v:
info["vcodec"] = "h264"
if reenc_a:
info["acodec"] = "aac"
return dest
def _download(job_id: int, asset_id: int, source_kind: str, source_ref: str, spec: dict) -> None:
staging = _staging_dir(asset_id)
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
_set_job(job_id, phase="downloading", progress=0)
adapter = adapters.resolve_adapter(source_kind, source_ref)
url = service.source_url(source_kind, source_ref)
info = _media_info(_extract_with_fallback(job_id, asset_id, url, spec, staging, adapter))
# Re-resolve now that yt-dlp's info reveals the true source (a bare/redirected url only
# identifies post-extraction) so the .nfo uniqueid type + uploader_url agree on one source.
adapter = adapters.resolve_adapter(source_kind, source_ref, info)
produced = _produced_file(info, staging)
produced = _ensure_browser_playable(job_id, produced, spec, info)
ext = produced.suffix.lstrip(".").lower()
meta = storage.MediaMeta(
video_id=info.get("id") or source_ref,
title=normalize_title(info.get("title")) or source_ref,
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
upload_date=_parse_upload_date(info.get("upload_date")),
uniqueid_type=adapter.nfo_uniqueid_type(),
duration_s=int(info["duration"]) if info.get("duration") else None,
description=info.get("description"),
thumbnail_url=info.get("thumbnail"),
)
layout, retention_days = _download_settings()
rel = storage.rel_path(meta, ext, layout, asset_id=asset_id)
thumb = _find_thumbnail(staging, produced)
# Convert the raw thumbnail (webp/png/jpeg) to a real JPEG ourselves — robustly — for the
# poster + og:image sidecar, now that yt-dlp's fatal thumbnail convertor is gone. On failure
# this is None and write_sidecars writes no poster, so ensure_poster cuts a frame instead.
poster_jpg = editmod.thumbnail_to_jpg(thumb, staging / "poster.jpg")
storage.place_file(produced, settings.download_root, rel)
nfo_ok = storage.write_sidecars(settings.download_root, rel, meta, poster_jpg)
final = storage.abs_path(settings.download_root, rel)
size = final.stat().st_size if final.exists() else None
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = size
asset.duration_s = meta.duration_s
asset.width = int(info["width"]) if info.get("width") else None
asset.height = int(info["height"]) if info.get("height") else None
asset.container = ext
asset.vcodec = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
asset.acodec = info.get("acodec") if info.get("acodec") not in (None, "none") else None
asset.title = meta.title
asset.uploader = meta.uploader
asset.uploader_url = adapter.uploader_url(info)
asset.upload_date = meta.upload_date
asset.thumbnail_url = meta.thumbnail_url
# Record a self-hosted poster for every download: it's the reliable link-preview
# (og:image) source — a remote thumbnail, esp. Facebook's signed CDN URL, expires
# and can't be relied on cross-origin. write_sidecars already wrote <base>.jpg from
# the thumbnail; ensure_poster returns that, or cuts a frame if there was none.
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.source_webpage_url = info.get("webpage_url")
asset.nfo_written = nfo_ok
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture the owner before commit expires the row (avoids a refresh SELECT just to notify).
owner_id = job.user_id if job else None
fields = {"status": JobStatus.DONE, "progress": 100, "phase": None, "speed_bps": None,
"eta_s": None}
if job and not job.display_name and meta.title:
# display_name is VARCHAR(255) — clip a long title (see _fill_asset_meta_early).
fields["display_name"] = meta.title[:255]
# Atomic terminal write: finalizes ONLY while the row is still running, so a child that
# finishes just after a route cancel/pause can't resurrect it (see finalize_if_running).
finalized = service.finalize_if_running(db, job_id, **fields)
db.commit()
if finalized:
_notify_download_done(db, owner_id, meta.title, job_id, kind="download")
log.info("job %s done: %s", job_id, rel)
except (_Aborted, editmod.EditAborted):
# Paused/canceled mid-download or mid-transcode: free the asset for a later retry; keep the
# job's status (the route set it to paused/canceled). ref_count is adjusted by the route.
_reset_asset_pending(asset_id)
log.info("job %s aborted (pause/cancel)", job_id)
except Exception as exc: # noqa: BLE001 — surface any failure to the user, keep worker alive
msg = str(exc)[:500]
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "error"
asset.error = msg
# Atomic fail: only while still running, so a route-set cancel/pause is never overwritten
# with `error` (error also holds the asset → would re-open the double-release class).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
)
db.commit()
log.warning("job %s failed: %s", job_id, msg)
finally:
shutil.rmtree(staging, ignore_errors=True)
def _reset_asset_pending(asset_id: int) -> None:
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset and asset.status == "downloading":
asset.status = "pending"
db.commit()
# --- editor derivatives (phase 2: trim/crop via ffmpeg) -------------------------------------
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
def _fail_edit(job_id: int, asset_id: int, msg: str) -> None:
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "error"
asset.error = msg
# Atomic fail: only while still running (see the download error path).
service.finalize_if_running(
db, job_id, status=JobStatus.ERROR, error=msg,
error_code=dlerrors.classify(msg), phase=None,
)
db.commit()
log.warning("edit job %s failed: %s", job_id, msg)
def _process_edit(job_id: int, asset_id: int) -> None:
"""Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
src = db.get(MediaAsset, job.source_asset_id) if job and job.source_asset_id else None
edit_spec = dict(job.edit_spec or {}) if job else {}
user_id = job.user_id if job else None
src_ready = src is not None and src.status == "ready" and bool(src.rel_path)
src_rel = src.rel_path if src else None
src_w, src_h, src_dur = (src.width, src.height, src.duration_s) if src else (None, None, None)
if not src_ready or not src_rel:
_fail_edit(job_id, asset_id, "The source download is no longer available.")
return
src_path = storage.abs_path(settings.download_root, src_rel)
if not src_path.exists():
_fail_edit(job_id, asset_id, "The source file is missing.")
return
src_ext = src_path.suffix.lstrip(".").lower() or "mp4"
crop = editmod.has_crop(edit_spec)
out_ext = "mp4" if editmod.needs_reencode(edit_spec) else src_ext
total_s = editmod.clip_duration(edit_spec, src_dur) or src_dur or 0
staging = _edit_staging_dir(asset_id)
try:
if staging.exists():
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True, exist_ok=True)
dest_staging = staging / f"out.{out_ext}"
_set_job(job_id, phase="editing", progress=0, speed_bps=None, eta_s=None)
if edit_spec.get("segments"):
cmd, prep = editmod.build_concat_plan(src_path, dest_staging, edit_spec, out_ext, staging)
for p, content in prep.items():
p.write_text(content, encoding="utf-8")
else:
cmd = editmod.build_edit_cmd(src_path, dest_staging, edit_spec, out_ext)
editmod.run_ffmpeg(
cmd,
total_s,
on_progress=_ffmpeg_progress_writer(job_id, "editing"),
should_cancel=_cancel_poll(job_id),
)
if not dest_staging.exists():
raise RuntimeError("ffmpeg produced no output file")
rel = storage.edit_rel_path(user_id, asset_id, out_ext)
storage.place_file(dest_staging, settings.download_root, rel)
final = storage.abs_path(settings.download_root, rel)
size = final.stat().st_size if final.exists() else None
_, retention_days = _download_settings()
expires = datetime.now(timezone.utc) + timedelta(days=retention_days)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
if asset:
asset.status = "ready"
asset.rel_path = rel
asset.size_bytes = size
asset.container = out_ext
asset.duration_s = total_s or asset.duration_s
if crop:
asset.width = edit_spec["crop"]["w"]
asset.height = edit_spec["crop"]["h"]
else:
asset.width, asset.height = src_w, src_h
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
asset.expires_at = expires
job = db.get(DownloadJob, job_id)
# Capture owner + title before commit expires the rows (avoids a refresh SELECT).
owner_id = job.user_id if job else None
dl_title = asset.title if asset else None
# Atomic terminal write (see the download path): don't resurrect a route-stopped job.
finalized = service.finalize_if_running(
db, job_id, status=JobStatus.DONE, progress=100, phase=None
)
db.commit()
if finalized:
_notify_download_done(db, owner_id, dl_title, job_id, kind="edit")
log.info("edit job %s done: %s", job_id, rel)
except editmod.EditAborted:
# Paused/canceled mid-encode: free the asset for a later retry (ref_count is the route's job).
_reset_asset_pending(asset_id)
log.info("edit job %s aborted (pause/cancel)", job_id)
except Exception as exc: # noqa: BLE001 — surface the failure, keep the worker alive
_fail_edit(job_id, asset_id, str(exc)[:500])
finally:
shutil.rmtree(staging, ignore_errors=True)
# --- loop ----------------------------------------------------------------------------------- # --- loop -----------------------------------------------------------------------------------
def _worker_loop(idx: int) -> None: def _worker_loop(idx: int) -> None:
@@ -0,0 +1,38 @@
"""E3 S3 (B-core-10): the config/state mutators flush but do NOT self-commit, so the caller owns the
transaction and a change lands together with its audit row. These lock that contract — a rollback
before the caller commits discards the change (proving no hidden internal commit), while a same-session
read still sees it (flush visibility). DB lane (`db` fixture) — skipped without Postgres.
"""
from app import state, sysconfig
def test_set_value_flushes_but_leaves_the_commit_to_the_caller(db):
default = sysconfig.get_int(db, "quota_daily_budget")
new = default + 4321
sysconfig.set_value(db, "quota_daily_budget", new)
assert sysconfig.get_int(db, "quota_daily_budget") == new # visible in-session (flushed)
db.rollback() # the caller never committed
assert sysconfig.get_int(db, "quota_daily_budget") == default # discarded — no hidden self-commit
def test_reset_flushes_but_leaves_the_commit_to_the_caller(db):
default = sysconfig.get_int(db, "quota_daily_budget")
sysconfig.set_value(db, "quota_daily_budget", default + 10)
db.commit() # a persisted override to reset
sysconfig.reset(db, "quota_daily_budget")
assert sysconfig.get_int(db, "quota_daily_budget") == default # reset visible in-session
db.rollback()
assert sysconfig.get_int(db, "quota_daily_budget") == default + 10 # reset discarded until committed
def test_set_sync_paused_flushes_but_leaves_the_commit_to_the_caller(db):
assert state.is_sync_paused(db) is False
state.set_sync_paused(db, True)
assert state.is_sync_paused(db) is True # visible in-session
db.rollback()
assert state.is_sync_paused(db) is False # discarded — the caller owns the commit
+33 -2
View File
@@ -9,13 +9,14 @@
Uses the `db` fixture (real Postgres) — skipped when no database is reachable. `_claim_job` runs in Uses the `db` fixture (real Postgres) — skipped when no database is reachable. `_claim_job` runs in
its OWN session, so every test COMMITS its setup before calling it. its OWN session, so every test COMMITS its setup before calling it.
""" """
from app.models import DownloadJob, DownloadQuota, User from app.models import DownloadJob, DownloadQuota, MediaAsset, User
from app.worker import _claim_job from app.worker import _claim_job
def _job(db, user_id, queue_pos, status="queued"): def _job(db, user_id, queue_pos, status="queued", asset_id=None):
j = DownloadJob( j = DownloadJob(
user_id=user_id, user_id=user_id,
asset_id=asset_id,
source_kind="youtube", source_kind="youtube",
source_ref="x" * 11, source_ref="x" * 11,
status=status, status=status,
@@ -26,6 +27,15 @@ def _job(db, user_id, queue_pos, status="queued"):
return j return j
def _asset(db, status, ref):
a = MediaAsset(
source_kind="url", source_ref=ref, format_sig="s", status=status, ref_count=1
)
db.add(a)
db.flush()
return a
def _quota(db, user_id, max_concurrent, unlimited=False): def _quota(db, user_id, max_concurrent, unlimited=False):
db.add( db.add(
DownloadQuota( DownloadQuota(
@@ -104,6 +114,27 @@ def test_stored_zero_concurrency_is_floored_not_starving(db):
assert db.get(DownloadJob, claimed).user_id == a.id # A's own first job runs (floored to 1) assert db.get(DownloadJob, claimed).user_id == a.id # A's own first job runs (floored to 1)
def test_job_whose_asset_is_already_downloading_is_skipped(db):
"""B-sub-3: a queued job whose shared asset is ALREADY `downloading` (another worker owns it) must
not be claimed — otherwise a second worker claims it, finds the asset busy, requeues, and ping-
pongs on it, starving everyone behind. The claim SQL excludes it, so the job behind it (on a
different, free asset) is served instead."""
u = User(email="asset-excl@example.com")
db.add(u)
db.flush()
_quota(db, u.id, max_concurrent=5) # high → the ASSET rule, not the per-user rule, does the excluding
busy = _asset(db, status="downloading", ref="busy")
free = _asset(db, status="pending", ref="free")
_job(db, u.id, queue_pos=1, status="running", asset_id=busy.id) # the holder actually downloading it
competitor = _job(db, u.id, queue_pos=2, asset_id=busy.id) # same asset — must be skipped
behind = _job(db, u.id, queue_pos=3, asset_id=free.id) # different asset — should be served
db.commit()
assert _claim_job() == behind.id # the competitor on the busy asset was skipped
db.expire_all()
assert db.get(DownloadJob, competitor.id).status == "queued" # still queued, never claimed
def test_unlimited_user_is_never_excluded(db): def test_unlimited_user_is_never_excluded(db):
u = User(email="unlimited@example.com") u = User(email="unlimited@example.com")
db.add(u) db.add(u)
+1 -1
View File
@@ -6,7 +6,7 @@ with "PosixPath('.') has an empty name" instead of downloading."""
import pytest import pytest
from app.downloads.edit import _concat_quote, parse_probe_streams from app.downloads.edit import _concat_quote, parse_probe_streams
from app.worker import _media_info, _produced_file from app.downloads.pipeline import _media_info, _produced_file
from pathlib import Path, PurePosixPath from pathlib import Path, PurePosixPath
+116
View File
@@ -0,0 +1,116 @@
"""E3 S2 characterization net — MediaAsset.ref_count enqueue/release accounting (B-sub-5).
`ref_count` tracks how many DownloadJobs still hold a shared cache asset; when it hits zero a READY
asset's file + row are freed. Enqueue +1's it and release_asset -1's it. Today those are Python
read-modify-writes (`asset.ref_count = (asset.ref_count or 0) ± 1`), which lose an update under two
concurrent enqueues/releases and can delete a file out from under a live holder. The fix makes the
increment/decrement atomic at the DB; this net locks the OBSERVABLE single-threaded semantics so the
conversion is provably behaviour-equivalent (concurrency itself can't be exercised in one session).
DB lane (`db` fixture) — skipped without Postgres.
"""
from sqlalchemy import select, text
from app.downloads import service
from app.models import DownloadJob, MediaAsset, User
def _user(db, email="ref@example.com"):
u = User(email=email)
db.add(u)
db.flush()
return u
def _ref_count(db, asset_id) -> int | None:
# Read straight from the row — the atomic-UPDATE fix leaves the ORM copy stale on purpose.
return db.execute(
text("SELECT ref_count FROM media_assets WHERE id = :id"), {"id": asset_id}
).scalar()
def _holding_asset(db, *, status="ready", ref_count=1, rel_path=None) -> MediaAsset:
asset = MediaAsset(
source_kind="url", source_ref=f"r{ref_count}-{status}", format_sig="sig",
status=status, ref_count=ref_count, rel_path=rel_path,
)
db.add(asset)
db.flush()
return asset
def _job(db, user_id, asset_id, status="done") -> DownloadJob:
j = DownloadJob(
user_id=user_id, asset_id=asset_id, source_kind="url", source_ref="r",
status=status, queue_pos=1, profile_snapshot={},
)
db.add(j)
db.flush()
return j
def test_two_enqueues_share_one_asset_and_stack_ref_count(db):
u = _user(db)
j1 = service.enqueue(db, u.id, "url", "http://example.com/v", {})
j2 = service.enqueue(db, u.id, "url", "http://example.com/v", {})
# One shared cache asset, two jobs on it, ref_count == 2.
assert j1.asset_id == j2.asset_id
assets = db.execute(select(MediaAsset)).scalars().all()
assert len(assets) == 1
assert _ref_count(db, j1.asset_id) == 2
def test_release_keeps_asset_until_last_holder_then_deletes_row(db):
u = _user(db)
asset = _holding_asset(db, status="ready", ref_count=2)
a = _job(db, u.id, asset.id, status="done")
b = _job(db, u.id, asset.id, status="done")
db.commit()
service.release_asset(db, a)
db.commit()
assert _ref_count(db, asset.id) == 1
assert db.get(MediaAsset, asset.id) is not None # still held by b
service.release_asset(db, b)
db.commit()
# Last holder gone → the ready asset's row is freed (rel_path None, so no file to unlink).
assert db.execute(select(MediaAsset)).scalars().all() == []
def test_release_is_a_noop_for_a_non_holding_job(db):
u = _user(db)
asset = _holding_asset(db, status="ready", ref_count=1)
canceled = _job(db, u.id, asset.id, status="canceled") # canceled ∉ HOLDING
db.commit()
service.release_asset(db, canceled)
db.commit()
assert _ref_count(db, asset.id) == 1 # untouched — a canceled job already released while holding
def test_release_never_drives_ref_count_negative(db):
u = _user(db)
# error-status asset (kept at zero, unlike ready) so we isolate the clamp from the delete branch.
asset = _holding_asset(db, status="error", ref_count=0) # already at zero (defensive)
j = _job(db, u.id, asset.id, status="error") # error ∈ HOLDING, so release runs
db.commit()
service.release_asset(db, j)
db.commit()
# The count clamps at 0 (never negative) even when released with nothing left to release.
assert _ref_count(db, asset.id) == 0
assert db.get(MediaAsset, asset.id) is not None
def test_errored_asset_is_not_deleted_at_zero(db):
u = _user(db)
asset = _holding_asset(db, status="error", ref_count=1)
j = _job(db, u.id, asset.id, status="error")
db.commit()
service.release_asset(db, j)
db.commit()
# An errored row is cheap (no file) and may be re-enqueued → never deleted here, only ready is.
assert _ref_count(db, asset.id) == 0
assert db.get(MediaAsset, asset.id) is not None
@@ -0,0 +1,86 @@
"""E3 S2 characterization net — download job state transitions (B-sub-8).
pause / resume / cancel / delete / redownload currently live inline in the routes, each hand-rolling
the asset bookkeeping (release, error-reset, fresh-asset rebuild). This pins their OBSERVABLE HTTP
behaviour so the logic can move into `service.*` methods (with the routes as thin callers) provably
behaviour-equivalently. DB lane (`db` fixture) — skipped without Postgres.
"""
from sqlalchemy import text
from app.models import DownloadJob, MediaAsset, User
def _seed(db, *, job_status, asset_status="ready", ref_count=1, rel_path=None):
u = User(email=f"tr-{job_status}-{asset_status}-{ref_count}@example.com")
db.add(u)
db.flush()
asset = MediaAsset(
source_kind="url", source_ref=f"r-{u.id}", format_sig="s",
status=asset_status, ref_count=ref_count, rel_path=rel_path,
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=u.id, asset_id=asset.id, source_kind="url", source_ref="r",
status=job_status, queue_pos=1, profile_snapshot={},
)
db.add(job)
db.commit()
return u, asset, job
def _ref(db, asset_id):
return db.execute(
text("SELECT ref_count FROM media_assets WHERE id = :id"), {"id": asset_id}
).scalar()
def test_cancel_releases_the_asset_and_marks_canceled(db, as_user):
u, asset, job = _seed(db, job_status="running", asset_status="downloading", ref_count=1)
r = as_user(u).post(f"/api/downloads/{job.id}/cancel")
assert r.status_code == 200, r.text
db.expire_all()
assert db.get(DownloadJob, job.id).status == "canceled"
assert _ref(db, asset.id) == 0 # the hold was released while the job still counted as holding
def test_pause_then_resume_round_trips_queued(db, as_user):
u, asset, job = _seed(db, job_status="queued", asset_status="pending")
c = as_user(u)
assert c.post(f"/api/downloads/{job.id}/pause").status_code == 200
db.expire_all()
assert db.get(DownloadJob, job.id).status == "paused"
assert c.post(f"/api/downloads/{job.id}/resume").status_code == 200
db.expire_all()
assert db.get(DownloadJob, job.id).status == "queued"
def test_resume_resets_a_failed_shared_asset_to_pending(db, as_user):
u, asset, job = _seed(db, job_status="error", asset_status="error")
r = as_user(u).post(f"/api/downloads/{job.id}/resume")
assert r.status_code == 200, r.text
db.expire_all()
assert db.get(DownloadJob, job.id).status == "queued"
# A failed shared asset is reset so the worker actually re-downloads instead of re-inheriting the error.
assert db.get(MediaAsset, asset.id).status == "pending"
def test_delete_releases_and_removes_the_job(db, as_user):
u, asset, job = _seed(db, job_status="done", asset_status="ready", ref_count=1, rel_path=None)
r = as_user(u).request("DELETE", f"/api/downloads/{job.id}")
assert r.status_code == 200, r.text
db.expire_all()
assert db.get(DownloadJob, job.id) is None
assert db.get(MediaAsset, asset.id) is None # last holder gone → ready asset freed
def test_redownload_rebuilds_into_a_fresh_asset(db, as_user):
u, asset, job = _seed(db, job_status="done", asset_status="ready", ref_count=1, rel_path=None)
old_asset_id = asset.id
r = as_user(u).post(f"/api/downloads/{job.id}/redownload")
assert r.status_code == 200, r.text
db.expire_all()
j = db.get(DownloadJob, job.id)
assert j.status == "queued"
assert j.progress == 0
assert j.asset_id != old_asset_id # a fresh private asset, not the old (now-released) one
+5 -3
View File
@@ -41,14 +41,16 @@ def _seed(db):
Subscription(user_id=user.id, channel_id="UCa", priority=5, hidden=False), Subscription(user_id=user.id, channel_id="UCa", priority=5, hidden=False),
Subscription(user_id=user.id, channel_id="UCb", priority=0, hidden=False), Subscription(user_id=user.id, channel_id="UCb", priority=0, hidden=False),
]) ])
# 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2 and day 3 x2, view_count # 7 normal/organic videos. Deliberate duplicates (published_at day 1 x2, day 2 x2 and day 3 x2,
# 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the sort columns. # view_count 100 x2, duration 60 x2 / 30 x2) and NULLs spread across the still-nullable sort columns
# (view_count / duration_seconds / title / subscriber_count) to exercise nulls_last. published_at is
# NOT NULL (migration 0068), so no NULL there — the duplicates keep the id tiebreaker load-bearing.
rows = [ rows = [
# id ch published views dur title # id ch published views dur title
("vid00", "UCa", _dt(1), 100, 60, "alpha"), ("vid00", "UCa", _dt(1), 100, 60, "alpha"),
("vid01", "UCa", _dt(1), 100, 60, "beta"), ("vid01", "UCa", _dt(1), 100, 60, "beta"),
("vid02", "UCa", _dt(2), None, None, "gamma"), ("vid02", "UCa", _dt(2), None, None, "gamma"),
("vid03", "UCb", None, 50, 120, None), ("vid03", "UCb", _dt(2), 50, 120, None),
("vid04", "UCb", _dt(3), 200, 30, "delta"), ("vid04", "UCb", _dt(3), 200, 30, "delta"),
("vid05", "UCb", _dt(3), None, 30, "epsilon"), ("vid05", "UCb", _dt(3), None, 30, "epsilon"),
("vid06", "UCa", _dt(28), None, None, "zeta"), ("vid06", "UCa", _dt(28), None, None, "zeta"),
+168
View File
@@ -0,0 +1,168 @@
"""Feed search + watch-state views (E2 S3), DB-lane.
Regression nets for two query rewrites that must stay behaviour-equivalent:
- PERF-12: the search channel-name branch moved from an ORed `unaccent(channels.title) ILIKE` (which
defeated the search_vector GIN) to a pre-resolved `channel_id IN (...)`. A search must STILL return
both videos whose text matches AND videos whose CHANNEL name matches.
- PERF-16: the watched/hidden/in_progress views moved from an outer join + `coalesce(state.status,'new')`
to an inner join on video_states + the raw column. Each view must return exactly the same set.
"""
from datetime import datetime, timezone
import pytest
from app.models import Channel, ChannelTag, Subscription, Tag, Video, VideoState
def _dt(day: int) -> datetime:
return datetime(2024, 1, day, tzinfo=timezone.utc)
def _video(vid: str, ch: str, title: str) -> Video:
return Video(
id=vid, channel_id=ch, published_at=_dt(1), title=title,
is_short=False, live_status="none", via_search=False, via_explore=False,
)
def _feed_ids(client, query: str) -> set[str]:
resp = client.get(f"/api/feed?{query}&limit=50")
assert resp.status_code == 200, resp.text
return {it["id"] for it in resp.json()["items"]}
def test_search_matches_both_video_text_and_channel_name(db, as_user, make_user):
# Two subscribed channels: one whose NAME matches the query (its video's text does not), and one
# whose video TEXT matches (its name does not). The pre-resolved channel_id branch must still OR in
# the name match, so both videos come back — and an unrelated video on a matching-name-less channel
# stays out.
user = make_user(email="search@example.com")
db.add_all([
Channel(id="UCkube", title="Kubernetes Weekly", subscriber_count=10),
Channel(id="UCother", title="Some Other Channel", subscriber_count=10),
])
db.add_all([
Subscription(user_id=user.id, channel_id="UCkube", priority=0, hidden=False),
Subscription(user_id=user.id, channel_id="UCother", priority=0, hidden=False),
])
db.add_all([
_video("vidChan", "UCkube", "Totally unrelated content"), # channel NAME matches
_video("vidText", "UCother", "kubernetes deep dive"), # video TEXT matches
_video("vidNone", "UCother", "cooking pasta at home"), # neither matches
])
db.commit()
ids = _feed_ids(as_user(user), "q=kubernetes&show=all")
assert ids == {"vidChan", "vidText"}
def test_watch_state_views_return_the_right_sets(db, as_user, make_user):
user = make_user(email="views@example.com")
db.add(Channel(id="UCv", title="Channel V", subscriber_count=10))
db.add(Subscription(user_id=user.id, channel_id="UCv", priority=0, hidden=False))
db.add_all([
_video("vWatch", "UCv", "watched one"),
_video("vHide", "UCv", "hidden one"),
_video("vProg", "UCv", "in progress one"),
_video("vNew", "UCv", "brand new one"),
])
db.flush()
db.add_all([
VideoState(user_id=user.id, video_id="vWatch", status="watched"),
VideoState(user_id=user.id, video_id="vHide", status="hidden"),
# in progress = a stored resume position while still "new" (not watched/hidden)
VideoState(user_id=user.id, video_id="vProg", status="new", position_seconds=30),
# vNew has NO state row -> "new"/unwatched
])
db.commit()
client = as_user(user)
assert _feed_ids(client, "show=watched") == {"vWatch"}
assert _feed_ids(client, "show=hidden") == {"vHide"}
assert _feed_ids(client, "show=in_progress") == {"vProg"}
# unwatched excludes watched + hidden, keeps new AND in-progress
assert _feed_ids(client, "show=unwatched") == {"vProg", "vNew"}
# "all" shows everything except hidden
assert _feed_ids(client, "show=all") == {"vWatch", "vProg", "vNew"}
def _seed_tagged(db, make_user):
"""A user subscribed to 4 channels with a spread of language/topic/other tags and differing video
counts, so the count sum and the per-category facet drill-down are all non-trivial."""
user = make_user(email="summary@example.com")
tags = {
"L1": Tag(user_id=None, name="Hungarian", category="language"),
"L2": Tag(user_id=None, name="English", category="language"),
"T1": Tag(user_id=None, name="Tech", category="topic"),
"T2": Tag(user_id=None, name="Gaming", category="topic"),
"O1": Tag(user_id=None, name="Favourite", category="other"),
"O2": Tag(user_id=None, name="Docs", category="other"),
# A USER-owned tag (+ a user-owned ChannelTag link below) so the visibility branch
# `ChannelTag.user_id == user.id` is exercised, not only the system `user_id IS NULL` one.
"U1": Tag(user_id=user.id, name="MyPin", category="other"),
}
db.add_all(tags.values())
chans = {c: Channel(id=c, title=c, subscriber_count=10) for c in ("chA", "chB", "chC", "chD")}
db.add_all(chans.values())
db.flush()
# channel -> (system) tag keys. chA carries two languages (L1+L2); chC two "other" tags (O1+O2)
# AND chA carries O1 too — so the per-category any/all match narrows NON-TRIVIALLY (AND != OR) for
# language and other, not only topic: e.g. other {O1,O2} → OR {chA,chC} vs AND {chC}; language
# {L1,L2} → OR {chA,chB,chC} vs AND {chA}. (For topic, chA and chB each carry a DIFFERENT topic tag
# — T1 vs T2 — so topic {T1,T2} → OR {chA,chB,chC} vs AND {} empty.)
links = {
"chA": ["L1", "L2", "T1", "O1"],
"chB": ["L1", "T2"],
"chC": ["L2", "T1", "O1", "O2"],
"chD": [],
}
for cid, tkeys in links.items():
db.add(Subscription(user_id=user.id, channel_id=cid, priority=0, hidden=False))
for tk in tkeys:
db.add(ChannelTag(channel_id=cid, tag_id=tags[tk].id, user_id=None))
# The user's own private link (visible only to them): chB carries U1 for this user.
db.add(ChannelTag(channel_id="chB", tag_id=tags["U1"].id, user_id=user.id))
# differing video counts per channel (chA:3, chB:2, chC:1, chD:2)
counts = {"chA": 3, "chB": 2, "chC": 1, "chD": 2}
for cid, n in counts.items():
db.add_all([_video(f"{cid}v{i}", cid, f"{cid} clip {i}") for i in range(n)])
db.commit()
return user, {k: v.id for k, v in tags.items()}
# Query-param combos (as a template with {L1}/{T1}/… placeholders) that stress every facet path:
# no tags, single-category, cross-category, disjunctive + conjunctive topic modes, and a search filter.
_SUMMARY_CASES = [
"show=all",
"show=all&tags={L1}",
"show=all&tags={T1}",
"show=all&tags={L1}&tags={T1}",
"show=all&tags={T1}&tags={T2}&tag_mode_topic=or",
"show=all&tags={T1}&tags={T2}&tag_mode_topic=and",
# Per-category any/all is NOT topic-only — language and other must drill down identically too.
"show=all&tags={L1}&tags={L2}&tag_mode_language=or",
"show=all&tags={L1}&tags={L2}&tag_mode_language=and",
"show=all&tags={O1}&tags={O2}&tag_mode_other=and",
"show=all&tags={U1}", # user-owned tag (visibility == user.id branch)
"show=all&tags={U1}&tags={L1}", # user tag AND a system tag, cross-category
"show=all&q=clip",
"show=unwatched",
]
@pytest.mark.parametrize("case", _SUMMARY_CASES)
def test_feed_summary_matches_count_plus_facets(db, as_user, make_user, case):
# /feed/summary must return byte-identical numbers to the old /feed/count + /facets it replaces,
# across the full matrix of tag filters — it reimplements the tag drill-down in Python, so this is
# the equivalence net.
user, tag_ids = _seed_tagged(db, make_user)
client = as_user(user)
qs = case.format(**tag_ids)
summary = client.get(f"/api/feed/summary?{qs}")
count = client.get(f"/api/feed/count?{qs}")
facets = client.get(f"/api/facets?{qs}")
assert summary.status_code == count.status_code == facets.status_code == 200
assert summary.json()["count"] == count.json()["count"], qs
assert summary.json()["facets"] == facets.json()["counts"], qs
+47 -15
View File
@@ -4,20 +4,20 @@ run_ffmpeg calls `on_progress`/`should_cancel` once per ffmpeg output LINE (`-pr
emits a block of ~12 every half second), and both callbacks are database round-trips: `_set_job` emits a block of ~12 every half second), and both callbacks are database round-trips: `_set_job`
opens a session and UPDATE+COMMITs, `_job_status` opens one and SELECTs. Unthrottled, a long opens a session and UPDATE+COMMITs, `_job_status` opens one and SELECTs. Unthrottled, a long
re-encode is ~10^5 of each.""" re-encode is ~10^5 of each."""
from app import worker import app.downloads.pipeline as pipeline
class TestProgressWriter: class TestProgressWriter:
def test_writes_the_first_value_immediately(self, monkeypatch): def test_writes_the_first_value_immediately(self, monkeypatch):
writes = [] writes = []
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f)) monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
worker._ffmpeg_progress_writer(7, "processing")(12.4) pipeline._ffmpeg_progress_writer(7, "processing")(12.4)
assert writes == [{"progress": 12, "phase": "processing"}] assert writes == [{"progress": 12, "phase": "processing"}]
def test_a_burst_of_lines_costs_one_write(self, monkeypatch): def test_a_burst_of_lines_costs_one_write(self, monkeypatch):
writes = [] writes = []
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f)) monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
write = worker._ffmpeg_progress_writer(7, "processing") write = pipeline._ffmpeg_progress_writer(7, "processing")
for pct in range(0, 40): # what one second of ffmpeg output looks like for pct in range(0, 40): # what one second of ffmpeg output looks like
write(pct) write(pct)
assert len(writes) == 1 assert len(writes) == 1
@@ -25,8 +25,8 @@ class TestProgressWriter:
def test_the_next_second_writes_again(self, monkeypatch): def test_the_next_second_writes_again(self, monkeypatch):
writes = [] writes = []
clock = {"t": 1000.0} clock = {"t": 1000.0}
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f)) monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
write = worker._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"]) write = pipeline._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(10) write(10)
write(11) # same second → dropped write(11) # same second → dropped
clock["t"] += 1.5 clock["t"] += 1.5
@@ -36,19 +36,51 @@ class TestProgressWriter:
def test_an_unchanged_whole_percent_is_never_rewritten(self, monkeypatch): def test_an_unchanged_whole_percent_is_never_rewritten(self, monkeypatch):
writes = [] writes = []
clock = {"t": 1000.0} clock = {"t": 1000.0}
monkeypatch.setattr(worker, "_set_job", lambda job_id, **f: writes.append(f)) monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
write = worker._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"]) monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
write = pipeline._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(10.1) write(10.1)
clock["t"] += 5 clock["t"] += 5
write(10.9) # still 10% — the bar shows integers write(10.9) # still 10% — the bar shows integers
assert [w["progress"] for w in writes] == [10] assert [w["progress"] for w in writes] == [10]
def test_a_slow_same_percent_stretch_heartbeats_the_watchdog(self, monkeypatch):
"""B-sub-4: a re-encode that sits on one integer percent for minutes must NOT be killed as
stalled — the writer touches the row (heartbeat) each interval even though it writes no new
progress, so the stall watchdog keeps seeing a fresh updated_at."""
writes, touches = [], []
clock = {"t": 1000.0}
monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: touches.append(job_id))
write = pipeline._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(10.1) # first real write → progress 10
for _ in range(3): # three more intervals stuck on 10%
clock["t"] += 1.5
write(10.4)
assert [w["progress"] for w in writes] == [10] # progress never rewritten
assert touches == [7, 7, 7] # …but each interval heartbeats the watchdog
def test_no_redundant_heartbeat_when_a_progress_write_just_happened(self, monkeypatch):
"""A progress write already refreshes updated_at, so an unchanged-percent call in the SAME
interval must NOT also heartbeat — that would be a redundant DB write (~2x during an active
download). The heartbeat only fires once no write/beat touched the row for an interval."""
writes, touches = [], []
clock = {"t": 1000.0}
monkeypatch.setattr(pipeline, "_set_job", lambda job_id, **f: writes.append(f))
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: touches.append(job_id))
write = pipeline._ffmpeg_progress_writer(7, "processing", now=lambda: clock["t"])
write(6.0) # progress write at t=1000 → refreshes updated_at
clock["t"] += 0.3
write(6.4) # same percent, SAME interval → the write already touched the row, so no heartbeat
assert [w["progress"] for w in writes] == [6]
assert touches == []
class TestCancelPoll: class TestCancelPoll:
def test_asks_once_then_reuses_the_answer(self, monkeypatch): def test_asks_once_then_reuses_the_answer(self, monkeypatch):
calls = [] calls = []
monkeypatch.setattr(worker, "_job_status", lambda job_id: calls.append(job_id) or "running") monkeypatch.setattr(pipeline, "_job_status", lambda job_id: calls.append(job_id) or "running")
cancelled = worker._cancel_poll(7) cancelled = pipeline._cancel_poll(7)
for _ in range(40): for _ in range(40):
assert cancelled() is False assert cancelled() is False
assert len(calls) == 1 assert len(calls) == 1
@@ -61,8 +93,8 @@ class TestCancelPoll:
calls.append(job_id) calls.append(job_id)
return "cancelled" return "cancelled"
monkeypatch.setattr(worker, "_job_status", status) monkeypatch.setattr(pipeline, "_job_status", status)
cancelled = worker._cancel_poll(7, now=lambda: clock["t"]) cancelled = pipeline._cancel_poll(7, now=lambda: clock["t"])
assert cancelled() is True assert cancelled() is True
clock["t"] += 60 # cancellation is sticky: no re-query, and no un-cancelling clock["t"] += 60 # cancellation is sticky: no re-query, and no un-cancelling
assert cancelled() is True assert cancelled() is True
@@ -71,8 +103,8 @@ class TestCancelPoll:
def test_a_cancel_arriving_later_is_seen_on_the_next_tick(self, monkeypatch): def test_a_cancel_arriving_later_is_seen_on_the_next_tick(self, monkeypatch):
clock = {"t": 1000.0} clock = {"t": 1000.0}
state = {"status": "running"} state = {"status": "running"}
monkeypatch.setattr(worker, "_job_status", lambda job_id: state["status"]) monkeypatch.setattr(pipeline, "_job_status", lambda job_id: state["status"])
cancelled = worker._cancel_poll(7, now=lambda: clock["t"]) cancelled = pipeline._cancel_poll(7, now=lambda: clock["t"])
assert cancelled() is False assert cancelled() is False
state["status"] = "paused" state["status"] = "paused"
clock["t"] += 1.0 # the throttle must not delay a cancel by more than its interval clock["t"] += 1.0 # the throttle must not delay a cancel by more than its interval
+31
View File
@@ -0,0 +1,31 @@
"""Plex card serializers (app.plex.cards) — pure model->dict projections, no DB needed.
E5 S1: movie/show cards now also emit `art` (the wide backdrop, for a Discover rail's landscape tile
and the crossfading hero), but ONLY when the item actually has an `art_key` — otherwise the URL would
resolve to a blank image on every card that tried to use it, so it must be null and let the client fall
back to the poster."""
from app.models import PlexItem, PlexShow
from app.plex.cards import movie_card, show_card
def test_movie_card_emits_art_when_art_key_present():
it = PlexItem(rating_key="m1", kind="movie", title="M", year=2020, art_key="/library/art/1")
card = movie_card(it, None)
assert card["thumb"] == "/api/plex/image/m1"
assert card["art"] == "/api/plex/image/m1?variant=art"
def test_movie_card_art_is_none_without_art_key():
it = PlexItem(rating_key="m2", kind="movie", title="M2")
assert movie_card(it, None)["art"] is None
def test_show_card_emits_art_when_art_key_present():
sh = PlexShow(rating_key="s1", title="S", art_key="/library/art/2")
assert show_card(sh)["art"] == "/api/plex/image/s1?variant=art"
def test_show_card_art_is_none_without_art_key():
sh = PlexShow(rating_key="s2", title="S2")
assert show_card(sh)["art"] is None
@@ -0,0 +1,301 @@
"""Piece 1a: the Plex collection filter is multi-select with any/all combine (a title can belong to
several collections). `collections=A,B` = ANY (OR, default); `&collection_mode=all` = ALL (AND).
DB lane (`db` fixture) — skipped without Postgres.
"""
from datetime import datetime, timezone
from fastapi.testclient import TestClient
from sqlalchemy import select
import app.state as state
from app.auth import admin_user, current_user
from app.db import get_db
from app.main import app
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexShow, User
from app.plex.client import PlexError
from app.routes.plex import collections as coll_routes
def _seed(db):
"""Three movies: one in collection A, one in B, one in both — so any/all are distinguishable."""
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
common = dict(
library_id=lib.id,
kind="movie",
people_enriched_at=datetime(2024, 1, 1, tzinfo=timezone.utc),
)
db.add_all([
PlexItem(rating_key="mA", title="Only A", collection_keys=["A"],
added_at=datetime(2024, 1, 1, tzinfo=timezone.utc), **common),
PlexItem(rating_key="mB", title="Only B", collection_keys=["B"],
added_at=datetime(2024, 1, 2, tzinfo=timezone.utc), **common),
PlexItem(rating_key="mAB", title="A and B", collection_keys=["A", "B"],
added_at=datetime(2024, 1, 3, tzinfo=timezone.utc), **common),
])
user = User(email="plexcoll@example.com")
db.add(user)
db.commit()
return user
def _get(db, user, monkeypatch, path):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
return TestClient(app).get(path)
finally:
app.dependency_overrides.clear()
def _titles(resp):
return {i["title"] for i in resp.json()["items"]}
def test_single_collection_filters_to_its_members(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/library?scope=movie&collections=A")
assert _titles(resp) == {"Only A", "A and B"}
def test_multiple_collections_any_is_the_union(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/library?scope=movie&collections=A,B")
assert _titles(resp) == {"Only A", "Only B", "A and B"}
def test_multiple_collections_all_is_the_intersection(db, monkeypatch):
user = _seed(db)
resp = _get(
db, user, monkeypatch,
"/api/plex/library?scope=movie&collections=A,B&collection_mode=all",
)
assert _titles(resp) == {"A and B"}
def test_collection_cards_carry_montage_art_and_total_runtime(db, monkeypatch):
# The browse-tile montage (top films, best-rated first) + the collection's total runtime.
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="A", library_id=lib.id, title="Coll A", child_count=2))
common = dict(library_id=lib.id, kind="movie", collection_keys=["A"])
db.add_all([
PlexItem(rating_key="m1", title="M1", duration_s=3600, rating=8.0,
added_at=datetime(2024, 1, 1, tzinfo=timezone.utc), **common),
PlexItem(rating_key="m2", title="M2", duration_s=1800, rating=9.0,
added_at=datetime(2024, 1, 2, tzinfo=timezone.utc), **common),
])
user = User(email="collart@example.com")
db.add(user)
db.commit()
card = next(c for c in _get(db, user, monkeypatch, "/api/plex/collections").json()["collections"]
if c["id"] == "A")
assert card["art"] == ["m2", "m1"] # best-rated first (m2 9.0 > m1 8.0)
assert card["total_runtime_s"] == 5400 # 3600 + 1800
def test_create_collection_rejects_duplicate_name(db, monkeypatch):
# Same-named collections conflate in Plex, so a create with an existing name (case-insensitive) is
# rejected 409 — BEFORE any Plex call (the guard runs first, so the DB lane can assert it).
monkeypatch.setattr(state, "_configured_cache", True)
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="A", library_id=lib.id, title="Marvel"))
db.add(PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="M1"))
user = User(email="dup@example.com")
db.add(user)
db.commit()
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
app.dependency_overrides[admin_user] = lambda: user
try:
r = TestClient(app).post(
"/api/plex/collections",
json={"library": "1", "title": "marvel", "item_rating_key": "m1"},
)
assert r.status_code == 409, r.text
finally:
app.dependency_overrides.clear()
def test_collection_detail_hero_and_ordered_members(db, monkeypatch):
# The Collection info page: hero aggregates (year range, most-restrictive cert, total runtime) +
# members ordered by release year then title.
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="A", library_id=lib.id, title="Coll A", summary="S", child_count=2))
common = dict(library_id=lib.id, kind="movie", collection_keys=["A"])
db.add_all([
PlexItem(rating_key="later", title="Later", year=1996, duration_s=3600, content_rating="PG", **common),
PlexItem(rating_key="early", title="Early", year=1992, duration_s=1800, content_rating="G", **common),
])
user = User(email="cd@example.com")
db.add(user)
db.commit()
d = _get(db, user, monkeypatch, "/api/plex/collections/A").json()
assert d["title"] == "Coll A" and d["summary"] == "S"
assert d["year_min"] == 1992 and d["year_max"] == 1996
assert d["content_rating"] == "PG" # most restrictive of {G, PG}
assert d["total_runtime_s"] == 5400
assert [i["id"] for i in d["items"]] == ["early", "later"] # release-year order
# --- Write-through collection mutations (create/add/remove/rename write to Plex, then resync the local
# mirror). We stub PlexClient so the DB lane can drive the happy path, the 422-on-mutation-failure
# rollback, and the 502-on-resync-failure split (a successful write whose follow-up refresh read fails
# must NOT be reported as a write failure). ------------------------------------------------------------
class _CollStub:
"""A PlexClient stand-in recording calls; `fail_on` makes one method raise PlexError."""
def __init__(self, fail_on: str | None = None):
self.fail_on = fail_on
self.calls: list[str] = []
def __enter__(self):
return self
def __exit__(self, *a):
return False
def _maybe(self, name: str):
self.calls.append(name)
if self.fail_on == name:
raise PlexError("boom")
def create_collection(self, section_key, title, item_rating_key):
self._maybe("create_collection")
return {"ratingKey": "NEW"}
def add_collection_item(self, rk, item):
self._maybe("add_collection_item")
def remove_collection_item(self, rk, item):
self._maybe("remove_collection_item")
def rename_collection(self, section_key, rk, title):
self._maybe("rename_collection")
def delete_collection(self, rk):
self._maybe("delete_collection")
def _resync_ok(db, plex, lib, rk):
"""Stand in for the post-write refresh: return the existing local row (no live Plex read)."""
return db.execute(select(PlexCollection).where(PlexCollection.rating_key == rk)).scalars().first()
def _seed_editable(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
# Plain, admin-editable collection (title isn't a provider word, not smart → source "collection").
db.add(PlexCollection(rating_key="A", library_id=lib.id, title="My Coll", editable=True,
child_count=1))
db.add(PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="M1",
collection_keys=["A"]))
user = User(email="mut@example.com")
db.add(user)
db.commit()
return lib, user
def _admin_client(db, user, monkeypatch, stub, resync=_resync_ok):
monkeypatch.setattr(state, "_configured_cache", True)
monkeypatch.setattr(coll_routes, "PlexClient", lambda db: stub)
monkeypatch.setattr(coll_routes.plex_sync, "resync_collection", resync)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
app.dependency_overrides[admin_user] = lambda: user
return TestClient(app)
def test_rename_writes_through_to_plex(db, monkeypatch):
_lib, user = _seed_editable(db)
stub = _CollStub()
client = _admin_client(db, user, monkeypatch, stub)
try:
r = client.patch("/api/plex/collections/A", json={"title": "Renamed"})
assert r.status_code == 200, r.text
assert "rename_collection" in stub.calls
finally:
app.dependency_overrides.clear()
def test_remove_item_writes_through_to_plex(db, monkeypatch):
_lib, user = _seed_editable(db)
stub = _CollStub()
client = _admin_client(db, user, monkeypatch, stub)
try:
r = client.delete("/api/plex/collections/A/items/m1")
assert r.status_code == 200, r.text
assert "remove_collection_item" in stub.calls
finally:
app.dependency_overrides.clear()
def test_mutation_failure_returns_422_write_failed(db, monkeypatch):
# The Plex mutation itself raised → the write never landed → 422 "Plex write failed", and resync
# is never reached (stub.calls holds only the mutating call).
_lib, user = _seed_editable(db)
stub = _CollStub(fail_on="rename_collection")
client = _admin_client(db, user, monkeypatch, stub)
try:
r = client.patch("/api/plex/collections/A", json={"title": "Renamed"})
assert r.status_code == 422, r.text
assert "write failed" in r.json()["detail"].lower()
finally:
app.dependency_overrides.clear()
def test_resync_failure_after_successful_write_is_422_and_rolls_back(db, monkeypatch):
# The mutation landed on Plex; only the follow-up refresh READ failed. We must (a) NOT claim a
# write failure and NOT use a 5xx (which the frontend treats as offline) → 422 with a "reconcile
# on the next sync" detail, and (b) roll back any partial local upsert the resync had flushed.
_lib, user = _seed_editable(db)
stub = _CollStub()
def _resync_boom(db_, plex, lib_, rk):
# Simulate resync having flushed a partial local change before its Plex read blew up — the
# route's rollback must undo it (this is what makes the rollback assertion non-vacuous).
row = db_.execute(select(PlexCollection).where(PlexCollection.rating_key == rk)).scalars().first()
row.title = "PARTIAL_SHOULD_ROLL_BACK"
db_.flush()
raise PlexError("read failed")
client = _admin_client(db, user, monkeypatch, stub, resync=_resync_boom)
try:
r = client.patch("/api/plex/collections/A", json={"title": "Renamed"})
assert r.status_code == 422, r.text
assert "reconcile" in r.json()["detail"].lower()
assert "rename_collection" in stub.calls # the write DID happen
finally:
app.dependency_overrides.clear()
row = db.execute(select(PlexCollection).where(PlexCollection.rating_key == "A")).scalars().first()
assert row.title == "My Coll" # the partial resync upsert was rolled back
def test_show_collection_gets_montage_art_from_shows(db, monkeypatch):
# A show-library collection must get its montage from PlexShow members (else it renders blank) — the
# A3 fix that unions movie + show art. Runtime stays 0 (shows have no single duration).
lib = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="S", library_id=lib.id, title="Show Coll", child_count=2))
db.add_all([
PlexShow(rating_key="s1", library_id=lib.id, title="S1", collection_keys=["S"], rating=7.0,
added_at=datetime(2024, 1, 1, tzinfo=timezone.utc)),
PlexShow(rating_key="s2", library_id=lib.id, title="S2", collection_keys=["S"], rating=9.0,
added_at=datetime(2024, 1, 2, tzinfo=timezone.utc)),
])
user = User(email="showcoll@example.com")
db.add(user)
db.commit()
card = next(c for c in _get(db, user, monkeypatch, "/api/plex/collections").json()["collections"]
if c["id"] == "S")
assert card["art"] == ["s2", "s1"] # show members, best-rated first
assert card["total_runtime_s"] == 0
+6 -6
View File
@@ -13,7 +13,7 @@ from types import SimpleNamespace
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
import app.routes.plex.browse as browse import app.routes.plex.detail as detail_routes
import app.state as state import app.state as state
from app.auth import current_user from app.auth import current_user
from app.db import get_db from app.db import get_db
@@ -216,7 +216,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch):
user = _user(db, "outage@example.com") user = _user(db, "outage@example.com")
db.commit() db.commit()
monkeypatch.setattr(browse, "PlexClient", lambda db: _StubClient(raise_=True)) monkeypatch.setattr(detail_routes, "PlexClient", lambda db: _StubClient(raise_=True))
resp = _get(db, user, monkeypatch, "/api/plex/item/m1") resp = _get(db, user, monkeypatch, "/api/plex/item/m1")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
body = resp.json() body = resp.json()
@@ -240,7 +240,7 @@ def test_item_detail_movie_lazy_fills_when_unenriched(db, monkeypatch):
user = _user(db, "lazy@example.com") user = _user(db, "lazy@example.com")
db.commit() db.commit()
monkeypatch.setattr(browse, "PlexClient", lambda db: _StubClient(RICH_META)) monkeypatch.setattr(detail_routes, "PlexClient", lambda db: _StubClient(RICH_META))
resp = _get(db, user, monkeypatch, "/api/plex/item/m1") resp = _get(db, user, monkeypatch, "/api/plex/item/m1")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
assert resp.json()["imdb_id"] == "tt0068646" assert resp.json()["imdb_id"] == "tt0068646"
@@ -267,7 +267,7 @@ def test_item_detail_episode_serves_rich_from_live(db, monkeypatch):
user = _user(db, "ep@example.com") user = _user(db, "ep@example.com")
db.commit() db.commit()
monkeypatch.setattr(browse, "PlexClient", lambda db: _StubClient(RICH_META)) monkeypatch.setattr(detail_routes, "PlexClient", lambda db: _StubClient(RICH_META))
resp = _get(db, user, monkeypatch, "/api/plex/item/e1") resp = _get(db, user, monkeypatch, "/api/plex/item/e1")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
body = resp.json() body = resp.json()
@@ -298,7 +298,7 @@ def test_show_detail_serves_from_row(db, monkeypatch):
def _boom(db_, rk): def _boom(db_, rk):
raise RuntimeError("plex unavailable") raise RuntimeError("plex unavailable")
monkeypatch.setattr(browse, "_show_enrichment", _boom) monkeypatch.setattr(detail_routes, "_show_enrichment", _boom)
resp = _get(db, user, monkeypatch, "/api/plex/show/s1") resp = _get(db, user, monkeypatch, "/api/plex/show/s1")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
body = resp.json()["show"] body = resp.json()["show"]
@@ -322,7 +322,7 @@ def test_show_detail_lazy_fills_when_unenriched(db, monkeypatch):
db.commit() db.commit()
# _show_enrichment returns (live meta, no related) → the fill branch runs. # _show_enrichment returns (live meta, no related) → the fill branch runs.
monkeypatch.setattr(browse, "_show_enrichment", lambda db_, rk: (RICH_META, [])) monkeypatch.setattr(detail_routes, "_show_enrichment", lambda db_, rk: (RICH_META, []))
resp = _get(db, user, monkeypatch, "/api/plex/show/s1") resp = _get(db, user, monkeypatch, "/api/plex/show/s1")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
assert resp.json()["show"]["imdb_id"] == "tt0068646" assert resp.json()["show"]["imdb_id"] == "tt0068646"
+267
View File
@@ -0,0 +1,267 @@
"""E5 S4: the Plex Discover rail-surface. GET /api/plex/discover returns the consolidated rail
payload (continue watching / recently added / genre tiles / because-you-watched) served from the DB,
and GET /api/plex/surprise returns a scope-bounded random sample. DB lane (`db` fixture) — skipped
without Postgres.
"""
from datetime import datetime, timedelta, timezone
from fastapi.testclient import TestClient
from sqlalchemy import select
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, PlexShow, PlexState, User
NOW = datetime(2024, 6, 1, tzinfo=timezone.utc)
def _seed(db):
"""A movie + show library. m1 is a watched, TMDB-enriched anchor whose one recommendation is m2
(in library by guid); m3 is a movie in-progress; sh1 is a show with an in-progress episode."""
lib_m = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
lib_s = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add_all([lib_m, lib_s])
db.flush()
m1 = PlexItem(
rating_key="m1", library_id=lib_m.id, kind="movie", title="Anchor Movie",
guid="tmdb:111", genres=["Drama"], year=2001,
added_at=NOW - timedelta(days=1),
tmdb_enriched_at=NOW - timedelta(days=1),
recommendations=[
{"tmdb_id": 555, "title": "Rec Movie", "poster_path": None, "media_type": "movie"}
],
)
m2 = PlexItem(
rating_key="m2", library_id=lib_m.id, kind="movie", title="Rec Movie",
guid="tmdb:555", genres=["Comedy"], year=2002,
added_at=NOW - timedelta(days=2),
)
m3 = PlexItem(
rating_key="m3", library_id=lib_m.id, kind="movie", title="Half-Watched Movie",
guid="tmdb:333", genres=["Drama"], year=2003, duration_s=6000,
added_at=NOW - timedelta(days=3),
)
sh1 = PlexShow(rating_key="sh1", library_id=lib_s.id, title="A Show", child_count=1,
added_at=NOW - timedelta(hours=1)) # newest → leads recently-added
db.add_all([m1, m2, m3, sh1])
db.flush()
ep1 = PlexItem(
rating_key="sh1e1", library_id=lib_s.id, kind="episode", title="Ep 1",
show_id=sh1.id, season_number=1, episode_number=1,
)
ep2 = PlexItem(
rating_key="sh1e2", library_id=lib_s.id, kind="episode", title="Ep 2",
show_id=sh1.id, season_number=1, episode_number=2,
)
db.add_all([ep1, ep2])
user = User(email="discover@example.com")
db.add(user)
db.flush()
db.add_all([
# m1 watched most recently → the because-you-watched anchor.
PlexState(user_id=user.id, item_id=m1.id, status="watched", watched_at=NOW),
# m3 in-progress → continue watching.
PlexState(user_id=user.id, item_id=m3.id, status="new", position_seconds=1200,
progress_updated_at=NOW - timedelta(hours=1)),
# sh1's ep1 in-progress → the show is in continue watching.
PlexState(user_id=user.id, item_id=ep1.id, status="new", position_seconds=300,
progress_updated_at=NOW - timedelta(minutes=30)),
])
db.commit()
return user
def _get(db, user, monkeypatch, path):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
client = TestClient(app) # no context manager: skip lifespan
return client.get(path)
finally:
app.dependency_overrides.clear()
def _ids(cards):
return [c["id"] for c in cards]
def test_discover_shape_and_recently_added_order(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/discover?scope=both")
assert resp.status_code == 200, resp.text
d = resp.json()
assert set(d) >= {"continue_watching", "recently_added", "genres", "because_you_watched"}
# Recently added is newest-first; the show (added 1d ago) leads, then m1, m2, m3.
assert _ids(d["recently_added"])[:4] == ["sh1", "m1", "m2", "m3"]
# Genre tiles carry counts; Drama (m1 + m3) outweighs Comedy (m2).
genres = {g["value"]: g["count"] for g in d["genres"]}
assert genres.get("Drama") == 2 and genres.get("Comedy") == 1
def test_genre_tiles_carry_poster_art(db, monkeypatch):
# Each genre tile carries up to 4 poster rating_keys for the 2×2 collage (top_art_keys): the top
# movies in that genre, best-rated then newest. No ratings here → newest-first; shows are excluded.
user = _seed(db)
d = _get(db, user, monkeypatch, "/api/plex/discover?scope=both").json()
art = {g["value"]: g["art"] for g in d["genres"]}
assert art["Drama"] == ["m1", "m3"] # both Drama movies, m1 (newer) first
assert art["Comedy"] == ["m2"]
def test_continue_watching_movies_and_shows(db, monkeypatch):
user = _seed(db)
d = _get(db, user, monkeypatch, "/api/plex/discover?scope=both").json()
ids = _ids(d["continue_watching"])
# The in-progress movie and the show with an in-progress episode are both present; the watched
# anchor movie (m1) is not. Order is most-recently-progressed first (the show's ep beats m3).
assert ids == ["sh1", "m3"]
assert "m1" not in ids
def test_continue_watching_null_status_in_hidden_out(db, monkeypatch):
# A started movie whose PlexState.status is NULL is still "in progress": `NULL NOT IN (...)` is NULL
# (not TRUE), so a bare notin_ would wrongly drop it — the or_(is_(None), notin_) branch keeps it.
# A hidden started movie must be excluded.
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
m_null = PlexItem(rating_key="mn", library_id=lib.id, kind="movie", title="Null Status", guid="tmdb:1")
m_hidden = PlexItem(rating_key="mh", library_id=lib.id, kind="movie", title="Hidden", guid="tmdb:2")
db.add_all([m_null, m_hidden])
db.flush()
user = User(email="cw-null@example.com")
db.add(user)
db.flush()
db.add_all([
PlexState(user_id=user.id, item_id=m_null.id, status=None, position_seconds=500,
progress_updated_at=NOW),
PlexState(user_id=user.id, item_id=m_hidden.id, status="hidden", position_seconds=500,
progress_updated_at=NOW - timedelta(hours=1)),
])
db.commit()
ids = _ids(_get(db, user, monkeypatch, "/api/plex/discover?scope=movie").json()["continue_watching"])
assert "mn" in ids # NULL-status started movie IS in progress
assert "mh" not in ids # hidden is excluded
def test_continue_watching_excludes_fully_watched_show(db, monkeypatch):
# A show whose every episode is watched is DONE, not in-progress: the `watched < total` guard must
# keep it off the Continue-watching rail (a bare `watched > 0` would wrongly resurface it).
lib_s = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(lib_s)
db.flush()
sh = PlexShow(rating_key="shw", library_id=lib_s.id, title="All Watched", child_count=1)
db.add(sh)
db.flush()
ep = PlexItem(rating_key="shwe1", library_id=lib_s.id, kind="episode", title="E1",
show_id=sh.id, season_number=1, episode_number=1)
db.add(ep)
db.flush()
user = User(email="fullywatched@example.com")
db.add(user)
db.flush()
db.add(PlexState(user_id=user.id, item_id=ep.id, status="watched", position_seconds=0,
progress_updated_at=NOW))
db.commit()
ids = _ids(_get(db, user, monkeypatch, "/api/plex/discover?scope=show").json()["continue_watching"])
assert "shw" not in ids # fully watched → excluded from continue-watching
def test_because_you_watched_resolves_in_library_recs(db, monkeypatch):
user = _seed(db)
d = _get(db, user, monkeypatch, "/api/plex/discover?scope=both").json()
bw = d["because_you_watched"]
assert bw is not None
assert bw["anchor"]["id"] == "m1" and bw["anchor"]["title"] == "Anchor Movie"
# m1's single recommendation (tmdb:555) is m2 in our library → it becomes the one rail card.
assert _ids(bw["items"]) == ["m2"]
def test_because_you_watched_null_without_history(db, monkeypatch):
_seed(db)
fresh = User(email="fresh-discover@example.com")
db.add(fresh)
db.commit()
d = _get(db, fresh, monkeypatch, "/api/plex/discover?scope=both").json()
# A user with no watched-and-enriched movie gets no recommendation rail.
assert d["because_you_watched"] is None
assert d["continue_watching"] == []
def test_surprise_scope_and_bound(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/surprise?scope=movie&limit=2")
assert resp.status_code == 200, resp.text
items = resp.json()["items"]
assert len(items) <= 2
assert all(c["type"] == "movie" for c in items) # scope=movie excludes shows
def test_recently_watched_movies_and_episodes(db, monkeypatch):
# The rail lists last-fully-watched titles newest-first: a watched MOVIE as its own card, a show as
# the specific watched EPISODE (with its show title). _seed watches m1 (movie); add a watched episode.
user = _seed(db)
ep2 = db.execute(
select(PlexItem).where(PlexItem.rating_key == "sh1e2")
).scalars().one()
db.add(PlexState(user_id=user.id, item_id=ep2.id, status="watched", watched_at=NOW))
db.commit()
rw = _get(db, user, monkeypatch, "/api/plex/discover?scope=both").json()["recently_watched"]
by_id = {c["id"]: c for c in rw}
assert "m1" in by_id and by_id["m1"]["type"] == "movie"
assert "sh1e2" in by_id and by_id["sh1e2"]["type"] == "episode"
assert by_id["sh1e2"]["show_title"] == "A Show" # episode carries its show's title
# ep2 was watched at NOW (newest) → it leads over m1 (watched earlier in _seed... same NOW ok).
assert set(_ids(rw)) == {"m1", "sh1e2"}
def test_surprise_excludes_watched_and_caps_per_kind(db, monkeypatch):
# Surprise surfaces only UNSEEN titles: the watched movie (m1) is out; the unwatched (m2), the
# in-progress movie (m3) and the partially-watched show (sh1) are eligible. Capped at 3 per kind.
user = _seed(db)
items = _get(db, user, monkeypatch, "/api/plex/surprise?scope=both").json()["items"]
ids = _ids(items)
assert "m1" not in ids # watched → excluded
assert set(ids) <= {"m2", "m3", "sh1"} # only unseen titles
assert len([c for c in items if c["type"] == "movie"]) <= 3
assert len([c for c in items if c["type"] == "show"]) <= 3
def test_discover_scope_movie_excludes_shows(db, monkeypatch):
user = _seed(db)
d = _get(db, user, monkeypatch, "/api/plex/discover?scope=movie").json()
assert all(c["type"] == "movie" for c in d["recently_added"])
# The show drops out of continue watching under the movie scope.
assert _ids(d["continue_watching"]) == ["m3"]
def test_filters_narrow_every_discover_rail(db, monkeypatch):
# The sidebar filters constrain the Discover rails just like the Browse grid (ask 3). A genre=Drama
# filter keeps only the Drama titles across every rail; the genre-less show and the Comedy rec drop.
user = _seed(db)
d = _get(db, user, monkeypatch, "/api/plex/discover?scope=both&genres=Drama").json()
# Recently-added: m1 + m3 (Drama) survive; Comedy m2 and the genre-less show sh1 are filtered out.
assert set(_ids(d["recently_added"])) == {"m1", "m3"}
# Continue-watching: only the in-progress Drama movie remains (the show has no genres → excluded).
assert _ids(d["continue_watching"]) == ["m3"]
# Genre tiles reflect the filtered set — only Drama, count 2 (m1 + m3).
assert {g["value"]: g["count"] for g in d["genres"]} == {"Drama": 2}
# Because-you-watched: the anchor m1 is watched, but its only rec (m2) is Comedy → filtered → no rail.
assert d["because_you_watched"] is None
def test_filters_narrow_surprise(db, monkeypatch):
user = _seed(db)
items = _get(
db, user, monkeypatch, "/api/plex/surprise?scope=movie&genres=Comedy"
).json()["items"]
# Only the one Comedy movie qualifies under the filter.
assert _ids(items) == ["m2"]
+27 -1
View File
@@ -12,7 +12,33 @@ import app.state as state
from app.auth import current_user from app.auth import current_user
from app.db import get_db from app.db import get_db
from app.main import app from app.main import app
from app.models import PlexLibrary, PlexShow, User from app.models import PlexItem, PlexLibrary, PlexShow, User
def test_facets_returns_studio_vocabulary(db, monkeypatch):
"""E5 S7: /facets returns the studio vocabulary (distinct studio + count) for the sidebar picker."""
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add_all([
PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="A", studio="Pixar"),
PlexItem(rating_key="m2", library_id=lib.id, kind="movie", title="B", studio="Pixar"),
PlexItem(rating_key="m3", library_id=lib.id, kind="movie", title="C", studio="A24"),
PlexItem(rating_key="m4", library_id=lib.id, kind="movie", title="D"), # no studio
])
user = User(email="studiofacets@example.com")
db.add(user)
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
resp = TestClient(app).get("/api/plex/facets?scope=movie")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 200, resp.text
studios = {s["value"]: s["count"] for s in resp.json()["studios"]}
assert studios == {"Pixar": 2, "A24": 1}
def test_facets_survives_json_null_genres(db, monkeypatch): def test_facets_survives_json_null_genres(db, monkeypatch):
@@ -0,0 +1,148 @@
"""E3 S1 characterization net — /library grid ⟷ /facets equivalence across watch-state scopes.
The Plex god-module split (E3 S1) extracts the movie/show watch-state predicates that are currently
HAND-DUPLICATED in two places: `unified_library` (the grid) and `facets.wsq` (the facet aggregation).
The facets docstring promises the facets "reflect exactly the visible result set" — i.e. with NO
metadata filter active (only the `show` state scope), a facet's genre/content-rating counts must equal
the same counts taken over the grid's visible items. This test derives the expected counts from the
grid's own response and asserts /facets matches, for every (scope × show) combination.
That locks the two duplicated branch-sets together BEFORE they are merged into one shared helper, so
the merge is provably behaviour-equivalent (and a drift between grid and facets fails here). DB lane
(`db` fixture) — skipped without Postgres.
"""
from collections import Counter
from datetime import datetime, timezone
from app.models import (
PlexItem,
PlexLibrary,
PlexSeason,
PlexShow,
PlexState,
User,
)
SCOPES = ("movie", "show", "both")
SHOW_STATES = ("all", "watched", "in_progress", "unwatched")
def _dt(day: int) -> datetime:
return datetime(2024, 1, day, tzinfo=timezone.utc)
def _seed(db):
"""A corpus spanning every watch state, each row carrying genres + a content rating so the facet
aggregation has something to count. Movies get per-item state directly; a show's state is the
rollup of its episodes' state (new / in-progress / fully watched)."""
movie_lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
show_lib = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add_all([movie_lib, show_lib])
db.flush()
# Movies — new / in-progress / watched / hidden, with overlapping genres.
m_new = PlexItem(rating_key="m_new", library_id=movie_lib.id, kind="movie", title="New Movie",
year=2001, added_at=_dt(1), genres=["Drama"], content_rating="PG")
m_prog = PlexItem(rating_key="m_prog", library_id=movie_lib.id, kind="movie", title="Prog Movie",
year=2002, added_at=_dt(2), genres=["Drama", "Comedy"], content_rating="PG-13")
m_watch = PlexItem(rating_key="m_watch", library_id=movie_lib.id, kind="movie", title="Watched Movie",
year=2003, added_at=_dt(3), genres=["Comedy"], content_rating="R")
m_hidden = PlexItem(rating_key="m_hidden", library_id=movie_lib.id, kind="movie", title="Hidden Movie",
year=2004, added_at=_dt(4), genres=["Drama"], content_rating="R")
# Shows — new / in-progress / watched (state comes from the episodes below).
sh_new = PlexShow(rating_key="sh_new", library_id=show_lib.id, title="New Show", year=2005,
added_at=_dt(5), child_count=1, genres=["Sci-Fi"], content_rating="TV-14")
sh_prog = PlexShow(rating_key="sh_prog", library_id=show_lib.id, title="Prog Show", year=2006,
added_at=_dt(6), child_count=1, genres=["Sci-Fi", "Drama"], content_rating="TV-MA")
sh_watch = PlexShow(rating_key="sh_watch", library_id=show_lib.id, title="Watched Show", year=2007,
added_at=_dt(7), child_count=1, genres=["Drama"], content_rating="TV-PG")
db.add_all([m_new, m_prog, m_watch, m_hidden, sh_new, sh_prog, sh_watch])
db.flush()
def _episode(rk, show, day, n):
se = PlexSeason(rating_key=f"se_{rk}", show_id=show.id, season_number=1)
db.add(se)
db.flush()
ep = PlexItem(rating_key=rk, library_id=show_lib.id, kind="episode", title=f"Ep {rk}",
show_id=show.id, season_id=se.id, season_number=1, episode_number=n, added_at=_dt(day))
db.add(ep)
db.flush()
return ep
_episode("e_new", sh_new, 8, 1) # no state -> show "new"
e_prog = _episode("e_prog", sh_prog, 9, 1) # in-progress episode -> show "in_progress"
_episode("e_prog2", sh_prog, 10, 2) # second episode, untouched (so watched < total)
e_watch = _episode("e_watch", sh_watch, 11, 1) # watched episode -> show "watched"
user = User(email="plexparity@example.com")
db.add(user)
db.flush()
db.add_all([
PlexState(user_id=user.id, item_id=m_prog.id, position_seconds=120),
PlexState(user_id=user.id, item_id=m_watch.id, status="watched"),
PlexState(user_id=user.id, item_id=m_hidden.id, status="hidden"),
PlexState(user_id=user.id, item_id=e_prog.id, position_seconds=90),
PlexState(user_id=user.id, item_id=e_watch.id, status="watched"),
])
db.commit()
# rating_key -> (table, genres, content_rating) so the test can total the grid's visible set.
meta = {}
for it in (m_new, m_prog, m_watch, m_hidden):
meta[it.rating_key] = (it.genres, it.content_rating)
for sh in (sh_new, sh_prog, sh_watch):
meta[sh.rating_key] = (sh.genres, sh.content_rating)
return {"user": user, "meta": meta}
def _facet_counts(body):
genres = {g["value"]: g["count"] for g in body["genres"]}
crs = {c["value"]: c["count"] for c in body["content_ratings"]}
return genres, crs
def test_facets_mirror_the_grid_visible_set(db, as_user):
"""For every (scope, show) the facet genre / content-rating counts equal the counts over the
grid's visible items — the invariant the extracted shared watch-state predicate must preserve."""
seed = _seed(db)
meta = seed["meta"]
client = as_user(seed["user"])
for scope in SCOPES:
for show in SHOW_STATES:
grid = client.get(f"/api/plex/library?scope={scope}&show={show}&limit=100")
assert grid.status_code == 200, grid.text
ids = [i["id"] for i in grid.json()["items"]]
exp_genres: Counter = Counter()
exp_crs: Counter = Counter()
for rk in ids:
genres, cr = meta[rk]
for g in genres or []:
exp_genres[g] += 1
if cr is not None:
exp_crs[cr] += 1
facets = client.get(f"/api/plex/facets?scope={scope}&show={show}")
assert facets.status_code == 200, facets.text
got_genres, got_crs = _facet_counts(facets.json())
ctx = f"scope={scope} show={show} grid_ids={ids}"
assert got_genres == dict(exp_genres), f"genre facet != grid ({ctx})"
assert got_crs == dict(exp_crs), f"content-rating facet != grid ({ctx})"
def test_hidden_movie_absent_from_grid_and_facets(db, as_user):
"""A hidden movie must not appear in the default grid NOR inflate its facets — the single most
important equivalence between the two duplicated 'exclude hidden' branches."""
seed = _seed(db)
client = as_user(seed["user"])
grid = client.get("/api/plex/library?scope=movie&show=all&limit=100")
assert "m_hidden" not in [i["id"] for i in grid.json()["items"]]
# Only the hidden movie carries content_rating "R" among the *visible* movies? No — m_watch is
# also "R". The hidden one must not add to the "R" count: visible R = {m_watch} = 1.
facets = client.get("/api/plex/facets?scope=movie&show=all")
crs = {c["value"]: c["count"] for c in facets.json()["content_ratings"]}
assert crs.get("R") == 1, crs
+61
View File
@@ -0,0 +1,61 @@
"""E5 S5: the TMDB keyword filter. A keyword chip on the detail page sets ?keywords=<kw>, which
narrows the unified library to rows whose persisted `keywords` JSONB array contains it. DB lane
(`db` fixture) — skipped without Postgres.
"""
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, User
def _seed(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add_all([
PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="Heist Movie",
keywords=["heist", "bank robbery"]),
PlexItem(rating_key="m2", library_id=lib.id, kind="movie", title="Space Movie",
keywords=["space", "aliens"]),
PlexItem(rating_key="m3", library_id=lib.id, kind="movie", title="No Keywords Movie"),
])
user = User(email="kwfilter@example.com")
db.add(user)
db.commit()
return user
def _get(db, user, monkeypatch, path):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
return TestClient(app).get(path)
finally:
app.dependency_overrides.clear()
def _titles(resp):
return {i["title"] for i in resp.json()["items"]}
def test_keyword_filter_narrows_to_matching_rows(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/library?scope=movie&keywords=heist")
assert resp.status_code == 200, resp.text
assert _titles(resp) == {"Heist Movie"}
def test_no_keyword_filter_returns_all(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/library?scope=movie")
assert _titles(resp) == {"Heist Movie", "Space Movie", "No Keywords Movie"}
def test_unknown_keyword_matches_nothing(db, monkeypatch):
user = _seed(db)
resp = _get(db, user, monkeypatch, "/api/plex/library?scope=movie&keywords=nonesuch")
assert resp.json()["items"] == []
+1 -1
View File
@@ -11,7 +11,7 @@ import app.plex.sync as syncmod
from app import sysconfig from app import sysconfig
from app.models import PlexItem, PlexLibrary from app.models import PlexItem, PlexLibrary
from app.plex.sync import _apply_item, _canonical_guid, _sync_overlay from app.plex.sync import _apply_item, _canonical_guid, _sync_overlay
from app.routes.plex.browse import _content_libs from app.routes.plex._shared import _content_libs
class _StubPlex: class _StubPlex:
+15 -2
View File
@@ -19,7 +19,7 @@ from app.models import (
PlexState, PlexState,
User, User,
) )
from app.routes.plex.browse import _content_libs from app.routes.plex._shared import _content_libs
NAME = "Robert De Niro" NAME = "Robert De Niro"
@@ -100,7 +100,20 @@ def test_person_detail_unknown_name_is_empty(db, monkeypatch):
resp = _get(db, user, monkeypatch, "/api/plex/person/Ghost%20Actor") resp = _get(db, user, monkeypatch, "/api/plex/person/Ghost%20Actor")
assert resp.status_code == 200, resp.text assert resp.status_code == 200, resp.text
body = resp.json() body = resp.json()
assert body == {"name": "Ghost Actor", "photo": None, "movies": [], "shows": []} # An unknown name has no plex_people row → no bio, no titles (E5 S6 added the bio fields, all null).
assert body == {
"name": "Ghost Actor",
"photo": None,
"biography": None,
"birthday": None,
"deathday": None,
"place_of_birth": None,
"known_for_department": None,
"also_known_as": [],
"known_for": [],
"movies": [],
"shows": [],
}
def test_content_libs_partitions_only_enabled(db): def test_content_libs_partitions_only_enabled(db):
+292
View File
@@ -0,0 +1,292 @@
"""E5 S6: person page TMDB bio enrichment. `parse_person` shapes the /person payload (bio + a
popularity-ranked, deduped `known_for`); `enrich_person` lazily fetches + persists once (guarded on a
key + a tmdb_person_id); and /person/{name} serves the bio + resolves `known_for` to library rows.
DB lane for the endpoint/enrich tests; parse_person is pure. Skipped without Postgres where DB-backed.
"""
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import PlexItem, PlexLibrary, PlexPerson, User
from app.plex import tmdb as plex_tmdb
PAYLOAD = {
"biography": " A celebrated actor. ",
"birthday": "1970-01-01",
"deathday": None,
"place_of_birth": "Springfield",
"known_for_department": "Acting",
"also_known_as": ["AKA One", "AKA Two"],
"combined_credits": {
"cast": [
{"id": 10, "media_type": "movie", "title": "Mid Movie", "popularity": 50},
{"id": 20, "media_type": "tv", "name": "Top Show", "popularity": 80},
{"id": 10, "media_type": "movie", "title": "dup", "popularity": 5}, # dup id
{"id": 30, "media_type": "movie", "title": "Low Movie", "popularity": 1},
{"id": None, "media_type": "movie", "title": "No id", "popularity": 99}, # dropped
]
},
}
def test_parse_person_shapes_bio_and_ranked_known_for():
p = plex_tmdb.parse_person(PAYLOAD)
assert p["biography"] == "A celebrated actor." # stripped
assert p["birthday"] == "1970-01-01"
assert p["known_for_department"] == "Acting"
assert p["also_known_as"] == ["AKA One", "AKA Two"]
# Popularity desc, deduped by id, None-id dropped: Top Show (80) → Mid Movie (50) → Low Movie (1).
assert [k["tmdb_id"] for k in p["known_for"]] == [20, 10, 30]
assert p["known_for"][0]["media_type"] == "tv"
def test_parse_person_empty_payload():
p = plex_tmdb.parse_person({})
assert p["biography"] is None and p["known_for"] is None
def test_parse_person_keeps_same_id_across_media_types():
# TMDB movie ids and tv ids are independent, so a movie-42 and a tv-42 credit are two distinct works
# and BOTH must survive the dedup (keyed on (id, media_type), not the bare id).
payload = {
"combined_credits": {
"cast": [
{"id": 42, "media_type": "movie", "title": "Movie 42", "popularity": 90},
{"id": 42, "media_type": "tv", "name": "Show 42", "popularity": 80},
{"id": 42, "media_type": "movie", "title": "dup movie 42", "popularity": 5}, # real dup
]
}
}
kf = plex_tmdb.parse_person(payload)["known_for"]
assert [(k["tmdb_id"], k["media_type"]) for k in kf] == [(42, "movie"), (42, "tv")] # both kept, dup dropped
class _FakeTmdb:
def __init__(self, payload):
self.payload = payload
def __enter__(self):
return self
def __exit__(self, *a):
return False
def fetch_person(self, pid):
return self.payload
def test_enrich_person_persists_and_stamps(db, monkeypatch):
person = PlexPerson(name="Actor X", tmdb_person_id=555)
db.add(person)
db.commit()
monkeypatch.setattr(plex_tmdb.sysconfig, "get_str", lambda d, k: "key" if k == "tmdb_api_key" else "")
monkeypatch.setattr(plex_tmdb, "TmdbClient", lambda d, timeout=None: _FakeTmdb(PAYLOAD))
assert plex_tmdb.enrich_person(db, person) is True
assert person.biography == "A celebrated actor."
assert person.tmdb_enriched_at is not None
assert [k["tmdb_id"] for k in person.known_for] == [20, 10, 30]
class _FakeTmdbResult:
"""Context-manager TmdbClient stand-in whose fetch_person raises / returns a fixed payload."""
def __init__(self, *, raises=False, payload=None):
self._raises = raises
self._payload = payload
def __enter__(self):
return self
def __exit__(self, *a):
return False
def fetch_person(self, pid):
if self._raises:
raise plex_tmdb.TmdbError("429 rate limited")
return self._payload
def test_enrich_person_transient_error_leaves_watermark_null(db, monkeypatch):
# A transient TmdbError must NOT stamp the watermark, so the next open retries (never permanently
# blank a person after one blip).
person = PlexPerson(name="Transient P", tmdb_person_id=555)
db.add(person)
db.commit()
monkeypatch.setattr(plex_tmdb.sysconfig, "get_str", lambda d, k: "key" if k == "tmdb_api_key" else "")
monkeypatch.setattr(plex_tmdb, "TmdbClient", lambda d, timeout=None: _FakeTmdbResult(raises=True))
assert plex_tmdb.enrich_person(db, person) is False
assert person.tmdb_enriched_at is None # left NULL → retried next open
def test_enrich_person_404_stamps_watermark_without_bio(db, monkeypatch):
# A 404 (deleted person → None payload) stamps the watermark so it isn't refetched every open.
person = PlexPerson(name="Gone P", tmdb_person_id=555)
db.add(person)
db.commit()
monkeypatch.setattr(plex_tmdb.sysconfig, "get_str", lambda d, k: "key" if k == "tmdb_api_key" else "")
monkeypatch.setattr(plex_tmdb, "TmdbClient", lambda d, timeout=None: _FakeTmdbResult(payload=None))
assert plex_tmdb.enrich_person(db, person) is True
assert person.tmdb_enriched_at is not None # stamped
assert person.biography is None # nothing fetched
def test_enrich_person_noop_without_key(db, monkeypatch):
person = PlexPerson(name="Actor Y", tmdb_person_id=555)
db.add(person)
db.commit()
monkeypatch.setattr(plex_tmdb.sysconfig, "get_str", lambda d, k: "") # no key
assert plex_tmdb.enrich_person(db, person) is False
assert person.tmdb_enriched_at is None
def test_enrich_person_noop_without_tmdb_id(db, monkeypatch):
person = PlexPerson(name="Actor Z", tmdb_person_id=None)
db.add(person)
db.commit()
monkeypatch.setattr(plex_tmdb.sysconfig, "get_str", lambda d, k: "key")
assert plex_tmdb.enrich_person(db, person) is False
def _get(db, user, monkeypatch, path):
monkeypatch.setattr(state, "_configured_cache", True)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user
try:
return TestClient(app).get(path)
finally:
app.dependency_overrides.clear()
def test_person_detail_schedules_enrich_off_request_path(db, monkeypatch):
# The bio fetch runs OFF the request path (a background task after the response), so the serve never
# makes a live TMDB call. An un-enriched person (tmdb_person_id set + tmdb_enriched_at None) schedules
# the enrich and the page still renders immediately with whatever's in the row (bio still null).
from app.routes.plex import people
lib = PlexLibrary(plex_key="9", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(PlexPerson(name="Lazy P", tmdb_person_id=777))
user = User(email="lazybg@example.com")
db.add(user)
db.commit()
scheduled: list[int] = []
monkeypatch.setattr(people, "_enrich_person_bg", lambda pid: scheduled.append(pid))
people._enrich_inflight.clear() # ensure the per-person cooldown guard doesn't suppress this open
resp = _get(db, user, monkeypatch, "/api/plex/person/Lazy P")
assert resp.status_code == 200, resp.text
assert resp.json()["biography"] is None # not enriched yet — the bio lands by the next open
assert len(scheduled) == 1 # the enrich was scheduled off the request path
def test_enrich_person_bg_closes_session_and_isolates_errors(monkeypatch):
# The background-task body must open a fresh session, fetch the person, and ALWAYS close it — rolling
# back (never propagating) if enrich_person raises. Pure-unit via a fake session (no DB needed).
from app.routes.plex import people
calls = {"rollback": 0, "close": 0, "enriched": 0}
class _FakeSession:
def __init__(self, person):
self._person = person
def get(self, model, pid):
return self._person
def rollback(self):
calls["rollback"] += 1
def close(self):
calls["close"] += 1
# Success path: person found → enrich called, session closed, no rollback.
monkeypatch.setattr(people, "SessionLocal", lambda: _FakeSession(object()))
monkeypatch.setattr(
people.plex_tmdb, "enrich_person", lambda db, p: calls.__setitem__("enriched", calls["enriched"] + 1)
)
people._enrich_person_bg(5)
assert calls == {"rollback": 0, "close": 1, "enriched": 1}
# Error path: enrich raises → rollback + close, no exception escapes.
calls.update({"rollback": 0, "close": 0, "enriched": 0})
def _boom(db, p):
raise RuntimeError("enrich blew up")
monkeypatch.setattr(people.plex_tmdb, "enrich_person", _boom)
people._enrich_person_bg(5) # must not raise
assert calls["rollback"] == 1 and calls["close"] == 1
# Missing person (deleted id): None-guard skips enrich, session still closed.
calls.update({"rollback": 0, "close": 0, "enriched": 0})
monkeypatch.setattr(people, "SessionLocal", lambda: _FakeSession(None))
people._enrich_person_bg(9)
assert calls == {"rollback": 0, "close": 1, "enriched": 0}
def test_should_schedule_enrich_dedups_within_cooldown():
# Concurrent/rapid opens of the same person must schedule the enrich only once per cooldown window.
from app.routes.plex import people
people._enrich_inflight.clear()
assert people._should_schedule_enrich(12345) is True # first open reserves
assert people._should_schedule_enrich(12345) is False # within cooldown → deduped
assert people._should_schedule_enrich(99999) is True # a different person is independent
people._enrich_inflight.clear()
def test_person_detail_no_enrich_when_already_enriched(db, monkeypatch):
# An already-enriched person (tmdb_enriched_at set) must NOT schedule another background fetch.
from datetime import datetime, timezone
from app.routes.plex import people
lib = PlexLibrary(plex_key="8", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(PlexPerson(name="Done P", tmdb_person_id=888,
tmdb_enriched_at=datetime(2024, 1, 1, tzinfo=timezone.utc), biography="Bio."))
user = User(email="donebg@example.com")
db.add(user)
db.commit()
scheduled: list[int] = []
monkeypatch.setattr(people, "_enrich_person_bg", lambda pid: scheduled.append(pid))
resp = _get(db, user, monkeypatch, "/api/plex/person/Done P")
assert resp.status_code == 200, resp.text
assert resp.json()["biography"] == "Bio."
assert scheduled == [] # already enriched → no background work
def test_person_detail_serves_bio_and_resolves_known_for(db, monkeypatch):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
# A library movie that IS one of the person's known_for credits (matched by guid tmdb:100).
db.add(PlexItem(rating_key="km", library_id=lib.id, kind="movie", title="Known Movie",
guid="tmdb:100", cast_names=["Actor X"]))
# Already-enriched person (tmdb_enriched_at set → no lazy fetch), with a known_for pool.
from datetime import datetime, timezone
db.add(PlexPerson(name="Actor X", tmdb_person_id=555,
tmdb_enriched_at=datetime(2024, 1, 1, tzinfo=timezone.utc),
biography="Bio here.", birthday="1970-01-01", known_for_department="Acting",
also_known_as=["Alt Name"],
known_for=[{"tmdb_id": 100, "media_type": "movie", "title": "Known Movie"},
{"tmdb_id": 999, "media_type": "movie", "title": "Not In Library"}]))
user = User(email="personbio@example.com")
db.add(user)
db.commit()
resp = _get(db, user, monkeypatch, "/api/plex/person/Actor X")
assert resp.status_code == 200, resp.text
j = resp.json()
assert j["biography"] == "Bio here."
assert j["birthday"] == "1970-01-01"
assert j["also_known_as"] == ["Alt Name"]
# Only the in-library known_for credit resolves to a card; the out-of-library one is dropped.
assert [c["title"] for c in j["known_for"]] == ["Known Movie"]
+3 -1
View File
@@ -149,7 +149,9 @@ class TestRollup:
def _fake_paginate(by_type): def _fake_paginate(by_type):
def _p(_plex, _key, typ): def _p(_plex, _key, typ, progress=None):
if progress is not None:
progress["complete"] = True # the fake returns the whole list — a complete listing
return list(by_type.get(typ, [])) return list(by_type.get(typ, []))
return _p return _p
+162
View File
@@ -0,0 +1,162 @@
"""E3 S1 catalog-prune (B-sub-6): a sync now deletes rows for titles that disappeared from Plex,
diffing each section's freshly-listed rating_keys against the stored ones. The prune is GUARDED on a
non-empty listing so a transient Plex glitch (an empty page) can never wipe a whole library.
DB lane (`db` fixture) — skipped without Postgres. Drives `_sync_movies` / `_sync_shows` directly
with a stub Plex (the same seam the overlay test uses), so no live server is needed.
"""
from sqlalchemy import select
from app.models import (
PlexItem,
PlexLibrary,
PlexSeason,
PlexShow,
PlexState,
User,
)
from app.plex.sync import (
_TYPE_EPISODE,
_TYPE_MOVIE,
_TYPE_SEASON,
_TYPE_SHOW,
_sync_movies,
_sync_shows,
)
class _StubPlex:
"""Returns canned section listings per Plex item type — the current 'truth' the sync reconciles
the stored rows against. `_paginate` is the only method the sync calls on it."""
def __init__(self, by_type: dict[int, list[dict]]):
self.by_type = by_type
def section_items(self, key, item_type, start, size):
items = self.by_type.get(item_type, [])
return items[start : start + size], len(items)
class _TruncatedPlex:
"""Simulates a truncated listing (a transient Plex hiccup): it reports a large `total` but returns
only one short page, so pagination stops BEFORE reaching total. Prune must treat this as an
incomplete fetch and delete nothing — the un-listed rows are present, just not mentioned."""
def __init__(self, items: list[dict], claimed_total: int):
self._items = items
self._total = claimed_total
def section_items(self, key, item_type, start, size):
if start == 0:
return list(self._items), self._total # one short page, total says there's more
return [], self._total
def _seed_movies(db):
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
m1 = PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="Kept Movie")
m2 = PlexItem(rating_key="m2", library_id=lib.id, kind="movie", title="Gone Movie")
db.add_all([m1, m2])
user = User(email="prune@example.com")
db.add(user)
db.flush()
# A watch-state on EACH movie: the kept one's must survive, the pruned one's must cascade away.
db.add_all([
PlexState(user_id=user.id, item_id=m1.id, status="watched"),
PlexState(user_id=user.id, item_id=m2.id, status="watched"),
])
db.commit()
return lib, m1.id, m2.id, user.id
def test_movie_prune_deletes_disappeared_titles_and_cascades_state(db):
lib, m1_id, _m2_id, _user_id = _seed_movies(db)
# Plex now lists only m1 — m2 disappeared.
stub = _StubPlex({_TYPE_MOVIE: [{"ratingKey": "m1", "title": "Kept Movie"}]})
stats = {"movies": 0, "pruned": 0}
_sync_movies(db, stub, lib, stats)
db.commit()
live = {r.rating_key for r in db.execute(select(PlexItem)).scalars()}
assert live == {"m1"}, "the disappeared movie must be pruned"
assert stats["pruned"] == 1
# The kept movie's watch-state survives; the pruned movie's cascaded away.
states = {s.item_id for s in db.execute(select(PlexState)).scalars()}
assert states == {m1_id}
def test_empty_listing_prunes_nothing(db):
"""The safety guard: a section that lists ZERO items (a real empty library, or a transient Plex
glitch we can't distinguish) must leave the stored rows untouched — never mass-delete."""
lib, _m1_id, _m2_id, _user_id = _seed_movies(db)
stub = _StubPlex({_TYPE_MOVIE: []}) # Plex returned nothing this run
stats = {"movies": 0, "pruned": 0}
_sync_movies(db, stub, lib, stats)
db.commit()
live = {r.rating_key for r in db.execute(select(PlexItem)).scalars()}
assert live == {"m1", "m2"}, "an empty listing must not prune anything"
assert stats["pruned"] == 0
def test_truncated_listing_prunes_nothing(db):
"""A partial fetch (Plex reports 5 items but the listing returns only 1 then stops) must NOT prune
the 4 it failed to mention — the completeness guard, not just the empty-listing guard."""
lib, m1_id, _m2_id, _user_id = _seed_movies(db)
# Only m1 is listed, but total claims 5 → the walk never reaches total → incomplete → no prune.
stub = _TruncatedPlex([{"ratingKey": "m1", "title": "Kept Movie"}], claimed_total=5)
stats = {"movies": 0, "pruned": 0}
_sync_movies(db, stub, lib, stats)
db.commit()
live = {r.rating_key for r in db.execute(select(PlexItem)).scalars()}
assert live == {"m1", "m2"}, "a truncated listing must not prune the un-listed rows"
assert stats["pruned"] == 0
def test_show_prune_removes_stale_show_season_and_episode(db):
lib = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(lib)
db.flush()
# Stored: one kept show (s1 -> se1 -> e1) and one stale show (s2 -> se2 -> e2) that Plex dropped.
s1 = PlexShow(rating_key="s1", library_id=lib.id, title="Kept Show")
s2 = PlexShow(rating_key="s2", library_id=lib.id, title="Gone Show")
db.add_all([s1, s2])
db.flush()
se1 = PlexSeason(rating_key="se1", show_id=s1.id, season_number=1)
se2 = PlexSeason(rating_key="se2", show_id=s2.id, season_number=1)
db.add_all([se1, se2])
db.flush()
e1 = PlexItem(rating_key="e1", library_id=lib.id, kind="episode", title="Kept Ep",
show_id=s1.id, season_id=se1.id, season_number=1, episode_number=1)
e2 = PlexItem(rating_key="e2", library_id=lib.id, kind="episode", title="Gone Ep",
show_id=s2.id, season_id=se2.id, season_number=1, episode_number=1)
db.add_all([e1, e2])
db.commit()
# Plex now lists only s1's subtree.
stub = _StubPlex({
_TYPE_SHOW: [{"ratingKey": "s1", "title": "Kept Show"}],
_TYPE_SEASON: [{"ratingKey": "se1", "parentRatingKey": "s1", "index": 1, "title": "Season 1"}],
_TYPE_EPISODE: [{
"ratingKey": "e1", "grandparentRatingKey": "s1", "parentRatingKey": "se1",
"parentIndex": 1, "index": 1, "title": "Kept Ep",
}],
})
stats = {"shows": 0, "seasons": 0, "episodes": 0, "pruned": 0}
_sync_shows(db, stub, lib, stats)
db.commit()
shows = {r.rating_key for r in db.execute(select(PlexShow)).scalars()}
seasons = {r.rating_key for r in db.execute(select(PlexSeason)).scalars()}
eps = {r.rating_key for r in db.execute(select(PlexItem)).scalars()}
assert shows == {"s1"}, "the stale show must be pruned"
assert seasons == {"se1"}, "the stale season must be pruned"
assert eps == {"e1"}, "the stale episode must be pruned"
assert stats["pruned"] == 3 # s2 + se2 + e2
+365
View File
@@ -0,0 +1,365 @@
"""TMDB enrichment engine (E5 S2): the pure parser + persister, guid→tmdb-id resolution, and the
DB-lane batch/serve integration (enrich stamps the watermark + links people; serve_block resolves
recommendations to library rows; the whole thing no-ops without an API key).
The DB tests use the `db` fixture (real Postgres) — skipped when none is reachable."""
from datetime import datetime, timezone
import httpx
import pytest
from app.models import PlexItem, PlexLibrary, PlexPerson, PlexShow
from app.plex import tmdb
# A TMDB /movie/{id}?append_to_response=keywords,credits,recommendations,images,external_ids payload.
MOVIE_PAYLOAD = {
"id": 238,
"vote_average": 8.7,
"vote_count": 20000,
"budget": 6000000,
"revenue": 245066411,
"keywords": {"keywords": [{"id": 1, "name": "mafia"}, {"id": 2, "name": "crime family"}]},
"credits": {
"cast": [
{"id": 3084, "name": "Marlon Brando", "character": "Vito", "profile_path": "/a.jpg", "order": 0},
{"id": 1158, "name": "Al Pacino", "character": "Michael", "profile_path": "/b.jpg", "order": 1},
],
"crew": [
{"id": 1776, "name": "Francis Ford Coppola", "job": "Director", "profile_path": "/c.jpg"},
{"id": 1776, "name": "Francis Ford Coppola", "job": "Writer", "profile_path": "/c.jpg"},
{"id": 9, "name": "Some Gaffer", "job": "Gaffer"}, # not a key job → dropped
],
},
"production_companies": [{"name": "Paramount"}, {"name": "Alfran"}],
"production_countries": [{"name": "United States of America"}],
"recommendations": {
"results": [
{"id": 240, "title": "The Godfather Part II", "poster_path": "/p.jpg", "media_type": "movie"},
{"id": 999, "title": "Not In Library", "poster_path": "/q.jpg", "media_type": "movie"},
]
},
"images": {"backdrops": [{"file_path": "/bd1.jpg"}, {"file_path": "/bd2.jpg"}]},
"external_ids": {"imdb_id": "tt0068646"},
}
# A TMDB /tv payload: keywords under `results`, aggregate_credits with per-person `jobs`, no budget.
TV_PAYLOAD = {
"id": 1396,
"vote_average": 9.0,
"vote_count": 12000,
"created_by": [{"id": 66633, "name": "Vince Gilligan", "profile_path": "/vg.jpg"}],
"keywords": {"results": [{"name": "drugs"}, {"name": "new mexico"}]},
"aggregate_credits": {
"cast": [{"id": 17419, "name": "Bryan Cranston"}],
"crew": [
{"id": 66633, "name": "Vince Gilligan", "jobs": [{"job": "Executive Producer"}]},
{"id": 5, "name": "Someone", "jobs": [{"job": "Sound"}]}, # dropped
],
},
"production_companies": [{"name": "Sony Pictures Television"}],
"production_countries": [{"name": "United States of America"}],
"recommendations": {"results": []},
"images": {"backdrops": []},
}
class FakeTmdb:
"""Stand-in for TmdbClient in the parse/batch tests — no network."""
def __init__(self, payloads=None, find_map=None):
self.payloads = payloads or {} # (media_type, tmdb_id) -> payload
self.find_map = find_map or {} # external_id -> /find result
def fetch(self, tmdb_id, media_type):
return self.payloads.get((media_type, tmdb_id))
def find(self, external_id, source):
return self.find_map.get(external_id)
# ---- pure parse/persist --------------------------------------------------------------------------
def test_parse_movie_pulls_all_fields():
d = tmdb.parse_tmdb(MOVIE_PAYLOAD, "movie")
assert d["tmdb_id"] == 238
assert d["tmdb_rating"] == 8.7
assert d["tmdb_vote_count"] == 20000
assert d["budget"] == 6000000
assert d["revenue"] == 245066411
assert d["keywords"] == ["mafia", "crime family"]
assert d["crew"] == [
{"name": "Francis Ford Coppola", "job": "Director", "tmdb_id": 1776, "profile_path": "/c.jpg"},
{"name": "Francis Ford Coppola", "job": "Writer", "tmdb_id": 1776, "profile_path": "/c.jpg"},
] # the Gaffer is filtered; the two Coppola jobs survive (dedup is by name+job)
assert d["studios"] == ["Paramount", "Alfran"]
assert d["production_countries"] == ["United States of America"]
assert [r["tmdb_id"] for r in d["recommendations"]] == [240, 999]
assert d["backdrops"] == ["/bd1.jpg", "/bd2.jpg"]
# cast + crew names → tmdb person ids for the plex_people foundation
links = {(p["name"], p["tmdb_id"]) for p in d["people"]}
assert ("Marlon Brando", 3084) in links
assert ("Al Pacino", 1158) in links
assert ("Francis Ford Coppola", 1776) in links
def test_parse_tv_uses_aggregate_credits_and_no_money():
d = tmdb.parse_tmdb(TV_PAYLOAD, "tv")
assert d["budget"] is None and d["revenue"] is None # TV has neither
assert d["keywords"] == ["drugs", "new mexico"] # from keywords.results
# The showrunner (created_by → Creator) leads; the aggregate-credits EP follows; the Sound person is
# filtered; TV jobs are read from jobs[0]. Crew is display-priority ordered (Creator before EP).
assert d["crew"] == [
{"name": "Vince Gilligan", "job": "Creator", "tmdb_id": 66633, "profile_path": "/vg.jpg"},
{"name": "Vince Gilligan", "job": "Executive Producer", "tmdb_id": 66633, "profile_path": None},
]
assert d["tmdb_rating"] == 9.0
assert ("Bryan Cranston", 17419) in {(p["name"], p["tmdb_id"]) for p in d["people"]}
def test_parse_zero_votes_and_money_are_unknown():
d = tmdb.parse_tmdb({"id": 1, "vote_average": 0.0, "vote_count": 0, "budget": 0, "revenue": 0}, "movie")
assert d["tmdb_rating"] is None # 0.0 with no votes is meaningless
assert d["tmdb_vote_count"] is None
assert d["budget"] is None and d["revenue"] is None
def test_apply_writes_row_fields_without_touching_plex_owned():
it = PlexItem(rating_key="m1", kind="movie", title="The Godfather", imdb_rating=8.7, full_cast=[{"x": 1}])
tmdb.apply_tmdb_detail(it, tmdb.parse_tmdb(MOVIE_PAYLOAD, "movie"))
assert it.tmdb_id == 238
assert it.budget == 6000000
assert it.keywords == ["mafia", "crime family"]
assert it.imdb_rating == 8.7 and it.full_cast == [{"x": 1}] # Plex-owned, untouched
def test_guid_to_tmdb_id_direct_find_and_bad():
fake = FakeTmdb(find_map={"tt0068646": {"movie_results": [{"id": 238}], "tv_results": []}})
assert tmdb.guid_to_tmdb_id("tmdb:238", "movie", fake) == 238
assert tmdb.guid_to_tmdb_id("imdb:tt0068646", "movie", fake) == 238 # resolved via /find
assert tmdb.guid_to_tmdb_id("imdb:tt0000000", "movie", fake) is None # not in find_map
assert tmdb.guid_to_tmdb_id("tmdb:notanumber", "movie", fake) is None
assert tmdb.guid_to_tmdb_id(None, "movie", fake) is None
assert tmdb.guid_to_tmdb_id("local://abc", "movie", fake) is None
def test_guid_to_tmdb_id_tvdb_source_and_tv_media_type():
# A tvdb guid for a TV item goes through /find and must pick the tv_results list (not movie_results),
# and a tvdb source maps to the tvdb_id external_source. The media_type branch is the whole point.
fake = FakeTmdb(find_map={"81189": {"movie_results": [], "tv_results": [{"id": 1396}]}})
assert tmdb.guid_to_tmdb_id("tvdb:81189", "tv", fake) == 1396
# The SAME /find result, asked for a movie, reads the (empty) movie_results → None.
assert tmdb.guid_to_tmdb_id("tvdb:81189", "movie", fake) is None
# A direct tmdb guid for a tv item is just the int (no /find).
assert tmdb.guid_to_tmdb_id("tmdb:1396", "tv", fake) == 1396
# An unknown provider (only imdb/tvdb are /find sources) is unresolvable.
assert tmdb.guid_to_tmdb_id("plex:abc", "tv", fake) is None
# ---- DB-lane: batch enrich + serve --------------------------------------------------------------
def _lib(db, key="1", kind="movie"):
lib = PlexLibrary(plex_key=key, title="L", kind=kind, enabled=True)
db.add(lib)
db.flush()
return lib
def test_enrich_batch_persists_stamps_and_links_people(db):
lib = _lib(db)
mv = PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="The Godfather", guid="tmdb:238")
unmatched = PlexItem(rating_key="m2", library_id=lib.id, kind="movie", title="Nope", guid="tmdb:777")
db.add_all([mv, unmatched])
db.commit()
fake = FakeTmdb(payloads={("movie", 238): MOVIE_PAYLOAD}) # 777 has no payload → definitive no-match
n = tmdb.enrich_batch(db, fake, 50)
assert n == 2 # both processed (one matched, one definitively unmatched)
db.refresh(mv)
assert mv.tmdb_id == 238
assert mv.keywords == ["mafia", "crime family"]
assert mv.revenue == 245066411
assert mv.tmdb_enriched_at is not None
db.refresh(unmatched)
assert unmatched.tmdb_enriched_at is not None # stamped so it isn't re-fetched every run
assert unmatched.tmdb_id is None # but nothing stored
brando = db.execute(
PlexPerson.__table__.select().where(PlexPerson.name == "Marlon Brando")
).mappings().first()
assert brando is not None and brando["tmdb_person_id"] == 3084
def test_serve_block_none_until_enriched_and_resolves_recs(db):
lib = _lib(db)
# A recommendation target that IS in our library (guid tmdb:240) → deep-linkable.
sequel = PlexItem(rating_key="m240", library_id=lib.id, kind="movie", title="Part II", guid="tmdb:240")
fresh = PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="The Godfather", guid="tmdb:238")
db.add_all([sequel, fresh])
db.commit()
assert tmdb.serve_block(db, fresh) is None # not enriched yet
tmdb.apply_tmdb_detail(fresh, tmdb.parse_tmdb(MOVIE_PAYLOAD, "movie"))
fresh.tmdb_enriched_at = datetime.now(timezone.utc)
db.commit()
block = tmdb.serve_block(db, fresh)
assert block is not None
assert block["budget"] == 6000000
assert block["keywords"] == ["mafia", "crime family"]
recs = {r["tmdb_id"]: r for r in block["recommendations"]}
assert recs[240]["in_library"] is True and recs[240]["rating_key"] == "m240"
assert recs[999]["in_library"] is False and recs[999]["rating_key"] is None
def test_get_never_leaks_api_key_in_error_messages(monkeypatch):
# The api_key rides in the request URL query string, so an error message built from the httpx
# exception (which embeds that URL) would leak the secret to logs (enrich_batch logs it). Both the
# HTTP-status and the network error paths must report status/path only — never the key.
monkeypatch.setattr(
tmdb.sysconfig, "get_str", lambda d, k: "SECRETKEY" if k == "tmdb_api_key" else "en-US"
)
client = tmdb.TmdbClient(None)
req = httpx.Request(
"GET", "https://api.themoviedb.org/3/movie/1?api_key=SECRETKEY&language=en-US"
)
class _StatusHttp:
def get(self, url, params=None):
return httpx.Response(401, request=req) # non-404 → raise_for_status → HTTPStatusError
def close(self):
pass
client._http = _StatusHttp()
with pytest.raises(tmdb.TmdbError) as ei:
client._get("/movie/1")
assert "SECRETKEY" not in str(ei.value) and "401" in str(ei.value)
class _NetHttp:
def get(self, url, params=None):
raise httpx.ConnectError("connect failed", request=req)
def close(self):
pass
client._http = _NetHttp()
with pytest.raises(tmdb.TmdbError) as ei2:
client._get("/movie/1")
assert "SECRETKEY" not in str(ei2.value)
client.close()
def test_run_enrich_noops_without_key(db):
# No tmdb_api_key configured (default "") → clean no-op, never an error.
out = tmdb.run_enrich(db)
assert out["enriched"] == 0 and "skipped" in out
class _RaisingTmdb:
"""A client whose fetch raises a transient TmdbError (e.g. a 429), to exercise the retry path."""
def find(self, external_id, source):
return None
def fetch(self, tmdb_id, media_type):
raise tmdb.TmdbError("429 rate limited")
def test_enrich_batch_transient_error_leaves_watermark_null(db):
# A transient TmdbError mid-batch must leave the row's tmdb_enriched_at NULL so the next run retries
# it (unlike a definitive no-match, which stamps the watermark). The row is NOT counted as processed.
lib = _lib(db)
mv = PlexItem(rating_key="m1", library_id=lib.id, kind="movie", title="Godfather", guid="tmdb:238")
db.add(mv)
db.commit()
n = tmdb.enrich_batch(db, _RaisingTmdb(), 50)
assert n == 0 # the transient failure is skipped, not processed
db.refresh(mv)
assert mv.tmdb_enriched_at is None # left NULL → retried next run
assert mv.tmdb_id is None
def test_enrich_batch_skips_shows_without_guid(db):
lib = _lib(db, key="2", kind="show")
sh = PlexShow(rating_key="s1", library_id=lib.id, title="No Guid Show") # guid None → not selected
db.add(sh)
db.commit()
assert tmdb.enrich_batch(db, FakeTmdb(), 50) == 0
db.refresh(sh)
assert sh.tmdb_enriched_at is None
def test_resolve_library_guids_scopes_to_given_libraries(db):
# The scoped path (person page) must keep a credit that lives in a NON-scoped library (disabled or an
# HU-overlay twin) OUT of the result; the unscoped path (detail recs) matches across all libraries;
# and an explicit empty list skips that media side entirely (distinct from None = match all).
lib_on = _lib(db, key="on", kind="movie")
lib_off = _lib(db, key="off", kind="movie")
m_on = PlexItem(rating_key="on1", library_id=lib_on.id, kind="movie", title="In Scope", guid="tmdb:100")
m_off = PlexItem(rating_key="off1", library_id=lib_off.id, kind="movie", title="Out", guid="tmdb:200")
db.add_all([m_on, m_off])
db.commit()
entries = [{"tmdb_id": 100, "media_type": "movie"}, {"tmdb_id": 200, "media_type": "movie"}]
# Scoped to lib_on: the row in lib_off is excluded.
movies, shows = tmdb.resolve_library_guids(
db, entries, movie_lib_ids=[lib_on.id], show_lib_ids=[]
)
assert set(movies) == {"tmdb:100"}
assert shows == {}
# Unscoped (None defaults): matches across all libraries.
movies_all, _ = tmdb.resolve_library_guids(db, entries)
assert set(movies_all) == {"tmdb:100", "tmdb:200"}
# Empty list for a media side skips its bucket even with candidates present (vs None = match all).
_, shows_empty = tmdb.resolve_library_guids(
db, [{"tmdb_id": 100, "media_type": "tv"}], show_lib_ids=[]
)
assert shows_empty == {}
def test_resolve_recs_matches_by_media_type_not_bare_id(db):
# TMDB movie ids and tv ids are independent, so tmdb:500 can be BOTH a show (in our library) and a
# movie recommendation. The match must respect media_type — a bare-id match would deep-link the movie
# rec to the unrelated show (and vice-versa).
mlib = _lib(db, key="10", kind="movie")
slib = _lib(db, key="11", kind="show")
show500 = PlexShow(rating_key="s500", library_id=slib.id, title="Show 500", guid="tmdb:500")
movie900 = PlexItem(
rating_key="m900", library_id=mlib.id, kind="movie", title="Movie 900", guid="tmdb:900"
)
db.add_all([show500, movie900])
db.commit()
# A movie rec for id 500 must NOT match show s500; a tv rec for id 900 must NOT match movie m900.
miss = {
r["tmdb_id"]: r
for r in tmdb.resolve_recs(
db,
[
{"tmdb_id": 500, "title": "Movie 500", "media_type": "movie"},
{"tmdb_id": 900, "title": "Show 900", "media_type": "tv"},
],
)
}
assert miss[500]["in_library"] is False and miss[500]["rating_key"] is None
assert miss[900]["in_library"] is False and miss[900]["rating_key"] is None
# The right media_type DOES resolve to the right row.
hit = {
r["tmdb_id"]: r
for r in tmdb.resolve_recs(
db,
[
{"tmdb_id": 900, "media_type": "movie"},
{"tmdb_id": 500, "media_type": "tv"},
],
)
}
assert hit[900]["in_library"] is True and hit[900]["rating_key"] == "m900"
assert hit[500]["in_library"] is True and hit[500]["rating_key"] == "s500"
+75
View File
@@ -0,0 +1,75 @@
"""ask 4: saved views are per-module (`source`). Feed and Plex views are separate lists, may share a
name, and each has its own single default. DB lane (`db` fixture) — skipped without Postgres.
"""
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import User
def _client(db, user, monkeypatch):
monkeypatch.setattr(state, "_configured_cache", True) # the request gate 503s an unconfigured app
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[current_user] = lambda: user # require_human resolves through this
return TestClient(app)
def _user(db):
u = User(email="views@example.com")
db.add(u)
db.commit()
return u
def test_views_are_scoped_by_source(db, monkeypatch):
user = _user(db)
try:
c = _client(db, user, monkeypatch)
# Same NAME under two sources is allowed (unique is per user+source now).
f = c.post("/api/saved-views", json={"name": "Fresh", "filters": {"a": 1}, "source": "feed"})
p = c.post("/api/saved-views", json={"name": "Fresh", "filters": {"b": 2}, "source": "plex"})
assert f.status_code == 200 and p.status_code == 200, (f.text, p.text)
feed = c.get("/api/saved-views?source=feed").json()
plex = c.get("/api/saved-views?source=plex").json()
assert [v["name"] for v in feed] == ["Fresh"]
assert [v["name"] for v in plex] == ["Fresh"]
assert feed[0]["filters"] == {"a": 1} and plex[0]["filters"] == {"b": 2}
# A source-less GET keeps the old feed-only behaviour.
assert [v["id"] for v in c.get("/api/saved-views").json()] == [v["id"] for v in feed]
finally:
app.dependency_overrides.clear()
def test_default_is_per_source(db, monkeypatch):
user = _user(db)
try:
c = _client(db, user, monkeypatch)
fid = c.post(
"/api/saved-views", json={"name": "F", "filters": {}, "source": "feed"}
).json()["id"]
pid = c.post(
"/api/saved-views", json={"name": "P", "filters": {}, "source": "plex"}
).json()["id"]
c.patch(f"/api/saved-views/{fid}", json={"is_default": True})
c.patch(f"/api/saved-views/{pid}", json={"is_default": True})
# Making the Plex view default must NOT clear the feed view's default (different source).
assert c.get("/api/saved-views?source=feed").json()[0]["is_default"] is True
assert c.get("/api/saved-views?source=plex").json()[0]["is_default"] is True
finally:
app.dependency_overrides.clear()
def test_duplicate_name_within_a_source_conflicts(db, monkeypatch):
user = _user(db)
try:
c = _client(db, user, monkeypatch)
a = c.post("/api/saved-views", json={"name": "Dup", "filters": {}, "source": "plex"})
b = c.post("/api/saved-views", json={"name": "Dup", "filters": {}, "source": "plex"})
assert a.status_code == 200
assert b.status_code == 409 # same (user, source, name)
finally:
app.dependency_overrides.clear()
+155
View File
@@ -0,0 +1,155 @@
"""Regression: live YouTube search ingest must survive the scrape source carrying no publish date.
`videos.published_at` is NOT NULL (migration 0068, PERF-15). The scrape source exposes only a
relative time ("3 years ago"), so its stub has `published_at=None`; the ingest used to pass that
None straight through, which the NOT NULL constraint rejected with an IntegrityError. That error
is raised OUTSIDE the route's page-fetch try/except, so the whole `/api/search/youtube` request
500'd and the UI showed the generic "YouTube search failed." — every scrape search was broken.
The stub now falls back to now() (a non-null placeholder), and the enrich overwrites it with the
video's real snippet date. These DB-lane tests lock both halves: no crash on a date-less stub, and
a placeholder never becomes the video's permanent date.
"""
from datetime import datetime, timezone
from app import sysconfig
from app.models import User, Video
from app.routes.search import _ingest_candidates
class _FakeYT:
"""Offline stand-in for YouTubeClient. Channel enrich is a no-op (the from_search stub survives);
video enrich returns a real publishedAt plus a long duration so Shorts classification finalises
without the youtube.com/shorts probe (no network in the DB lane)."""
def __init__(self, enrich: dict[str, dict]):
self._enrich = enrich
def get_channels(self, ids):
return []
def get_videos(self, ids):
return [self._enrich[i] for i in ids if i in self._enrich]
def _candidate(vid: str, ch: str) -> dict:
# The scrape shape (see search_scrape._video_item): published_at is ALWAYS None.
return {
"id": vid,
"channel_id": ch,
"channel_title": f"Chan {ch}",
"title": f"Title {vid}",
"published_at": None,
"thumbnail_url": None,
"live_broadcast": "none",
}
def _enrich_item(vid: str, published_iso: str) -> dict:
# PT30M keeps the video well above shorts_probe_max_seconds so no probe fires.
return {
"id": vid,
"snippet": {"title": f"Title {vid}", "publishedAt": published_iso},
"contentDetails": {"duration": "PT30M"},
"statistics": {},
"status": {},
}
def test_scrape_ingest_dateless_stub_ingests_and_takes_real_enrich_date(db):
# Small cap so the 1800s enrich duration keeps the shorts step offline.
sysconfig.set_value(db, "shorts_probe_max_seconds", 60)
user = User(email="searcher@example.com")
db.add(user)
db.commit()
candidates = [_candidate("vid_aaaaaaa", "UC_chan_a"), _candidate("vid_bbbbbbb", "UC_chan_a")]
enrich = {
"vid_aaaaaaa": _enrich_item("vid_aaaaaaa", "2021-02-12T19:00:07Z"),
"vid_bbbbbbb": _enrich_item("vid_bbbbbbb", "2023-10-30T17:00:08Z"),
}
kept = _ingest_candidates(db, _FakeYT(enrich), user, candidates)
# No IntegrityError, and both non-Short/non-live videos are kept in input order.
assert kept == ["vid_aaaaaaa", "vid_bbbbbbb"]
a = db.get(Video, "vid_aaaaaaa")
# The now() placeholder was replaced by the video's real snippet date, not left as "now".
assert a.published_at == datetime(2021, 2, 12, 19, 0, 7, tzinfo=timezone.utc)
assert a.via_search is True
assert a.enriched_at is not None
def test_scrape_ingest_enrich_miss_keeps_nonnull_placeholder(db):
# If enrich yields nothing (proxy blip / omitted row), the stub must STILL satisfy the NOT NULL
# constraint (a non-null placeholder) — it simply isn't kept (enriched_at stays None).
user = User(email="searcher2@example.com")
db.add(user)
db.commit()
kept = _ingest_candidates(db, _FakeYT({}), user, [_candidate("vid_ccccccc", "UC_chan_c")])
assert kept == []
c = db.get(Video, "vid_ccccccc")
assert c.published_at is not None # placeholder wrote a real value → no crash
assert c.enriched_at is None
def test_search_youtube_route_returns_422_when_ingest_fails(db, monkeypatch):
# Route-level guard (B-core-1): an ingest failure with nothing yet collected must surface as a 422
# + message on this JSON endpoint (the FE hides 5xx behind a generic "connection lost" + auto-retry,
# burning a second quota-costed search), and the session is rolled back. Here the page fetch succeeds
# (one candidate) and _ingest_candidates raises → the endpoint returns 422.
from fastapi.testclient import TestClient
import app.state as state
from app.auth import require_human
from app.db import get_db
from app.main import app
from app.routes import search as search_mod
user = User(email="route422@example.com")
db.add(user)
db.commit()
monkeypatch.setattr(state, "_configured_cache", True)
monkeypatch.setattr(
search_mod.sysconfig, "get_str", lambda d, k: "scrape" if k == "search_source" else ""
)
monkeypatch.setattr(search_mod.sysconfig, "get_int", lambda d, k: 100) # daily cap not reached
monkeypatch.setattr(search_mod.quota, "actions_today", lambda d, uid, action: 0)
class _FakeYTCM:
def __init__(self, *a, **k):
pass
def __enter__(self):
return self
def __exit__(self, *a):
return False
monkeypatch.setattr(search_mod, "YouTubeClient", _FakeYTCM)
monkeypatch.setattr(
search_mod.search_scrape,
"search",
lambda term, continuation=None: {
"items": [_candidate("vid_route12", "UC_route_a")],
"next_page_token": None,
},
)
def _boom(db_, yt, user_, candidates):
raise RuntimeError("ingest exploded")
monkeypatch.setattr(search_mod, "_ingest_candidates", _boom)
app.dependency_overrides[get_db] = lambda: db
app.dependency_overrides[require_human] = lambda: user
try:
resp = TestClient(app).get("/api/search/youtube?q=test+term")
finally:
app.dependency_overrides.clear()
assert resp.status_code == 422, resp.text
assert "YouTube search failed" in resp.json()["detail"]
+28 -1
View File
@@ -1,7 +1,7 @@
"""iso(): the single date/datetime→JSON serializer the routes share (R6 S1a).""" """iso(): the single date/datetime→JSON serializer the routes share (R6 S1a)."""
from datetime import date, datetime, timezone from datetime import date, datetime, timezone
from app.utils import iso from app.utils import iso, normalize_description
def test_none_maps_to_none(): def test_none_maps_to_none():
@@ -28,3 +28,30 @@ def test_matches_datetime_isoformat_contract():
# hand-roll `x.isoformat() if x else None`. # hand-roll `x.isoformat() if x else None`.
dt = datetime(2020, 1, 2, 3, 4, 5, 678901, tzinfo=timezone.utc) dt = datetime(2020, 1, 2, 3, 4, 5, 678901, tzinfo=timezone.utc)
assert iso(dt) == dt.isoformat() assert iso(dt) == dt.isoformat()
def test_normalize_description_none_stays_none():
assert normalize_description(None) is None
def test_normalize_description_drops_blank_lines():
# A YouTube contact block padded with runs of blank lines — the waste we strip before storing.
raw = "Contact us\n\n\nEmail: a@b.com\n \nInstagram: @x\n\n\n"
assert normalize_description(raw) == "Contact us\nEmail: a@b.com\nInstagram: @x"
def test_normalize_description_right_strips_and_trims():
assert normalize_description(" hi there \n\n bye ") == " hi there\n bye"
def test_normalize_description_all_blank_collapses_to_empty():
assert normalize_description("\n \n\t\n") == ""
def test_normalize_description_single_line_unchanged():
assert normalize_description("just one line") == "just one line"
def test_normalize_description_folds_crlf_line_endings():
# splitlines() folds CR/CRLF to \n — a documented, benign line-ending normalization.
assert normalize_description("a\r\nb\r\n\r\nc") == "a\nb\nc"
+61
View File
@@ -0,0 +1,61 @@
# Development
How to run Siftlode locally and pass the gate. Conventions for the backend code itself live in
[`backend/app/README.md`](../backend/app/README.md).
## Run it locally
The whole stack (Postgres + API + worker) comes up from the localdev compose file:
```bash
docker compose -f docker-compose.localdev.yml up --build
```
Browse at **http://localhost:8080** (Google login works — the OAuth redirect is
`http://localhost:8080/auth/callback`). The API runs `alembic upgrade head` on startup, so the schema
is always current.
For a fast frontend edit loop, run the Vite dev server with HMR against that API:
```bash
cd frontend && npm install && npm run dev # http://localhost:5173, proxying the API on :8080
```
## The gate
A change is done when all of this is green. CI runs the same checks server-side
(`.gitea/workflows/check.yml`), so "main is always green".
**Backend**
```bash
# lint
python -m ruff check --config backend/ruff.toml backend scripts
# tests — pure-logic + the DB-backed integration lane, against an ephemeral Postgres.
# The backend tree is bind-mounted, so it always tests your working files.
docker compose -f backend/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test
docker compose -f backend/docker-compose.test.yml down -v
```
The test compose sets `SIFTLODE_TEST_REQUIRE_DB=1`, which turns a would-be silent skip (no DB, or no
ffmpeg) into a hard failure — so the gate can't print green while the DB-backed tests were quietly not
run. Tests that need a database take the `db` fixture (`backend/tests/conftest.py`); the rest are
pure-logic and run without one.
**Frontend** (from `frontend/`)
```bash
npm run typecheck # tsc x3 (app / node / e2e)
npm run lint # eslint
npm run format:check # prettier
npm run knip # unused exports/deps
npm run test # vitest
```
## Branching
Trunk-based: branch off `main`, keep the branch gate-green, and land the whole change back to `main`
in one reviewed step. Migrations are Alembic revisions under `backend/alembic/versions`; the API
applies them on startup, and the DB-lane rebuilds the schema via `alembic upgrade head`, so a broken
migration chain fails the tests.
+11 -27
View File
@@ -1,5 +1,5 @@
import { test, expect } from "@playwright/test"; import { test, expect } from "@playwright/test";
import { clickVisibleChannelLink, openApp } from "./helpers"; import { clickVisibleChannelLink, expectContentType, openApp, toggleContentType } from "./helpers";
// Locks channel-view entry + exit: opening a channel from a feed card replaces the view, and // Locks channel-view entry + exit: opening a channel from a feed card replaces the view, and
// both the in-page Back and the browser Back return to the feed. (This orthogonal channel branch // both the in-page Back and the browser Back return to the feed. (This orthogonal channel branch
@@ -68,29 +68,22 @@ test.describe("channel view", () => {
}) => { }) => {
await openApp(page); await openApp(page);
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
const feedLive = page.getByTestId("feed-content-includeLive"); await expectContentType(page, "includeLive", false);
await expect(feedLive).toHaveAttribute("aria-pressed", "false");
// Inherit OFF: the channel toolbar's Live/Upcoming stays off, not forced on. // Inherit OFF: the channel toolbar's Live/Upcoming stays off, not forced on.
await page.getByTestId("video-card-channel").first().click(); await page.getByTestId("video-card-channel").first().click();
await expect(page.getByTestId("channel-title")).toBeVisible(); await expect(page.getByTestId("channel-title")).toBeVisible();
await expect(page.getByTestId("feed-content-includeLive")).toHaveAttribute( await expectContentType(page, "includeLive", false);
"aria-pressed",
"false",
);
await page.getByTestId("channel-back").click(); await page.getByTestId("channel-back").click();
await expect(page.getByTestId("channel-title")).toHaveCount(0); await expect(page.getByTestId("channel-title")).toHaveCount(0);
// Inherit ON: flip the feed's Live/Upcoming on, reopen the channel → it inherits the on state. // Inherit ON: flip the feed's Live/Upcoming on, reopen the channel → it inherits the on state.
await feedLive.click(); await toggleContentType(page, "includeLive");
await expect(feedLive).toHaveAttribute("aria-pressed", "true"); await expectContentType(page, "includeLive", true);
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
await page.getByTestId("video-card-channel").first().click(); await page.getByTestId("video-card-channel").first().click();
await expect(page.getByTestId("channel-title")).toBeVisible(); await expect(page.getByTestId("channel-title")).toBeVisible();
await expect(page.getByTestId("feed-content-includeLive")).toHaveAttribute( await expectContentType(page, "includeLive", true);
"aria-pressed",
"true",
);
}); });
// Bug 2 (Platform S5): the channel view is its OWN filter scope — a content toggle made there // Bug 2 (Platform S5): the channel view is its OWN filter scope — a content toggle made there
@@ -99,27 +92,18 @@ test.describe("channel view", () => {
test("a channel's content toggle does not leak into the feed", async ({ page }) => { test("a channel's content toggle does not leak into the feed", async ({ page }) => {
await openApp(page); await openApp(page);
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
await expect(page.getByTestId("feed-content-includeLive")).toHaveAttribute( await expectContentType(page, "includeLive", false);
"aria-pressed",
"false",
);
// Turn Live/Upcoming ON inside the channel view. // Turn Live/Upcoming ON inside the channel view.
await page.getByTestId("video-card-channel").first().click(); await page.getByTestId("video-card-channel").first().click();
await expect(page.getByTestId("channel-title")).toBeVisible(); await expect(page.getByTestId("channel-title")).toBeVisible();
await page.getByTestId("feed-content-includeLive").click(); await toggleContentType(page, "includeLive");
await expect(page.getByTestId("feed-content-includeLive")).toHaveAttribute( await expectContentType(page, "includeLive", true);
"aria-pressed",
"true",
);
// Back on the feed, its Live/Upcoming is still off — the channel's change stayed local. // Back on the feed, its Live/Upcoming is still off — the channel's change stayed local.
await page.getByTestId("channel-back").click(); await page.getByTestId("channel-back").click();
await expect(page.getByTestId("channel-title")).toHaveCount(0); await expect(page.getByTestId("channel-title")).toHaveCount(0);
await expect(page.getByTestId("feed-content-includeLive")).toHaveAttribute( await expectContentType(page, "includeLive", false);
"aria-pressed",
"false",
);
}); });
// Platform S4b: the whole channel chrome (banner, identity, tabs, toolbar) is fixed — it stays // Platform S4b: the whole channel chrome (banner, identity, tabs, toolbar) is fixed — it stays
@@ -127,7 +111,7 @@ test.describe("channel view", () => {
test("the channel chrome stays fixed while the video grid scrolls", async ({ page }) => { test("the channel chrome stays fixed while the video grid scrolls", async ({ page }) => {
await openApp(page); await openApp(page);
await page.getByTestId("video-card-channel").first().click(); await page.getByTestId("video-card-channel").first().click();
const count = page.getByTestId("feed-result-count"); const count = page.getByTestId("item-count");
const title = page.getByTestId("channel-title"); const title = page.getByTestId("channel-title");
await expect(count).toBeVisible(); await expect(count).toBeVisible();
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
+10 -6
View File
@@ -8,15 +8,18 @@ test.describe("feed", () => {
await openApp(page); await openApp(page);
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
const count = page.getByTestId("feed-result-count"); const count = page.getByTestId("item-count");
await expect(count).toHaveText(/\d/); await expect(count).toHaveText(/\d/);
const baseline = await count.innerText(); const baseline = await count.innerText();
// Clean baseline: default filters, so no chips and "unwatched" is the active show. // Clean baseline: default filters, so no chips and "unwatched" is the active show (the Watch-state
// pill summarises the current pick).
await expect(page.getByTestId("active-filter-chip")).toHaveCount(0); await expect(page.getByTestId("active-filter-chip")).toHaveCount(0);
await expect(page.getByTestId("feed-show-unwatched")).toHaveAttribute("aria-pressed", "true"); await expect(page.getByTestId("feed-show")).toContainText("Unwatched");
// Narrow to "watched" — the seed guarantees exactly one, a real deterministic change. // Narrow to "watched" — open the Watch-state dropdown and pick it (single-select closes on pick).
// The seed guarantees exactly one, a real deterministic change.
await page.getByTestId("feed-show").click();
await page.getByTestId("feed-show-watched").click(); await page.getByTestId("feed-show-watched").click();
await expect(page.getByTestId("active-filter-chip")).toHaveCount(1); await expect(page.getByTestId("active-filter-chip")).toHaveCount(1);
await expect(count).not.toHaveText(baseline); await expect(count).not.toHaveText(baseline);
@@ -24,7 +27,7 @@ test.describe("feed", () => {
// Remove the chip → default show restored, chips gone. // Remove the chip → default show restored, chips gone.
await page.getByTestId("active-filter-chip").click(); await page.getByTestId("active-filter-chip").click();
await expect(page.getByTestId("active-filter-chip")).toHaveCount(0); await expect(page.getByTestId("active-filter-chip")).toHaveCount(0);
await expect(page.getByTestId("feed-show-unwatched")).toHaveAttribute("aria-pressed", "true"); await expect(page.getByTestId("feed-show")).toContainText("Unwatched");
}); });
test("scrolling virtualizes in fresh cards", async ({ page }) => { test("scrolling virtualizes in fresh cards", async ({ page }) => {
@@ -52,7 +55,7 @@ test.describe("feed", () => {
// pins in the shell's toolbar band above the scroller, so it stays put while the cards scroll. // pins in the shell's toolbar band above the scroller, so it stays put while the cards scroll.
test("the feed toolbar stays pinned while cards scroll", async ({ page }) => { test("the feed toolbar stays pinned while cards scroll", async ({ page }) => {
await openApp(page); await openApp(page);
const count = page.getByTestId("feed-result-count"); const count = page.getByTestId("item-count");
await expect(count).toBeVisible(); await expect(count).toBeVisible();
const topBefore = await count.evaluate((el) => el.getBoundingClientRect().top); const topBefore = await count.evaluate((el) => el.getBoundingClientRect().top);
@@ -73,6 +76,7 @@ test.describe("feed", () => {
expect(await page.getByRole("main").evaluate((el) => el.scrollTop)).toBeGreaterThan(500); expect(await page.getByRole("main").evaluate((el) => el.scrollTop)).toBeGreaterThan(500);
// Switch the show filter → a different result set → back to the top. // Switch the show filter → a different result set → back to the top.
await page.getByTestId("feed-show").click();
await page.getByTestId("feed-show-all").click(); await page.getByTestId("feed-show-all").click();
await expect.poll(() => page.getByRole("main").evaluate((el) => el.scrollTop)).toBeLessThan(50); await expect.poll(() => page.getByRole("main").evaluate((el) => el.scrollTop)).toBeLessThan(50);
}); });
+24
View File
@@ -33,6 +33,30 @@ export async function clickVisibleChannelLink(page: Page): Promise<boolean> {
return false; return false;
} }
/**
* Assert a content-type (e.g. "includeLive") is on/off via the Content-types FilterSelect dropdown.
* The options only exist while the dropdown is open, so this opens it, checks the option's
* `aria-checked` (the multi FilterSelect renders its rows as menuitemcheckbox), then closes it
* (Escape) so the floating panel can't block a later click.
*/
export async function expectContentType(page: Page, value: string, on: boolean): Promise<void> {
await page.getByTestId("feed-content").click();
await expect(page.getByTestId(`feed-content-${value}`)).toHaveAttribute(
"aria-checked",
on ? "true" : "false",
);
await page.keyboard.press("Escape");
await expect(page.getByTestId(`feed-content-${value}`)).toHaveCount(0);
}
/** Toggle a content-type via the Content-types FilterSelect dropdown, then close it. */
export async function toggleContentType(page: Page, value: string): Promise<void> {
await page.getByTestId("feed-content").click();
await page.getByTestId(`feed-content-${value}`).click();
await page.keyboard.press("Escape");
await expect(page.getByTestId(`feed-content-${value}`)).toHaveCount(0);
}
/** Open the in-app player from the first feed card (clicks its thumbnail link). */ /** Open the in-app player from the first feed card (clicks its thumbnail link). */
export async function openFirstVideo(page: Page): Promise<void> { export async function openFirstVideo(page: Page): Promise<void> {
await expect(page.getByTestId("video-card").first()).toBeVisible(); await expect(page.getByTestId("video-card").first()).toBeVisible();
+43 -52
View File
@@ -4,7 +4,7 @@ import { useTranslation } from "react-i18next";
import { useQuery, useQueryClient } from "@tanstack/react-query"; import { useQuery, useQueryClient } from "@tanstack/react-query";
import { api, getActiveAccount, setActiveAccount, setUnauthorizedHandler } from "./lib/api"; import { api, getActiveAccount, setActiveAccount, setUnauthorizedHandler } from "./lib/api";
import { setLanguage, isSupported, type LangCode } from "./i18n"; import { setLanguage, isSupported, type LangCode } from "./i18n";
import { hasFilterParams, stripUrlParams } from "./lib/urlState"; import { clearParams } from "./lib/urlState";
import type { Page } from "./lib/urlState"; import type { Page } from "./lib/urlState";
import { import {
loadLayout, loadLayout,
@@ -15,7 +15,8 @@ import {
type PanelLayout, type PanelLayout,
} from "./lib/panelLayout"; } from "./lib/panelLayout";
import { notify, removeByMetaKind } from "./lib/notifications"; import { notify, removeByMetaKind } from "./lib/notifications";
import { LS, readAccount, setAccountRaw, writeAccount } from "./lib/storage"; import { LS, setAccountRaw } from "./lib/storage";
import { usePref } from "./lib/usePref";
import { useNavigation, useNavigationActions } from "./components/NavigationProvider"; import { useNavigation, useNavigationActions } from "./components/NavigationProvider";
import { useFeedFilters, useFeedFiltersActions } from "./components/FeedFiltersProvider"; import { useFeedFilters, useFeedFiltersActions } from "./components/FeedFiltersProvider";
import { useChannelLayout } from "./components/ChannelsProvider"; import { useChannelLayout } from "./components/ChannelsProvider";
@@ -90,37 +91,20 @@ export default function App() {
} }
// Collapse state for the two full-height panels (left nav + filter sidebar). Persisted to the // Collapse state for the two full-height panels (left nav + filter sidebar). Persisted to the
// user's preferences so it follows the account across devices; a localStorage cache seeds the // user's preferences (via usePref) so it follows the account across devices; the localStorage mirror
// initial render synchronously so nothing flashes open before the server prefs arrive. // seeds the initial render synchronously so nothing flashes open before the server prefs arrive.
// Default unpinned (tab + hover-expand overlay); accounts that explicitly pin keep their choice. // Default unpinned (tab + hover-expand overlay); accounts that explicitly pin keep their choice.
const [filterCollapsed, setFilterCollapsedState] = useState<boolean>(() => const [filterCollapsed, setFilterCollapsed] = usePref<boolean>("filterCollapsed", true, {
Boolean(readAccount(LS.filterCollapsed, true)), localKey: LS.filterCollapsed,
); parse: Boolean,
});
// The Playlists rail has its own collapse flag (it's a list/navigator, not filters). // The Playlists rail has its own collapse flag (it's a list/navigator, not filters).
const [playlistsCollapsed, setPlaylistsCollapsedState] = useState<boolean>(() => const [playlistsCollapsed, setPlaylistsCollapsed] = usePref<boolean>("playlistsCollapsed", true, {
Boolean(readAccount(LS.playlistsCollapsed, true)), localKey: LS.playlistsCollapsed,
); parse: Boolean,
function setFilterCollapsed(next: boolean) { });
setFilterCollapsedState(next); // (The feed filters + page/channel now LIVE in the address bar — FeedFiltersProvider and
writeAccount(LS.filterCollapsed, next); // NavigationProvider read them on load and keep them mirrored — so there's no query to strip here.)
api.savePrefs({ filterCollapsed: next }).catch(() => {});
}
function setPlaylistsCollapsed(next: boolean) {
setPlaylistsCollapsedState(next);
writeAccount(LS.playlistsCollapsed, next);
api.savePrefs({ playlistsCollapsed: next }).catch(() => {});
}
// Filters live in localStorage, not the address bar. If we arrived via a "Share view"
// link, its params have already hydrated the initial state — persist them and strip the
// query so the URL stays clean from here on.
useEffect(() => {
const params = new URLSearchParams(window.location.search);
if (hasFilterParams(params) || params.has("page")) {
// A share link's filters are held in state; they're persisted to the account's key by the
// post-login effect (which knows the account id). Here we only clean the address bar.
stripUrlParams();
}
}, []);
const queryClient = useQueryClient(); const queryClient = useQueryClient();
// First-run install gate: a fresh instance reports configured=false and runs in setup mode. // First-run install gate: a fresh instance reports configured=false and runs in setup mode.
@@ -152,11 +136,24 @@ export default function App() {
if (meQuery.data && getActiveAccount() == null) setActiveAccount(meQuery.data.id); if (meQuery.data && getActiveAccount() == null) setActiveAccount(meQuery.data.id);
}, [meQuery.data?.id]); }, [meQuery.data?.id]);
// Once signed in, drop any leftover query string from the pre-login flow (e.g. // Measure the scrollbar's gutter width once and publish it as --sbw, so the floating header and the
// ?access=requested, ?verify, ?reset). The logged-in app reads nothing from the URL, so the // toolbar band can pad their right edge to line up with the scrolled cards (the app scroller reserves
// address bar stays clean. Gated on auth so the logged-out Welcome page still reads its params. // that gutter via scrollbar-gutter: stable). It's a device constant, so a one-shot mount measure is enough.
useEffect(() => { useEffect(() => {
if (meQuery.data) stripUrlParams(); const probe = document.createElement("div");
probe.style.cssText =
"position:absolute;top:-9999px;width:100px;height:100px;overflow:scroll;scrollbar-gutter:stable";
document.body.appendChild(probe);
const w = probe.offsetWidth - probe.clientWidth;
document.body.removeChild(probe);
document.documentElement.style.setProperty("--sbw", `${w}px`);
}, []);
// Once signed in, drop any leftover ONE-TIME query params from the pre-login flow (?access, ?verify,
// ?reset) — but NOT the nav/filter params, which the logged-in app now reads from and keeps in the
// URL. Gated on auth so the logged-out Welcome page still reads its params first.
useEffect(() => {
if (meQuery.data) clearParams("access", "verify", "reset");
}, [meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps }, [meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps
// Any request that 401s means the session ended server-side. If we were signed in (cached // Any request that 401s means the session ended server-side. If we were signed in (cached
@@ -188,7 +185,7 @@ export default function App() {
const key = result === "conflict" || result === "mismatch" ? result : "error"; const key = result === "conflict" || result === "mismatch" ? result : "error";
notify({ level: "error", message: t(`settings.account.googleLink.${key}`) }); notify({ level: "error", message: t(`settings.account.googleLink.${key}`) });
} }
stripUrlParams(); clearParams("link");
}, []); // eslint-disable-line react-hooks/exhaustive-deps }, []); // eslint-disable-line react-hooks/exhaustive-deps
// A Google OAuth flow started while already signed in (add-another-account, link, or upgrade) was // A Google OAuth flow started while already signed in (add-another-account, link, or upgrade) was
@@ -197,12 +194,12 @@ export default function App() {
useEffect(() => { useEffect(() => {
if (new URLSearchParams(window.location.search).get("oauth") !== "cancelled") return; if (new URLSearchParams(window.location.search).get("oauth") !== "cancelled") return;
notify({ level: "info", message: t("settings.account.googleCancelled") }); notify({ level: "info", message: t("settings.account.googleCancelled") });
stripUrlParams(); clearParams("oauth");
}, []); // eslint-disable-line react-hooks/exhaustive-deps }, []); // eslint-disable-line react-hooks/exhaustive-deps
// Deep-link from the "new access request" admin email (?admin=access-requests) → jump straight to // Deep-link from the "new access request" admin email (?admin=access-requests) → jump straight to
// the admin Users → Access requests view. Snapshot the param so the pre-login URL strip doesn't // the admin Users → Access requests view. Snapshot the param up front (read once at mount); act once
// drop it; act once `me` is known, and only for an admin (others just land on their default page). // `me` is known, and only for an admin (others just land on their default page).
const [adminDeepLink] = useState(() => new URLSearchParams(window.location.search).get("admin")); const [adminDeepLink] = useState(() => new URLSearchParams(window.location.search).get("admin"));
useEffect(() => { useEffect(() => {
if (adminDeepLink !== "access-requests" || !meQuery.data) return; if (adminDeepLink !== "access-requests" || !meQuery.data) return;
@@ -210,12 +207,13 @@ export default function App() {
focusAccessRequestsTab(); focusAccessRequestsTab();
setPage("users"); setPage("users");
} }
stripUrlParams(); clearParams("admin");
}, [adminDeepLink, meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps }, [adminDeepLink, meQuery.data?.id]); // eslint-disable-line react-hooks/exhaustive-deps
// On login, adopt the server-stored SHELL preferences (panel layouts, rail collapse flags, // On login, adopt the server-stored SHELL preferences (panel layouts + language). The rail-collapse
// language). The Settings-page prefs (theme/perf/hints/notifications) are adopted by PrefsProvider // flags (filterCollapsed / playlistsCollapsed) are adopted by their own usePref hooks, the
// and the feed view by FeedViewProvider — each via its own passive qk.me() observer. // Settings-page prefs (theme/perf/hints/notifications) by PrefsProvider, and the feed view by
// FeedViewProvider — each via its own passive qk.me() observer.
useEffect(() => { useEffect(() => {
const prefs = meQuery.data?.preferences; const prefs = meQuery.data?.preferences;
if (!prefs) return; if (!prefs) return;
@@ -227,14 +225,6 @@ export default function App() {
saveLayoutLocal(p, l); saveLayoutLocal(p, l);
} }
}); });
if (typeof prefs.filterCollapsed === "boolean") {
setFilterCollapsedState(prefs.filterCollapsed);
writeAccount(LS.filterCollapsed, prefs.filterCollapsed);
}
if (typeof prefs.playlistsCollapsed === "boolean") {
setPlaylistsCollapsedState(prefs.playlistsCollapsed);
writeAccount(LS.playlistsCollapsed, prefs.playlistsCollapsed);
}
const lang = prefs.language; const lang = prefs.language;
if (typeof lang === "string" && isSupported(lang)) setLanguage(lang); if (typeof lang === "string" && isSupported(lang)) setLanguage(lang);
// (Per-account filters — incl. the demo account's whole-library default — are loaded by the // (Per-account filters — incl. the demo account's whole-library default — are loaded by the
@@ -323,7 +313,7 @@ export default function App() {
const me = meQuery.data!; const me = meQuery.data!;
const pageGate = moduleDef(page).gate; const pageGate = moduleDef(page).gate;
const activePage: Page = !pageGate || pageGate(me) ? page : "feed"; const activePage: Page = !pageGate || pageGate(me) ? page : "feed";
const { Component: PageContent, fadeTop } = PAGE_CONTENT[activePage]; const { Component: PageContent, fadeTop, maxWidth } = PAGE_CONTENT[activePage];
const railDef = channelView ? undefined : PAGE_RAIL[activePage]; const railDef = channelView ? undefined : PAGE_RAIL[activePage];
return ( return (
@@ -389,6 +379,7 @@ export default function App() {
) : ( ) : (
<PageShell <PageShell
scrollKey={normalScrollKey} scrollKey={normalScrollKey}
maxWidth={maxWidth}
// Channels: the registry disables the top fade for its sticky-header table, but the card // Channels: the registry disables the top fade for its sticky-header table, but the card
// layout has no sticky header, so re-enable it there. // layout has no sticky header, so re-enable it there.
fadeTop={activePage === "channels" ? channelLayout === "cards" : (fadeTop ?? true)} fadeTop={activePage === "channels" ? channelLayout === "cards" : (fadeTop ?? true)}
+8 -1
View File
@@ -28,6 +28,10 @@ export default function AboutCell({ text, channelId }: { text: string | null; ch
offset: 6, offset: 6,
// Pop instantly like the old cell did; the close grace lets the cursor reach the card. // Pop instantly like the old cell did; the close grace lets the cursor reach the card.
hoverDelay: { open: 0, close: 160 }, hoverDelay: { open: 0, close: 160 },
// Cap the panel to the space between the trigger and the viewport edge (publishes --fit-max-h) so a
// long About never runs its top off-screen — at high browser zoom the short viewport made a fixed
// max-height overflow above the top edge, where the scroll container can't reach.
fitHeight: true,
}); });
const { data: ch } = useQuery({ const { data: ch } = useQuery({
@@ -51,7 +55,10 @@ export default function AboutCell({ text, channelId }: { text: string | null; ch
</span> </span>
<PopoverSurface <PopoverSurface
popover={pop} popover={pop}
className="glass w-[480px] max-h-112 overflow-y-auto px-3.5 py-3 rounded-lg animate-[fadeIn_0.12s_ease]" className="glass w-[480px] overflow-y-auto px-3.5 py-3 rounded-lg animate-[fadeIn_0.12s_ease]"
// min(design cap, available space): keeps the 28rem cap on a tall viewport, shrinks to fit on a
// short one (high zoom). --fit-max-h is set by the fitHeight `size` middleware once positioned.
style={{ maxHeight: "min(28rem, var(--fit-max-h, 28rem))" }}
> >
{/* Full About once the detail loads; until then the list's truncated description. */} {/* Full About once the detail loads; until then the list's truncated description. */}
{ch ? ( {ch ? (
+234 -27
View File
@@ -1,51 +1,258 @@
import { useLayoutEffect, useRef, useState } from "react";
import { useTranslation } from "react-i18next"; import { useTranslation } from "react-i18next";
import { X } from "lucide-react"; import { Filter, X } from "lucide-react";
import type { ActiveFilter } from "../lib/useActiveFilters"; import type { ActiveFilter } from "../lib/useActiveFilters";
import { decideChipLayout } from "../lib/chipOverflow";
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./AnchoredPopover";
// The applied-filter row above the feed: every narrowing currently in force, each removable on its // The applied-filter row above the feed: every narrowing currently in force, each removable on its
// own. It exists because the filter panel now sits collapsed as a tab — without this you'd have to // own. It exists because the filter panel now sits collapsed as a tab — without this you'd have to
// open the panel just to see what you'd filtered by. // open the panel just to see what you'd filtered by.
//
// The row is a single flex line that shares its width with the Sort/View controls pinned to its right,
// so the chips must never push those off-screen. When they'd overflow it trims responsively:
// - wide -> show as many chips as fit, then a "+N" pill whose popover holds the rest;
// - narrow -> collapse every chip into one "N filters" pill whose popover holds them all.
// Fit is measured off a hidden mirror of the full chip set (so slicing the visible lane can't feed
// back into the measurement and oscillate).
// Below this lane width an overflow collapses all the way to a single count pill (mobile), instead of
// the "fit some + N more" treatment used on wider lanes.
const NARROW_PX = 420;
const chipCls =
"chip group inline-flex items-center gap-1 pl-2.5 pr-1.5 py-1 text-sm whitespace-nowrap hover:border-accent hover:bg-accent/20 transition";
function ChipButton({ f, testId }: { f: ActiveFilter; testId?: string }) {
const { t } = useTranslation();
return (
<button
data-testid={testId}
data-filter-key={f.key}
onClick={f.remove}
title={t("feed.chips.remove", { name: f.label })}
aria-label={t("feed.chips.remove", { name: f.label })}
className={chipCls}
>
<span className="max-w-56 truncate">{f.label}</span>
<X className="w-3 h-3 shrink-0 opacity-60 group-hover:opacity-100" />
</button>
);
}
// The "+N" / "N filters" pill and its popover listing the (overflowed / all) chips, each removable.
function OverflowPopover({
label,
ariaLabel,
chips,
onClearAll,
icon,
}: {
label: string;
/** Accessible name for the trigger + popover — describes what's inside (e.g. "3 more filters"),
* since the visible label can be a bare "+3". Defaults to `label`. */
ariaLabel?: string;
chips: ActiveFilter[];
onClearAll: () => void;
icon?: boolean;
}) {
const { t } = useTranslation();
const [open, setOpen] = useState(false);
const pop = useAnchoredPopover({
open,
onOpenChange: setOpen,
mode: "menu",
placement: "bottom-end",
});
const name = ariaLabel ?? label;
return (
<>
<button
type="button"
ref={pop.refs.setReference}
{...pop.getReferenceProps()}
aria-label={name}
className={`chip inline-flex shrink-0 items-center gap-1 px-2.5 py-1 text-sm whitespace-nowrap transition ${
open ? "border-accent text-accent" : "hover:border-accent"
}`}
>
{icon && <Filter className="w-3.5 h-3.5 shrink-0" aria-hidden="true" />}
{label}
</button>
<PopoverSurface
popover={pop}
ariaLabel={name}
className="glass-menu min-w-48 max-w-72 rounded-xl p-1.5 shadow-lg animate-[popIn_0.16s_ease]"
>
<div role="group" aria-label={name} className="max-h-72 overflow-y-auto no-scrollbar">
{chips.map((f) => (
<PopoverItem
key={f.key}
role="menuitem"
onSelect={f.remove}
label={f.label}
className="group flex w-full items-center gap-2 rounded-md px-2 py-1.5 text-left text-sm text-fg transition"
>
<span className="flex-1 truncate">{f.label}</span>
<X className="w-3.5 h-3.5 shrink-0 opacity-60 group-hover:opacity-100" />
</PopoverItem>
))}
</div>
<div className="mt-1 border-t border-border px-1 pt-1.5">
<button
type="button"
data-testid="active-filter-clear-all"
onClick={() => {
onClearAll();
setOpen(false);
}}
className="text-xs text-muted hover:text-fg underline underline-offset-2 transition"
>
{t("sidebar.clearAll")}
</button>
</div>
</PopoverSurface>
</>
);
}
export default function ActiveFilterChips({ export default function ActiveFilterChips({
filters, filters,
onClearAll, onClearAll,
className,
}: { }: {
filters: ActiveFilter[]; filters: ActiveFilter[];
onClearAll: () => void; onClearAll: () => void;
className?: string;
}) { }) {
const { t } = useTranslation(); const { t } = useTranslation();
if (filters.length === 0) return null; const laneRef = useRef<HTMLDivElement | null>(null);
const labelRef = useRef<HTMLSpanElement | null>(null);
const chipRefs = useRef<Array<HTMLSpanElement | null>>([]);
const plusRef = useRef<HTMLSpanElement | null>(null);
const clearRef = useRef<HTMLSpanElement | null>(null);
// How the row is currently laid out: all chips, a fit-count + "+N", or a single count pill.
const [layout, setLayout] = useState<{ mode: "all" | "some" | "count"; visible: number }>({
mode: "all",
visible: filters.length,
});
const n = filters.length;
// Re-measure whenever the chip set changes (labels or count) — captures both in one dep.
const chipsKey = filters.map((f) => f.key + f.label).join("|");
useLayoutEffect(() => {
const lane = laneRef.current;
const labelEl = labelRef.current;
if (!lane || !labelEl || n === 0) return;
const measure = () => {
const rightOf = (el: HTMLElement | null | undefined) =>
el ? el.offsetLeft + el.offsetWidth : 0;
const chips = chipRefs.current.slice(0, n);
const plusEl = plusRef.current;
// Measure the real rendered gap off the box between the last chip and the "+N" sample, so it
// scales with the text-size slider (gap-1.5 is rem) rather than being frozen in px.
const lastChip = chips[n - 1];
const gap = plusEl && lastChip ? Math.max(0, plusEl.offsetLeft - rightOf(lastChip)) : 6;
// "all" mode also renders a Clear all button after the chips (when n > 1), so reserve its width.
const clearW = n > 1 ? gap + (clearRef.current?.offsetWidth ?? 0) : 0;
setLayout(
decideChipLayout({
n,
labelRight: rightOf(labelEl),
chipRights: chips.map((c) => rightOf(c)),
plusW: plusEl?.offsetWidth ?? 0,
clearW,
avail: lane.clientWidth,
gap,
narrowPx: NARROW_PX,
}),
);
};
measure();
const ro = new ResizeObserver(measure);
ro.observe(lane);
return () => ro.disconnect();
}, [n, chipsKey]);
if (n === 0) return null;
const hiddenCount = n - layout.visible;
const overflowChips = filters.slice(layout.visible);
return ( return (
<div <div
ref={laneRef}
data-testid="active-filter-chips" data-testid="active-filter-chips"
className="flex flex-wrap items-center gap-1.5 pb-3" className={`relative flex min-w-0 items-center gap-1.5 ${className ?? ""}`}
aria-label={t("feed.chips.label")} aria-label={t("feed.chips.label")}
> >
<span className="text-2xs uppercase tracking-wider text-muted mr-0.5"> {/* Hidden mirror of the full set, used only to measure natural widths. Kept out of the a11y tree
{t("feed.chips.label")} and without test ids so it never doubles the visible chip count. */}
</span> <div
{filters.map((f) => ( aria-hidden="true"
<button className="pointer-events-none invisible absolute left-0 top-0 flex items-center gap-1.5"
key={f.key} >
data-testid="active-filter-chip" <span ref={labelRef} className="text-xs uppercase tracking-wider text-muted mr-0.5">
data-filter-key={f.key} {t("feed.chips.label")}
onClick={f.remove} </span>
title={t("feed.chips.remove", { name: f.label })} {filters.map((f, i) => (
aria-label={t("feed.chips.remove", { name: f.label })} <span
className="chip group pl-2.5 pr-1.5 py-1 text-xs hover:border-accent hover:bg-accent/20 transition" key={f.key}
> ref={(el) => {
<span className="max-w-56 truncate">{f.label}</span> chipRefs.current[i] = el;
<X className="w-3 h-3 shrink-0 opacity-60 group-hover:opacity-100" /> }}
</button> className={chipCls}
))} >
{filters.length > 1 && ( <span className="max-w-56 truncate">{f.label}</span>
<button <X className="w-3 h-3 shrink-0" />
data-testid="active-filter-clear-all" </span>
onClick={onClearAll} ))}
className="ml-0.5 text-xs text-muted hover:text-fg underline underline-offset-2 transition" <span ref={plusRef} className="chip inline-flex items-center gap-1 px-2.5 py-1 text-sm">
> +{n}
</span>
<span ref={clearRef} className="ml-0.5 text-sm underline underline-offset-2">
{t("sidebar.clearAll")} {t("sidebar.clearAll")}
</button> </span>
</div>
{/* Visible lane. */}
{layout.mode === "count" ? (
<OverflowPopover
label={t("feed.chips.count", { count: n })}
chips={filters}
onClearAll={onClearAll}
icon
/>
) : (
<>
<span className="text-xs uppercase tracking-wider text-muted mr-0.5 shrink-0">
{t("feed.chips.label")}
</span>
{filters.slice(0, layout.visible).map((f) => (
<ChipButton key={f.key} f={f} testId="active-filter-chip" />
))}
{layout.mode === "some" && hiddenCount > 0 && (
<OverflowPopover
label={`+${hiddenCount}`}
ariaLabel={t("feed.chips.moreCount", { count: hiddenCount })}
chips={overflowChips}
onClearAll={onClearAll}
/>
)}
{layout.mode === "all" && n > 1 && (
<button
data-testid="active-filter-clear-all"
onClick={onClearAll}
className="ml-0.5 shrink-0 text-sm text-muted hover:text-fg underline underline-offset-2 transition"
>
{t("sidebar.clearAll")}
</button>
)}
</>
)} )}
</div> </div>
); );
+1 -1
View File
@@ -34,7 +34,7 @@ export default function AdminUsers() {
<> <>
{/* The tab bar is fixed chrome; the tab content scrolls under it. */} {/* The tab bar is fixed chrome; the tab content scrolls under it. */}
<PageToolbar> <PageToolbar>
<div className="px-4 pt-3 max-w-3xl w-full mx-auto"> <div className="px-4 pt-2 max-w-3xl w-full mx-auto">
<Tabs tabs={tabs} active={active} onChange={setTab} /> <Tabs tabs={tabs} active={active} onChange={setTab} />
</div> </div>
</PageToolbar> </PageToolbar>
@@ -0,0 +1,60 @@
// @vitest-environment jsdom
//
// Gates the shared PopoverItem ARIA contract: aria-checked for menuitemradio/menuitemcheckbox and
// aria-selected for option are rendered as explicit attributes AFTER the floating-ui item props (so they
// win over getItemProps' single-select guess). Six live menus (LanguageSwitcher, ViewSwitcher,
// AddToPlaylist, ChannelTags, tableHeader, plexDetailUi) depend on this; without a test, a regression in
// the shared path would silently drop the checked indicator for screen-reader users. Floating-ui recipe:
// stub the two observers autoUpdate wires, wrap in a FloatingTree, query the hidden surface.
import { describe, it, expect, beforeAll, afterEach } from "vitest";
import { useState } from "react";
import { render, screen, cleanup } from "@testing-library/react";
import { FloatingTree } from "@floating-ui/react";
import { useAnchoredPopover, PopoverSurface, PopoverItem } from "./AnchoredPopover";
beforeAll(() => {
class Stub {
observe() {}
unobserve() {}
disconnect() {}
takeRecords() {
return [];
}
}
globalThis.ResizeObserver ??= Stub as unknown as typeof ResizeObserver;
globalThis.IntersectionObserver ??= Stub as unknown as typeof IntersectionObserver;
});
afterEach(cleanup);
function Menu() {
const [open, setOpen] = useState(true);
const pop = useAnchoredPopover({ open, onOpenChange: setOpen, mode: "menu" });
return (
<PopoverSurface popover={pop} ariaLabel="Choices">
<PopoverItem role="menuitemradio" checked onSelect={() => {}}>
Newest
</PopoverItem>
<PopoverItem role="menuitemradio" checked={false} onSelect={() => {}}>
Oldest
</PopoverItem>
<PopoverItem role="menuitemcheckbox" checked onSelect={() => {}}>
Show shorts
</PopoverItem>
</PopoverSurface>
);
}
describe("PopoverItem ARIA contract", () => {
it("exposes aria-checked on menuitemradio and menuitemcheckbox items", () => {
render(
<FloatingTree>
<Menu />
</FloatingTree>,
);
const radios = screen.getAllByRole("menuitemradio", { hidden: true });
expect(radios.map((r) => r.getAttribute("aria-checked"))).toEqual(["true", "false"]);
const checkbox = screen.getByRole("menuitemcheckbox", { hidden: true });
expect(checkbox.getAttribute("aria-checked")).toBe("true");
});
});
+118 -30
View File
@@ -2,6 +2,7 @@ import {
createContext, createContext,
useContext, useContext,
useEffect, useEffect,
useId,
useLayoutEffect, useLayoutEffect,
useRef, useRef,
useState, useState,
@@ -61,14 +62,29 @@ import { useFullscreenElement } from "../lib/fullscreen";
// right role (menuitem / menuitemradio+aria-checked / option+aria-selected). A menu that also holds a // right role (menuitem / menuitemradio+aria-checked / option+aria-selected). A menu that also holds a
// free-text input (the tag creator) must stopPropagation non-Escape keys on that input so ↑/↓ and // free-text input (the tag creator) must stopPropagation non-Escape keys on that input so ↑/↓ and
// typeahead act on the text, not the list — see ChannelTags. // typeahead act on the text, not the list — see ChannelTags.
//
// S? adds a "combobox" mode — the ONE right way to build a typeable combobox (a search/filter input
// with a listbox of options below), so a hand-rolled floating-ui engine (the old MultiSelectChips /
// PlexSearchBox pattern) is no longer needed. It differs from menu/listbox in exactly the focus model:
// - the trigger IS a text input, so open is driven by the caller (focus/typing), NOT useClick;
// - focus STAYS on the input — the surface is NOT focus-trapped and list items are never DOM-focused;
// arrow-nav is VIRTUAL (useListNavigation `virtual:true` → the ↑/↓/Home/End handler rides on the
// input's getReferenceProps and moves an `aria-activedescendant` pointer, not real focus).
// Because `role:"combobox"` is passed to useRole, Floating UI wires the whole ARIA tree for free: the
// reference input gets role=combobox + aria-expanded/controls/autocomplete/activedescendant, the surface
// gets role=listbox, and each <PopoverItem role="option"> gets its option id (so activedescendant has a
// target). The caller mounts the INPUT as `refs.setReference` (makes it a "typeable combobox reference")
// and the field box as `refs.setPositionReference` (so the panel anchors under the whole chips+input
// field). The universal <Combobox> component wraps all of this — call sites use it, not this mode raw.
// Internal to the platform; call sites pass the string literal via `mode`, they don't name the type. // Internal to the platform; call sites pass the string literal via `mode`, they don't name the type.
type PopoverMode = "menu" | "listbox" | "hover" | "dialog"; type PopoverMode = "menu" | "listbox" | "hover" | "dialog" | "combobox";
export interface AnchoredPopoverOptions { export interface AnchoredPopoverOptions {
open: boolean; open: boolean;
onOpenChange: (open: boolean, event?: Event) => void; onOpenChange: (open: boolean, event?: Event) => void;
/** "menu"/"listbox"/"dialog" open on click; "hover" opens on hover+focus (tooltips, hover cards). */ /** "menu"/"listbox"/"dialog" open on click; "hover" opens on hover+focus (tooltips, hover cards);
* "combobox" is a typeable input + listbox where focus stays in the field (use via <Combobox>). */
mode?: PopoverMode; mode?: PopoverMode;
placement?: Placement; placement?: Placement;
/** Gap between trigger and panel, px. Default 4. */ /** Gap between trigger and panel, px. Default 4. */
@@ -99,9 +115,14 @@ type AnchoredPopover = ReturnType<typeof useAnchoredPopover>;
export function useAnchoredPopover(opts: AnchoredPopoverOptions) { export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
const mode = opts.mode ?? "menu"; const mode = opts.mode ?? "menu";
const isHover = mode === "hover"; const isHover = mode === "hover";
const isCombobox = mode === "combobox";
// Roving-keyboard list surfaces: a menu of actions or a listbox of options. A "dialog" popover holds // Roving-keyboard list surfaces: a menu of actions or a listbox of options. A "dialog" popover holds
// arbitrary focusable content (its own Tab order), and "hover" is a passive tooltip — neither roves. // arbitrary focusable content (its own Tab order), and "hover" is a passive tooltip — neither roves.
const isNav = mode === "menu" || mode === "listbox"; const isNav = mode === "menu" || mode === "listbox";
// Surfaces that host a navigable item list (registry + list-navigation): the roving menu/listbox AND
// the virtual-nav combobox. The combobox navigates WITHOUT moving DOM focus (see below), so it is kept
// out of `isNav` (which also gates the focus trap + typeahead + roving tabindex) and folded in here.
const roves = isNav || isCombobox;
// null when not inside a <FloatingTree> — the hook still works, just without nested-layer awareness. // null when not inside a <FloatingTree> — the hook still works, just without nested-layer awareness.
const nodeId = useFloatingNodeId(); const nodeId = useFloatingNodeId();
@@ -131,7 +152,27 @@ export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
size({ size({
padding: 8, padding: 8,
apply({ availableHeight, elements }) { apply({ availableHeight, elements }) {
// Cap the positioned element, AND publish the available space as a custom property so
// the INNER styled/scroll wrapper (PopoverSurface renders `className` on a child, not on
// the floating element) can cap itself too — e.g. `maxHeight: min(28rem, var(--fit-max-h))`
// keeps a caller's design cap while still shrinking to fit at high zoom / short viewports.
// Capping only the outer element would leave the inner overflow-y-auto box at its fixed
// height, spilling its top off-screen (the whole point of fitHeight).
elements.floating.style.maxHeight = `${availableHeight}px`; elements.floating.style.maxHeight = `${availableHeight}px`;
elements.floating.style.setProperty("--fit-max-h", `${availableHeight}px`);
},
}),
]
: []),
// A combobox panel matches the field's width (with a sane floor for a very narrow rail); its inner
// scroll box owns the height cap. `rects.reference` is the POSITION reference — the whole chips+input
// field box (the caller mounts it via setPositionReference), not the bare input.
...(isCombobox
? [
size({
padding: 8,
apply({ rects, elements }) {
elements.floating.style.width = `${Math.max(rects.reference.width, 192)}px`;
}, },
}), }),
] ]
@@ -149,7 +190,9 @@ export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
} }
}, [opts.open]); }, [opts.open]);
const click = useClick(context, { enabled: !isHover }); // A combobox opens from the field's own focus/typing (wired by <Combobox>), never a click-to-toggle —
// a click on the input must place the caret, not close the just-opened list.
const click = useClick(context, { enabled: !isHover && !isCombobox });
const hover = useHover(context, { const hover = useHover(context, {
enabled: isHover, enabled: isHover,
move: false, move: false,
@@ -163,9 +206,19 @@ export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
// stopPropagation the Escape away from a Modal/player underneath, or double-close both.) Outside- // stopPropagation the Escape away from a Modal/player underneath, or double-close both.) Outside-
// press stays with floating-ui (tree-aware). // press stays with floating-ui (tree-aware).
const dismiss = useDismiss(context, { escapeKey: false }); const dismiss = useDismiss(context, { escapeKey: false });
const role = useRole(context, { role: opts.role ?? (isHover ? "tooltip" : "menu") }); // `role:"combobox"` makes Floating UI wire the whole typeable-combobox ARIA tree: the reference input
// → role=combobox + aria-expanded/controls/autocomplete + aria-activedescendant (from the virtual nav),
// the surface → role=listbox, the items → role=option + their option ids. (`combobox` maps to the
// `listbox` aria-role internally.) So <Combobox> sets NO aria-* by hand.
const role = useRole(context, {
role: opts.role ?? (isHover ? "tooltip" : isCombobox ? "combobox" : "menu"),
});
const listNav = useListNavigation(context, { const listNav = useListNavigation(context, {
enabled: isNav, enabled: roves,
// Virtual navigation for a combobox: DOM focus stays on the text input while ↑/↓ move an
// `aria-activedescendant` pointer over the options (the handler rides on getReferenceProps → the
// input). A menu/listbox uses real roving focus instead.
virtual: isCombobox,
listRef: elementsRef, listRef: elementsRef,
activeIndex, activeIndex,
// Keyboard (and pointer, when floating-ui owns it) navigation takes the visual highlight back from // Keyboard (and pointer, when floating-ui owns it) navigation takes the visual highlight back from
@@ -181,12 +234,15 @@ export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
// `isListIndexDisabled`.) No explicit index list needed. // `isListIndexDisabled`.) No explicit index list needed.
loop: true, loop: true,
// Focus the first item on open for BOTH click and keyboard (the default 'auto' skips pointer-opens, // Focus the first item on open for BOTH click and keyboard (the default 'auto' skips pointer-opens,
// which strands focus on the menu container — arrows then can't enter the roving list at all). // which strands focus on the menu container — arrows then can't enter the roving list at all). A
focusItemOnOpen: true, // combobox opens with NO active item — the caret is in the input, and the first ↓ lands on the first
// A menu that hosts a free-text input must NOT pull focus onto a list item when the pointer merely // option; auto-highlighting item 0 on open would fight the typed query.
// passes over it (the library default) — that blurs the input mid-typing and mis-routes the next focusItemOnOpen: isCombobox ? false : true,
// Enter to the hovered item. Turned off together with typeahead for such menus. // A menu that hosts a free-text input, and every combobox, must NOT pull focus onto a list item when
focusItemOnHover: !opts.hostsInput, // the pointer merely passes over it (the library default) — that blurs the input mid-typing and
// mis-routes the next Enter to the hovered item. The visual hover highlight is driven off hoverIndex
// instead (see PopoverItem), so the pointer still highlights without stealing the caret.
focusItemOnHover: isCombobox ? false : !opts.hostsInput,
}); });
const typeahead = useTypeahead(context, { const typeahead = useTypeahead(context, {
enabled: isNav && (opts.typeahead ?? true) && !opts.hostsInput, enabled: isNav && (opts.typeahead ?? true) && !opts.hostsInput,
@@ -215,6 +271,8 @@ export function useAnchoredPopover(opts: AnchoredPopoverOptions) {
nodeId, nodeId,
mode, mode,
isNav, isNav,
isCombobox,
roves,
open: opts.open, open: opts.open,
selectedIndex: opts.selectedIndex ?? null, selectedIndex: opts.selectedIndex ?? null,
refs, refs,
@@ -238,6 +296,9 @@ interface PopoverListContextValue {
highlightIndex: number | null; highlightIndex: number | null;
setHoverIndex: (index: number | null) => void; setHoverIndex: (index: number | null) => void;
getItemProps: AnchoredPopover["getItemProps"]; getItemProps: AnchoredPopover["getItemProps"];
/** Combobox surface: items are navigated VIRTUALLY (aria-activedescendant), so a <PopoverItem> never
* takes DOM focus (roving tabindex off) and keeps the caret in the input on click. */
virtual: boolean;
} }
const PopoverListContext = createContext<PopoverListContextValue | null>(null); const PopoverListContext = createContext<PopoverListContextValue | null>(null);
@@ -295,6 +356,12 @@ export function PopoverItem({
label?: string; label?: string;
} & Omit<ButtonHTMLAttributes<HTMLButtonElement>, "onSelect" | "role" | "children" | "className">) { } & Omit<ButtonHTMLAttributes<HTMLButtonElement>, "onSelect" | "role" | "children" | "className">) {
const ctx = useContext(PopoverListContext); const ctx = useContext(PopoverListContext);
const virtual = ctx?.virtual ?? false;
// A virtual (combobox) option needs a STABLE DOM id from first render: the input's aria-activedescendant
// is set to the active option's `.id` when the arrow handler reads it, so the id must already be there
// (Floating UI's own on-activation id assignment races the re-render and lands empty through our
// indirection). A menu/listbox item uses roving DOM focus instead and needs no id.
const generatedId = useId();
const textLabel = label ?? (typeof children === "string" ? children : undefined); const textLabel = label ?? (typeof children === "string" ? children : undefined);
const { ref, index } = useListItem({ label: textLabel }); const { ref, index } = useListItem({ label: textLabel });
// Roving tabindex / DOM focus follow the KEYBOARD position (activeIndex); the visible highlight // Roving tabindex / DOM focus follow the KEYBOARD position (activeIndex); the visible highlight
@@ -302,12 +369,12 @@ export function PopoverItem({
const focusActive = ctx?.activeIndex === index; const focusActive = ctx?.activeIndex === index;
const visualActive = ctx?.highlightIndex === index; const visualActive = ctx?.highlightIndex === index;
const ariaProps = // The checked/selected state is rendered as an EXPLICIT attribute after {...handlers} below (so it
role === "menuitemradio" || role === "menuitemcheckbox" // wins): for a combobox `role:"combobox"` makes getItemProps emit its own `aria-selected` off the
? { role, "aria-checked": !!checked } // single selectedIndex, which is always false for a MULTI-select — our per-value truth must override.
: role === "option" const ariaChecked =
? { role, "aria-selected": !!selected } role === "menuitemradio" || role === "menuitemcheckbox" ? !!checked : undefined;
: { role }; const ariaSelected = role === "option" ? !!selected : undefined;
// Route the caller's own props (onClick, onFocus, …) THROUGH floating-ui's item-prop getter, which // Route the caller's own props (onClick, onFocus, …) THROUGH floating-ui's item-prop getter, which
// composes handlers rather than dropping them — spreading `{...rest}` separately would let the // composes handlers rather than dropping them — spreading `{...rest}` separately would let the
@@ -325,19 +392,37 @@ export function PopoverItem({
rest.onMouseMove?.(e); rest.onMouseMove?.(e);
if (ctx && !disabled && index != null && ctx.highlightIndex !== index) ctx.setHoverIndex(index); if (ctx && !disabled && index != null && ctx.highlightIndex !== index) ctx.setHoverIndex(index);
}; };
// In a combobox, clicking an option must NOT move DOM focus off the input (that would close the list
// and drop the caret) — preventDefault the mousedown so focus stays put; the onClick still selects.
const mergedMouseDown = (e: React.MouseEvent<HTMLButtonElement>) => {
rest.onMouseDown?.(e);
if (virtual) e.preventDefault();
};
const handlers = ctx const handlers = ctx
? ctx.getItemProps({ ...rest, onClick: mergedClick, onMouseMove: mergedMouseMove }) ? ctx.getItemProps({
: { ...rest, onClick: mergedClick }; ...rest,
onClick: mergedClick,
onMouseMove: mergedMouseMove,
onMouseDown: mergedMouseDown,
})
: { ...rest, onClick: mergedClick, onMouseDown: mergedMouseDown };
return ( return (
<button <button
ref={ref} ref={ref}
type="button" type="button"
{...ariaProps} role={role}
{...handlers} {...handlers}
// These win over anything in `handlers` — the roving tabindex must not be a caller override, // A virtual option carries its own stable id (the aria-activedescendant target); it wins over any
// and disabled/className/data-active are the platform's to set. // id Floating UI's getItemProps would inject on activation.
tabIndex={focusActive ? 0 : -1} id={virtual ? generatedId : undefined}
// These win over anything in `handlers` — the roving tabindex must not be a caller override, the
// checked/selected truth must beat getItemProps' single-select guess, and disabled/className/
// data-active are the platform's to set. A virtual (combobox) item never holds DOM focus, so its
// tabindex is always -1; navigation is by aria-activedescendant.
aria-checked={ariaChecked}
aria-selected={ariaSelected}
tabIndex={virtual ? -1 : focusActive ? 0 : -1}
data-active={visualActive ? "true" : undefined} data-active={visualActive ? "true" : undefined}
disabled={disabled} disabled={disabled}
className={`outline-hidden ${ className={`outline-hidden ${
@@ -402,6 +487,8 @@ function PopoverSurfaceInner({
isPositioned, isPositioned,
mode, mode,
isNav, isNav,
isCombobox,
roves,
selectedIndex, selectedIndex,
activeIndex, activeIndex,
highlightIndex, highlightIndex,
@@ -434,9 +521,10 @@ function PopoverSurfaceInner({
if (ariaLabel) delete floatingProps["aria-labelledby"]; if (ariaLabel) delete floatingProps["aria-labelledby"];
const zClass = mode === "hover" ? "z-tooltip" : "z-popover"; const zClass = mode === "hover" ? "z-tooltip" : "z-popover";
// Everything but a passive hover card manages focus by default (trap into the surface, return to // A menu/listbox/dialog traps focus into the surface (and returns it on close). A hover card is
// the trigger on close). // passive, and a COMBOBOX must leave focus in the input (its list is navigated virtually) — neither
const managed = focusManaged ?? mode !== "hover"; // is focus-managed.
const managed = focusManaged ?? (mode !== "hover" && !isCombobox);
const surface = ( const surface = (
<div <div
@@ -470,11 +558,11 @@ function PopoverSurfaceInner({
</div> </div>
); );
// Menu/listbox surfaces host the item registry (FloatingList) + active-state context so nested // Menu/listbox AND combobox surfaces host the item registry (FloatingList) + active-state context so
// <PopoverItem>s can self-register and rove. // nested <PopoverItem>s can self-register and rove/aria-activedescendant.
const positioned = isNav ? ( const positioned = roves ? (
<PopoverListContext.Provider <PopoverListContext.Provider
value={{ activeIndex, highlightIndex, setHoverIndex, getItemProps }} value={{ activeIndex, highlightIndex, setHoverIndex, getItemProps, virtual: isCombobox }}
> >
<FloatingList elementsRef={elementsRef} labelsRef={labelsRef}> <FloatingList elementsRef={elementsRef} labelsRef={labelsRef}>
{surface} {surface}
@@ -0,0 +1,93 @@
// @vitest-environment jsdom
//
// Contract of the shared horizontal rail. jsdom lays out nothing (scrollWidth/clientWidth are 0), so
// the overflow-driven scroll buttons only appear once we feed the scroller fake metrics and fire a
// scroll — which is exactly the affordance under test: a button shows ONLY on the clipped edge, and
// clicking it pages the scroller. i18n is key-passthrough, so asserted names are the i18n keys.
import { describe, it, expect, vi, beforeAll, afterEach } from "vitest";
import { render, screen, fireEvent, cleanup } from "@testing-library/react";
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return { ...actual, useTranslation: () => ({ t: (k: string) => k }) };
});
import { Carousel } from "./Carousel";
beforeAll(() => {
// useHScroll wires a ResizeObserver (watch-the-children); jsdom ships none.
class Stub {
observe() {}
unobserve() {}
disconnect() {}
takeRecords() {
return [];
}
}
globalThis.ResizeObserver ??= Stub as unknown as typeof ResizeObserver;
});
afterEach(cleanup);
function setMetrics(
el: HTMLElement,
m: { scrollWidth: number; clientWidth: number; scrollLeft: number },
) {
Object.defineProperty(el, "scrollWidth", { configurable: true, value: m.scrollWidth });
Object.defineProperty(el, "clientWidth", { configurable: true, value: m.clientWidth });
Object.defineProperty(el, "scrollLeft", {
configurable: true,
writable: true,
value: m.scrollLeft,
});
}
describe("Carousel", () => {
it("renders children and an optional header with a see-all action", () => {
const onSeeAll = vi.fn();
render(
<Carousel title="Recently added" onSeeAll={onSeeAll}>
<div>card-a</div>
<div>card-b</div>
</Carousel>,
);
expect(screen.getByText("card-a")).toBeTruthy();
expect(screen.getByRole("heading", { name: "Recently added" })).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "common.seeAll" }));
expect(onSeeAll).toHaveBeenCalledOnce();
});
it("omits the header when no title is given and names the scroll region", () => {
render(
<Carousel ariaLabel="Cast">
<div>x</div>
</Carousel>,
);
expect(screen.queryByRole("heading")).toBeNull();
expect(screen.getByRole("group", { name: "Cast" })).toBeTruthy();
});
it("shows only the clipped edge's button and pages the scroller on click", () => {
render(
<Carousel ariaLabel="rail">
<div>x</div>
</Carousel>,
);
const scroller = screen.getByRole("group", { name: "rail" });
const scrollBy = vi.fn();
scroller.scrollBy = scrollBy as unknown as typeof scroller.scrollBy;
// At the left boundary with more content to the right → only the right button.
setMetrics(scroller, { scrollWidth: 1000, clientWidth: 300, scrollLeft: 0 });
fireEvent.scroll(scroller);
expect(screen.queryByRole("button", { name: "common.scrollLeft" })).toBeNull();
fireEvent.click(screen.getByRole("button", { name: "common.scrollRight" }));
expect(scrollBy).toHaveBeenCalled();
// Scrolled into the middle → both edges clipped → both buttons.
setMetrics(scroller, { scrollWidth: 1000, clientWidth: 300, scrollLeft: 350 });
fireEvent.scroll(scroller);
expect(screen.getByRole("button", { name: "common.scrollLeft" })).toBeTruthy();
expect(screen.getByRole("button", { name: "common.scrollRight" })).toBeTruthy();
});
});
+135
View File
@@ -0,0 +1,135 @@
import { useCallback, type ReactNode } from "react";
import { ChevronLeft, ChevronRight } from "lucide-react";
import { useTranslation } from "react-i18next";
import clsx from "clsx";
import { useScrollEdges } from "../lib/useScrollEdges";
// Shared horizontal rail — the one place the app's "row of cards you scroll sideways" lives, replacing
// the hand-rolled `flex gap overflow-x-auto` strips (Cast / Related / collections) and the future
// Plex-Discover rows. Gives every consumer the same affordances for free: edge fade, hover/focus scroll
// buttons that appear only on the side that actually has more content, a native scrollbar hidden behind
// `.no-scrollbar`, and reduced-motion-aware paging. Purely presentational — the caller owns the cards.
// Prefer an instant jump when the viewer asked for reduced motion; smooth otherwise. Guarded for jsdom,
// where matchMedia is often absent (the button paging is still exercised — it just uses "auto" there).
function scrollBehavior(): ScrollBehavior {
const reduce =
typeof window !== "undefined" &&
typeof window.matchMedia === "function" &&
window.matchMedia("(prefers-reduced-motion: reduce)").matches;
return reduce ? "auto" : "smooth";
}
// Horizontal scroll-affordance: whether the scroller is clipped on the left / right, so the rail shows
// a button + fade only where there's more to see. The X-axis face of the shared `useScrollEdges` core
// (the same watch-the-scroller-and-its-children discipline as the vertical `useScrollFade`), plus the
// element itself so the paging buttons can scroll it.
function useHScroll(): {
ref: (node: HTMLElement | null) => void;
el: HTMLElement | null;
left: boolean;
right: boolean;
} {
const { ref, el, start, end } = useScrollEdges("x");
return { ref, el, left: start, right: end };
}
const FADE_PX = 28;
export function Carousel({
title,
onSeeAll,
seeAllLabel,
ariaLabel,
gap = "gap-3",
className,
children,
}: {
/** Optional built-in header (rails). Omit for a bare scroller inside a card that owns its own heading. */
title?: ReactNode;
/** When set, a "see all" affordance appears in the header (needs `title`). */
onSeeAll?: () => void;
seeAllLabel?: string;
/** Names the scroll region for AT; falls back to the string `title` when that's a plain string. */
ariaLabel?: string;
/** Tailwind gap class for the scroller (e.g. "gap-4"). */
gap?: string;
className?: string;
children: ReactNode;
}) {
const { t } = useTranslation();
const { ref, el, left, right } = useHScroll();
const page = useCallback(
(dir: -1 | 1) => {
if (!el) return;
// ~one viewport-worth, but never a hair scroll on a narrow rail (or 0 in a zero-width jsdom box).
const amount = Math.max(Math.round(el.clientWidth * 0.85), 200);
el.scrollBy({ left: dir * amount, behavior: scrollBehavior() });
},
[el],
);
// Fade only the clipped edge — no fade at a boundary, none when it all fits (matches useScrollFade).
const mask = `linear-gradient(to right, transparent, #000 ${left ? FADE_PX : 0}px, #000 calc(100% - ${
right ? FADE_PX : 0
}px), transparent)`;
const label = ariaLabel ?? (typeof title === "string" ? title : undefined);
const arrowBtn =
"absolute top-1/2 -translate-y-1/2 z-10 grid h-9 w-9 place-items-center rounded-full " +
"bg-black/55 text-white backdrop-blur-xs border border-white/15 shadow-lg transition " +
"opacity-0 group-hover/car:opacity-100 focus-visible:opacity-100 hover:bg-black/75";
return (
<div className={clsx("group/car relative", className)}>
{title != null && (
<div className="mb-3 flex items-baseline gap-2">
<h2 className="text-sm font-semibold">{title}</h2>
{onSeeAll && (
<button
onClick={onSeeAll}
className="ml-auto inline-flex items-center gap-0.5 text-xs text-muted transition hover:text-accent"
>
{seeAllLabel ?? t("common.seeAll")}
<ChevronRight className="h-3.5 w-3.5" />
</button>
)}
</div>
)}
<div className="relative">
{left && (
<button
type="button"
onClick={() => page(-1)}
aria-label={t("common.scrollLeft")}
className={clsx(arrowBtn, "left-1")}
>
<ChevronLeft className="h-5 w-5" />
</button>
)}
<div
ref={ref}
className={clsx("no-scrollbar flex overflow-x-auto pb-1", gap)}
style={{ maskImage: mask, WebkitMaskImage: mask }}
role="group"
aria-label={label}
>
{children}
</div>
{right && (
<button
type="button"
onClick={() => page(1)}
aria-label={t("common.scrollRight")}
className={clsx(arrowBtn, "right-1")}
>
<ChevronRight className="h-5 w-5" />
</button>
)}
</div>
</div>
);
}
+1 -1
View File
@@ -200,7 +200,7 @@ export default function ChannelDiscovery({
<> <>
{/* Intro + the table controls are fixed chrome; only the table rows scroll (sticky header). */} {/* Intro + the table controls are fixed chrome; only the table rows scroll (sticky header). */}
<PageToolbar> <PageToolbar>
<div className="px-4 pt-3 max-w-384 mx-auto flex items-start justify-between gap-4"> <div className="px-4 pt-2 max-w-384 mx-auto flex items-start justify-between gap-4">
<p className="text-xs text-muted leading-relaxed">{t("channels.discovery.intro")}</p> <p className="text-xs text-muted leading-relaxed">{t("channels.discovery.intro")}</p>
{/* Cards has no column headers, so a dropdown drives the sort + a pager (table uses its own). */} {/* Cards has no column headers, so a dropdown drives the sort + a pager (table uses its own). */}
{layout === "cards" && ( {layout === "cards" && (
+2 -2
View File
@@ -27,7 +27,7 @@ import { useFeedFilters } from "./FeedFiltersProvider";
import { useConfirm } from "./ConfirmProvider"; import { useConfirm } from "./ConfirmProvider";
import { notify } from "../lib/notifications"; import { notify } from "../lib/notifications";
import { notifyYouTubeActionError } from "../lib/youtubeErrors"; import { notifyYouTubeActionError } from "../lib/youtubeErrors";
import { api, type FeedFilters, type Me, type ChannelDetail } from "../lib/api"; import { api, DEFAULT_TAG_MODES, type FeedFilters, type Me, type ChannelDetail } from "../lib/api";
import type { FeedView } from "../lib/feedView"; import type { FeedView } from "../lib/feedView";
import { channelYouTubeUrl, formatViews } from "../lib/format"; import { channelYouTubeUrl, formatViews } from "../lib/format";
import { ChannelTagsControl } from "./ChannelTags"; import { ChannelTagsControl } from "./ChannelTags";
@@ -231,7 +231,7 @@ export default function ChannelPage({
// the explored, per-user-private videos show — the backend gates them to this explorer). // the explored, per-user-private videos show — the backend gates them to this explorer).
const [filters, setFilters] = useState<FeedFilters>(() => ({ const [filters, setFilters] = useState<FeedFilters>(() => ({
tags: [], tags: [],
tagMode: "or", tagModes: DEFAULT_TAG_MODES,
q: "", q: "",
sort: "newest", sort: "newest",
scope: "all", scope: "all",
+16 -76
View File
@@ -1,11 +1,8 @@
import { useState } from "react";
import { useTranslation } from "react-i18next"; import { useTranslation } from "react-i18next";
import { useQuery } from "@tanstack/react-query"; import { useQuery } from "@tanstack/react-query";
import { qk } from "../lib/queryKeys"; import { qk } from "../lib/queryKeys";
import { X } from "lucide-react";
import { api, type FeedFilters } from "../lib/api"; import { api, type FeedFilters } from "../lib/api";
import { useScrollFade } from "../lib/useScrollFade"; import Combobox, { type ComboboxOption } from "./Combobox";
import { inputCls } from "./ui/form";
// Pick channels to narrow the feed to — several OR together, so every other filter then applies // Pick channels to narrow the feed to — several OR together, so every other filter then applies
// across all of them. Until this existed the filter was settable only via a shared "?channel=" link: // across all of them. Until this existed the filter was settable only via a shared "?channel=" link:
@@ -22,86 +19,29 @@ export default function ChannelPicker({
setFilters: (f: FeedFilters) => void; setFilters: (f: FeedFilters) => void;
}) { }) {
const { t } = useTranslation(); const { t } = useTranslation();
const [query, setQuery] = useState("");
// Same edge-fade affordance the nav rail and the panel body use to hint at more content.
const fade = useScrollFade();
const channelsQuery = useQuery({ const channelsQuery = useQuery({
queryKey: qk.channels(), queryKey: qk.channels(),
queryFn: api.channels, queryFn: api.channels,
staleTime: 5 * 60_000, staleTime: 5 * 60_000,
}); });
const q = query.trim().toLowerCase();
const all = channelsQuery.data ?? []; const all = channelsQuery.data ?? [];
const matches = q ? all.filter((c) => (c.title ?? c.id).toLowerCase().includes(q)) : all; // The list order isn't alphabetical — it's the user's channel priority — and Combobox preserves the
const picked = filters.channelIds; // options order, so it stays meaningful. selectedLabels keeps a picked channel named even if a later
const nameOf = (id: string) => all.find((c) => c.id === id)?.title ?? id; // narrowing ever drops it from the visible rows.
const toggle = (id: string) => const options: ComboboxOption[] = all.map((c) => ({ value: c.id, label: c.title ?? c.id }));
setFilters({ const selectedLabels = Object.fromEntries(all.map((c) => [c.id, c.title ?? c.id]));
...filters,
channelIds: picked.includes(id) ? picked.filter((x) => x !== id) : [...picked, id],
});
return ( return (
<> <Combobox
{/* The picks sit above the list so they stay visible once the list scrolls away. */} label={t("sidebar.channelSearch")}
{picked.length > 0 && ( placeholder={t("sidebar.channelSearch")}
<div className="flex flex-wrap gap-1.5 pb-2"> options={options}
{picked.map((id) => ( selected={filters.channelIds}
<button onChange={(channelIds) => setFilters({ ...filters, channelIds })}
key={id} loading={channelsQuery.isLoading}
onClick={() => toggle(id)} emptyText={t("sidebar.noChannelMatch")}
title={t("sidebar.channelRemove", { name: nameOf(id) })} selectedLabels={selectedLabels}
className="chip-solid inline-flex items-center gap-1 max-w-full pl-2.5 pr-1.5 py-1 rounded-full text-xs shadow-xs hover:opacity-90 transition" />
>
<span className="truncate">{nameOf(id)}</span>
<X className="w-3 h-3 shrink-0" />
</button>
))}
</div>
)}
<input
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder={t("sidebar.channelSearch")}
aria-label={t("sidebar.channelSearch")}
className={inputCls}
/>
{/* The order isn't arbitrary and isn't alphabetical — say so, or the top of the list looks
random to anyone who hasn't set a priority. Only while unfiltered: a search reorders nothing
but does drop the rows that explain it. */}
{!q && all.length > 0 && (
<div className="px-2 pt-1.5 text-2xs text-muted">{t("sidebar.channelOrder")}</div>
)}
<div
ref={fade.ref}
style={fade.style}
className="mt-1.5 max-h-44 overflow-y-auto no-scrollbar flex flex-col gap-0.5"
>
{matches.length === 0 ? (
<div className="px-2 py-1.5 text-xs text-muted">
{channelsQuery.isLoading ? t("sidebar.loading") : t("sidebar.noChannelMatch")}
</div>
) : (
matches.map((c) => {
const on = picked.includes(c.id);
return (
<button
key={c.id}
onClick={() => toggle(c.id)}
aria-pressed={on}
// shrink-0 is load-bearing: these are flex children in a max-height box, so without
// them they squash to ~12px each and clip their own text instead of the box scrolling.
className={`w-full shrink-0 text-left text-sm px-2 py-1.5 rounded-lg truncate transition ${
on ? "text-accent font-medium" : "text-muted hover:text-fg hover:bg-card"
}`}
>
{c.title ?? c.id}
</button>
);
})
)}
</div>
</>
); );
} }
+2 -2
View File
@@ -115,7 +115,7 @@ export default function Channels() {
// FB2: syncing subscriptions can change the feed immediately — newly-followed channels may // FB2: syncing subscriptions can change the feed immediately — newly-followed channels may
// already have videos in the shared catalog, so refresh the feed too (not just the manager). // already have videos in the shared catalog, so refresh the feed too (not just the manager).
qc.invalidateQueries({ queryKey: qk.feed() }); qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() }); qc.invalidateQueries({ queryKey: qk.feedSummary() });
}; };
const userTags = (tagsQuery.data ?? []).filter((t) => !t.system); const userTags = (tagsQuery.data ?? []).filter((t) => !t.system);
@@ -131,7 +131,7 @@ export default function Channels() {
qc.invalidateQueries({ queryKey: qk.channels() }); qc.invalidateQueries({ queryKey: qk.channels() });
qc.invalidateQueries({ queryKey: qk.myStatus() }); qc.invalidateQueries({ queryKey: qk.myStatus() });
qc.invalidateQueries({ queryKey: qk.feed() }); qc.invalidateQueries({ queryKey: qk.feed() });
qc.invalidateQueries({ queryKey: qk.feedCount() }); qc.invalidateQueries({ queryKey: qk.feedSummary() });
}, },
}); });
// Priority is updated optimistically in place and NOT re-sorted/refetched, so the list doesn't // Priority is updated optimistically in place and NOT re-sorted/refetched, so the list doesn't
@@ -0,0 +1,249 @@
// @vitest-environment jsdom
//
// The universal combobox (chips + typed search field over the AnchoredPopover `combobox` mode). Covers
// chip removal, open-on-focus, client filter, toggling, Backspace-removes-last, the big-vocabulary cap,
// single-select, and the combobox ARIA the old hand-rolled control lacked (role=combobox, aria-expanded,
// and a virtual aria-activedescendant that keeps DOM focus on the input). i18n is key-passthrough; the
// floating-ui recipe (ActionMenu's): stub the two observers autoUpdate wires, wrap in a FloatingTree,
// and query the hidden surface with `{ hidden: true }` since jsdom never positions it.
import { describe, it, expect, vi, afterEach, beforeAll } from "vitest";
import { render, screen, fireEvent, cleanup } from "@testing-library/react";
import { FloatingTree } from "@floating-ui/react";
beforeAll(() => {
class Stub {
observe() {}
unobserve() {}
disconnect() {}
takeRecords() {
return [];
}
}
globalThis.ResizeObserver ??= Stub as unknown as typeof ResizeObserver;
globalThis.IntersectionObserver ??= Stub as unknown as typeof IntersectionObserver;
});
vi.mock("react-i18next", async (importOriginal) => {
const actual = await importOriginal<typeof import("react-i18next")>();
return { ...actual, useTranslation: () => ({ t: (k: string) => k }) };
});
import Combobox from "./Combobox";
afterEach(cleanup);
const OPTIONS = [
{ value: "Drama", count: 120 },
{ value: "Action", count: 90 },
{ value: "Comedy", count: 60 },
];
function wrap(ui: React.ReactElement) {
return render(<FloatingTree>{ui}</FloatingTree>);
}
describe("Combobox", () => {
it("renders selected values as removable chips and removes on ✕", () => {
const onChange = vi.fn();
wrap(<Combobox label="Genre" options={OPTIONS} selected={["Drama"]} onChange={onChange} />);
expect(screen.getByText("Drama")).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "common.remove" }));
expect(onChange).toHaveBeenCalledWith([]); // Drama removed
});
it("exposes combobox ARIA: role=combobox, aria-expanded toggling, aria-controls when open", () => {
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={vi.fn()} />);
const input = screen.getByRole("combobox");
expect(input.getAttribute("aria-expanded")).toBe("false");
expect(input.getAttribute("aria-controls")).toBeNull();
fireEvent.focus(input);
expect(input.getAttribute("aria-expanded")).toBe("true");
// aria-controls now points at the listbox surface.
const controls = input.getAttribute("aria-controls");
expect(controls).toBeTruthy();
expect(document.getElementById(controls!)?.getAttribute("role")).toBe("listbox");
});
it("focuses the input when the field chrome is pressed (merged ref, not a dead inputRef)", () => {
// Regression: the input carried only floating-ui's ref, so `inputRef` stayed null. The field's
// onMouseDown then preventDefault()-ed every press (its `e.target !== inputRef.current` guard was
// always true) and its compensating inputRef.current?.focus() was a no-op → the box couldn't be
// typed into with the mouse. A merged ref feeds the node to both, so pressing the chrome focuses it.
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={vi.fn()} />);
const input = screen.getByRole("combobox");
const field = input.parentElement as HTMLElement;
expect(document.activeElement).not.toBe(input);
fireEvent.mouseDown(field); // press the field padding, not the input itself
expect(document.activeElement).toBe(input);
});
it("keeps the option list unmounted until the field is focused", () => {
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={vi.fn()} />);
expect(screen.queryByRole("option", { hidden: true })).toBeNull();
fireEvent.focus(screen.getByRole("combobox"));
expect(screen.getAllByRole("option", { hidden: true }).length).toBe(3);
});
it("filters the options by the typed query", () => {
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={vi.fn()} />);
const field = screen.getByRole("combobox");
fireEvent.focus(field);
fireEvent.change(field, { target: { value: "com" } });
const items = screen.getAllByRole("option", { hidden: true });
expect(items.length).toBe(1);
expect(items[0]?.textContent).toContain("Comedy");
});
it("marks a selected option aria-selected and toggles another on", () => {
const onChange = vi.fn();
wrap(<Combobox label="Genre" options={OPTIONS} selected={["Drama"]} onChange={onChange} />);
fireEvent.focus(screen.getByRole("combobox"));
const drama = screen
.getAllByRole("option", { hidden: true })
.find((el) => el.textContent?.includes("Drama"));
expect(drama?.getAttribute("aria-selected")).toBe("true");
fireEvent.click(screen.getByText("Action")); // clicking the option's label toggles it on
expect(onChange).toHaveBeenCalledWith(["Drama", "Action"]);
});
it("moves an aria-activedescendant on ArrowDown while DOM focus stays on the input", () => {
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={vi.fn()} />);
const input = screen.getByRole("combobox");
input.focus();
fireEvent.focus(input);
fireEvent.keyDown(input, { key: "ArrowDown" });
// Virtual navigation: the input keeps DOM focus, the active option is referenced by id.
expect(document.activeElement).toBe(input);
expect(input.getAttribute("aria-activedescendant")).toBeTruthy();
});
it("removes the last chip on Backspace in an empty field", () => {
const onChange = vi.fn();
wrap(
<Combobox
label="Genre"
options={OPTIONS}
selected={["Drama", "Action"]}
onChange={onChange}
/>,
);
fireEvent.keyDown(screen.getByRole("combobox"), { key: "Backspace" });
expect(onChange).toHaveBeenCalledWith(["Drama"]); // last chip dropped
});
it("caps the rendered rows for a large vocabulary and shows a refine hint", () => {
// A studio-sized list (hundreds) must not render every row — cap at 50 + a "N more" footer.
const many = Array.from({ length: 60 }, (_, i) => ({ value: `Studio ${i}`, count: 60 - i }));
wrap(<Combobox label="Studio" options={many} selected={[]} onChange={vi.fn()} />);
fireEvent.focus(screen.getByRole("combobox"));
expect(screen.getAllByRole("option", { hidden: true }).length).toBe(50);
expect(screen.getByText("plex.filter.moreRefine")).toBeTruthy(); // 10 hidden behind search
});
it("Enter commits the pointer-highlighted option, not the arrow-active one", () => {
// The visible highlight follows the pointer (hoverIndex); Enter must commit THAT row, not the bare
// keyboard activeIndex — otherwise arrow-then-hover-then-Enter toggles a different option than shown.
const onChange = vi.fn();
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={onChange} />);
const input = screen.getByRole("combobox");
input.focus();
fireEvent.focus(input);
fireEvent.keyDown(input, { key: "ArrowDown" }); // activeIndex → 0 (Drama)
const action = screen
.getAllByRole("option", { hidden: true })
.find((el) => el.textContent?.includes("Action"))!;
fireEvent.mouseMove(action); // hoverIndex → Action (the visibly highlighted row)
fireEvent.keyDown(input, { key: "Enter" });
expect(onChange).toHaveBeenCalledWith(["Action"]); // the highlighted (hovered) row, not Drama
});
it("Enter commits a hover-highlighted option even with no prior arrow key", () => {
const onChange = vi.fn();
wrap(<Combobox label="Genre" options={OPTIONS} selected={[]} onChange={onChange} />);
const input = screen.getByRole("combobox");
input.focus();
fireEvent.focus(input);
const comedy = screen
.getAllByRole("option", { hidden: true })
.find((el) => el.textContent?.includes("Comedy"))!;
fireEvent.mouseMove(comedy); // highlight via hover, activeIndex still null
fireEvent.keyDown(input, { key: "Enter" });
expect(onChange).toHaveBeenCalledWith(["Comedy"]); // no longer a silent no-op
});
it("async mode: controlled query + onQueryChange/onOpenChange, no client filter, selectedLabels chip", () => {
const onQueryChange = vi.fn();
const onOpenChange = vi.fn();
const onChange = vi.fn();
const serverOpts = [{ value: "id1", label: "Harry Potter", count: 8 }];
wrap(
<Combobox
label="Collection"
multiple={false}
clientFilter={false}
options={serverOpts}
selected={["stored-id"]}
selectedLabels={{ "stored-id": "Stored Collection" }}
query="har"
onQueryChange={onQueryChange}
onOpenChange={onOpenChange}
onChange={onChange}
/>,
);
// The selected value's option isn't in the (server-narrowed) list, so its chip uses selectedLabels.
expect(screen.getByText("Stored Collection")).toBeTruthy();
const input = screen.getByRole("combobox");
fireEvent.focus(input);
expect(onOpenChange).toHaveBeenCalledWith(true);
fireEvent.change(input, { target: { value: "harr" } });
expect(onQueryChange).toHaveBeenCalledWith("harr"); // controlled query propagates out
// clientFilter=false: the single server option renders verbatim (not re-filtered against "har").
const opts = screen.getAllByRole("option", { hidden: true });
expect(opts.length).toBe(1);
expect(opts[0]?.textContent).toContain("Harry Potter");
fireEvent.click(screen.getByText("Harry Potter"));
expect(onChange).toHaveBeenCalledWith(["id1"]); // single-select replaces
});
it("shows the any/all match footer only at 2+ selections and reports a mode change", () => {
// The per-category any/all toggle: meaningless below 2 selections (any === all), so it appears
// only at >= 2, and clicking 'all' reports the mode change out.
const onMatchModeChange = vi.fn();
const props = {
label: "Topic",
options: OPTIONS,
onChange: vi.fn(),
matchMode: "or" as const,
onMatchModeChange,
};
// One selection: no footer.
const { rerender } = wrap(<Combobox {...props} selected={["Drama"]} />);
fireEvent.focus(screen.getByRole("combobox"));
expect(screen.queryByText("sidebar.all")).toBeNull();
// Two selections: footer appears with an any/all segment.
rerender(<FloatingTree>{<Combobox {...props} selected={["Drama", "Action"]} />}</FloatingTree>);
fireEvent.focus(screen.getByRole("combobox"));
const all = screen.getByText("sidebar.all");
expect(all).toBeTruthy();
fireEvent.click(all);
expect(onMatchModeChange).toHaveBeenCalledWith("and");
});
it("single-select: picking replaces the value and closes the list", () => {
const onChange = vi.fn();
wrap(
<Combobox
label="Collection"
multiple={false}
options={OPTIONS}
selected={["Drama"]}
onChange={onChange}
/>,
);
fireEvent.focus(screen.getByRole("combobox"));
fireEvent.click(screen.getByText("Action"));
expect(onChange).toHaveBeenCalledWith(["Action"]); // replaced, not appended
expect(screen.getByRole("combobox").getAttribute("aria-expanded")).toBe("false"); // closed
});
});
+325
View File
@@ -0,0 +1,325 @@
import { useCallback, useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { useTranslation } from "react-i18next";
import { Check, ChevronDown, X } from "lucide-react";
import { useAnchoredPopover, PopoverSurface, PopoverItem } from "./AnchoredPopover";
import { useScrollFade } from "../lib/useScrollFade";
// The universal combobox: a react-select / Overseerr-style "combi input" — selected values are
// removable chips that live INSIDE a text field you type into, and a checklist of options opens below.
// It is the ONE control for "pick value(s) from a (possibly large, possibly async) vocabulary": the E5
// Plex filters (genre / age-rating / studio, multi-select) and the collection picker (single-select,
// server-searched) are both instances, and the header search typeahead can move onto it later.
//
// All the fiddly parts live in the AnchoredPopover `combobox` mode, NOT here: floating-ui positioning,
// the paint-before-position + transform-clobber flash guards, the topmost-only Escape layer, and the
// FULL combobox ARIA tree (input role=combobox + aria-expanded/controls/activedescendant, listbox
// surface, option ids, virtual ↑/↓ arrow-nav that keeps the caret in the field). This component only
// adds the field chrome (chips + input + chevron), the query/selection state, and the big-vocabulary
// render cap. That is what makes it accessible where the old hand-rolled MultiSelectChips was not.
export interface ComboboxOption {
value: string;
/** Display text; defaults to `value`. */
label?: string;
/** A facet count shown at the row's end. */
count?: number;
disabled?: boolean;
}
export interface ComboboxProps {
/** Accessible name for the input (the section heading usually names it visibly too). */
label: string;
options: readonly ComboboxOption[];
/** Selected values. Single-select carries 0 or 1. */
selected: string[];
onChange: (next: string[]) => void;
/** Several values at once (chips accumulate) vs one (picking replaces + closes). Default true. */
multiple?: boolean;
/** Notified when the list opens/closes — a server-searched caller gates its fetch on `open`. */
onOpenChange?: (open: boolean) => void;
placeholder?: string;
/** Controlled query text — pass with `onQueryChange` for a server-searched (async) vocabulary; omit
* to let the combobox own the query state and filter `options` on the client. */
query?: string;
onQueryChange?: (q: string) => void;
/** Filter `options` by the query on the client (default). Set false when the caller already narrows
* the list server-side (then `options` arrives pre-filtered). */
clientFilter?: boolean;
loading?: boolean;
/** Shown when there are no options to list. */
emptyText?: string;
/** Cap on rendered rows for a huge vocabulary; the rest hide behind the search. Default 50. */
maxVisible?: number;
/** Custom row body (icon + label + count …); defaults to a checkbox + label + count. */
renderOption?: (option: ComboboxOption, isSelected: boolean) => ReactNode;
/** Chip/label text for a selected value whose option is not in the current (e.g. server-narrowed)
* list — the collection picker passes the stored title here so a reloaded filter still names itself. */
selectedLabels?: Record<string, string>;
/** Optional "any/all" match segment in the popover footer (like SortControl's direction segment):
* "or" = a match on ANY selected value, "and" = must match ALL. Provide BOTH to show it; it appears
* only once 2+ values are selected (below that, any and all are the same). */
matchMode?: "or" | "and";
onMatchModeChange?: (mode: "or" | "and") => void;
}
export default function Combobox({
label,
options,
selected,
onChange,
multiple = true,
onOpenChange,
placeholder,
query: queryProp,
onQueryChange,
clientFilter = true,
loading = false,
emptyText,
maxVisible = 50,
renderOption,
selectedLabels,
matchMode,
onMatchModeChange,
}: ComboboxProps) {
const { t } = useTranslation();
const [open, setOpen] = useState(false);
// The query is controlled when the caller drives it (async search), else owned here.
const [queryState, setQueryState] = useState("");
const query = queryProp ?? queryState;
const setQuery = (q: string) => {
if (queryProp === undefined) setQueryState(q);
onQueryChange?.(q);
};
const inputRef = useRef<HTMLInputElement | null>(null);
const fade = useScrollFade();
// Remember each value's display label as options flow through, so a selected chip can name itself
// even after the option leaves the query-narrowed list (async single-select). Caller-supplied
// `selectedLabels` win (they survive a page reload where nothing has been fetched yet).
const labelCache = useRef<Record<string, string>>({});
useEffect(() => {
for (const o of options) if (o.label != null) labelCache.current[o.value] = o.label;
}, [options]);
const labelFor = (v: string) => selectedLabels?.[v] ?? labelCache.current[v] ?? v;
const setOpenAnd = (next: boolean) => {
setOpen(next);
onOpenChange?.(next);
};
const pop = useAnchoredPopover({ open, onOpenChange: setOpenAnd, mode: "combobox" });
// The input is the combobox reference for floating-ui AND our own focus target (the field's
// click-to-focus handler, and commit() re-focusing after a pick). A merged callback ref feeds the
// node to BOTH — without it `inputRef` stays null, so the field's onMouseDown guard (`e.target !==
// inputRef.current`) is always true and preventDefault() cancels the input's own click-focus, making
// the field impossible to type into with the mouse. Memoised so floating-ui's ref isn't re-attached
// every render.
const setInputRef = useCallback(
(node: HTMLInputElement | null) => {
pop.refs.setReference(node);
inputRef.current = node;
},
[pop.refs],
);
// `highlightIndex` = the row the user actually SEES highlighted (pointer hover, else keyboard); Enter
// commits THAT, not the bare keyboard `activeIndex` — otherwise arrow-then-hover-then-Enter would toggle
// a different option than the highlighted one, and a hover with no prior arrow (activeIndex still null)
// would make Enter a no-op over a highlighted row.
const { activeIndex, highlightIndex } = pop;
// Clear the typed query when the list closes.
useEffect(() => {
if (!open) setQuery("");
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [open]);
const filtered = useMemo(() => {
if (!clientFilter) return options;
const q = query.trim().toLowerCase();
if (!q) return options;
return options.filter((o) => (o.label ?? o.value).toLowerCase().includes(q));
}, [options, query, clientFilter]);
// Options arrive count-desc (facet order), so the capped default view is the most-common values; a
// footer flags how many more are reachable through the search.
const visible = filtered.slice(0, maxVisible);
const hiddenCount = filtered.length - visible.length;
const commit = (value: string) => {
if (multiple) {
onChange(
selected.includes(value) ? selected.filter((x) => x !== value) : [...selected, value],
);
// Keep the field focused so the user can keep picking / typing; the list stays open.
inputRef.current?.focus();
} else {
// Single-select is final: close and DON'T re-focus — a refocus fires onFocus, which would
// immediately re-open the list (now that the merged ref makes focus() actually work).
onChange([value]);
setOpenAnd(false);
}
};
return (
<div>
{/* The field: removable chips + a free-text input + a chevron, styled as one input box. The whole
box is the POSITION anchor (the panel matches its width); the input is the combobox reference. */}
<div
ref={pop.refs.setPositionReference}
onMouseDown={(e) => {
// A press anywhere but the input (padding / a chip gap) focuses the field and opens the list —
// focus drives "open", so a caret click never toggles it shut.
if (e.target !== inputRef.current) {
e.preventDefault();
inputRef.current?.focus();
setOpenAnd(true);
}
}}
className="glass-card flex min-h-9 w-full cursor-text flex-wrap items-center gap-1 rounded-lg border border-border px-1.5 py-1 transition focus-within:border-accent focus-within:ring-2 focus-within:ring-accent/30"
>
{selected.map((v) => (
<span key={v} className="chip inline-flex items-center gap-1 px-2 py-0.5 text-sm">
<span className="max-w-36 truncate">{labelFor(v)}</span>
<button
type="button"
onMouseDown={(e) => {
// Keep the box's focus-the-field handler from firing when removing a chip.
e.stopPropagation();
e.preventDefault();
}}
onClick={() => onChange(selected.filter((x) => x !== v))}
aria-label={t("common.remove", { defaultValue: "Remove" })}
className="hover:opacity-70"
>
<X className="h-3 w-3" />
</button>
</span>
))}
<input
ref={setInputRef}
{...pop.getReferenceProps({
onChange: (e) => {
setQuery((e.target as HTMLInputElement).value);
setOpenAnd(true);
},
onFocus: () => setOpenAnd(true),
onKeyDown: (e) => {
const idx = highlightIndex ?? activeIndex;
if (e.key === "Enter" && open && idx != null && visible[idx]) {
e.preventDefault(); // select the highlighted option, don't submit a surrounding form
if (!visible[idx].disabled) commit(visible[idx].value);
} else if (e.key === "Backspace" && query === "" && selected.length > 0) {
// Backspace on an empty field removes the last chip (react-select behaviour).
onChange(selected.slice(0, -1));
}
},
})}
value={query}
aria-label={label}
placeholder={
selected.length === 0
? (placeholder ?? t("common.search", { defaultValue: "Search" }))
: ""
}
// min-w-8 (not -16): flex line-packing sizes the input at its MIN-width, so a wide 64px floor
// made a single chip + the reserved input + the chevron overflow a narrow rail field — wrapping
// the chevron to a second line while the (empty) input filled the first. A small floor keeps the
// chip + input + chevron on one line; flex-1 still grows the input to fill whatever room is left.
className="min-w-8 flex-1 bg-transparent px-1 py-0.5 text-sm outline-hidden placeholder:text-muted/60"
/>
<ChevronDown className="pointer-events-none mr-0.5 h-3.5 w-3.5 shrink-0 text-muted" />
</div>
<PopoverSurface
popover={pop}
ariaLabel={label}
className="glass-menu w-full rounded-xl p-1.5 shadow-lg animate-[popIn_0.16s_ease]"
>
{/* The scroll box owns the height cap AND the scroll-fade mask (ref+style on the same element). */}
<div ref={fade.ref} style={fade.style} className="max-h-64 overflow-y-auto no-scrollbar">
{loading && filtered.length === 0 ? (
<p className="px-2 py-1.5 text-xs text-muted">
{t("plex.loading", { defaultValue: "Loading…" })}
</p>
) : filtered.length === 0 ? (
<p className="px-2 py-1.5 text-xs text-muted">
{emptyText ?? t("common.noResults", { defaultValue: "No matches" })}
</p>
) : (
<>
{visible.map((o) => {
const on = selected.includes(o.value);
return (
<PopoverItem
key={o.value}
role="option"
selected={on}
disabled={o.disabled}
onSelect={() => commit(o.value)}
label={o.label ?? o.value}
className={`flex w-full items-center gap-2 rounded-md px-2 py-1.5 text-left text-sm transition ${
on ? "text-fg" : "text-muted"
}`}
>
{renderOption ? (
renderOption(o, on)
) : (
<>
<span
className={`flex h-3.5 w-3.5 shrink-0 items-center justify-center rounded border ${
on ? "border-accent bg-accent text-accent-fg" : "border-border"
}`}
>
{on && <Check className="h-2.5 w-2.5" />}
</span>
<span className="flex-1 truncate">{o.label ?? o.value}</span>
{o.count != null && (
<span className="shrink-0 text-2xs tabular-nums text-muted">
{o.count}
</span>
)}
</>
)}
</PopoverItem>
);
})}
{hiddenCount > 0 && (
<p className="px-2 py-1.5 text-2xs text-muted">
{t("plex.filter.moreRefine", {
count: hiddenCount,
defaultValue: "{{count}} more — refine with search",
})}
</p>
)}
</>
)}
</div>
{/* Any/all match segment (like SortControl's direction footer) — only meaningful, so only shown,
once 2+ values are picked. mousedown-preventDefault keeps the input focused so the list stays
open in the combobox's focus model. */}
{matchMode != null && onMatchModeChange && selected.length >= 2 && (
<div className="mt-1 flex items-center gap-2 border-t border-border px-1 pt-2">
<span className="text-2xs text-muted">{t("sidebar.match")}</span>
<div className="ml-auto flex gap-1">
{(["or", "and"] as const).map((m) => (
<button
key={m}
type="button"
aria-pressed={matchMode === m}
onMouseDown={(e) => e.preventDefault()}
onClick={() => onMatchModeChange(m)}
className={`rounded-md border px-2 py-0.5 text-2xs transition ${
matchMode === m
? "border-accent bg-accent/15 text-accent"
: "border-border text-muted hover:border-accent/50 hover:text-fg"
}`}
>
{t(m === "or" ? "sidebar.any" : "sidebar.all")}
</button>
))}
</div>
</div>
)}
</PopoverSurface>
</div>
);
}
@@ -0,0 +1,191 @@
import { useLayoutEffect, useState } from "react";
import { useTranslation } from "react-i18next";
import { CalendarDays, Check, ChevronDown } from "lucide-react";
import { PopoverItem, PopoverSurface, useAnchoredPopover } from "./AnchoredPopover";
import { type FeedFilters } from "../lib/api";
import { deriveDateKey, selectDateChip, type DatePreset } from "../lib/dateFilter";
// The feed rail's "Upload date" control: a single-select dropdown (the SortControl/FilterSelect pill,
// but full-width for the rail) over "Any time" + the day presets + a "Custom range". Picking Custom
// reveals the from/to fields in the popover footer (reveal-on-select) and the trigger then reads the
// chosen range; the presets and Any time close on pick. Selection logic is the pure lib/dateFilter so
// it stays testable without mounting this. The popover hosts free-text (date) inputs, so it opens as a
// `hostsInput` menu and the inputs stop non-Escape keys from reaching the list navigation.
// yyyy-mm-dd -> a local Date at midnight (split so it isn't parsed as UTC, which would shift a day).
function parseDay(s: string): Date | null {
const m = /^(\d{4})-(\d{2})-(\d{2})$/.exec(s);
if (!m) return null;
return new Date(Number(m[1]), Number(m[2]) - 1, Number(m[3]));
}
export default function DateFilterSelect({
filters,
setFilters,
presets,
}: {
filters: FeedFilters;
setFilters: (f: FeedFilters) => void;
presets: readonly DatePreset[];
}) {
const { t, i18n } = useTranslation();
const [open, setOpen] = useState(false);
const [customOpen, setCustomOpen] = useState<boolean>(!!(filters.dateFrom || filters.dateTo));
// If the range is cleared elsewhere (chip-row Clear all / removing the date chip) while this popover
// is closed, drop the custom reveal so the trigger reverts to "Any time" instead of lingering on
// "Custom range" with empty fields. Guarded on !open so it never closes a freshly-picked Custom the
// user hasn't filled in yet — that stays open until they close it (handled by onOpenChange below).
// useLayoutEffect (not useEffect) so the reset commits BEFORE paint — otherwise the trigger flashes
// one stale "Custom range" frame between the external clear and this reset.
useLayoutEffect(() => {
if (!open && customOpen && !filters.dateFrom && !filters.dateTo) setCustomOpen(false);
}, [open, customOpen, filters.dateFrom, filters.dateTo]);
const key = deriveDateKey(filters, customOpen, presets);
const fmt = (s: string) => {
const d = parseDay(s);
return d
? new Intl.DateTimeFormat(i18n.language, { month: "short", day: "numeric" }).format(d)
: s;
};
const rangeSummary = () => {
const { dateFrom: f, dateTo: to } = filters;
if (f && to) return `${fmt(f)} – ${fmt(to)}`;
if (f) return t("sidebar.date.fromShort", { date: fmt(f) });
if (to) return t("sidebar.date.untilShort", { date: fmt(to) });
return t("sidebar.date.customRange");
};
const summary =
key === ""
? t("sidebar.date.anyTime")
: key === "custom"
? rangeSummary()
: t("sidebar.datePreset." + presets.find((p) => String(p.days) === key)!.key);
const options = [
{ value: "any", label: t("sidebar.date.anyTime") },
...presets.map((p) => ({ value: String(p.days), label: t("sidebar.datePreset." + p.key) })),
{ value: "custom", label: t("sidebar.date.customRange") },
];
const pop = useAnchoredPopover({
open,
onOpenChange: (next) => {
setOpen(next);
// Opened Custom but committed no range → drop the reveal on close so it doesn't stick.
if (!next && customOpen && !filters.dateFrom && !filters.dateTo) setCustomOpen(false);
},
mode: "menu",
placement: "bottom-start",
hostsInput: true,
});
const pick = (value: string) => {
const change = selectDateChip(value, filters);
setCustomOpen(change.customOpen);
setFilters(change.filters);
if (value !== "custom") setOpen(false); // Custom stays open to reveal the range fields
};
const dateInputCls =
"bg-surface border border-border rounded-md px-2 py-1 text-xs focus:outline-hidden focus:ring-2 focus:ring-accent/40";
return (
<>
<button
type="button"
ref={pop.refs.setReference}
{...pop.getReferenceProps()}
aria-label={`${t("sidebar.datePresetLabel")}: ${summary}`}
className={`glass-card glass-hover flex w-full items-center gap-1.5 rounded-lg border px-2.5 py-1.5 text-sm transition ${
open ? "border-accent" : "border-border"
}`}
>
<CalendarDays className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden="true" />
<span className="flex-1 truncate text-left text-fg">{summary}</span>
<ChevronDown className="h-3.5 w-3.5 shrink-0 text-muted" aria-hidden="true" />
</button>
<PopoverSurface
popover={pop}
ariaLabel={t("sidebar.datePresetLabel")}
className="glass-menu min-w-52 rounded-xl p-1.5 shadow-lg animate-[popIn_0.16s_ease]"
>
<div role="group" aria-label={t("sidebar.datePresetLabel")}>
{options.map((o) => {
const on = o.value === key || (o.value === "any" && key === "");
return (
<PopoverItem
key={o.value}
role="menuitemradio"
checked={on}
onSelect={() => pick(o.value)}
label={o.label}
className={`flex w-full items-center gap-2 rounded-md px-2 py-1.5 text-left text-sm transition ${
on ? "text-fg" : "text-muted"
}`}
>
<Check className={`h-3.5 w-3.5 shrink-0 ${on ? "text-accent" : "opacity-0"}`} />
<span className="flex-1 truncate">{o.label}</span>
</PopoverItem>
);
})}
</div>
{/* Reveal-on-select: the from/to fields appear once Custom is the chosen option. The inputs
stop non-Escape keys so the date-segment arrows act on the field, not the menu's list nav. */}
{key === "custom" && (
<div className="mt-1 flex flex-col gap-1.5 border-t border-border px-1 pt-2">
<label className="flex items-center justify-between gap-2 text-xs">
<span className="text-muted">{t("sidebar.from")}</span>
<input
type="date"
value={filters.dateFrom ?? ""}
onKeyDown={(e) => {
if (e.key !== "Escape") e.stopPropagation();
}}
onChange={(e) =>
setFilters({
...filters,
dateFrom: e.target.value || undefined,
maxAgeDays: undefined,
})
}
className={dateInputCls}
/>
</label>
<label className="flex items-center justify-between gap-2 text-xs">
<span className="text-muted">{t("sidebar.to")}</span>
<input
type="date"
value={filters.dateTo ?? ""}
onKeyDown={(e) => {
if (e.key !== "Escape") e.stopPropagation();
}}
onChange={(e) =>
setFilters({
...filters,
dateTo: e.target.value || undefined,
maxAgeDays: undefined,
})
}
className={dateInputCls}
/>
</label>
{(filters.dateFrom || filters.dateTo) && (
<button
type="button"
onClick={() => setFilters({ ...filters, dateFrom: undefined, dateTo: undefined })}
className="text-2xs text-muted hover:text-accent self-end"
>
{t("sidebar.clearDates")}
</button>
)}
</div>
)}
</PopoverSurface>
</>
);
}
+5 -726
View File
@@ -1,14 +1,11 @@
import { lazy, Suspense, useEffect, useMemo, useState } from "react"; import { lazy, Suspense, useEffect, useState } from "react";
import { qk } from "../lib/queryKeys"; import { qk } from "../lib/queryKeys";
import { useTranslation } from "react-i18next"; import { useTranslation } from "react-i18next";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { import {
Ban, Ban,
CalendarClock,
ClipboardPaste, ClipboardPaste,
Copy,
Download, Download,
Link2,
Pause, Pause,
Play, Play,
Plus, Plus,
@@ -19,11 +16,9 @@ import {
Pencil, Pencil,
Trash2, Trash2,
} from "lucide-react"; } from "lucide-react";
import clsx from "clsx";
import Tabs from "./Tabs"; import Tabs from "./Tabs";
import { PageToolbar } from "./PageShell"; import { PageToolbar } from "./PageShell";
import { LoadingState, StateMessage } from "./QueryState"; import { LoadingState, StateMessage } from "./QueryState";
import Modal from "./Modal";
// Lazy: these dialogs/editors open on demand, so they stay out of the Downloads page chunk until // Lazy: these dialogs/editors open on demand, so they stay out of the Downloads page chunk until
// used (the video editor in particular is heavy — filmstrip + scrubber). // used (the video editor in particular is heavy — filmstrip + scrubber).
const DownloadDialog = lazy(() => import("./DownloadDialog")); const DownloadDialog = lazy(() => import("./DownloadDialog"));
@@ -35,726 +30,10 @@ import { useLiveQuery } from "../lib/useLiveQuery";
import { useMe } from "../lib/useMe"; import { useMe } from "../lib/useMe";
import { api, type DownloadJob } from "../lib/api"; import { api, type DownloadJob } from "../lib/api";
import { invalidateDownloads } from "../lib/downloads"; import { invalidateDownloads } from "../lib/downloads";
import { notify } from "../lib/notifications";
import { copyText } from "../lib/clipboard";
import { formatBytes, formatDuration, formatEta, formatSpeed, relativeTime } from "../lib/format";
import { inputCls } from "./ui/form"; import { inputCls } from "./ui/form";
import { DownloadRow, IconBtn, UsageBar } from "./downloadRow";
const GiB = 1024 ** 3; import { MetaEditModal, ErrorModal } from "./downloadModals";
import { AdminSystem } from "./downloadAdmin";
const STATUS_CLS: Record<string, string> = {
queued: "text-muted",
running: "text-accent",
paused: "text-warn",
done: "text-success",
error: "text-danger",
canceled: "text-muted",
};
// Byte-progress phases (show a % bar) vs ffmpeg post-steps (no %, indeterminate pulse).
// "editing" (the phase-2 editor) reports a real % from ffmpeg's out_time, so it gets a bar.
const DOWNLOAD_PHASES = ["video", "audio", "editing"];
const PHASE_KEYS = [
"video",
"audio",
"editing",
"merging",
"audio_extract",
"thumbnail",
"sponsorblock",
"metadata",
"processing",
];
function StatusPill({ job }: { job: DownloadJob }) {
const { t } = useTranslation();
// While downloading, show the concrete phase (video / audio / merging) so the user isn't
// confused by the bar resetting between the separate video and audio streams.
if (job.status === "running" && job.phase && PHASE_KEYS.includes(job.phase)) {
return (
<span className="text-xs font-medium text-accent">{t(`downloads.phase.${job.phase}`)}</span>
);
}
const key = job.expired ? "expired" : job.status;
return (
<span className={clsx("text-xs font-medium", STATUS_CLS[job.status] ?? "text-muted")}>
{t(`downloads.status.${key}`)}
</span>
);
}
function Thumb({ job }: { job: DownloadJob }) {
// Prefer the source thumbnail; fall back to our generated poster for thumbnail-less sources.
const primary = job.asset?.thumbnail_url || job.poster_url;
// A remote i.ytimg.com thumbnail 404s once the source video is deleted/privated, leaving a
// broken image. On load failure fall back to our always-present self-hosted poster.
const [src, setSrc] = useState(primary);
useEffect(() => setSrc(primary), [primary]);
return (
<div className="w-28 aspect-video shrink-0 rounded-lg overflow-hidden bg-surface border border-border">
{src ? (
<img
src={src}
alt=""
loading="lazy"
className="w-full h-full object-cover"
onError={() => {
if (job.poster_url && src !== job.poster_url) setSrc(job.poster_url);
}}
/>
) : null}
</div>
);
}
function jobTitle(job: DownloadJob): string {
return job.display_name || job.asset?.title || job.source_ref;
}
// The channel to show under the title: a user override wins over the auto-extracted asset value.
// `url` is null for sources without a channel link (e.g. reddit) — then it renders as plain text.
function jobChannel(job: DownloadJob): { name: string; url: string | null } {
return {
name: job.display_uploader || job.asset?.uploader || job.source_ref,
url: job.display_uploader_url || job.asset?.uploader_url || null,
};
}
function IconBtn({
onClick,
title,
danger,
children,
}: {
onClick: () => void;
title: string;
danger?: boolean;
children: React.ReactNode;
}) {
return (
<button
onClick={onClick}
title={title}
className={clsx(
"p-1.5 rounded-md hover:bg-surface text-muted transition",
danger ? "hover:text-danger" : "hover:text-fg",
)}
>
{children}
</button>
);
}
// Compact display of a URL: drop the protocol + trailing slash so "youtube.com/watch?v=…" fits.
function displayUrl(url: string): string {
try {
const u = new URL(url);
return (u.host + u.pathname + u.search).replace(/\/$/, "");
} catch {
return url;
}
}
// "Downloaded from" reference: opens the original page in a new tab, or copies the link.
function SourceRef({ url }: { url: string }) {
const { t } = useTranslation();
const copy = async () => {
if (await copyText(url)) {
notify({ level: "success", message: t("downloads.source.copied") });
} else {
notify({ level: "error", message: t("downloads.source.copyFailed") });
}
};
return (
<div className="flex items-center gap-1 mt-1 text-xs text-muted min-w-0">
<Link2 className="w-3 h-3 shrink-0" />
<a
href={url}
target="_blank"
rel="noopener noreferrer"
title={`${t("downloads.source.open")} — ${url}`}
className="truncate hover:text-accent hover:underline"
>
{displayUrl(url)}
</a>
<button
onClick={copy}
aria-label={t("downloads.source.copy")}
title={t("downloads.source.copy")}
className="shrink-0 p-0.5 rounded-sm hover:bg-surface hover:text-fg transition"
>
<Copy className="w-3 h-3" />
</button>
</div>
);
}
// Channel under the title — a clickable link when a channel URL is known, plain text otherwise.
function ChannelLine({ job }: { job: DownloadJob }) {
const { name, url } = jobChannel(job);
if (url) {
return (
<div className="text-xs mt-0.5 truncate">
<a
href={url}
target="_blank"
rel="noopener noreferrer"
title={url}
className="text-muted hover:text-accent hover:underline"
>
{name}
</a>
</div>
);
}
return <div className="text-xs text-muted truncate mt-0.5">{name}</div>;
}
function DownloadRow({
job,
onShowError,
children,
}: {
job: DownloadJob;
onShowError?: (job: DownloadJob) => void;
children?: React.ReactNode;
}) {
const { t } = useTranslation();
const running = job.status === "running";
return (
<div className="flex gap-3 p-2.5 rounded-xl glass-card glass-hover transition">
<Thumb job={job} />
<div className="min-w-0 flex-1">
<div className="font-medium leading-snug line-clamp-1 flex items-center gap-1.5">
{job.job_kind === "edit" && (
<span className="inline-flex items-center gap-1 shrink-0 px-1.5 py-0.5 rounded-sm text-3xs font-semibold uppercase tracking-wide bg-accent/15 text-accent">
<Scissors className="w-3 h-3" /> {t("downloads.clipBadge")}
</span>
)}
<span className="truncate">{jobTitle(job)}</span>
</div>
<ChannelLine job={job} />
<div className="flex items-center gap-2 mt-1 text-xs">
<StatusPill job={job} />
{job.asset?.size_bytes ? (
<span className="text-muted">· {formatBytes(job.asset.size_bytes)}</span>
) : null}
{job.asset?.duration_s ? (
<span className="text-muted">· {formatDuration(job.asset.duration_s)}</span>
) : null}
{job.status === "done" && job.created_at ? (
<span
className="text-muted inline-flex items-center gap-1"
title={new Date(job.created_at).toLocaleString()}
>
· <CalendarClock className="w-3 h-3" /> {relativeTime(job.created_at)}
</span>
) : null}
{job.status === "error" ? (
<button
onClick={() => onShowError?.(job)}
className="text-danger hover:opacity-80 inline-flex items-center gap-1 truncate"
title={t("downloads.errors.fail.title")}
>
· {t(`downloads.errors.fail.cat.${job.error_code || "unknown"}.short`)}
<span className="text-muted underline">{t("downloads.errors.fail.detailsLink")}</span>
</button>
) : null}
</div>
{job.source_url && <SourceRef url={job.source_url} />}
{job.extra_links?.map((url) => (
<SourceRef key={url} url={url} />
))}
{running && (
<div className="mt-1.5">
{job.phase && !DOWNLOAD_PHASES.includes(job.phase) ? (
// ffmpeg post-steps aren't byte-progress — show an indeterminate pulse, no %.
<div className="h-1.5 rounded-full bg-surface overflow-hidden">
<div className="h-full w-full bg-accent/60 animate-pulse" />
</div>
) : (
<>
<div className="h-1.5 rounded-full bg-surface overflow-hidden">
<div
className="h-full bg-accent transition-all"
style={{ width: `${job.progress}%` }}
/>
</div>
<div className="text-2xs text-muted mt-0.5 flex gap-2">
<span>{job.progress}%</span>
{job.speed_bps ? <span>{formatSpeed(job.speed_bps)}</span> : null}
{job.eta_s ? <span>{formatEta(job.eta_s)}</span> : null}
</div>
</>
)}
</div>
)}
</div>
<div className="flex items-start gap-0.5 shrink-0">{children}</div>
</div>
);
}
// --- Rename + Share small modals ------------------------------------------------------------
// Edit a download's display metadata: title, channel (name + link), and extra reference links.
// All are display-only overrides — the on-disk file is never renamed.
function MetaEditModal({ job, onClose }: { job: DownloadJob; onClose: () => void }) {
const { t } = useTranslation();
const qc = useQueryClient();
const [name, setName] = useState(job.display_name || job.asset?.title || "");
const [channel, setChannel] = useState(job.display_uploader || job.asset?.uploader || "");
const [channelUrl, setChannelUrl] = useState(
job.display_uploader_url || job.asset?.uploader_url || "",
);
// Editable extra-link rows; keep one blank row so there's always somewhere to type.
const [links, setLinks] = useState<string[]>(() => [...(job.extra_links ?? []), ""]);
const setLink = (i: number, v: string) => setLinks((ls) => ls.map((l, j) => (j === i ? v : l)));
const addLink = () => setLinks((ls) => [...ls, ""]);
const removeLink = (i: number) => setLinks((ls) => ls.filter((_, j) => j !== i));
const save = useMutation({
mutationFn: () =>
api.updateDownloadMeta(job.id, {
display_name: name.trim(),
display_uploader: channel.trim(),
display_uploader_url: channelUrl.trim(),
extra_links: links.map((l) => l.trim()).filter(Boolean),
}),
onSuccess: () => {
qc.invalidateQueries({ queryKey: qk.downloads() });
onClose();
},
});
return (
<Modal title={t("downloads.edit.title")} onClose={onClose}>
<div className="space-y-3">
<label className="block">
<span className="block text-sm font-medium mb-1">{t("downloads.edit.name")}</span>
<input
value={name}
onChange={(e) => setName(e.target.value)}
className={inputCls}
autoFocus
/>
</label>
<label className="block">
<span className="block text-sm font-medium mb-1">{t("downloads.edit.channel")}</span>
<input
value={channel}
onChange={(e) => setChannel(e.target.value)}
className={inputCls}
/>
</label>
<label className="block">
<span className="block text-sm font-medium mb-1">{t("downloads.edit.channelUrl")}</span>
<input
value={channelUrl}
onChange={(e) => setChannelUrl(e.target.value)}
placeholder="https://…"
className={inputCls}
/>
</label>
<div>
<span className="block text-sm font-medium mb-1">{t("downloads.edit.extraLinks")}</span>
<div className="space-y-2">
{links.map((l, i) => (
<div key={i} className="flex gap-2">
<input
value={l}
onChange={(e) => setLink(i, e.target.value)}
placeholder="https://…"
className={inputCls}
/>
<IconBtn
onClick={() => removeLink(i)}
title={t("downloads.edit.removeLink")}
danger
>
<Trash2 className="w-4 h-4" />
</IconBtn>
</div>
))}
</div>
<button
onClick={addLink}
className="mt-2 inline-flex items-center gap-1.5 text-sm text-muted hover:text-fg transition"
>
<Plus className="w-4 h-4" /> {t("downloads.edit.addLink")}
</button>
</div>
<p className="text-xs text-muted">{t("downloads.edit.hint")}</p>
<div className="flex justify-end">
<button
onClick={() => save.mutate()}
disabled={!name.trim() || save.isPending}
className="px-3 py-1.5 rounded-lg text-sm font-medium btn-primary disabled:opacity-50 transition"
>
{t("downloads.edit.save")}
</button>
</div>
</div>
</Modal>
);
}
// --- Error detail modal ---------------------------------------------------------------------
// Transient failures worth an in-modal Retry button; the rest are permanent (private/geo/DRM/…),
// where a retry can't help — the queue row still keeps its own retry control regardless.
const RETRYABLE_ERRORS = ["postprocess", "network", "unknown"];
// A readable "why did this fail" window for a failed job: a category headline + plain explanation,
// the raw yt-dlp/ffmpeg text behind a disclosure, and a Retry when the failure may be transient.
function ErrorModal({
job,
onRetry,
onClose,
}: {
job: DownloadJob;
onRetry: () => void;
onClose: () => void;
}) {
const { t } = useTranslation();
const code = job.error_code || "unknown";
const [showRaw, setShowRaw] = useState(false);
const retryable = RETRYABLE_ERRORS.includes(code);
return (
<Modal title={t("downloads.errors.fail.title")} onClose={onClose}>
<div className="space-y-3">
<div className="font-medium">{t(`downloads.errors.fail.cat.${code}.short`)}</div>
<p className="text-sm text-muted">{t(`downloads.errors.fail.cat.${code}.body`)}</p>
{job.error && (
<div>
<button
onClick={() => setShowRaw((v) => !v)}
className="text-xs text-muted hover:text-fg transition"
>
{t("downloads.errors.fail.detailsToggle")}
</button>
{showRaw && (
<pre className="mt-1.5 text-xs bg-surface rounded-lg p-2.5 overflow-x-auto whitespace-pre-wrap wrap-break-word max-h-40 text-muted">
{job.error}
</pre>
)}
</div>
)}
<div className="flex justify-end gap-2 pt-1">
{retryable && (
<button
onClick={() => {
onRetry();
onClose();
}}
className="px-3 py-1.5 rounded-lg text-sm font-medium btn-primary transition"
>
{t("downloads.actions.retry")}
</button>
)}
<button
onClick={onClose}
className="px-3 py-1.5 rounded-lg text-sm text-muted hover:text-fg hover:bg-surface transition"
>
{t("common.close")}
</button>
</div>
</div>
</Modal>
);
}
// --- Usage bar ------------------------------------------------------------------------------
function UsageBar() {
const { t } = useTranslation();
// Poll live (like the library list) so the footprint updates the moment the worker finishes an
// edit/download — enqueue only creates a queued job (0 bytes), the size lands asynchronously, so
// a one-shot fetch at enqueue time would show a stale 0 B until a manual reload.
const usage = useLiveQuery(qk.downloadUsage(), api.downloadUsage, { intervalMs: 2000 });
const u = usage.data;
if (!u) return null;
const pct =
u.unlimited || !u.max_bytes ? 0 : Math.min(100, (u.footprint_bytes / u.max_bytes) * 100);
return (
<div className="rounded-xl glass-card p-3 mb-4">
<div className="flex justify-between text-sm mb-1.5">
<span className="font-medium">{t("downloads.usage.title")}</span>
<span className="text-muted">
{formatBytes(u.footprint_bytes)}
{u.unlimited ? "" : ` / ${formatBytes(u.max_bytes)}`} ·{" "}
{u.unlimited
? t("downloads.usage.unlimited")
: t("downloads.usage.items", { used: u.active_jobs, max: u.max_jobs })}
</span>
</div>
{!u.unlimited && (
<div className="h-2 rounded-full bg-surface overflow-hidden">
<div
className={clsx("h-full transition-all", pct > 90 ? "bg-danger" : "bg-accent")}
style={{ width: `${pct}%` }}
/>
</div>
)}
</div>
);
}
// --- Admin system tab -----------------------------------------------------------------------
function QuotaModal({
userId,
email,
onClose,
}: {
userId: number;
email: string;
onClose: () => void;
}) {
const { t } = useTranslation();
const qc = useQueryClient();
const q = useQuery({
queryKey: qk.dlQuota(userId),
queryFn: () => api.adminGetDownloadQuota(userId),
});
const [form, setForm] = useState<{
gb: number;
jobs: number;
conc: number;
unlimited: boolean;
} | null>(null);
const cur = q.data;
const state =
form ??
(cur
? {
gb: Math.round((cur.max_bytes / GiB) * 10) / 10,
jobs: cur.max_jobs,
conc: cur.max_concurrent,
unlimited: cur.unlimited,
}
: null);
const done = () => {
qc.invalidateQueries({ queryKey: qk.dlQuota(userId) });
qc.invalidateQueries({ queryKey: qk.adminStorage() });
onClose();
};
const save = useMutation({
mutationFn: () =>
api.adminSetDownloadQuota(userId, {
max_bytes: Math.round(state!.gb * GiB),
max_jobs: state!.jobs,
max_concurrent: state!.conc,
unlimited: state!.unlimited,
}),
onSuccess: done,
});
const reset = useMutation({
mutationFn: () => api.adminResetDownloadQuota(userId),
onSuccess: done,
});
if (!state) return null;
const upd = (p: Partial<typeof state>) => setForm({ ...state, ...p });
return (
<Modal title={t("downloads.admin.quotaTitle", { email })} onClose={onClose}>
<div className="space-y-3">
<div className="grid grid-cols-3 gap-3">
<label className="text-sm">
{t("downloads.admin.maxBytes")}
<input
type="number"
value={state.gb}
onChange={(e) => upd({ gb: Number(e.target.value) })}
className={inputCls}
disabled={state.unlimited}
/>
</label>
<label className="text-sm">
{t("downloads.admin.maxJobs")}
<input
type="number"
value={state.jobs}
onChange={(e) => upd({ jobs: Number(e.target.value) })}
className={inputCls}
/>
</label>
<label className="text-sm">
{t("downloads.admin.maxConcurrent")}
<input
type="number"
value={state.conc}
onChange={(e) => upd({ conc: Number(e.target.value) })}
className={inputCls}
/>
</label>
</div>
<label className="flex items-center gap-2 text-sm cursor-pointer">
<input
type="checkbox"
checked={state.unlimited}
onChange={(e) => upd({ unlimited: e.target.checked })}
/>
{t("downloads.admin.unlimited")}
</label>
<div className="flex justify-between pt-1">
<button
onClick={() => reset.mutate()}
className="px-3 py-1.5 rounded-lg text-sm text-muted hover:text-fg hover:bg-surface transition"
>
{t("downloads.admin.reset")}
</button>
<button
onClick={() => save.mutate()}
disabled={save.isPending}
className="px-3 py-1.5 rounded-lg text-sm font-medium btn-primary disabled:opacity-50 transition"
>
{t("downloads.admin.save")}
</button>
</div>
</div>
</Modal>
);
}
// Pick ANY user to set a download quota for — even one who hasn't downloaded anything yet (the
// footprint list below only shows users who already have files).
function QuotaUserPicker({ onPick }: { onPick: (u: { id: number; email: string }) => void }) {
const { t } = useTranslation();
const [q, setQ] = useState("");
const [open, setOpen] = useState(false);
const users = useQuery({ queryKey: qk.adminUsers(), queryFn: api.adminUsers });
const list = (users.data ?? []).filter((u) => !u.is_demo);
const filtered = useMemo(() => {
const s = q.trim().toLowerCase();
const base = s
? list.filter(
(u) =>
u.email.toLowerCase().includes(s) || (u.display_name ?? "").toLowerCase().includes(s),
)
: list;
return base.slice(0, 8);
}, [q, list]);
return (
<div className="relative w-full sm:w-72">
<input
value={q}
onChange={(e) => {
setQ(e.target.value);
setOpen(true);
}}
onFocus={() => setOpen(true)}
onBlur={() => setTimeout(() => setOpen(false), 150)}
placeholder={t("downloads.admin.quotaPickPlaceholder")}
className={inputCls}
/>
{open && filtered.length > 0 && (
<div className="absolute z-base mt-1 w-full rounded-lg glass-menu overflow-hidden">
{filtered.map((u) => (
<button
key={u.id}
onClick={() => {
onPick({ id: u.id, email: u.email });
setQ("");
setOpen(false);
}}
className="w-full text-left px-3 py-2 text-sm hover:bg-surface transition flex flex-col"
>
<span className="truncate">{u.display_name || u.email}</span>
{u.display_name && <span className="text-xs text-muted truncate">{u.email}</span>}
</button>
))}
</div>
)}
</div>
);
}
function AdminSystem() {
const { t } = useTranslation();
const storage = useLiveQuery(qk.adminStorage(), api.adminDownloadStorage, { intervalMs: 5000 });
const jobs = useLiveQuery(qk.adminDownloads(), api.adminDownloads, { intervalMs: 5000 });
const [quotaFor, setQuotaFor] = useState<{ id: number; email: string } | null>(null);
const s = storage.data;
return (
<div>
{s && (
<div className="grid grid-cols-2 sm:grid-cols-4 gap-3 mb-5">
{[
[t("downloads.admin.readyFiles"), String(s.ready_files)],
[t("downloads.admin.totalSize"), formatBytes(s.total_bytes)],
[
t("downloads.admin.cap"),
s.total_cap_bytes ? formatBytes(s.total_cap_bytes) : t("downloads.admin.noCap"),
],
].map(([label, val]) => (
<div key={label} className="rounded-xl glass-card p-3">
<div className="text-xs text-muted">{label}</div>
<div className="text-lg font-semibold mt-0.5">{val}</div>
</div>
))}
</div>
)}
{s && (
<div className="mb-6">
<div className="flex items-center justify-between gap-3 mb-2 flex-wrap">
<div className="text-sm font-medium">{t("downloads.admin.perUser")}</div>
{/* Set a quota for any user, regardless of whether they've downloaded anything. */}
<QuotaUserPicker onPick={setQuotaFor} />
</div>
<div className="space-y-1">
{s.per_user.map((u) => (
<div
key={u.user_id}
className="flex items-center gap-2 text-sm px-3 py-1.5 rounded-lg bg-card/40"
>
<span className="flex-1 truncate">{u.email}</span>
<span className="text-muted">{formatBytes(u.footprint_bytes)}</span>
<button
onClick={() => setQuotaFor({ id: u.user_id, email: u.email ?? "" })}
className="p-1 rounded-sm hover:bg-surface text-muted hover:text-fg"
title={t("downloads.admin.editQuota")}
>
<Settings2 className="w-4 h-4" />
</button>
</div>
))}
{s.per_user.length === 0 && (
<div className="text-xs text-muted px-3 py-2">{t("downloads.admin.noFootprint")}</div>
)}
</div>
</div>
)}
<div className="text-sm font-medium mb-2">{t("downloads.tabs.system")}</div>
<div className="space-y-1.5">
{(jobs.data ?? []).map((j) => (
<div
key={j.id}
className="flex items-center gap-3 text-sm px-3 py-1.5 rounded-lg bg-card/40"
>
<span className="flex-1 truncate">{jobTitle(j)}</span>
<span className="text-muted truncate w-40">{j.user_email}</span>
<StatusPill job={j} />
<span className="text-muted w-16 text-right">
{j.asset?.size_bytes ? formatBytes(j.asset.size_bytes) : ""}
</span>
</div>
))}
{jobs.data && jobs.data.length === 0 && (
<div className="text-sm text-muted py-6 text-center">{t("downloads.empty.admin")}</div>
)}
</div>
{quotaFor && (
<QuotaModal userId={quotaFor.id} email={quotaFor.email} onClose={() => setQuotaFor(null)} />
)}
</div>
);
}
// --- Main -----------------------------------------------------------------------------------
export default function DownloadCenter() { export default function DownloadCenter() {
const me = useMe(); const me = useMe();
@@ -856,7 +135,7 @@ export default function DownloadCenter() {
<> <>
{/* Page title + subtitle + tabs are fixed chrome; the tab content scrolls under them. */} {/* Page title + subtitle + tabs are fixed chrome; the tab content scrolls under them. */}
<PageToolbar> <PageToolbar>
<div className="px-4 pt-3 max-w-4xl w-full mx-auto"> <div className="px-4 pt-2 max-w-4xl w-full mx-auto">
<div className="flex items-center justify-between gap-3 mb-1"> <div className="flex items-center justify-between gap-3 mb-1">
<h1 className="text-xl font-semibold">{t("downloads.page.title")}</h1> <h1 className="text-xl font-semibold">{t("downloads.page.title")}</h1>
<button <button
+137 -116
View File
@@ -13,6 +13,7 @@ import {
Grid3x3, Grid3x3,
Info, Info,
LayoutGrid, LayoutGrid,
Library,
List, List,
RefreshCw, RefreshCw,
Rows3, Rows3,
@@ -36,11 +37,13 @@ import { type SortOption } from "../lib/columnSort";
import { FEED_VIEWS, type FeedView } from "../lib/feedView"; import { FEED_VIEWS, type FeedView } from "../lib/feedView";
import { clearedFilters, useActiveFilters } from "../lib/useActiveFilters"; import { clearedFilters, useActiveFilters } from "../lib/useActiveFilters";
import ActiveFilterChips from "./ActiveFilterChips"; import ActiveFilterChips from "./ActiveFilterChips";
import SelectedViewChip from "./SelectedViewChip";
import { LoadingState, StateMessage } from "./QueryState"; import { LoadingState, StateMessage } from "./QueryState";
import { PageToolbar } from "./PageShell"; import { PageToolbar } from "./PageShell";
import { useWizard } from "./WizardProvider"; import { useWizard } from "./WizardProvider";
import SortControl from "./SortControl"; import SortControl from "./SortControl";
import FilterChipGroup, { filterChipClass } from "./FilterChipGroup"; import FilterSelect from "./FilterSelect";
import ItemCount from "./ItemCount";
import ViewSwitcher from "./ViewSwitcher"; import ViewSwitcher from "./ViewSwitcher";
import VirtualFeed from "./VirtualFeed"; import VirtualFeed from "./VirtualFeed";
// Lazy: the in-app YouTube player (IFrame API) loads only when a video is first opened. // Lazy: the in-app YouTube player (IFrame API) loads only when a video is first opened.
@@ -226,9 +229,11 @@ export default function Feed({
} }
}, [query.dataUpdatedAt]); }, [query.dataUpdatedAt]);
const countQuery = useQuery({ // One shared query with the sidebar (same key + filters -> React Query dedupes to a single request):
queryKey: qk.feedCount(queryFilters), // the feed reads `.count`, the sidebar reads `.facets` (PERF-13, replaces feedCount + facets).
queryFn: () => api.feedCount(queryFilters), const summaryQuery = useQuery({
queryKey: qk.feedSummary(queryFilters),
queryFn: () => api.feedSummary(queryFilters),
staleTime: 30_000, staleTime: 30_000,
enabled: !ytActive, enabled: !ytActive,
placeholderData: keepPreviousData, placeholderData: keepPreviousData,
@@ -252,7 +257,11 @@ export default function Feed({
api api
.setState(id, status) .setState(id, status)
.then(() => { .then(() => {
qc.invalidateQueries({ queryKey: qk.feed() }); // Mark stale WITHOUT refetching: the optimistic `overrides` layer + the matchesView filter
// already show the correct state (a hidden/watched card drops out immediately), so refetching
// every loaded infinite-query page here was pure waste — 10 pages x rapid marks = a refetch
// storm (PERF-2). The next filter change / remount refetches fresh from the server.
qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" });
if (status === "hidden") { if (status === "hidden") {
const v = loadedRef.current.find((x) => x.id === id); const v = loadedRef.current.find((x) => x.id === id);
notify({ notify({
@@ -309,7 +318,8 @@ export default function Feed({
setOverrides((o) => ({ ...o, [id]: "new" })); setOverrides((o) => ({ ...o, [id]: "new" }));
api api
.clearState(id) .clearState(id)
.then(() => qc.invalidateQueries({ queryKey: qk.feed() })) // Stale-mark only (see setState above): the optimistic override already reset the card.
.then(() => qc.invalidateQueries({ queryKey: qk.feed(), refetchType: "none" }))
.catch(() => .catch(() =>
setOverrides((o) => { setOverrides((o) => {
const next = { ...o }; const next = { ...o };
@@ -358,8 +368,7 @@ export default function Feed({
// pre-search cache. Drop them so the feed re-fetches fresh on return. // pre-search cache. Drop them so the feed re-fetches fresh on return.
const dropFeedCaches = () => { const dropFeedCaches = () => {
qc.removeQueries({ queryKey: qk.feed() }); qc.removeQueries({ queryKey: qk.feed() });
qc.removeQueries({ queryKey: qk.feedCount() }); qc.removeQueries({ queryKey: qk.feedSummary() });
qc.removeQueries({ queryKey: qk.facets() });
}; };
// --- Live YouTube search mode ------------------------------------------------------------- // --- Live YouTube search mode -------------------------------------------------------------
@@ -561,124 +570,109 @@ export default function Feed({
: SORT_DEFAULT_DIR[k as Exclude<SortKey, "shuffle" | "relevance">], : SORT_DEFAULT_DIR[k as Exclude<SortKey, "shuffle" | "relevance">],
})); }));
// Sort + view mode are tightly bound to the grid below, so they live pinned to the right of the
// applied-filter row (directly above the cards), not up in the filter toolbar. Extracted here so the
// same controls render on both the feed and channel-scoped rows.
const sortView = (
<>
<SortControl
options={sortOptions}
sort={{ key: sortKey, dir: sortDir }}
onChange={(next) => {
if (!next) return; // the feed is always sorted — no "no sort" option is offered
setFilters({
...filters,
sort: buildSort(next.key as SortKey, next.dir),
seed: next.key === "shuffle" ? rollSeed() : undefined,
});
}}
label={t("feed.sortLabel")}
ascLabel={t("feed.dirAsc")}
descLabel={t("feed.dirDesc")}
trailing={
sortKey === "shuffle" ? (
<button
onClick={() => setFilters({ ...filters, seed: rollSeed() })}
title={t("sidebar.reshuffle")}
aria-label={t("sidebar.reshuffle")}
className="shrink-0 p-1.5 rounded-lg border border-border bg-card text-fg hover:border-accent hover:text-accent active:translate-y-px transition"
>
<RefreshCw className="w-4 h-4" />
</button>
) : undefined
}
/>
<span className="mx-0.5 h-5 w-px bg-border" aria-hidden="true" />
<ViewSwitcher
value={view}
options={FEED_VIEWS.map((id) => ({
id,
label: t("feed.view." + id),
icon: VIEW_ICON[id],
}))}
onChange={setView}
label={t("feed.viewLabel")}
// The mode's name only earns its place once the window is wide.
labelClassName="hidden xl:inline"
/>
</>
);
const toolbar = ( const toolbar = (
<div className="pb-3"> <div className="pb-1.5">
<div className="flex items-center gap-1.5 gap-y-2 flex-wrap"> <div className="flex items-center gap-1.5 gap-y-2 flex-wrap">
<FilterChipGroup {/* The item count is the fixed leading element — a fixed-width pill so a changing number
ariaLabel={t("feed.showLabel")} never shoves the controls beside it (see ItemCount). */}
options={SHOW_IDS.map((id) => ({ <ItemCount count={summaryQuery.data?.count} />
value: id, {/* Watch state is single-select (always exactly one); content types are an independent
label: t("sidebar.show." + id), multi-select. Both compress to one labeled dropdown pill via FilterSelect. */}
testId: `feed-show-${id}`, <FilterSelect
}))} label={t("feed.showLabel")}
value={filters.show} testId="feed-show"
onChange={(show) => setFilters({ ...filters, show })} options={SHOW_IDS.map((id) => ({ value: id, label: t("sidebar.show." + id) }))}
selected={[filters.show]}
onChange={(next) => setFilters({ ...filters, show: next[0] as FeedFilters["show"] })}
/>
<FilterSelect
label={t("feed.contentLabel")}
testId="feed-content"
multiple
options={CONTENT.map((c) => ({ value: c.key, label: t(c.label) }))}
selected={CONTENT.filter((c) => filters[c.key]).map((c) => c.key)}
onChange={(next) => {
const patch = { ...filters };
for (const c of CONTENT) patch[c.key] = next.includes(c.key);
setFilters(patch);
}}
/> />
<span className="mx-1 h-5 w-px bg-border" aria-hidden="true" />
{/* Content-type toggles are MULTI-select (independent booleans) — same chip look via
filterChipClass, but not a single-select group. */}
<div className="flex flex-wrap gap-1.5" role="group" aria-label={t("feed.contentLabel")}>
{CONTENT.map((c) => {
const on = filters[c.key];
return (
<button
key={c.key}
data-testid={`feed-content-${c.key}`}
onClick={() => setFilters({ ...filters, [c.key]: !on })}
aria-pressed={on}
className={filterChipClass}
>
{t(c.label)}
</button>
);
})}
</div>
{/* The Source filter (organic / include-search / search-only) is a global-catalog concept — {/* The Source filter (organic / include-search / search-only) is a global-catalog concept —
it selects how videos ENTERED the library. On a channel-scoped view we already show all it selects how videos ENTERED the library. On a channel-scoped view we already show all
of this one channel's videos (librarySource is pinned to "all"), so the selector would of this one channel's videos (librarySource is pinned to "all"), so the selector would
only offer a near-empty, confusing slice — hide it here. */} only offer a near-empty, confusing slice — hide it here. */}
{!channelScoped && ( {!channelScoped && (
<> <FilterSelect
<span className="mx-1 h-5 w-px bg-border" aria-hidden="true" /> label={t("feed.source.label")}
<label className="inline-flex items-center gap-1.5 text-xs text-muted"> testId="feed-source"
<span>{t("feed.source.label")}</span> options={[
<select { value: "organic", label: t("feed.source.organic") },
value={filters.librarySource ?? "organic"} { value: "all", label: t("feed.source.all") },
onChange={(e) => { value: "search", label: t("feed.source.only") },
setFilters({ ]}
...filters, selected={[filters.librarySource ?? "organic"]}
librarySource: e.target.value as FeedFilters["librarySource"], onChange={(next) =>
})
}
title={t("feed.source.tip")}
className="bg-surface border border-border rounded-lg px-2 py-1 text-xs focus:outline-hidden focus:ring-2 focus:ring-accent/40"
>
<option value="organic">{t("feed.source.organic")}</option>
<option value="all">{t("feed.source.all")}</option>
<option value="search">{t("feed.source.only")}</option>
</select>
</label>
</>
)}
<span className="mx-1 h-5 w-px bg-border" aria-hidden="true" />
<span data-testid="feed-result-count" className="text-xs text-muted whitespace-nowrap">
{countQuery.data
? t("feed.videoCount", {
count: countQuery.data.count,
formattedCount: countQuery.data.count.toLocaleString(),
})
: " "}
</span>
<div className="ml-auto flex items-center gap-2">
<span className="text-xs text-muted">{t("feed.sortLabel")}</span>
<SortControl
options={sortOptions}
sort={{ key: sortKey, dir: sortDir }}
onChange={(next) => {
if (!next) return; // the feed is always sorted — no "no sort" option is offered
setFilters({ setFilters({
...filters, ...filters,
sort: buildSort(next.key as SortKey, next.dir), librarySource: next[0] as FeedFilters["librarySource"],
seed: next.key === "shuffle" ? rollSeed() : undefined, })
});
}}
label={t("feed.sortLabel")}
ascLabel={t("feed.dirAsc")}
descLabel={t("feed.dirDesc")}
trailing={
sortKey === "shuffle" ? (
<button
onClick={() => setFilters({ ...filters, seed: rollSeed() })}
title={t("sidebar.reshuffle")}
aria-label={t("sidebar.reshuffle")}
className="shrink-0 p-1.5 rounded-lg border border-border bg-card text-fg hover:border-accent hover:text-accent active:translate-y-px transition"
>
<RefreshCw className="w-4 h-4" />
</button>
) : undefined
} }
/> />
<span className="mx-0.5 h-5 w-px bg-border" aria-hidden="true" /> )}
<ViewSwitcher
value={view}
options={FEED_VIEWS.map((id) => ({
id,
label: t("feed.view." + id),
icon: VIEW_ICON[id],
}))}
onChange={setView}
label={t("feed.viewLabel")}
// This toolbar is packed — show/content chips, source, count, sort — so the mode's name
// only earns its place once the window is wide.
labelClassName="hidden xl:inline"
/>
</div>
</div> </div>
</div> </div>
); );
return ( return (
<div className={channelScoped ? "px-4 sm:px-6 pb-4" : "px-4 pb-4 pt-2"}> <div className={channelScoped ? "px-4 sm:px-6 pb-4" : "px-4 pb-4 pt-1"}>
{/* The toolbar (+ active-filter chips) is FIXED chrome: it portals into a PageToolbarSlot so {/* The toolbar (+ active-filter chips) is FIXED chrome: it portals into a PageToolbarSlot so
it stays put while the cards scroll (bug 4). On the feed page that slot is the shell's top it stays put while the cards scroll (bug 4). On the feed page that slot is the shell's top
band; on a channel page ChannelPage overrides the slot with its own sticky sub-header, so band; on a channel page ChannelPage overrides the slot with its own sticky sub-header, so
@@ -686,19 +680,46 @@ export default function Feed({
baseline (channel-scoped, whole-catalog, show-all), so each would read as a "filter you baseline (channel-scoped, whole-catalog, show-all), so each would read as a "filter you
applied" when it's just the page being itself. */} applied" when it's just the page being itself. */}
<PageToolbar> <PageToolbar>
<div className={channelScoped ? "px-4 sm:px-6" : "px-4 pt-4 pb-1"}> <div className={channelScoped ? "px-4 sm:px-6" : "px-4 pt-2 pb-1"}>
{toolbar} {toolbar}
{!channelScoped && ( {/* The controls row sits directly above the grid: the applied-filter chips flow from the left
<ActiveFilterChips (overflowing to a "+N"/count pill so they never crowd the controls), while Sort + View stay
filters={activeFilters} pinned to the right. Channel-scoped feeds show no chips (their baseline isn't a "filter you
onClearAll={() => setFilters(clearedFilters(filters))} applied"), but still get the right-pinned Sort/View. */}
/> <div className="flex items-center gap-2 pb-2">
)} {!channelScoped && (
<>
{/* The active personal view (if any) leads the applied-filter row. */}
<SelectedViewChip />
<ActiveFilterChips
filters={activeFilters}
onClearAll={() => setFilters(clearedFilters(filters))}
className="flex-1"
/>
</>
)}
<div className="ml-auto flex shrink-0 items-center gap-1.5">{sortView}</div>
</div>
</div> </div>
</PageToolbar> </PageToolbar>
{items.length === 0 ? ( {items.length === 0 ? (
<div className="py-16 text-center text-muted"> <div className="py-16 text-center text-muted">
<p>{t("feed.noMatches")}</p> <p>{t("feed.noMatches")}</p>
{/* A channel filter under the "Mine" scope returns nothing when that channel is hidden from
your subscriptions (or you don't subscribe to it) — its videos only live in the shared
Library. Point there rather than leaving a bare "no matches". */}
{filters.channelIds.length > 0 && filters.scope === "my" && (
<div className="mt-4">
<p className="text-sm">{t("feed.channelScopeHint")}</p>
<button
onClick={() => setFilters({ ...filters, scope: "all" })}
className="mt-3 inline-flex items-center gap-2 px-4 py-2 rounded-xl font-semibold btn-primary transition"
>
<Library className="w-4 h-4" />
{t("feed.viewInLibrary")}
</button>
</div>
)}
{filters.q.trim() && !isDemo && ( {filters.q.trim() && !isDemo && (
<button <button
onClick={() => onYtSearch(filters.q.trim())} onClick={() => onYtSearch(filters.q.trim())}
@@ -0,0 +1,45 @@
// @vitest-environment jsdom
//
// loadAccountFilters is the localStorage seed path for the feed filters. It must migrate a persisted
// legacy global `tagMode` (older builds wrote it) into the per-category `tagModes` — the same coercion
// the saved-view and URL paths do — so an upgrading user's topic any/all match isn't silently lost.
import { describe, it, expect, beforeEach } from "vitest";
import { loadAccountFilters } from "./FeedFiltersProvider";
import { accountKey, LS } from "../lib/storage";
const ID = 7;
beforeEach(() => localStorage.clear());
describe("loadAccountFilters — legacy tagMode migration on seed", () => {
it("migrates a persisted legacy tagMode:'and' from the last-session blob into topic", () => {
localStorage.setItem(accountKey(LS.filters, ID)!, JSON.stringify({ q: "hi", tagMode: "and" }));
const f = loadAccountFilters(ID);
expect(f.tagModes).toEqual({ language: "or", topic: "and", other: "or" });
expect(f.q).toBe("hi"); // other fields preserved through the merge
});
it("migrates a legacy tagMode:'and' from the default-view mirror when no last-session blob exists", () => {
localStorage.setItem(
accountKey(LS.defaultViewFilters, ID)!,
JSON.stringify({ tagMode: "and" }),
);
expect(loadAccountFilters(ID).tagModes.topic).toBe("and");
});
it("leaves tagModes at defaults when nothing is stored", () => {
expect(loadAccountFilters(ID).tagModes).toEqual({ language: "or", topic: "or", other: "or" });
});
it("preserves an already-migrated per-category tagModes blob", () => {
localStorage.setItem(
accountKey(LS.filters, ID)!,
JSON.stringify({ tagModes: { language: "and", topic: "or", other: "and" } }),
);
expect(loadAccountFilters(ID).tagModes).toEqual({
language: "and",
topic: "or",
other: "and",
});
});
});
+35 -10
View File
@@ -9,9 +9,17 @@ import {
} from "react"; } from "react";
import { qk } from "../lib/queryKeys"; import { qk } from "../lib/queryKeys";
import { useQuery } from "@tanstack/react-query"; import { useQuery } from "@tanstack/react-query";
import { api, getActiveAccount, type FeedFilters } from "../lib/api"; import {
import { hasFilterParams, paramsToFilters } from "../lib/urlState"; api,
import { accountKey, LS, readJSON, readMerged, writeJSON } from "../lib/storage"; getActiveAccount,
coerceTagModes,
DEFAULT_TAG_MODES,
type FeedFilters,
} from "../lib/api";
import { filterParamUpdates, hasFilterParams, paramsToFilters } from "../lib/urlState";
import { useUrlMirror } from "../lib/useUrlMirror";
import { accountKey, LS, readJSON, writeJSON } from "../lib/storage";
import { useNavigation } from "./NavigationProvider";
// Owns the GLOBAL feed-filter state (the feed's scope/tags/content/sort/show), lifted out of the App // Owns the GLOBAL feed-filter state (the feed's scope/tags/content/sort/show), lifted out of the App
// god component so consumers read it via a hook instead of prop-drilling `filters`/`setFilters` // god component so consumers read it via a hook instead of prop-drilling `filters`/`setFilters`
@@ -30,7 +38,7 @@ import { accountKey, LS, readJSON, readMerged, writeJSON } from "../lib/storage"
const DEFAULT_FILTERS: FeedFilters = { const DEFAULT_FILTERS: FeedFilters = {
tags: [], tags: [],
tagMode: "or", tagModes: DEFAULT_TAG_MODES,
channelIds: [], channelIds: [],
q: "", q: "",
sort: "newest", sort: "newest",
@@ -47,7 +55,7 @@ const FILTERS_KEY = LS.filters;
// default saved view wins over the last-session filters (SavedViewsWidget mirrors it here). When // default saved view wins over the last-session filters (SavedViewsWidget mirrors it here). When
// the account isn't known yet (first load, before the tab is pinned), start from defaults; the // the account isn't known yet (first load, before the tab is pinned), start from defaults; the
// post-login effect loads the real ones once the id arrives. // post-login effect loads the real ones once the id arrives.
function loadAccountFilters(id: number | null): FeedFilters { export function loadAccountFilters(id: number | null): FeedFilters {
// Your last-applied filters win, so a reload keeps whatever view you're on (a saved view you // Your last-applied filters win, so a reload keeps whatever view you're on (a saved view you
// picked, or manual tweaks) — setFilters persists them under LS.filters on every change. The // picked, or manual tweaks) — setFilters persists them under LS.filters on every change. The
// starred default view only SEEDS a fresh account that has never stored filters: it drives the // starred default view only SEEDS a fresh account that has never stored filters: it drives the
@@ -60,10 +68,19 @@ function loadAccountFilters(id: number | null): FeedFilters {
} catch { } catch {
/* localStorage may be unavailable */ /* localStorage may be unavailable */
} }
if (hasStored && fKey) return readMerged(fKey, DEFAULT_FILTERS); // Migrate a persisted legacy global `tagMode` into the per-category `tagModes` on every seed path,
// matching what the saved-view (asFilters) and URL (paramsToFilters) paths already do. coerceTagModes
// MUST run on the RAW stored blob, not the object merged over DEFAULT_FILTERS: the merge injects
// DEFAULT_TAG_MODES (all "or"), which coerceTagModes prefers over the legacy scalar and would mask it.
if (hasStored && fKey) {
const raw = readJSON<Partial<FeedFilters>>(fKey, {});
return { ...DEFAULT_FILTERS, ...raw, tagModes: coerceTagModes(raw) };
}
const dvKey = accountKey(LS.defaultViewFilters, id); const dvKey = accountKey(LS.defaultViewFilters, id);
const def = dvKey ? readJSON<FeedFilters | null>(dvKey, null) : null; const def = dvKey ? readJSON<Partial<FeedFilters> | null>(dvKey, null) : null;
if (def) return { ...DEFAULT_FILTERS, ...def }; if (def) {
return { ...DEFAULT_FILTERS, ...def, tagModes: coerceTagModes(def) };
}
return { ...DEFAULT_FILTERS }; return { ...DEFAULT_FILTERS };
} }
@@ -90,8 +107,8 @@ export function useFeedFiltersActions(): FeedFiltersActions {
export function FeedFiltersProvider({ children }: { children: ReactNode }) { export function FeedFiltersProvider({ children }: { children: ReactNode }) {
const [filters, setFiltersState] = useState<FeedFilters>(loadInitialFilters); const [filters, setFiltersState] = useState<FeedFilters>(loadInitialFilters);
// Whether this load arrived via a "Share view" link (captured once, before the URL is stripped) // Whether this load arrived with feed-filter params in the URL (a bookmark/shared link). Captured
// — such filters win over the account's stored view and are persisted to the account instead. // once at mount — such filters win over the account's stored view and are persisted to the account.
const [initialHadUrlFilters] = useState(() => const [initialHadUrlFilters] = useState(() =>
hasFilterParams(new URLSearchParams(window.location.search)), hasFilterParams(new URLSearchParams(window.location.search)),
); );
@@ -108,6 +125,14 @@ export function FeedFiltersProvider({ children }: { children: ReactNode }) {
if (key) writeJSON(key, next); if (key) writeJSON(key, next);
}, []); }, []);
// Mirror the feed filters into the address bar so every feed view is bookmarkable. Gated on
// page===feed: on other pages the filter keys are cleared from the URL (localStorage still holds
// them; they re-mirror on return to the feed). The channel detail page keeps its own isolated
// filters, so those never reach here. useUrlMirror preserves the nav params (?page/?chan/?v/?pl) the
// other providers own.
const { page } = useNavigation();
useUrlMirror(filterParamUpdates(page === "feed" ? filters : null));
// Load THIS account's own filters when the account first resolves or changes (login / switch / // Load THIS account's own filters when the account first resolves or changes (login / switch /
// add). Filters are per-account localStorage state, so another account's last view (incl. its // add). Filters are per-account localStorage state, so another account's last view (incl. its
// default saved view) must never carry over. A share link's filters (captured at mount) win and // default saved view) must never carry over. A share link's filters (captured at mount) win and
@@ -0,0 +1,39 @@
import { useTranslation } from "react-i18next";
import { Library, User } from "lucide-react";
import SegmentedToggle from "./SegmentedToggle";
import { useFeedFilters, useFeedFiltersActions } from "./FeedFiltersProvider";
// The feed's Mine/Library scope, relocated from the filter sidebar to sit just right of the header
// search box — mirroring PlexScopeToggle's placement, so the primary "what am I browsing" control is
// always in reach instead of buried in the (possibly collapsed) filter rail. Same `filters.scope`
// field, so behaviour is unchanged. The `pointer-events-auto`/`shrink-0` wrapper matches the other
// header pills: the header band is `pointer-events-none`, and the pill must not collapse in the flex row.
export default function FeedScopeToggle() {
const { t } = useTranslation();
const filters = useFeedFilters();
const { setFilters } = useFeedFiltersActions();
return (
<div className="pointer-events-auto shrink-0">
<SegmentedToggle
size="md"
ariaLabel={t("header.scope.label")}
options={[
{
value: "my",
label: t("header.scope.my"),
icon: <User className="w-3.5 h-3.5" />,
title: t("header.scope.myTip"),
},
{
value: "all",
label: t("header.scope.all"),
icon: <Library className="w-3.5 h-3.5" />,
title: t("header.scope.allTip"),
},
]}
value={filters.scope}
onChange={(scope) => setFilters({ ...filters, scope })}
/>
</div>
);
}
+11 -41
View File
@@ -1,26 +1,15 @@
import { import { createContext, useContext, useMemo, type ReactNode } from "react";
createContext,
useCallback,
useContext,
useEffect,
useMemo,
useState,
type ReactNode,
} from "react";
import { qk } from "../lib/queryKeys";
import { useQuery } from "@tanstack/react-query";
import { api } from "../lib/api";
import { parseFeedView, type FeedView } from "../lib/feedView"; import { parseFeedView, type FeedView } from "../lib/feedView";
import { LS, readAccount, writeAccount } from "../lib/storage"; import { LS } from "../lib/storage";
import { usePref } from "../lib/usePref";
// Owns the feed's view mode (the card/row/tile density switcher), lifted out of App so both the feed // Owns the feed's view mode (the card/row/tile density switcher), lifted out of App so both the feed
// page and the channel page read it via hooks instead of prop-drilling `view`/`setView` through App. // page and the channel page read it via hooks instead of prop-drilling `view`/`setView` through App.
// Split State + Actions contexts, mirroring the other providers; composed in main.tsx around <App/>. // Split State + Actions contexts, mirroring the other providers; composed in main.tsx around <App/>.
// //
// The view mode is an account PREFERENCE, not a filter: it persists to the server on pick (savePrefs) // The view mode is an account PREFERENCE, not a filter: it persists to the server on pick and is
// and is adopted from server prefs on login — so unlike the ephemeral list filters, this provider // adopted from server prefs on login. usePref carries that server-sync (plus the localStorage mirror
// carries the same server-sync the App god component used to. A localStorage mirror seeds the initial // that seeds the first render synchronously) so this provider is just the two contexts over it.
// render synchronously; the passive qk.me() observer adopts the server value once the account resolves.
interface FeedViewActions { interface FeedViewActions {
setView: (v: FeedView) => void; setView: (v: FeedView) => void;
@@ -37,30 +26,11 @@ export function useFeedViewActions(): FeedViewActions {
} }
export function FeedViewProvider({ children }: { children: ReactNode }) { export function FeedViewProvider({ children }: { children: ReactNode }) {
const [view, setViewState] = useState<FeedView>(() => // parseFeedView also maps the pre-0.45 "grid"/"list" values forward (idempotent for current values).
parseFeedView(readAccount(LS.feedView, null)), const [view, setView] = usePref<FeedView>("view", "cards", {
); localKey: LS.feedView,
// Passive observer of the account App fetches (enabled:false → reads cache, re-renders on update). parse: parseFeedView,
const { data: me } = useQuery({ queryKey: qk.me(), queryFn: api.me, enabled: false }); });
// A user pick persists both to the localStorage mirror (instant reload) and to the server.
const setView = useCallback((next: FeedView) => {
setViewState(next);
writeAccount(LS.feedView, next);
api.savePrefs({ view: next }).catch(() => {});
}, []);
// Adopt the server-stored view on login / account switch. Writes the mirror but does NOT savePrefs
// (that would echo the just-adopted value straight back). parseFeedView also maps the pre-0.45
// "grid" / "list" values forward.
useEffect(() => {
const prefs = me?.preferences;
if (!prefs) return;
const adopted = parseFeedView(prefs.view);
setViewState(adopted);
writeAccount(LS.feedView, adopted);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [me?.id]);
const actions = useMemo<FeedViewActions>(() => ({ setView }), [setView]); const actions = useMemo<FeedViewActions>(() => ({ setView }), [setView]);
+76
View File
@@ -0,0 +1,76 @@
import { useState, type ReactNode } from "react";
import { useTranslation } from "react-i18next";
import { defaultLayout, type PanelId, type PanelLayout } from "../lib/panelLayout";
import SidePanel from "./SidePanel";
import PanelGroups, { type PanelItem } from "./PanelGroups";
// The shared filter-rail shell, extracted from the three rails that all render the same SidePanel
// chrome around a PanelGroups body: the feed Sidebar, the PlexSidebar, and the PlaylistsRail. It owns
// the customize (reorder/hide) `editing` state and the fixed title-bar wiring (edit toggle, reset to
// the layout default, edit hint); each rail supplies only what actually differs — its identity
// (title/icon), the active-filter count + Clear-all action, its `items`, and any content below the
// groups (`children`, e.g. the feed's tag manager). `kind` is the PanelId (also the layout key), so
// `children` defaults to nothing for the rails that need none.
export default function FilterRail({
kind,
title,
icon,
count,
actions,
collapsed,
onToggleCollapse,
layout,
setLayout,
items,
collapsible = true,
iconExpanded,
onTitleClick,
titleActionLabel,
children,
}: {
kind: PanelId;
title: string;
icon: ReactNode;
count?: number;
actions?: ReactNode;
collapsed: boolean;
onToggleCollapse: () => void;
layout: PanelLayout;
setLayout: (l: PanelLayout) => void;
items: PanelItem[];
/** Passed to PanelGroups: false renders plain always-open label + control rows (no collapse). */
collapsible?: boolean;
iconExpanded?: ReactNode;
onTitleClick?: () => void;
titleActionLabel?: string;
children?: ReactNode;
}) {
const { t } = useTranslation();
const [editing, setEditing] = useState(false);
return (
<SidePanel
title={title}
icon={icon}
count={count}
collapsed={collapsed}
onToggleCollapse={onToggleCollapse}
editing={editing}
onToggleEditing={() => setEditing((e) => !e)}
onReset={() => setLayout(defaultLayout(kind))}
editHint={t("sidebar.editHint")}
actions={actions}
iconExpanded={iconExpanded}
onTitleClick={onTitleClick}
titleActionLabel={titleActionLabel}
>
<PanelGroups
items={items}
layout={layout}
setLayout={setLayout}
editing={editing}
collapsible={collapsible}
/>
{children}
</SidePanel>
);
}

Some files were not shown because too many files have changed in this diff Show More