Compare commits

..
90 Commits
Author SHA1 Message Date
peter 2a699e26ba release: 0.99.1 (improvement/drop-ci) 2026-10-04 06:45:48 +02:00
peter 13c8c05661 chore(release): 0.99.1 2026-10-04 06:36:06 +02:00
peter cc414de952 build: the gate is scripts/check.sh, and CI goes
The Gitea Actions gate repeated, after the merge, what a release already runs
before it, and its runner shared the git host's container: one run there
starved Gitea of memory until it stopped answering. The lanes now live in
scripts/check.sh, run the same by the release dispatcher and by hand, and the
gate-contract test holds its lines (the ruff and pip-audit checks read it
instead of the workflow). The two audits only CI ran join the gate: pip-audit
over the lock (pinned in the test image, blocking at any severity) and npm
audit of the production dependencies. The engine contracts drop the CI
containers; the register records that pushes have no server-side gate.
2026-10-04 06:20:58 +02:00
peter 0528b35f2d release: 0.99.0 (feature/e32-gate-integrity)
gate / backend (push) Successful in 8m5s
gate / frontend (push) Successful in 8m32s
2026-10-04 05:19:05 +02:00
peter cb46774851 chore(release): 0.99.0 2026-10-04 05:09:33 +02:00
peter f1cc6ce5dc fix: the reconcile scan's return type names its settled sections; the composer stays at its size ceiling 2026-10-04 05:04:14 +02:00
peter 4efe799eda fix(ui): the issue composer opens on its title, not on the Edit switch
A form dialog opens on the body's first control, and the composer's first
control is the Edit / Preview switch above the form: the focus landed there,
at open and when a retried load brought the form, and typing went nowhere. A
form can now mark the field to open on (data-initial-focus); every other
dialog keeps its first control. The Modal's comment on the re-armed watch now
says where the focus goes when the focused control is removed but others stay.
2026-10-04 04:56:57 +02:00
peter 40b27f2982 fix(sync): a channel with no uploads backfills as done, not as a failure
YouTube answers the uploads list of a channel with no uploads as
playlistNotFound. The recent and deep backfills raised on it, so the channel
was never stamped and was picked again, charged a quota unit and logged an
ERROR on every run. Both now read it as an empty last page, as explore already
does; any other failure still raises.
2026-10-04 04:51:25 +02:00
peter 0314c4cce2 fix(plex): the watch reconcile settles each library on its own walk
A mirrored library that answers "0 items" is not believed (its rows stay), but
that verdict stopped the whole reconcile: one library emptied on Plex with its
mirrored items kept made every run incomplete, and un-watches in every other
library never synced again. Each section now settles on its own walk, and only
the settled sections' rows are cleared; the unsettled ones are named in the
warning.
2026-10-04 04:49:55 +02:00
peter 182ec31120 test(gates): hold the lint and lane configs to a committed snapshot
The register test detected narrowings by modelling config shapes, and each
review round found one more shape it did not model (a basePath global ignore,
a rule src never declares, a widened glob filed under a narrow row, a substring
check on a CI line, knip.json beside knip.jsonc). It now compares instead:
gates.snapshot.json holds ESLint's effective config for every code file and the
lines that run each lane, and any difference fails until it is regenerated.
The register is read off the snapshot's configs, which surfaced the ui/ and
lib/ test scopes the sampling never reached.

The complexity ratchet also names a wrapped function by its binding
(const onSave = useCallback(() => ...)), not as "useCallback callback".
2026-10-04 04:47:10 +02:00
peter 12a39f5291 fix(plex): the watch reconcile clears only what it walked, and doubts a mirrored 0
A library left out of the allowlist is never scanned, so its watch states were cleared on
every reconcile; clearing is now limited to the sections walked. A section we mirror items
from that says it holds 0 is not believed complete. The page size is no longer read as the
section total when totalSize is missing (measured: the server always sends totalSize on a
paged request, 0 for an empty one).
2026-10-04 04:16:52 +02:00
peter 79704b0001 fix(youtube): "this playlist no longer exists" only where the user picked that list
playlistNotFound was mapped to "gone" everywhere: exploring a channel without uploads (its
uploads list answers not found) and a list deleted mid bulk pull both said a playlist no
longer exists. The reason is now "gone" only on single-playlist routes; explore reads it
as an empty back-catalogue, and the bulk pull skips that list and syncs the rest.
2026-10-04 04:16:52 +02:00
peter 4191189b4c fix(ui): a late form takes a dialog's focus once, even if it arrived elsewhere
The watch stayed armed when the form arrived while the focus was off the close button, so a
later change in the body pulled the focus off a close button the user went back to. It now
disarms on arrival and re-arms only when the focus is taken away.
2026-10-04 04:16:52 +02:00
peter 7dbda60815 fix(ui): an undo in the same tick as a rebase steps back to a rebased snapshot
rebase updated the state only, so the mirror the callbacks read still held the removed
row and an undo right after it wrote that row back for a moment.
2026-10-04 04:16:52 +02:00
peter d1df76e8e5 test(gates): the complexity ratchet names a function by what it is bound to
ESLint leaves an arrow bound to a const or a property unnamed, so three ratchet keys were
(anonymous) and ranked by position again. A tiny rule reports each unnamed function's
binding (const, property, assignment, else <callee> callback) at the head complexity
reports, and the keys line up by location.
2026-10-04 04:16:52 +02:00
peter 3f6a9efd57 test(gates): the register test parses ruff's TOML and holds pytest.ini line by line
Round 2 of the E32 review found the key whitelists blind: a line regex saw neither a
[lint.per-file-ignores] table nor a quoted key, and pytest.ini's continuation lines and
key: value form slipped past. Ruff's config is now parsed (smol-toml) and held to its
key paths; pytest.ini to its whole lines; the CI ruff steps to --config backend/ruff.toml,
which is what makes that file the whole configuration.

ESLint: every glob of a block must reach a sample (reactPlugin.ts and devOnlyGuard.ts
rode along unseen), a block's own severity is held to src, a no-restricted-* list is held
to the src block's declaration rather than Feed.tsx's effective config, named global
ignores count, an ignored sample is named, and linterOptions are checked. tsconfig.node
must keep allowJs + checkJs while it lists a .js file; knip's positive glob is pinned.
2026-10-04 04:16:52 +02:00
peter 64e8f1da3d fix(lint): E402 is on — the ignore meant for main.py had let 11 late imports in elsewhere
ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
2026-10-04 03:14:24 +02:00
peter a1224e9a8c test(gates): the repo-root scripts are linted and format-checked
scripts/check-node-modules.mjs and gen-changelog.mjs run in check and release but sat outside
every lane: the eslint and prettier lanes run from frontend/, and an ESLint config cannot reach
past its own directory. A contract test now holds them to the js-configs bar (recommended rules
under Node, the frontend's Prettier style); an unused variable and the old formatting are each
red. 0 lint hits; both files reformatted. Registered in docs/gates.md.
2026-10-04 03:00:49 +02:00
peter da4b19a39a docs(lint): the config comments say which rules can report, and count what is suppressed
eslint.config.js called every zero-hit React Compiler diagnostic a tripwire. Under
eslint-plugin-react-hooks 7.1.1 three cannot report anything: exhaustive-effect-dependencies (its
validation is compiled off), memoized-effect-dependencies (nothing emits its category) and
rule-suppression (lint mode passes no suppression rules). Probed: a missing effect dependency is
reported by exhaustive-deps only, and an inline disable is named by nothing. The header's
"33 suppressions" of exhaustive-deps is 0. reactPlugin.ts said one react-hooks rule was still
off; none is (three run at warn, which --max-warnings 0 fails).
2026-10-04 03:00:49 +02:00
peter 5a74cf86b7 test(api): the wire-key checks send every filter field, and tsc says when one is new
The Plex and feed filter fixtures were typed as the filter type itself, whose scalar filters are
optional: a new optional field compiled without a test value, so its wire key went unchecked.
Both are Required<…> now (a probe field fails typecheck on each). The feed set gained the fields
it omitted (year, librarySource, the scope and show switches) and goes out with and without year,
which wins over the other date filters; a misspelt year key is now red in the wire check, which
passed it before.
2026-10-04 02:57:09 +02:00
peter e432dc2780 test(gates): the complexity ratchet holds each function by name, and its scope is registered
The ceilings were a per-file ranked list, so one function could grow by what another in the same
file lost: Home.tsx's [34, 32] passed with Home at 32 and Hero at 34. Both twins now key every
offender as module::function (an unnamed one is (anonymous); a repeated name gets 2, 3 in source
order) and re-baseline at today's measured values. Swapping two ceilings is red on both sides
(Home 34 > 32; feedquery filtered_query 28 > 14).

The ratchet measures shipped code only. docs/gates.md said C901 covered backend + scripts; it
covers backend/app. The exclusion of tests, test kits, e2e and scripts/ is now a row with its
measured hits, and the stale counts are replaced (9 BE modules over 800, not 11).
2026-10-04 02:55:07 +02:00
peter 3741d1a46d fix(playlists): a list reopened from the cache keeps its rows
Leaving the Playlists page and coming back painted the selected list empty ("0 videos") while
the rail counted its rows. The page re-seeds its undoable order from the cached list in an
effect; StrictMode runs that effect twice on the remount, so a reset was followed in the same
tick by a rebase. Since E32 S3 a reset is state only and the rebase read the mirror ref, which
still held the empty initial history, and wrote it back over the reset.

rebase is now a state updater, so it builds on whatever an earlier call in the tick installed.
Both tests seen RED (hook: [] for [10, 20]; page: no rows), and the bug reproduced in the
browser on the old hook.
2026-10-04 02:48:10 +02:00
peter 98ec1abf25 test(gates): the register test reads every config block, option and key, not eight samples and severities
E32 review round 1 (T1–T5, s1). The test that holds docs/gates.md to the effective configuration
could itself be passed by a narrowing it did not look at:

- ESLint: besides each scope's sample, every config block is read on its own. A block that targets
  files no sample reaches fails as an "unsampled block"; a no-restricted-* list re-declared without
  an entry src has, an option that lets something through (allow…/ignore…/except…), a block-level
  `ignores` and EVERY global ignores object (not only the first) count. A scope switching a src rule
  at warn off now counts too (it skipped them). Two live narrowings surfaced and are registered:
  no-floating-promises' allowForKnownSafeCalls (src and e2e) and the three network owners'
  no-restricted-globals.
- TypeScript: a strict sub-flag set false beside `strict: true`, an extends/exclude/files, a changed
  include, and a frontend-root config file no project checks. eslint.config.js is that file: listed,
  with the 43 errors strict checkJs reports in it (measured, none a defect).
- Ruff, basedpyright, pytest and knip are read as key whitelists (an unknown key fails until someone
  decides whether it narrows); pyright's information/hint count as weak; ruff's extend-select is
  pinned; pytest's three keys are pinned exactly; knip's script must run both modes.

Each new check was probed with a planted narrowing (an unsampled block, a block ignores, a second
global ignores, a warn rule off in a scope, strictNullChecks false, a ruff extend-exclude, a pytest
-m) and failed on it.
2026-10-04 02:37:33 +02:00
peter e5b155ada9 refactor(ui): reuse the store, media-query and deleted-tag rules the review found rebuilt, and name a loading pick
E32 review round 1 (F3–F6).

- Channels' open-row store and the chat dock use lib/store's createStore instead of a hand-rolled
  listener set (the chat dock gets a test of its own).
- The Plex backdrop reads prefers-reduced-motion through useMediaQuery; the query string is one
  constant (REDUCED_MOTION) for the hook and the carousel's one-off read.
- A link-restored collection filter showed its raw rating_key in the picker while the titles
  loaded: usePlexCollectionTitles now says it is pending, the picker draws a placeholder
  (labelsPending, as the channel picker does) and so does the chip row.
- The "deleted tag" rule was written twice with two i18n keys and had drifted: the channel manager
  showed NO chip for a pick while the tag list loaded, or ever after a failed load, while the pick
  still narrowed the table. One rule now (lib/tagPicks), one key (common.deletedTag); the manager's
  row shows every pick, as a placeholder while loading and as its id after a failure.
2026-10-04 02:29:03 +02:00
peter e09d613b04 fix(ui): a reviewed log takes the focus back once, and a dialog's focus survives a Retry
E32 review round 1 (F1, F2).

- Issue composer: the review's focus-return token became state in S4 and was never spent, so every
  Preview → Edit round trip remounted the attachments and moved the focus from the toggle to the
  log's Review button. The token is cleared once it has been used, as the ref was.
- Modal (initialFocus "field"): the late-form watch moved the focus to the first control once and
  stopped. A failed load's Retry then gave way to the spinner, its focus fell to <body> inside the
  aria-modal dialog, and the form's first field never got it. The watch now re-arms when the focus
  falls to <body>: it waits on the ✕ and moves to the first field when the form arrives. Focus in a
  dialog stacked on top is left alone.
2026-10-04 02:29:03 +02:00
peter 38943fa93e fix(maintenance): one clock per run, so a video phase 1 recovers is not fetched again in phase 2
E32 review round 1 (B7). Each phase took its own now; phase 2's "not checked this run" test then
let a just-recovered video through and spent another videos.list call on it.
2026-10-04 02:14:04 +02:00
peter 887b4e546c fix(feed): a malformed saved view cannot break the autotag cleanup, and a foreign private tag reads as missing
E32 review round 1 (B5, B6).

- The autotag orphan cleanup reads every user's feed views. A view whose tag pick was not a list
  ("tags": 5) raised for everyone's job, True passed as tag id 1, and ~66k ids pushed the NOT IN
  past psycopg's 65535 bind parameters. The views are now read defensively and the kept ids
  filtered in Python; the saved-views route holds filters.tags to a list of at most 500 ids, on
  create and on update.
- tags_by_category resolved any tag id, so another user's private tag landed in its real category
  and the feed summary answered differently than for an id that does not exist, which told the
  caller which foreign ids exist. It is owner-scoped now (pre-existing; S2's missing-tag branch
  made the difference visible).
2026-10-04 02:14:04 +02:00
peter df4a79a1f1 fix(sync): an empty answer the remote's own count contradicts is not believed, one level down
E32 review round 1 (B1, B2, B3, B8).

- Plex watch reconcile: a section the server SAYS is empty (totalSize 0, nothing listed) is a
  complete listing. It read as "cut short" forever, so one empty library stopped un-watches from
  ever syncing. A listing with no total at all stays incomplete (the client now returns None for
  it instead of 0), and a whole scan that listed nothing still clears nothing. Tests now cover
  completeness across two sections, both orders.
- Plex collections: no members from a collection whose listing counts some (childCount) keeps
  the membership we hold, in the full sync and in the targeted resync after a write.
- YouTube playlists: no items for a playlist whose listing counts some (itemCount) keeps the
  mirror in the bulk pull, and is refused as retryable in "reset to YouTube" and in the push
  plan/merge, where it read as "YouTube deleted them all". The rule sits beside vanished().
- A playlist deleted on YouTube answered "reconnect your account"; it now says it is gone
  (errors.youtube.gone, HU+EN).
- The pure 3-way merge moved to sync/playlist_merge.py; playlists.py is back under 800 lines and
  off the size list.
2026-10-04 02:13:53 +02:00
peter 5d10a521a8 docs(gates): react-hooks/refs is on, the compiler skips one function, and the async-cleanup idiom that keeps it so 2026-10-04 00:05:37 +02:00
peter d7fb7075b8 refactor(ui): every function compiles under the React Compiler but VirtualGrid, and react-hooks/refs is on
E32 S4 — the burn-down from 29 skipped functions to 1:

- try/finally (26 sites): the cleanup runs after a try/catch; steps that return early moved into
  their own function (push, the setup wizard's next, the share page's unlock).
- A conditional, loop or optional call inside a try skips the function too, and react-hooks/todo
  does not see it: the value is computed before the try, or the try lives in a module-level helper
  (config writes, clip queueing, clipboard, frame counts, seekable end, YouTube player times).
- The two exhaustive-deps suppressions are gone. useYtProgress takes its per-item snapshot through
  one effect event, read once when the player is built; PlexPlayer's unmount relies on the
  leaving-the-item cleanup for the generation bump it duplicated.
- What the skips hid: PlexPlayer's 17 render-phase ref writes (useLatest), its tracks popover alias,
  the loadSession self-reference (declared early, set after commit); PlayerModal's session-position
  read during render (now where the player starts), navRef/modeRef writes, and a description helper
  that ran a ref-carrying callback during render (now PlayerDescription); SavedViewsProvider's
  latest-cfg ref, and its setState in the seed effect (preferSticky now prefers the starred default
  among same-filter views, so the seed needs no sticky write — test seen red on the old helper).
- A store's hook was called `use()`, which the compiler does not know as a hook: inside the newly
  compiled usePlaylistActions it was memoized, ran on the first render alone, and the playlist rail
  threw "rendered fewer hooks" on its second render. Renamed to `useValue()` everywhere; the old
  contract test claimed the call was safe inside any `use…` function, which was false — replaced by
  a render test that is red under the old name.
- PlexPlayer's palette ref was written and never read since the Backspace guard moved to the layer
  stack: removed.

react-hooks/refs is back at error with no exception. The ratchet lists only VirtualGrid
(IncompatibleLibrary) and pins both players as compiled. No size or complexity ceiling was raised.
2026-10-04 00:05:37 +02:00
peter a486c7521a docs(gates): the compiler burn-down after E32 S3, and the two idioms that keep it
react-hooks/refs 245 -> 31 hits and 93 -> 29 skipped functions in the register and the eslint
comment. The canonical-patterns table gets the latest-value row (useEffectEvent; a handler or timer:
useLatest / useStableCallback, never a render-phase ref write) and the popover / scroll-fade row
(destructure the ref setter).
2026-10-03 23:18:33 +02:00
peter 76358a0d77 refactor(ui): the popover engine and every render-phase ref write compile under the React Compiler
E32 S3. The compiler skipped 93 functions; it now skips 29 (try/finally shapes, two suppressions,
VirtualGrid). react-hooks/refs hits 245 -> 31.

- Popover engine: `ref={pop.refs.setReference}` made the compiler treat the whole popover handle as
  a ref, so every component holding one was skipped (21 call sites, the engine and its surface
  among them). The setter is destructured now; the same for useScrollFade and useReturnFocus.
- Render-phase `ref.current = x` writes moved to `useEffectEvent` where only effects read them,
  else to the new `useLatest` (written in a layout effect) / `useStableCallback`.
- FocusSafeButton (under every Button) read the focused element during render; it reads it through
  useSyncExternalStore, which React asks on every render instead of a cached memo.

Bugs this surfaced, fixed:
- Combobox: a chip could keep showing its raw id. The label cache was a ref an effect filled after
  the render that needed it; it is state merged during render now (test seen RED).
- useUndoable rendered from a ref plus a forced re-render; compiled, that memoizes a stale value.
  The history is state now, mirrored in a ref for the stable callbacks.
- ConfigPanel and the saved-views editor re-seeded their drafts in an effect, painting the stale
  draft for a frame; both adjust during render against the remembered key.
- Toaster's handover is written after commit instead of during render; a new test pins that the
  inherited set is taken once, at the takeover (seen RED on a per-render re-read).
- IssueComposer mutated a ref prop as a one-shot focus token; it is plain state.
- PlexBrowse wrote its info-scroll memory from an effect declared before the ref, past its own
  "one writer" rule; the writer comes first now and the effect goes through it.

The compiler ratchet drops to the measured 29 and now also requires FocusSafeButton and the popover
engine to compile.
2026-10-03 23:18:26 +02:00
peter dc00b55feb test(auth): warm the Welcome import once so the first case cannot time out under load 2026-10-03 21:53:06 +02:00
peter fb6b56f121 test(api): every filter key the client sends is one the backend reads
FastAPI ignores an unknown query parameter, so a misspelt or renamed key silently stops a
filter while its chip still shows. Contract tests read routes/feed.py _feed_params and
plex/query.py LibraryFilters. The superseded feedCount/facets calls and keys are removed.
2026-10-03 21:53:06 +02:00
peter 2e2d6f5746 fix(downloads): the quota editor opens at once and says it is loading or failed
It rendered nothing until its quota arrived and nothing at all when the load failed, so the
click looked dead. The first field still takes focus when the form arrives.
2026-10-03 21:53:06 +02:00
peter 2ab9ff6a42 fix(plex): a server without a machine id refuses the collection write by name
The missing identifier became server:/// in the write URI. Client tests over a MockTransport
(errors, paging, cutoff, write URIs, image transcode) and admin-route tests (connection test,
enabled-section sampling, no server URL in a 422, the one-time watch import).
2026-10-03 21:53:06 +02:00
peter 378c4daf0f fix(youtube): the scrape search never pins a stale key, and an unknown shape is an error
A failed config fetch or a keyless consent page cached the fallback InnerTube key for the life
of the process; once it went stale, search returned nothing with no error. The fallback is now
used for that call only. A 200 JSON without a result frame returned an empty page that read as
no results; it is a ScrapeError now. Parser, cursor and Shorts-probe tests added.
2026-10-03 21:53:05 +02:00
peter 8f1208fdfe fix(feed): a filter on a deleted tag matches nothing and its chip says so
A saved view or the stored filter outlives its tag. feedquery dropped the unknown id, so the
view showed the whole feed under its old name. The id now lands in a MISSING_TAG_CATEGORY that
filters to nothing in the SQL and in feed_summary's mirror alike; the feed chip reads "Deleted
tag" (a placeholder while the tag list loads, the id if it failed). Channels' persisted tag
filter gets the same chip, first in the row so the +N overflow cannot fold it away.
2026-10-03 21:53:05 +02:00
peter 52f12e5331 test(e2e): click only a channel link that is the element hit at its centre
A card inside the viewport can still sit under the chrome above the scroller (the update
banner); the covered click made Playwright scroll and retry, moving the feed the scroll-restore
test then measured. webkit failed 3 in 24 before, 0 in 48 after.
2026-10-03 21:10:27 +02:00
peter c4abbe48c1 refactor(sync): the empty-answer rule for a mirror is one function, vanished()
The YouTube playlist guard from the previous commit put sync/playlists.py over its size ceiling;
the rule now lives beside should_prune as vanished(existing, listed), which the mirror calls in
place of its own list. The pyright baseline also drops by one (187), from the plex/overlay split.
2026-10-03 19:45:14 +02:00
peter 4fc307c426 fix(autotag): the orphan cleanup keeps a system tag a saved view filters on
Feed views store tag ids. A system tag no channel carried for a while was deleted and, once a
channel earned it again, recreated under a new id — the view's filter then pointed at nothing, the
feed dropped the unknown id, and the view showed the whole feed under its old name. The cleanup now
skips tags a saved feed view references (test seen red without it).
2026-10-03 19:37:24 +02:00
peter 328d415cfe test(e2ee): the DM crypto round-trips, refuses the wrong key, and keeps its nonces apart
lib/e2ee was 14% covered and fails silently in every way it can break. Real WebCrypto and an
in-memory IndexedDB: a message decrypts for its partner and nobody else, every message gets its own
IV, a new device unlocks with the passphrase and refuses a wrong one, the key survives a reload
until reset, a partner's rotated key is used only after forgetConv, and the key check never shares
the wrap IV (GCM nonce reuse). Mutations seen red: the shared IV, a no-op forgetConv, no persist.
2026-10-03 19:35:36 +02:00
peter 43e4929b45 fix(sync): an empty answer no longer empties a mirror; maintenance checks each video once a run
The catalog prune and the subscription import have long refused to reconcile against an empty
listing that contradicts what we hold (should_prune: a 200 with no items is a blip far more often
than a user who removed everything). Two mirrors did not: the YouTube playlist mirrors (dropped
every mirror) and the Plex collections (dropped every collection with its user-set `editable`
flag, and every member's keys — the guard landed in plex/sync.py with the previous commit). Both
now hold the same rule; test_empty_listing_guards covers each way (seen red).

sync/maintenance (19% covered) gets a lifecycle test: recover, grace, delete + one notification
per user, live stream kept, abandoned premiere. It found the rolling revalidation re-fetching the
same videos until `checked` reached the batch (15000 by default) whenever fewer videos were
available — quota spent on answers it already had; a run now skips what it has checked.
2026-10-03 19:33:38 +02:00
peter 5806adda7e fix(plex): a listing cut short no longer clears watch state or Hungarian text
The daily watch reconcile reads "absent from Plex's scan" as "Plex un-watched it" and deletes the
row — but it walked the sections without asking _paginate whether the walk reached the server's
total, so a Plex hiccup mid-walk (an empty page early) un-watched every title it failed to reach
until the next complete run. The scan now reports completeness; an incomplete one still pulls in
what it saw, re-pushes dirty rows, and clears nothing (logged as plex.watch.reconcile_incomplete).

The HU overlay had the same shape: clear every row's HU text, then refill from the listing. It now
reads the listing first and clears only after a complete one. The overlay moved to plex/overlay.py
(plex/sync.py drops under the 800-line limit and off the size ratchet's list).

Tests: plex/paths (23% -> 100%: prefix boundary, longest prefix, dead mount), the reconcile's
clear/keep/re-push rules, the cut-short scan and listing (each seen red with the guard removed).
2026-10-03 19:33:30 +02:00
peter c73761ae39 refactor(plex): one cast row for a show's page and a title's info
CastStrip (a show's page) and PlexInfo (a title's page and the player's overlay) each drew the
cast carousel by hand, and the copies had drifted: the gap, lazy pictures, the name's tooltip, the
initial's size. Both now render CastRow, which takes the overlay's white-on-art palette as a tone;
each caller keeps its own frame and heading. A DOM test holds the palette per variant (seen red
with the tone wired to "panel"). jscpd over the shipped code: 1 clone >= 10 lines -> 0.
2026-10-03 19:12:17 +02:00
peter ca9ca85b18 test: no test throws a response away; the stream session is narrowed, not baselined
The 7 baselined Optional accesses in plex/stream.py were one fact the type did not carry: the
initial spawn (replace_only=None) always supersedes, so only the fallback respawn can get None.
An assert at the call site says so; the pyright baseline drops 195 -> 188.

test_abuse_limits: the public-URL and YouTube enqueue tests stopped at the profile lookup
(400 no_profile) and passed on that unrelated refusal. They now send an inline spec and assert
the job is queued. The same shape — a request whose response is discarded — sat at 53 more
places across the suite (all returned what they should, measured with a TestClient probe);
each now asserts its status, and test_response_checked keeps it that way (seen red).
2026-10-03 19:06:47 +02:00
peter 614d728561 test(e2e): the nav spec opens the admin pages from their rail panel
E31 put the admin pages behind one rail row ("Administration", nav-admin)
that opens a panel. The spec still clicked nav-scheduler, nav-config,
nav-users and nav-audit as rows of their own, and timed out on every engine.
That went unseen from E31 to now because e2e ran in no required gate, the
exact hole E32 closes: the first run of the new release e2e step found it.

The rail test keeps the rail rows. A new test opens each admin page from the
panel and checks that the group row is current and that the reopened panel
marks the page. Seen red before the fix; the whole suite then passes, 52/52.
2026-10-03 17:41:26 +02:00
peter 783f6408f9 docs: the gate register, held to the effective config by a test
docs/gates.md lists every place a gate deliberately checks less than it
could, each with its reason and the count measured when the row was
written:
- an ESLint rule off, at warn, or not applied, by scope;
- a weak compiler flag, a ruff ignore, the basedpyright mode and baseline;
- a knip exclusion, a lane that does not run everywhere;
- an accepted CVE.
gates.contract.test.ts reads the EFFECTIVE configuration and fails when a
narrowing has no row:
- ESLint's computed config per scope;
- the three tsconfigs, ruff.toml, pyrightconfig, pytest.ini and knip.jsonc;
- .trivyignore.yaml, where an entry also needs a statement and an expiry;
- CI's pip-audit ignores.
It also pins `--max-warnings 0`, which is what makes a warn-level rule fail.
Seen red on a missing row, a new `off` rule and an allowlist entry without
an expiry.

CLAUDE.md gains the rule E32 exists for: a gate with any real hit is never
switched off or narrowed, and an "enforced by" comment is checked by running
the check. The gate descriptions there, in docs/development.md and in
backend/app/README.md name the lanes as they now are, and the "directly"
recipe points at the full list.
2026-10-03 17:26:42 +02:00
peter 751e1ae5ed test: a size and complexity ratchet; coverage reported on both sides
The "class size" item from the quality checklist, measured:
- A shipped file over 800 lines may not grow past its line count today
  (13 frontend, 11 backend). No other file may cross 800.
- A function over the complexity limit may not rise, and no new one may
  appear. The limit is eslint `complexity` 20 on the frontend (64 functions
  over it today) and ruff C901 10 on the backend (32).
- Each is a per-file ceiling. Shrinking is free; an entry back under the
  limit must be deleted, so the lists only shrink.

C901 is measured in a test rather than switched on in ruff.toml: ruff has no
per-function baseline, and a noqa would let a function grow without limit.

Coverage (owner decision D4) is printed with no threshold:
`vitest --coverage` (v8) gives 80% of lines, and the pytest lane ends with
the total for app/, 78%. The bar stays "every regression test seen red".
.coverage is ignored by git and kept out of the image's build context.
2026-10-03 17:18:53 +02:00
peter da554df7c7 build(image): the image passes a Trivy scan: OS security updates, no runtime pip
The first Trivy scan of the shipped image (0.98.0) found four fixable HIGH
advisories. One was pcre2: a Debian fix newer than the base snapshot, now
taken by `apt-get upgrade` at build time. The others were urllib3 (bumped in
the lock), and msgpack, setuptools and urllib3 vendored inside pip itself.
pip is removed once it has installed the lock, since nothing runs it at
runtime.

A local build of this Dockerfile with --pull scans clean: 0 HIGH/CRITICAL
with a fix, no secrets. .trivyignore.yaml is the publish gate's allowlist,
empty today. Each entry must carry a statement and an expiry date.
2026-10-03 17:06:28 +02:00
peter b1ff4058ac build: CI builds the bundle and audits what ships; the toolchain advisories are cleared
The server gate adds `vite build` (and with it devOnlyGuard, which until now
first ran inside the image build, after main was pushed) and
`npm audit --omit=dev --audit-level=high`. pip-audit is blocking: when it was
finally read it held three urllib3 advisories with a fix available.
urllib3 goes 2.7.0 -> 2.8.0 as a single in-place pin, passing pip check,
rather than a regeneration that would move every package.

The dev toolchain's 18 advisories are gone:
- vite 5 -> 6.4.3 (path traversal, fs.deny bypass, and esbuild <= 0.24.2
  through it);
- vitest 3 -> 4.1.11 (mocker path traversal);
- typescript-eslint 8.46 -> 8.71;
- `npm audit fix` for the rest.
tsc, eslint, knip, all 2269 tests and the build pass unchanged.

scripts/check-node-modules.mjs checks that frontend/node_modules holds
exactly what package-lock.json pins. The local gate's frontend lanes judge
the host tree, and nothing tied it to the lock the image installs.
2026-10-03 16:50:56 +02:00
peter 7ed2ac4e95 build(frontend): knip also checks what ships; strict indexing in every project
`npm run knip` now runs a second, --production pass, which asks whether the
code that ships uses an export, not just whether anything does. Of the 47
exports only tests reached:
- 34 are exported for their test and say so with @internal, naming the test.
- 11 were module-private names with a stray `export`. Two of them were not
  used at all and are deleted.
- The unused alreadyReported re-export is gone.
Test kits, the e2e suite and the two build-tooling modules are excluded from
the production project, each named in knip.jsonc.

noUncheckedIndexedAccess is now on for the node and e2e tsconfigs too (0
errors), matching the app project. tsconfig.node.json listed a
tailwind.config.js that has not existed since Tailwind 4. That entry is now
playwright.config.ts, which the typed lint finds through this project.
2026-10-03 16:44:28 +02:00
peter cff8fe9a6a test(frontend): a React Compiler ratchet over everything the build ships
compilerCoverage.contract.test.ts used to check two files. It now runs the
real compiler over all of src (minus tests) and holds the exact set it skips
today, file -> function -> category: 93 functions. A new skip or a category
change is red, and so is a listed skip that has gone, so the list only
shrinks. Channels and DataTable must also show up as compiled, so a harness
that compiled nothing cannot pass.

Measured against the audit's scan: no function this sprint touched started
to skip. PlexPersonView and useOverlayOrder now compile.

reactPlugin.ts claimed the compiler's rules were linted, which was false
while react-hooks/refs was off. That comment and the refs block in
eslint.config.js now say what holds the line and why refs is still off:
245 real hits, pinned by the ratchet, burned down from S3.
2026-10-03 16:38:46 +02:00
peter 471846d784 build(frontend): close the lint gaps the E32 audit measured
eslint now also enforces:
- the React Compiler diagnostics that `recommended` leaves out (todo,
  memo-dependencies and ten zero-hit tripwires);
- a description on every suppression, and no blanket disables;
- no focused or skipped tests (vitest and playwright);
- the typed pair no-floating-promises and no-deprecated.

Lint also covers e2e/ and the TypeScript build configs, which `eslint .`
had been skipping in silence.

Every hit is fixed or suppressed inline with a named reason:
- The 31 try statements the compiler cannot lower yet carry a suppression
  until S3 rewrites them. Five other compiler skips were plain fixes:
  a module counter bumped inside a hook, a ref read in a default
  parameter, and a local named like the <section> it renders.
- The real defects behind floating promises are fixed. ChatDock threw at
  nobody when the device key store could not be opened. VideoEditor's
  play() rejection went unhandled. loadYouTubeApi never rejected, so a
  blocked IFrame API left the player on an empty stage forever; it now
  fails on a script error or after 15 s, shows its own error with the
  way out, and lets the next open try again.
- MutableRefObject, FormEvent and lucide's deprecated brand icon are
  replaced. The YouTube glyph now lives in lib/brandIcons.
- The lint-probe harness lets its virtual file names into TypeScript's
  default project, so the contract tests still run the real config.
2026-10-03 16:34:20 +02:00
peter 3c16012524 build(backend): close the lint gaps the E32 audit measured
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.

PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
2026-10-03 16:04:06 +02:00
peter cdfe7a8986 perf(channels): the row menu opens and closes without re-rendering the table
The React Compiler silently skipped Channels and DataTable (a ref written
during render; the refs lint is off), and even once compiled it dropped the
memo of the rows and columns (they reach filterRows/optionsFromColumns past
the page's hooks) and the columns read the open row. Every open, close and
reposition of the right-click menu rebuilt the whole table: on "All" rows
~0.9 s to open and ~0.6 s to close (dev). Now: effect events instead of the
render-written refs, explicit memos for rows, columns and their inputs, a
module-scope keyOf, and the open row reaches its "…" through a small store.
Measured on 397 rows (dev): ~75 ms to open, ~35 ms to close.

A contract test runs the real compiler over both files and fails on any
skipped function. The dropped memos have no automated guard: they need
the full page rendered with its providers; the measurement above is the
evidence.
2026-10-03 05:10:52 +02:00
peter 3e96211db0 fix(plex): the backdrop crossfade toggle wins over reduced motion once set
prefers-reduced-motion hard-disabled the rotation whatever the toggle said,
and Windows reports it whenever its animation effects are off — so the
owner never saw the crossfade. Untouched, the setting now follows the
system (and the toggle shows that state); a pick made on the toggle is
stored and wins on every device.
2026-10-03 04:38:50 +02:00
peter e9c15baae0 fix(feed): a channel filter's chips name their channels from the first paint
After a reload the chips (filter panel and the row above the feed) read as
raw UC… ids until the subscription list landed. The names seen last time
are now kept per account (useChannelNames) and name the chips at once; the
list then replaces the copy, so a renamed or dropped channel heals. A
channel never named on this browser draws a placeholder bone while the
list loads; a failed list still falls back to the id.
2026-10-03 03:29:54 +02:00
peter 6edb22988c release: 0.98.0 (feature/e31-design-review-2)
gate / frontend (push) Successful in 8m29s
gate / backend (push) Successful in 9m54s
2026-10-03 02:58:46 +02:00
peter f9c3714025 release: 0.98.0 notes — design review 2 2026-10-03 02:58:07 +02:00
peter a4775e1cd0 fix(ui): the filter rail shows no tag counts until the facets land
It printed the tag's whole-library count as the current filter's while the
facets loaded, and for good when they failed: live, "Music 238" beside a
search that matched 3 of its channels. Ordering still falls back to it.
2026-10-03 02:43:52 +02:00
peter b40440b825 fix(ui): review round 2 — reveal floor, failed next page, count states, row Play
- DataTable: the reveal floor holds only in the view it was made for (Channels
  never clears its follow, so every later sort mounted rows up to it)
- Feed + Plex browse: a failed next page stops the read-ahead and shows a row
  that retries, instead of re-requesting it once a round-trip
- the YouTube Load more click uses nextPageOnClick; a contract pins NEXT_PAGE
  to the read-aheads and the click sites to nextPageOnClick
- ItemCount: a failed count says so; a stale one says updating to a screen reader
- the narrow-picture Play lift applies only where a hover row is there to clear
- PlayerModal: one description ladder for the popover and the panel, both tested
- plexImage contract reads every img src, any attribute order
- Tabs comment names who persists the sub-tab
2026-10-03 02:25:15 +02:00
peter c4c442bc5b fix(feed): a hand-made relevance cursor is refused, not a crash
The page halves compare the cursor's timestamp with the window's in Python;
a null or naive one raised there, past the route's 400. decode_cursor now
refuses a null non-nullable key and a naive timestamp.
2026-10-03 02:25:14 +02:00
peter c74ceb73e3 fix(feed): the count pill dims while a new search's count loads
keepPreviousData held the last filter's count, and a broad word's count
takes seconds: the pill showed the old number as the new search's. It now
dims and is aria-busy until the new count lands, as AdminLogs does.
2026-10-03 01:01:46 +02:00
peter e0e28c006c perf(feed): relevance ranks the newest 2000 matches, the rest follow newest first
A broad word matched most of the library and ts_rank read every match:
"a" took 4.4 s a page on prod. The window's start is read once (0.1 s) and
rides the cursor, so a video ingested mid-scroll cannot repeat a row; the
page is read as its two halves, each on an index (80 ms + 5 ms).
2026-10-03 00:28:29 +02:00
peter f8f22fc985 test(i18n): a language test unmounts its reader before switching the language back
useTags reset the language in a finally while its hook was still mounted;
the switch scheduled a render, and in a loaded gate run it landed after
the jsdom environment was torn down ('window is not defined', two
unhandled errors). auditColumns and comfortPin had the same shape.
2026-10-02 23:48:37 +02:00
peter 90e0baa870 style(db): ruff format the re-runnable index migrations 2026-10-02 23:43:21 +02:00
peter dfd43ae89c docs: the folding channel header in its e2e, and three comments brought up to date
The e2e asserted the channel chrome never moves; it folds once now. The
ui catalog lists scrollFold and progressiveRows, Tabs names the
per-tab memory its callers use, and the welcome note says a removed
language orphans its stored copies too.
2026-10-02 23:42:32 +02:00
peter 608fde2d55 fix(db): re-running 0095/0096 rebuilds an invalid leftover index
A concurrent build that failed part-way leaves an INVALID index under its
name; IF NOT EXISTS kept it on the re-run, and 0095 then dropped the old
channel_id index, leaving none usable. The leftover is now dropped and
built again, and the old index goes only once the new one is valid.
2026-10-02 23:42:32 +02:00
peter 0302c466e8 fix(i18n): a failed RSS poll over one channel reads as one, and keeps its stops
rss_all_failed is a plural pair counted on the raw total (the values are
formatted strings, which pick no plural form); both sentences end with a
stop. The Hungarian 'show N more' uses the house's megjelenítése.
2026-10-02 23:42:32 +02:00
peter a50055577a fix(plex): every small Plex picture asks for its card-size cut
The playlist rows, the search box and the collection picker still
fetched the 400px poster for a 32-80px picture. A contract test pins the
set of bare Plex pictures, and the srcset widths to imgcache.SIZES.
2026-10-02 23:42:32 +02:00
peter e16cdbd16f fix(ui): a card's Play clears its hover row at a large text size; Downloads links and the player description
At text size 1.3 a 242px feed thumbnail put the 63px Play 13px into the
hover row (the Play grows with the root font, the picture's height does
not). A picture narrower than 14rem centres the Play and the resume pair
above the row; the picture box is a size container for it.

The Downloads title and channel links were 21px tall inline boxes; they
stand on a 24px line now (B1). The player's description says a failed
load, with a retry, instead of 'No description' (both the side panel
and the popover).
2026-10-02 23:42:31 +02:00
peter 15ec67b029 fix(ui): a read-ahead waits out a background refetch, and a revealed row is mounted before it is focused
NEXT_PAGE joins ANY fetch in flight: a read-ahead during an invalidation
or a poll joined that refetch and appended no page, and nothing asked
again until the reader moved the range (the channel page during a
backfill). The read-ahead now waits while the list fetches anything
(mayReadAhead, re-run on isFetching); a Load older click cancels a poll
instead (nextPageOnClick).

A page turn to a revealed row restarted the progressive mount at 30
rows; Channels focuses that row in the next commit, and at index 40 of
a 50-row page it was not there, so focus fell to <body>. The mount now
reaches through the revealed row.
2026-10-02 23:42:31 +02:00
peter ac53a3b757 perf(search): a one- or two-letter last word is matched whole, not as a prefix
A 1-2 letter prefix matched most of the library and ts_rank scored every
match: "m" took ~4-8 s on prod against ~150 ms now. The prefix match
starts at three characters (MIN_PREFIX_LEN); shared by the YouTube feed
and the Plex library search.
2026-10-02 21:52:47 +02:00
peter e6de18b8c0 perf(plex): the facets ask the episode rollup once per call, not once per show group
A watch-state scope (watched / in progress / unwatched) joined the per-user episode rollup into
each show facet query: five passes over the user's episodes per call, half of a filtered facets
request. The shows in scope are now resolved once and every group filters on their ids; the
grid-parity test holds the counts. Localdev: 99 -> 61 ms.
2026-10-02 21:32:49 +02:00
peter fb2e3d8490 perf(plex): a card fetches its picture at a card's size, not the detail page's
Cards fetched the full cuts: a 400x600 poster (~38 KB) for a ~144px picture and the 1280x720
backdrop (~162 KB) for a ~212px shelf picture. The image proxy now also cuts thumb_sm (240x360)
and art_sm (480x270), cached apart; card pictures (grid, shelves, collection tiles, Up next) carry
both in srcset with sizes=auto, so the browser picks by laid-out width and DPR. Localdev:
17 KB a poster, 56 KB a backdrop. The detail page keeps the full cuts.
2026-10-02 21:27:13 +02:00
peter d928d50686 perf(db): each feed sort rides an index, and an ICU upgrade rebuilds the title ones
The views, duration and title sorts had no index: every page read the whole videos heap and kept a
top-N (prod 350 ms a page, 550-780 for title). Migration 0096 adds one per direction and, for title,
per reader collation, in the exact ORDER BY feedquery builds. Localdev prod copy: plain sorts
180-310 ms -> ~1 ms, page 20 3-5 ms, narrow filters unchanged. view_count/duration are only written
by the enrich, which already rewrites indexed columns, so no HOT update is lost.

The title indexes are the first on an ICU collation, and postgres:16-alpine floats: collation.heal_drift
rebuilds them (CONCURRENTLY) at startup when Postgres reports the ICU version moved.
test_feed_sort_indexes holds every videos sort to an index and checks the rebuild.
2026-10-02 21:17:41 +02:00
peter 540822d2b0 perf(ui): a next-page fetch joins the one in flight instead of sending the cursor again
react-query's fetchNextPage cancels a next page still loading and sends it again by default.
Our requests ignore the abort signal, so the server ran both: the feed's read-ahead effect
re-runs on scroll frames before isFetchingNextPage reaches it, and on prod every feed page
went out 2-3 times (a 0.8 s title sort queued to 2.3 s). Every call now passes NEXT_PAGE
({cancelRefetch: false}), lint-enforced, with a contract test.
2026-10-02 20:46:30 +02:00
peter b4ef07d28e perf(ui): a long table page paints its first rows at once and the rest in steps
The channel manager's All page (~400 rows) blocked one frame for 0.7-0.9 s on prod to mount,
and every sort and search key re-rendered every row (300-400 ms). No single cell is to blame,
measured column by column. DataTable now mounts the first 30 rows at once and 40 more per
background transition; a new page, size, sort, filter or caller-side search (narrowedBy)
starts over from the first rows. A row is its own component, so the compiler keeps the rows a
step already holds.
2026-10-02 03:16:37 +02:00
peter e995c664c6 perf(db): the channel manager and the feed sidebar read an index, not the video heap
GET /api/channels summed every subscribed channel's videos by reading each video's heap page
(681 MB on prod): 51-61 ms warm, but 2.25 s on a first open and 6.4 s under EXPLAIN once the
table had left memory. /feed/summary and /feed/years did the same (~140 ms warm, ~800 ms cold).
Migration 0095 replaces ix_videos_channel_pub_desc with the same key INCLUDING the columns
those queries read, so they are Index Only Scans over a 25 MB index. A test holds the queries
and the index's column list together.

The admin Configuration page did a db.get per setting (two for an unset one): it now reads
every override in one query (~40 ms of ~55 on prod).
2026-10-02 03:16:37 +02:00
peter 20010178b0 fix(ui): the Plex detail poster keeps its 2:3, and a long crew shows its first names
The hero's grid row is as tall as the crew column beside the text, and the poster, a flex child,
stretched to it: a 2:3 picture drawn ~175x590 and cropped. It now aligns to the top, and a wide hero
gives the movie and show poster a bigger rung (w-56). The crew, the hero's tallest part, shows its
first 6 names and a 'Show N more' toggle; a list of 8 or fewer shows whole.
2026-10-02 02:37:20 +02:00
peter 78fdcda6bb feat(ui): E31 S4c — the channel header folds, the Plex pages fill their hero, the player gets a side panel
- Channel page: the banner and identity block fold away once the grid scrolls
  (ui/scrollFold: folds past 32px only if the page can stay scrolled, unfolds
  at the very top) and the name rides in the tab row. One Back button in both
  states keeps its focus; folded at a tablet width the actions wrap instead of
  pushing the page sideways.
- Plex movie and show pages: the TMDB facts over the key crew are the hero's
  second column (plexDetailAside), the keywords sit under the text; the show
  page shows them for the first time, with the same three toggles. A
  collection's hero is sized to its text.
- YouTube player, 1280px and wider: the title, channel, actions, description
  and queue sit in a side panel beside the video, which takes the window's
  height (1438x809 at 1920x911, was ~1008 wide). The wheel scrolls that panel
  while it can and turns the volume elsewhere.
- DESIGN.md decision log: the keywords' place under the text.
2026-10-02 02:13:17 +02:00
peter 98d0ca6eeb fix(i18n): every counted string is a plural pair, so one reads singular
"1 seasons", "Synced 1 subscriptions", "1 video(s) were skipped": 21 English
strings carried {{count}} in one plain key and read plural at one. Each is a
_one/_other pair now (Hungarian keeps one form for both), and so is every other
counted string, so the next wording change is a plural too. A contract test
holds the rule for every locale file.
2026-10-02 02:13:17 +02:00
peter e4c80fb7bb fix(ui): a page's sub-tab is remembered per browser tab, not forever
Settings, Configuration, Statistics and Users kept the last sub-tab in localStorage, so every
fresh tab opened Configuration on "Plex" as if that were its home. useTabPersistedState keeps it
in sessionStorage (account-scoped): a reload and a trip to another page keep it, a new tab starts
on the first sub-tab. The pre-selects (a Review link, the Google-link return) write the same store;
a value an older build left in localStorage is cleared on first read.
2026-10-02 00:49:46 +02:00
peter be156fc52b feat(ui): E31 S4b — the admin pages behind one rail row, Configuration as a sectioned list, Statistics as tiles and bars
- NavRailGroup: one "Admin" rail row opening a panel of the admin pages (links, so a
  middle-click still opens a tab); the rail fits 133% zoom without a hidden scroll
- sysconfig.SECTIONS orders the config groups under four headings (every group in exactly
  one, test-held); the page shows them as Settings' side list (Tabs `group` headings);
  the lone registration group reads "Registration" under "Access"
- StatTiles / StatTile / StatBars: the Statistics overview's figures as tiles, the
  per-operation quota use as horizontal bars (a zero draws none)
2026-10-02 00:07:00 +02:00
peter 25905aa786 fix(audit): fold a job's successful runs whatever ran between them
The owner read a fold that broke at every other job as the same row five
times. A job's successful runs now fold into one row at the newest run's
place across other jobs and admin actions; only that job's own failed or
skipped run breaks the fold (it stays its own row, and older successes
start a new fold below it).
2026-10-01 23:05:38 +02:00
peter 87d71e3762 feat(ui): E31 S4a — rail rows are links, the channel page opens on unwatched, users and audit tidied
- Nav rail page rows are real links (/?page=<id>): a middle-click or
  Ctrl-click opens the page in a new tab; a plain click stays in-app.
- A fresh channel page shows the unwatched videos; a picked watch state
  is a remembered preference (channelShow).
- Users: the role, suspend and delete move into the row's one menu
  (the "…", right-click, Shift+F10); locked ones keep their reason.
- Audit log: back-to-back successful runs of one job fold into one row
  that opens to the earlier ones; a failed run reads in the danger colour.
- DESIGN.md decision log: the E31 picks.
2026-10-01 22:51:38 +02:00
peter ab146211fa fix(scheduler,i18n): a failure a job names reads in the reader's language
The audit log showed a failed RSS poll as the server's English sentence
under a Hungarian UI. A job can now raise progress.JobFailed(reason,
message, **params): the scheduler stores the reason and params as the
run's data (dashboard, manual-run notice, audit row) beside the English
line, and the client renders scheduler.failure.<reason>. The RSS outage
names its first channel and the reason code (HTTP 404 vs unreachable).
An upstream's own error text stays as raised (diagnostic).
2026-10-01 22:51:37 +02:00
peter ec22eefd31 fix(ui): E31 S2 — the tab names the object, the channel header wraps, the player fits a short window
- The browser tab names the place and the object: PageHeading reports the
  view's own title (lib/documentTitle), so a channel, playlist or any Plex
  view reads "<object> · <module> · Siftlode" however it was opened; the
  Feed is named too; a season adds its show (tabTitle).
- The channel page's tab + action row wraps and lets the tag lane fold into
  "+N": at 1024 it ran past the edge and the page scrolled sideways.
- The YouTube player card's width is capped from the window height less the
  chrome measured under the video: at 133% zoom (685px) it cut its channel row.
- Plex certificate words (Not Rated, Approved, Passed) are named through the
  vocabulary on the info badge, show page, collection, filter and chips.
- DESIGN.md: the zoom target is 125% and 133%, not 200% (decision log).
2026-10-01 21:17:20 +02:00
peter 2543d44860 fix(messages): the welcome reads in the reader's language, not the one it was created in
E31 S1: the welcome is rendered once, at account creation, so an account that
switched language kept it in the old one (the owner's, English under a
Hungarian UI). A stored copy is recognised by its opening line (derived from
WELCOME) and served in the reader's language; other system messages stay as
stored.
2026-10-01 21:17:09 +02:00
446 changed files with 17235 additions and 4756 deletions
+1
View File
@@ -4,6 +4,7 @@
*.env.local
**/__pycache__/
**/*.pyc
**/.coverage
.venv/
venv/
# Docker patterns are anchored at the context root with NO implicit `**/` (unlike .gitignore, where
-119
View File
@@ -1,119 +0,0 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# They are declared side by side but they DO NOT run in parallel: the lab has a single act_runner
# with one slot, so they queue. Measured across runs #86-#91 — the backend job starts at the exact
# second the frontend one ends, every time, and the pair costs ~4 + ~6 = **~10 minutes from the
# push**. Anything that waits on this gate has to budget for the sum, not the max; the ops
# `release` read-back timed out on its first real run for believing otherwise.
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip.
#
# The frontend container's node major is NOT free: it must equal the prod build image's and the dev
# machine's, because vitest/tsc read the engine's own Intl and web-API implementations and those split
# between majors. `.nvmrc` is the single source of truth and
# `frontend/src/lib/nodeEngine.contract.test.ts` fails this gate if any of them drift apart.
name: gate
# Branch pushes only. A release pushes `main` and then the `v<ver>` tag, and a bare `on: [push]`
# treated the tag as a second push of the IDENTICAL commit — every release ran this whole gate
# twice, to the same verdict, for nothing (runs #81-#90 are five such pairs). `branches: ['**']`
# keeps every branch covered and drops tag refs.
#
# It has to be `branches`, NOT the `tags-ignore` this obviously wants to be. Gitea's matchPushEvent
# requires EVERY named filter to match, and each one bails on the wrong ref kind: `tags-ignore`
# starts with `if !refName.IsTag() { break }`, so a filter meant to skip tags rejects every BRANCH
# push and the gate stops running entirely. `branches` bails the same way on a tag, which is the
# direction we want. (Measured in go-gitea v1.27.3 modules/actions/workflows.go, not assumed — it
# differs from GitHub, where `tags-ignore` alone leaves branches unfiltered.)
#
# If that filter were ever wrong the gate would not run at all — which is why `siftlode release`
# now ends by reading the verdict back for the sha it pushed, and says so out loud when there is
# no run for it. A gate that silently stops running is the failure this repo already lived through.
on:
push:
branches: ["**"]
pull_request:
jobs:
frontend:
runs-on: ubuntu-latest
container: node:24-bookworm
defaults:
run:
working-directory: frontend
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run typecheck
- run: npm run lint
- run: npm run format:check
- run: npm run knip
- run: npm run test
backend:
runs-on: ubuntu-latest
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
#
# This node runs the checkout action, never our code, so it is deliberately OUTSIDE the engine
# contract the frontend job is held to — it tracks `.nvmrc` as hygiene (node 20 went EOL in April
# 2026), but a missing upstream tag here must never be able to block the gate.
container: nikolaik/python-nodejs:python3.12-nodejs24
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: siftlode
POSTGRES_PASSWORD: siftlode
POSTGRES_DB: siftlode_test
options: >-
--health-cmd "pg_isready -U siftlode -d siftlode_test"
--health-interval 2s
--health-timeout 3s
--health-retries 30
env:
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
SIFTLODE_TEST_REQUIRE_DB: "1"
steps:
- uses: actions/checkout@v4
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
# The ninth lane (E27 S3). Run from backend/ so it resolves exactly as the local lane does —
# pyrightconfig.json's include/extraPaths are relative to the config, but the baseline is
# found relative to the working directory. New errors fail; the ~200 pre-existing ones are in
# .basedpyright/baseline.json and burn down epic by epic. Matching is per (file, rule, column
# span) plus a count, so a new error of the SAME shape in a file where a sibling was just
# fixed is absorbed — burn the hot files down first.
#
# The second step is not redundant. basedpyright TIGHTENS the baseline in place when the count
# drops, with no --writebaseline, and CI throws its workspace away — so without this a
# burn-down is green here and red on the developer's `siftlode check`, which has the same
# guard. Two gates over one tree must not answer differently.
- name: basedpyright
run: python -m basedpyright --project pyrightconfig.json
working-directory: backend
- name: basedpyright baseline is committed
run: git diff --exit-code -- backend/.basedpyright/baseline.json
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
- name: pip-audit (advisory)
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
continue-on-error: true
+1
View File
@@ -9,6 +9,7 @@ __pycache__/
venv/
.mypy_cache/
.pytest_cache/
.coverage
.ruff_cache/
# Node / frontend
+15
View File
@@ -0,0 +1,15 @@
# The CVE gate's allowlist (E32). The publish gate scans the built image and the committed tree with
# Trivy and fails on any HIGH or CRITICAL finding that has a fix available. A finding that cannot be
# fixed yet is let through ONLY by an entry here, and every entry carries both:
# statement — why it is acceptable for now (what is exposed, what is not)
# expired_at — the date it stops being accepted; after it, the gate fails again on its own
# A contract test (frontend/src/lib/gates.contract.test.ts) refuses an entry without either, and
# docs/gates.md lists them.
#
# Example:
# vulnerabilities:
# - id: CVE-2026-00000
# statement: in libfoo's XML path, which nothing in the image calls; no fixed Debian build yet
# expired_at: 2026-12-31
vulnerabilities: []
secrets: []
+39
View File
@@ -7,6 +7,45 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.99.1 — 2026-10-04
### Fixed
- Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency.
## 0.99.0 — 2026-10-04
### New
- Nearly every screen is now compiled by the React Compiler, so lists and menus re-render less.
- The channel list's row menu opens and closes without redrawing the table.
### Fixed
- A failed or empty answer from YouTube or Plex no longer empties a playlist, a mirror or your Plex watch history: Siftlode keeps what it had until a complete answer arrives.
- Plex watch states un-watched on Plex now sync for each library on its own, even if another library is briefly unreachable.
- A channel with no uploads no longer fails its backfill on every run.
- “This playlist no longer exists” appears only for a list you picked yourself.
- A filter on a deleted tag matches nothing and its chip says so, and the tag cleanup keeps tags a saved view filters on.
- A channel filter names its channels from the first moment, and a playlist reopened from the cache keeps its rows.
- The bug report opens on its title, and a dialog whose form loads late gives the form the focus once.
- The Plex backdrop crossfade setting is respected even with reduced motion on.
- Smaller fixes to keyboard focus, undo and loading states.
## 0.98.0 — 2026-10-03
### New
- Search is much faster: a short or common word now answers in a fraction of a second, with the newest matches ranked by relevance.
- The feed, the channel list and the Plex pages load faster, and Plex cards fetch a picture sized to the card.
- While a new search counts its results, the count dims instead of showing the old number.
### Fixed
- A list whose next page failed to load says so and offers Retry, instead of asking again and again.
- The filter rail's tag counts follow the current search.
- Smaller fixes to layout, keyboard focus and wording.
## 0.97.0 — 2026-10-01
### New
+41 -36
View File
@@ -12,12 +12,15 @@ router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest + pip-audit + npm audit**, then a coverage report (no threshold). It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
@@ -77,37 +80,39 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE button | `ui/Button` (a `SIZES` rung) · `IconButton` · `Pressable` (a clickable card/row) · `Chip` · `DragHandle` | a raw `<button>` or `FocusSafeButton` outside the control primitives (lint-enforced) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE text field / checkbox / slider | `ui/Input` · `Textarea` · `InnerInput` in a `fieldClass` frame · `Checkbox`/`CheckRow` · `Slider` | a raw `<input>`/`<textarea>` anywhere but those primitives (lint-enforced) |
| FE glass surface | a role: `SURFACE.panel` / `card` / `menu` / `popover` / `tooltip` (`ui/surfaces.ts`) | a bare `glass`/`glass-card`/`glass-menu` beside a hand-picked radius (lint-enforced) |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+), else a latest-value ref | an excused dep list asserting the same thing in prose |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| Area | Use | Not |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE button | `ui/Button` (a `SIZES` rung) · `IconButton` · `Pressable` (a clickable card/row) · `Chip` · `DragHandle` | a raw `<button>` or `FocusSafeButton` outside the control primitives (lint-enforced) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE text field / checkbox / slider | `ui/Input` · `Textarea` · `InnerInput` in a `fieldClass` frame · `Checkbox`/`CheckRow` · `Slider` | a raw `<input>`/`<textarea>` anywhere but those primitives (lint-enforced) |
| FE glass surface | a role: `SURFACE.panel` / `card` / `menu` / `popover` / `tooltip` (`ui/surfaces.ts`) | a bare `glass`/`glass-card`/`glass-menu` beside a hand-picked radius (lint-enforced) |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
## House rules that bite (learned from real bugs)
+10 -4
View File
@@ -11,7 +11,7 @@ ARG BUILD_DATE=
# Stage 1: build the frontend SPA
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image, the CI gate's image and the dev machine must be ONE engine (see that test for why).
# this image and the dev machine the gate runs on must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
@@ -53,8 +53,11 @@ RUN adduser --disabled-password --gecos "" appuser \
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade` (E32): the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
@@ -76,11 +79,14 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls.
COPY backend/requirements.lock .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && pip install -r requirements.lock
pip install --upgrade pip && pip install -r requirements.lock && pip uninstall -y pip
COPY --chown=appuser backend/ .
# Read at runtime by config.py as the app_version (a committed single source of truth).
+1 -1
View File
@@ -1 +1 @@
0.97.0
0.99.1
-84
View File
@@ -548,64 +548,6 @@
}
}
],
"./app/plex/stream.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 51,
"endColumn": 55,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 23,
"endColumn": 25,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 18,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 75,
"endColumn": 79,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 23,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 12,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 17,
"lineCount": 1
}
}
],
"./app/plex/sync.py": [
{
"code": "reportArgumentType",
@@ -638,14 +580,6 @@
"endColumn": 89,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 17,
"endColumn": 18,
"lineCount": 1
}
}
],
"./app/plex/tmdb.py": [
@@ -1650,24 +1584,6 @@
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 46,
"endColumn": 52,
"lineCount": 1
}
}
],
"./app/youtube/rss.py": [
{
"code": "reportGeneralTypeIssues",
"range": {
"startColumn": 10,
"endColumn": 15,
"lineCount": 1
}
}
]
}
@@ -0,0 +1,83 @@
"""videos: the per-channel index covers what the channel manager and the feed sidebar read (E31 S4d)
The channel manager (`GET /api/channels`) sums each subscribed channel's videos: count, last upload, total
duration. The feed sidebar (`/feed/summary`, `/feed/years`) counts the reader's videos per channel and per
year under the default filters: available, not via explore, not a short, not live. Every one of them found
its rows through `ix_videos_channel_pub_desc` (channel_id, published_at DESC NULLS LAST, id) and then read
each video's heap page for the rest: ~270k rows at ~2.5 KB each, 681 MB on prod. That stays in cache only
while nothing else needs the memory. Measured on prod (2026-10-02): `/api/channels` 51–61 ms warm, but
2.25 s and 1.96 s on a first open in the browser and 6.4 s under EXPLAIN with 30k blocks read from disk;
`/feed/summary` and `/feed/years` ~140 ms warm, ~800 ms cold.
The same key with those columns INCLUDEd makes them Index Only Scans. On a prod-copy localdev, after a
VACUUM, the manager's sums read 0 heap pages and the summary reads the 25 MB index instead of the heap.
The visibility map is what lets them skip the heap; prod's autovacuum kept it complete (relallvisible ==
relpages). The key columns are unchanged, so every reader of the old index (the per-channel feed, the FK
cascade, the sync upsert, channel detail) rides this one. The old index is dropped.
tests/test_channel_list_index_only.py holds the queries and this column list together.
CONCURRENTLY (autocommit_block) builds it without blocking a live prod. IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent. A re-run after a build that failed part-way rebuilds
the INVALID leftover, and the old index is dropped only once the new one is valid: a re-run that kept the
leftover and dropped the old one left `videos` with no usable channel_id index (E31 S5 review).
"""
from alembic import op
from sqlalchemy import text
revision: str = "0095_videos_channel_cover_index"
down_revision: str | None = "0094_plex_genres_hu_everywhere"
branch_labels: str | None = None
depends_on: str | None = None
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def _swap(build: str, ddl: str, drop: str) -> None:
_build(build, ddl)
if not _valid(build):
raise RuntimeError(f"{build} did not build valid; {drop} is kept")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {drop}")
def upgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_cover",
"ON videos (channel_id, published_at DESC NULLS LAST, id) "
"INCLUDE (duration_seconds, unavailable_since, via_explore, is_short, live_status)",
"ix_videos_channel_pub_desc",
)
def downgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_desc",
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)",
"ix_videos_channel_pub_cover",
)
@@ -0,0 +1,86 @@
"""videos: an index per feed sort, so a sorted page stops after its rows (E31 S4e)
The feed's views, duration and title sorts had no index. Every page (the first and each next cursor)
read the whole videos heap, filtered it and kept a top-N: on prod ~350 ms a page for views/duration,
550–780 ms for title (the ICU comparison on every row), 2.3 s for page 2 when the copies of one request
queued. With an index in the exact ORDER BY (`feedquery.order_by`: the key, NULLS LAST, then id) the
planner walks it and stops after the page. On a prod-copy localdev every plain sort fell from 180–310 ms
to ~1 ms, deep pages (20) to 3–5 ms, and narrow filters (a tag, a channel, a year, nothing found) stayed
at 0.3–25 ms.
One index per direction: a backward scan of `DESC NULLS LAST` gives `ASC NULLS FIRST`, not the
`ASC NULLS LAST` the asc sorts ask for. The title pair is per reader language (`app.collation.ICU`):
an index only serves an ORDER BY in its own collation.
Cost: ~130 MB (localdev: 10 + 10 + 8 + 8 + 4 × 23). Writes stay cheap: view_count and duration are
written by the enrich, which rewrites indexed columns already (the revalidation job only touches
last_checked_at), so no HOT update is lost. A search combined with one of these sorts can now walk the
index too: a term matching ~2 % of the videos took 130–200 ms instead of 7 (still under the plain sort it
replaces); rarer and broader terms stay fast. Fencing the index off under a search made broad terms
(`m`, `the`) take 2 s, so it is left to the planner.
The title indexes are the schema's first ones on an ICU collation: an ICU upgrade in the Postgres image
reorders them, and `app.collation.heal_drift` rebuilds them at startup when it sees one.
tests/test_feed_sort_indexes.py holds each sort and its index together.
CONCURRENTLY (autocommit_block) builds them without blocking a live prod; IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent.
"""
from alembic import op
from sqlalchemy import text
revision: str = "0096_videos_sort_indexes"
down_revision: str | None = "0095_videos_channel_cover_index"
branch_labels: str | None = None
depends_on: str | None = None
# name → the indexed ORDER BY (feedquery.sort_keys + order_by: key NULLS LAST, then id).
INDEXES = {
"ix_videos_views_desc": "view_count DESC NULLS LAST, id",
"ix_videos_views_asc": "view_count ASC NULLS LAST, id",
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
"ix_videos_title_en": '(lower(title) COLLATE "en-x-icu") ASC NULLS LAST, id',
"ix_videos_title_en_desc": '(lower(title) COLLATE "en-x-icu") DESC NULLS LAST, id',
"ix_videos_title_hu": '(lower(title) COLLATE "hu-x-icu") ASC NULLS LAST, id',
"ix_videos_title_hu_desc": '(lower(title) COLLATE "hu-x-icu") DESC NULLS LAST, id',
}
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def upgrade() -> None:
with op.get_context().autocommit_block():
for name, columns in INDEXES.items():
_build(name, f"ON videos ({columns})")
def downgrade() -> None:
with op.get_context().autocommit_block():
for name in INDEXES:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
+6 -4
View File
@@ -72,7 +72,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
## The gate
Everything below must be green before a change is done (run via `siftlode check`):
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
`ruff` + `basedpyright` (backend) · the lock check + `tsc` ×3 + `eslint` + `prettier` + `knip` +
`vite build` (frontend) · `vitest` · `pytest`. The pytest lane runs in `backend/docker-compose.test.yml`
— an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the `db`
fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks
for pure-logic only with `SIFTLODE_TEST_REQUIRE_DB=0`. See `docs/development.md`, and `docs/gates.md`
for what each gate deliberately leaves out.
+1 -1
View File
@@ -219,7 +219,7 @@ class BodyLimitMiddleware:
try:
await self.app(scope, counting_receive, guarded_send)
except Exception: # noqa: BLE001 — only swallowed when WE caused it, see below
except Exception: # only swallowed when WE caused it, see below
# ClientDisconnect (or whatever a handler raises on a truncated read) is the expected
# consequence of the disconnect we just faked, and the caller must get the 413 rather
# than a 500. Anything raised when we did NOT refuse is a real failure and is re-raised
+46 -2
View File
@@ -7,17 +7,25 @@ sort names one: Hungarian puts "Ö" after "O" and "Cs" after "Cu", English folds
`collated` wraps an ORDER BY expression — and because a keyset cursor compares with the SAME expression
(`feedquery.sort_keys`), the cursor pages in the same order it sorted in. `sorted_words` orders a Python
list the same way through the database, so a list built in Python matches one built in SQL exactly."""
list the same way through the database, so a list built in Python matches one built in SQL exactly.
The feed's title sorts ride indexes built in these collations (migration 0096), and an index keeps the
order of the ICU that built it. `postgres:16-alpine` is a floating tag: an image update can bring a newer
ICU, after which those indexes disagree with the comparisons a query makes (a keyset page skips or
repeats a title). `heal_drift` rebuilds them at startup once Postgres reports the version moved."""
from typing import Any
from sqlalchemy import SQLColumnExpression, column, func, select, values
from sqlalchemy import Engine, SQLColumnExpression, column, func, select, text, values
from sqlalchemy.orm import Session
from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.types import String
from app import logs
from app.mailtext import normalize
log = logs.get_logger(__name__)
ICU: dict[str, str] = {"en": "en-x-icu", "hu": "hu-x-icu"}
@@ -37,3 +45,39 @@ def sorted_words(db: Session, words: list[str], lang: str | None) -> list[str]:
return []
w = values(column("w", String), name="words").data([(x,) for x in words])
return list(db.scalars(select(w.c.w).order_by(collated(w.c.w, lang), w.c.w)))
def heal_drift(engine: Engine) -> list[str]:
"""Rebuild every index on an ICU collation whose library version moved, then record the new version.
Returns the rebuilt indexes. CONCURRENTLY: the feed keeps reading while it runs."""
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
drifted = conn.execute(
text(
"SELECT oid, collname FROM pg_collation WHERE collname = ANY(:names) "
"AND collversion IS DISTINCT FROM pg_collation_actual_version(oid)"
),
{"names": list(ICU.values())},
).all()
if not drifted:
return []
indexes = list(
conn.execute(
text(
"SELECT DISTINCT d.objid::regclass::text FROM pg_depend d "
"JOIN pg_class c ON c.oid = d.objid AND c.relkind = 'i' "
"WHERE d.classid = 'pg_class'::regclass "
"AND d.refclassid = 'pg_collation'::regclass AND d.refobjid = ANY(:oids)"
),
{"oids": [row.oid for row in drifted]},
).scalars()
)
log.warning(
"ICU collation version changed (%s): rebuilding %d indexes",
", ".join(row.collname for row in drifted),
len(indexes),
)
for name in indexes:
conn.execute(text(f"REINDEX INDEX CONCURRENTLY {name}"))
for row in drifted:
conn.execute(text(f'ALTER COLLATION "{row.collname}" REFRESH VERSION'))
return indexes
+2 -2
View File
@@ -400,7 +400,7 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
try:
proc.wait(timeout=grace_s)
except subprocess.TimeoutExpired:
# noqa TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# `wait` call and nothing else — it would add three frames and no fact the line lacks.
log.error("%s pid %s survived SIGKILL", label, proc.pid) # noqa: TRY400
@@ -456,7 +456,7 @@ def run_ffmpeg(
try:
ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
except subprocess.TimeoutExpired: # pipes closed but the process lingers
raise StatedFailure("ffmpeg did not exit after closing its output")
raise StatedFailure("ffmpeg did not exit after closing its output") from None
except BaseException:
# ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
# function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
+2 -2
View File
@@ -57,7 +57,7 @@ def main() -> None:
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
# noqa T201: a usage line is for the person who typed the command. Logging is already
# T201: a usage line is for the person who typed the command. Logging is already
# configured here, and in a deployment it writes JSON records — the one reader of this
# line wants the bare sentence, not a record with a request id they do not have.
print( # noqa: T201
@@ -104,7 +104,7 @@ def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) ->
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
# noqa TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
# TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
log.error( # noqa: TRY400
"the download child for job %s failed",
job_id,
+100 -17
View File
@@ -12,7 +12,7 @@ import binascii
import json
from datetime import date, datetime, timedelta, timezone
from sqlalchemy import BigInteger, Select, and_, cast, false, func, or_, select
from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.collation import collated
@@ -106,15 +106,30 @@ def _tag_visible(user: User):
return or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
# The pseudo-category of a selected tag id no Tag row answers (a saved view or the feed state outlives
# a deleted tag). It filters like a category whose tags no channel carries — to nothing, in the SQL and
# in feed_summary's mirror alike — instead of the id being dropped, which showed the WHOLE feed under
# the view's old name (E32 S2). No real category is called this (language/topic/other).
MISSING_TAG_CATEGORY = "missing"
def tags_by_category(db: Session, user: User, tag_ids: list[int]) -> dict[str, list[int]]:
"""Group tag ids by their category (language/topic/other). The single source for the category split
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
two can't disagree on which selected tag belongs to which category."""
two can't disagree on which selected tag belongs to which category. An unknown id lands in
MISSING_TAG_CATEGORY — and so does another user's private tag: resolving it to its real category
told the caller which foreign ids exist (E32 review)."""
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
for tid, cat in db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids))).all():
visible = or_(Tag.user_id.is_(None), Tag.user_id == user.id)
for tid, cat in db.execute(
select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids), visible)
).all():
out.setdefault(cat, []).append(tid)
known = {tid for ids in out.values() for tid in ids}
if missing := [tid for tid in tag_ids if tid not in known]:
out[MISSING_TAG_CATEGORY] = missing
return out
@@ -265,7 +280,8 @@ def filtered_query(
# the queries that surfaced the video > description; see Video.search_vector). YouTube-like
# "fuzzy" matching: word-order-independent, multi-word AND, prefix on the word being typed,
# accent-insensitive (unaccent_simple). The channel name still matches as a substring so
# channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort.
# channel searches work. `rank_expr` (weight-aware ts_rank) drives the "relevance" sort, over the
# newest RANK_WINDOW matches only (see there).
rank_expr = None
if q:
# Resolve the channels whose NAME matches up front (there are only a few hundred channels, so
@@ -303,7 +319,7 @@ def filtered_query(
# ALL its selected tags. ⚠️ feed_summary.passes() reimplements this exact rule in Python for the
# sidebar count/facets — keep the two in lockstep (equivalence-tested by
# test_feed_summary_matches_count_plus_facets).
by_category = tags_by_category(db, tags)
by_category = tags_by_category(db, user, tags)
# Facet counting drops the category being counted so its own chips don't zero each
# other out (standard drill-down faceting: a category's count ignores its own
# selections but still honours the other categories' filters).
@@ -394,7 +410,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
return {"count": 0, "facets": {}}
# channel -> its visible tag ids, plus each tag's category (for grouping the facet output).
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
visible = _tag_visible(user)
channel_tags: dict[str, set[int]] = {}
tag_category: dict[int, str] = {}
for cid, tid, cat in db.execute(
@@ -409,7 +425,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
# base channel carries still filters everything out for its category — matching the SQL `IN (empty)`
# behaviour).
selected_by_cat: dict[str, set[int]] = {
cat: set(ids) for cat, ids in tags_by_category(db, params.get("tags") or []).items()
cat: set(ids) for cat, ids in tags_by_category(db, user, params.get("tags") or []).items()
}
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
@@ -437,7 +453,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
# Drop this category's own selections so its chips keep independent counts; a conjunctive
# topic-AND keeps them (each remaining chip narrows to channels that ALSO have the selected).
exclude = None if tag_modes.get(category, "or") == "and" else category
for cid, ch_tags in channel_tags.items():
for ch_tags in channel_tags.values():
if not passes(ch_tags, exclude):
continue
for tid in ch_tags:
@@ -543,6 +559,54 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
return table.get(sort) or table["newest"]
# Relevance ranks only the newest RANK_WINDOW matches (E31): ts_rank reads every match's tsvector, and
# a broad word matches most of the library ("a" ranked 193k rows in 6.8 s on prod, against 1.1 s to just
# find them). The rows past the window keep rank 0, so they follow the ranked ones newest first; a word
# that broad has no meaningful relevance order anyway, and a specific one fits the window whole.
RANK_WINDOW = 2000
def rank_window_start(db: Session, query: Select) -> datetime | None:
"""The published_at the ranked window starts at: the RANK_WINDOW-th newest match, None when fewer
match (rank them all). The newest-first walk stops after RANK_WINDOW matches (40 ms for "a")."""
return db.scalar(
query.with_only_columns(Video.published_at)
.order_by(Video.published_at.desc())
.offset(RANK_WINDOW - 1)
.limit(1)
)
def windowed_rank(rank_expr, since: datetime | None):
"""`rank_expr` inside the window, 0 past it. CASE evaluates ts_rank only for the rows inside."""
if since is None:
return rank_expr
return case((Video.published_at >= since, rank_expr), else_=0)
def windowed_relevance_rows(
db: Session, query: Select, keys: list[dict], window: datetime, cursor_id: str | None, n: int
) -> list:
"""Up to `n` rows of the windowed relevance order, read as its two halves: the window ranked, then
the rest newest first (every window row sorts before every row past it). As one ORDER BY the
planner computes the CASE for every match and sorts them all (2.7 s for "a" on prod); split, each
half rides an index (80 ms + 5 ms). With a cursor, `keys` carry its values (keyset_predicate)."""
pub_key = keys[1] # relevance: [rank, published_at]
rows: list = []
if cursor_id is None or pub_key["value"] >= window:
inside = query.where(Video.published_at >= window).order_by(*order_by(keys))
if cursor_id is not None:
inside = inside.where(keyset_predicate(keys, cursor_id))
rows = list(db.execute(inside.limit(n)).all())
if len(rows) < n:
newest = [pub_key] # past the window every rank is 0: the order is newest first
rest = query.where(Video.published_at < window).order_by(*order_by(newest))
if cursor_id is not None:
rest = rest.where(keyset_predicate(newest, cursor_id))
rows += db.execute(rest.limit(n - len(rows))).all()
return rows
def order_by(keys: list[dict]):
cols = []
for k in keys:
@@ -589,26 +653,45 @@ def keyset_predicate(keys: list[dict], cursor_id: str):
return rec(0)
def encode_cursor(keys: list[dict], row) -> str:
def encode_cursor(keys: list[dict], row, window: datetime | None = None) -> str:
"""`window` is the relevance window's start: the cursor carries it, so every page ranks against the
window page one did. Recomputed per page, a video ingested mid-scroll would push the window's oldest
row to rank 0, behind the cursor, and the reader would see it twice."""
vals = []
for i, k in enumerate(keys):
v = getattr(row, f"_k{i}")
vals.append(v.isoformat() if (v is not None and k["kind"] == "dt") else v)
payload = json.dumps({"k": vals, "id": row.id})
body: dict = {"k": vals, "id": row.id}
if window is not None:
body["w"] = window.isoformat()
payload = json.dumps(body)
return base64.urlsafe_b64encode(payload.encode()).decode()
def decode_cursor(keys: list[dict], cursor: str) -> tuple[list, str]:
def _cursor_dt(v):
"""A cursor's timestamp: aware or refused. The server writes only aware ones (timestamptz), and
windowed_relevance_rows compares them in Python, where a naive one beside an aware one raises — a
500 for a hand-made cursor instead of the 400 (E31 review 2)."""
if v is None:
return None
dt = datetime.fromisoformat(v)
if dt.tzinfo is None:
raise ValueError("naive cursor timestamp")
return dt
def decode_cursor(keys: list[dict], cursor: str) -> tuple[list, str, datetime | None]:
try:
payload = json.loads(base64.urlsafe_b64decode(cursor.encode()))
raw = payload["k"]
if len(raw) != len(keys):
raise ValueError("key arity mismatch")
vals = [
datetime.fromisoformat(v) if (v is not None and k["kind"] == "dt") else v
for k, v in zip(keys, raw)
]
return vals, payload["id"]
vals = [_cursor_dt(v) if k["kind"] == "dt" else v for k, v in zip(keys, raw, strict=True)]
for k, v in zip(keys, vals, strict=True):
if v is None and not k["nullable"]:
raise ValueError("null value for a non-nullable key")
window = payload.get("w")
return vals, payload["id"], _cursor_dt(window) if window else None
except (binascii.Error, json.JSONDecodeError, KeyError, ValueError, TypeError) as exc:
# Framework-free: the route handler translates this into an HTTP 400 so the
# read-model stays web-layer-agnostic (see routes/feed.get_feed).
+1 -1
View File
@@ -239,7 +239,7 @@ def deliver(
# already open on the tracker and this is what would have recorded its link. We
# still must not write — another send owns the row and is opening its own issue —
# so say which issue was left behind, because nothing else will.
# noqa TRY400: the traceback is one SendClaimLost carrying the session id and
# TRY400: the traceback is one SendClaimLost carrying the session id and
# nothing else; the issue number below is the whole news.
log.error( # noqa: TRY400
"an issue was created by a send that had already lost its claim",
+1 -1
View File
@@ -166,7 +166,7 @@ def bind(**fields: Any):
try:
yield
finally:
for key, token in zip(keys, tokens):
for key, token in zip(keys, tokens, strict=True):
_CONTEXT_VARS[key].reset(token)
+37 -25
View File
@@ -1,37 +1,36 @@
import asyncio
import math
import mimetypes
from contextlib import asynccontextmanager, suppress
from pathlib import Path
from fastapi import FastAPI, HTTPException, Request
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
# Logging is configured HERE, before the route modules are imported, so every logger they create at
# import time is already covered. ONE format for the api, the worker, the download child and alembic
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
from app import logs
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logs.get_logger(__name__)
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import bodylimit
from app import logsink
from app import state
from app import sysconfig
from app.config import settings
from app.db import SessionLocal
from app.plex import stream as plex_stream
from app.plex import subtitles as plex_subtitles
from app.routes import (
# Logging is configured HERE, before the app's own modules are imported, so a line one of them logs
# WHILE importing already has the format (a logger is configured whenever it is created —
# app/logs.py — but a line emitted before `configure_logging` goes out raw). ONE format for the api,
# the worker, the download child and alembic alike — this replaces the old uvicorn --log-config +
# the hand-rolled fallback that used to live here, and uvicorn's access log is replaced by the
# `http.request` middleware below. The imports after it are E402 on purpose, each one marked.
from app import logs
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logs.get_logger(__name__)
from app import bodylimit, collation, logsink, state, sysconfig # noqa: E402 — logging is configured first
from app.config import settings # noqa: E402 — logging is configured first
from app.db import SessionLocal, engine # noqa: E402 — logging is configured first
from app.plex import stream as plex_stream # noqa: E402 — logging is configured first
from app.plex import subtitles as plex_subtitles # noqa: E402 — logging is configured first
from app.routes import ( # noqa: E402 — logging is configured first
admin,
admin_logs as admin_logs_routes,
audit as audit_routes,
@@ -59,7 +58,7 @@ from app.routes import (
tags,
version,
)
from app.scheduler import shutdown_scheduler, start_scheduler
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
@asynccontextmanager
@@ -76,7 +75,7 @@ async def lifespan(app: FastAPI):
# booting — an unreachable or not-yet-migrated database has to start into the env level.
try:
set_level(sysconfig.get_str(db, "log_level"))
except Exception: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
except Exception: # any DB/config failure keeps the bootstrap level
# With the traceback: this catch spans a connection, a query and a coercion at boot,
# and `%s` of the exception does not say which of them raised.
log.exception("could not read the stored log level; staying at %s", current_level())
@@ -97,6 +96,9 @@ async def lifespan(app: FastAPI):
# owns, so racing it against early playback is safe.
sweep = asyncio.create_task(_sweep_hls_orphans())
reaper = asyncio.create_task(_reap_hls_sessions())
# Fire and forget as well: a rebuild after an ICU upgrade takes seconds, and the title sorts serve
# meanwhile (CONCURRENTLY) in the old index's order, as they did before the restart.
heal = asyncio.create_task(_heal_collation_drift())
start_scheduler()
try:
yield
@@ -106,7 +108,7 @@ async def lifespan(app: FastAPI):
# pending ("Task was destroyed but it is pending!" — noise that reads like a bug during
# triage). Cancelling does NOT interrupt the rmtree thread itself; this only makes our own
# bookkeeping honest.
for task in (sweep, reaper):
for task in (sweep, reaper, heal):
task.cancel()
with suppress(asyncio.CancelledError):
await task
@@ -139,6 +141,18 @@ async def _reap_hls_sessions() -> None:
log.exception("HLS session reap failed")
async def _heal_collation_drift() -> None:
"""Background startup check of the ICU collation the title-sort indexes are built in (see
`collation.heal_drift`). A failure is logged, never fatal: the sorts still answer, only possibly in
the old order."""
try:
await asyncio.to_thread(collation.heal_drift, engine)
except asyncio.CancelledError:
raise
except Exception:
log.exception("ICU collation drift check failed")
async def _sweep_hls_orphans() -> None:
"""Background half of the startup HLS sweep (see lifespan). Owns its own errors: a failure here
must never take the app down, and a cancelled shutdown is not one."""
@@ -316,8 +330,6 @@ app.include_router(setup_routes.router)
# Ensure modern image types resolve to the right Content-Type when FileResponse guesses from the
# filename (the runtime's mimetypes db doesn't always know .webp → it'd fall back to octet-stream).
import mimetypes
mimetypes.add_type("image/webp", ".webp")
mimetypes.add_type("image/avif", ".avif")
+10 -7
View File
@@ -23,6 +23,7 @@ from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.utils import canonical_email
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
# columns below are declared with it rather than with a literal: a name a person typed goes into
@@ -51,9 +52,6 @@ def display_text(raw: str | None) -> str | None:
return (raw or "").strip()[:DISPLAY_TEXT_MAX] or None
from app.utils import canonical_email
# The resume-position columns (VideoState / PlexState.position_seconds) are int4. The checkpoint
# routes bound their bodies by this, so an out-of-range position is a 422 rather than a 500.
POSITION_MAX_S = 2**31 - 1
@@ -330,10 +328,12 @@ class Video(Base, TimestampMixin):
__tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_desc`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0067 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) AND the per-channel feed
# sort. Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
# No standalone channel_id index: the composite `ix_videos_channel_pub_cover`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0095 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) and the per-channel feed
# sort. Its INCLUDEd columns make the channel manager's sums and the feed sidebar's counts Index Only
# Scans (no heap): a column those queries start reading belongs in it (test_channel_list_index_only).
# Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
@@ -347,6 +347,9 @@ class Video(Base, TimestampMixin):
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
# `feedquery.order_by` by test_feed_sort_indexes.
duration_seconds: Mapped[int | None] = mapped_column(Integer)
view_count: Mapped[int | None] = mapped_column(BigInteger)
like_count: Mapped[int | None] = mapped_column(BigInteger)
+14 -5
View File
@@ -78,9 +78,10 @@ class PlexClient:
# --- Catalog mirror (P1) ---
def section_items(
self, key: str, item_type: int, start: int = 0, size: int = 200
) -> tuple[list[dict], int]:
) -> tuple[list[dict], int | None]:
"""A page of a section's top-level items. type 1 = movie, 2 = show. Returns
(items, total_size)."""
(items, total_size); the total is None when the answer carries none, so a container that
says "0" (a genuinely empty section) stays distinguishable from one that says nothing."""
mc = self._get(
f"/library/sections/{key}/all",
params={
@@ -93,7 +94,10 @@ class PlexClient:
"X-Plex-Container-Size": size,
},
)
return mc.get("Metadata", []) or [], int(mc.get("totalSize", mc.get("size", 0)) or 0)
# `totalSize` only: `size` is THIS page's count, and read as the total it made a single
# page look like the whole section — "complete" to a prune or a reconcile (E32 review).
total = mc.get("totalSize")
return mc.get("Metadata", []) or [], None if total is None else int(total)
def children(self, rating_key: str) -> list[dict]:
"""Direct children (show → seasons, season → episodes)."""
@@ -136,9 +140,14 @@ class PlexClient:
# --- Collection editing (P2 — write-back to Plex; all shared/global, admin-gated in the routes) ---
@property
def machine_id(self) -> str:
"""Server machineIdentifier — needed in the `uri` of collection add/create calls. Cached."""
"""Server machineIdentifier — needed in the `uri` of collection add/create calls. Cached.
Without one the uri would read `server:///…`, a malformed write Plex answers in its own way;
it is refused here, by name (E32 S2)."""
if not getattr(self, "_machine_id", None):
self._machine_id = self._get("/identity").get("machineIdentifier") or ""
mid = self._get("/identity").get("machineIdentifier")
if not mid:
raise PlexError("Plex did not report its machine identifier")
self._machine_id = mid
return self._machine_id
def _write(self, method: str, path: str) -> dict:
+17 -3
View File
@@ -23,9 +23,23 @@ log = logs.get_logger(__name__)
_EXT_CT = [("jpg", "image/jpeg"), ("png", "image/png"), ("webp", "image/webp")]
_CT_EXT = {ct: ext for ext, ct in _EXT_CT}
# Target box per variant — Plex resizes to these server-side. Posters fill ≤176px cells (grid/info/
# strips), so 400px covers 2× DPR; the faint backdrop art only needs ~1280px. Keeps images ~tens of KB.
SIZES = {"thumb": (400, 600), "art": (1280, 720)}
# Target box per variant — Plex resizes to these server-side. The full poster (400px) covers a detail
# page's ≤224px poster at 2× DPR, the full art (1280px) a hero backdrop. A card is much smaller: a grid
# poster is ~144px and a shelf's landscape art ~212px, so the `_sm` pair serves those at 1.33× DPR
# (E31 S4e: cards fetched the 400 poster, 42 KB, and the 1280 art for a 212px picture). The browser
# picks between a pair itself (`srcset`, built by the frontend's `lib/plexImage` cardImage, whose
# widths plexImage.contract.test holds to these).
SIZES = {
"thumb": (400, 600),
"thumb_sm": (240, 360),
"art": (1280, 720),
"art_sm": (480, 270),
}
def picture(variant: str) -> str:
"""Which stored picture a size variant is cut from: the poster (`thumb`) or the backdrop (`art`)."""
return "art" if variant.startswith("art") else "thumb"
def key_for(db: Session, rating_key: str, variant: str) -> str | None:
+150
View File
@@ -0,0 +1,150 @@
"""The HU overlay: a Hungarian Plex section written onto its English twin's rows (E30), split out of
`plex/sync.py` (E32 S2), which runs it once per paired section during the full sync.
The pairing is admin-configured (`plex_overlay_map`, ``huKey:enKey,...``). An overlay section creates
no browsable rows of its own; it only fills the `*_hu` columns of the EN rows it matches by guid.
`plex/sync.py` and this module import each other as MODULES (attribute access at call time), which is
what lets either be imported first.
"""
from collections.abc import Iterable
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex import sync as plex_sync
from app.plex.client import PlexClient
from app.plex.genres import hu_genres
log = logs.get_logger(__name__)
def overlay_map(db: Session) -> dict[str, str]:
"""The configured HU-overlay pairing: {huSectionKey: enSectionKey} from `plex_overlay_map`
('huKey:enKey,...'). Each HU section is synced as a non-browsable overlay onto its EN twin.
Malformed pairs are skipped."""
raw = sysconfig.get_str(db, "plex_overlay_map")
out: dict[str, str] = {}
for pair in raw.split(","):
hu, sep, en = pair.partition(":")
hu, en = hu.strip(), en.strip()
if sep and hu and en:
out[hu] = en
return out
def sync_overlay(
db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict
) -> None:
"""Write an overlay section's Hungarian metadata onto its EN twin's rows, matched by guid. Creates
NO browsable rows — an overlay library is never listed on its own. Only title/summary/tagline/genres
differ by language; the language-neutral people_text feeds both search vectors. A show section's
episodes are overlaid too (E30): the HU section lists every episode with its Hungarian title and
summary, and the episode rows live in the EN show library.
This is a full RECONCILE: after a COMPLETE HU listing every guid-bearing EN row's HU fields are
cleared, then the matches are repopulated — so an EN row whose HU twin vanished (or was re-guid'd)
falls back to EN instead of keeping stale Hungarian metadata. A listing cut short only adds. HU
items with no EN match are skipped."""
if lib.kind == "movie":
passes = [
(
plex_sync._TYPE_MOVIE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "movie",
PlexItem.guid.is_not(None),
),
)
]
else:
passes = [
(
plex_sync._TYPE_SHOW,
select(PlexShow).where(
PlexShow.library_id == en_lib_id, PlexShow.guid.is_not(None)
),
),
(
plex_sync._TYPE_EPISODE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "episode",
PlexItem.guid.is_not(None),
),
),
]
for item_type, rows_q in passes:
_overlay_pass(db, plex, lib, item_type, db.execute(rows_q).scalars(), stats)
db.flush()
def _overlay_pass(
db: Session,
plex: PlexClient,
lib: PlexLibrary,
item_type: int,
rows: Iterable[PlexItem] | Iterable[PlexShow],
stats: dict,
) -> None:
"""One kind of the overlay (movies, shows or episodes) — see `sync_overlay`."""
# guid -> the EN rows carrying it (a list: two editions of one film can share a guid; both overlay).
by_guid: dict[str, list[PlexItem | PlexShow]] = {}
for r in rows:
if r.guid: # the passes select guid-bearing rows only
by_guid.setdefault(r.guid, []).append(r)
# The HU listing is read first and applied after, so the clear can wait for its verdict.
hu_by_guid, complete = _read_hu_listing(plex, lib, item_type, by_guid)
episodes = item_type == plex_sync._TYPE_EPISODE
if complete:
_clear_hu(by_guid, episodes)
else:
stats["overlay_incomplete"] = stats.get("overlay_incomplete", 0) + 1
log.warning(
"the HU listing ended before the server's total; existing HU text is kept",
extra=logs.event("plex.sync.overlay_incomplete", library=lib.plex_key),
)
for g, meta in hu_by_guid.items():
targets = by_guid[g]
title_hu = (meta.get("title") or "").strip() or None
summary_hu = meta.get("summary")
tagline_hu = (meta.get("tagline") or "").strip() or None
genres_hu = hu_genres(plex_sync._tags(meta, "Genre"))
for row in targets:
row.title_hu = title_hu
row.summary_hu = summary_hu
if not episodes:
row.tagline_hu = tagline_hu
row.genres_hu = genres_hu or row.genres_hu
stats["overlaid"] = stats.get("overlaid", 0) + 1
def _read_hu_listing(
plex: PlexClient, lib: PlexLibrary, item_type: int, by_guid: dict
) -> tuple[dict[str, dict], bool]:
"""The HU section's entries for the guids the EN side carries, and whether the walk reached the
server's total. The first HU entry for a guid wins; a dup HU item can't overwrite or double-count."""
progress: dict = {}
hu_by_guid: dict[str, dict] = {}
for meta in plex_sync._paginate(plex, lib.plex_key, item_type, progress):
g = plex_sync._canonical_guid(meta)
if g and g in by_guid and g not in hu_by_guid:
hu_by_guid[g] = meta
return hu_by_guid, bool(progress.get("complete"))
def _clear_hu(by_guid: dict, episodes: bool) -> None:
"""Reconcile step: clear HU fields on all guid-bearing rows; the matches are re-set after. Only
rows that actually held HU data become dirty (SQLAlchemy skips a no-net-change UPDATE), so this is
cheap. The genres are the exception: every row gets Hungarian ones, from its HU twin or, lacking
one, from its own English list — otherwise the HU genre list mixed "Drama" (no twin) with "Dráma".
Only after a COMPLETE listing: one a Plex hiccup cut short would clear the HU text of every title
it failed to reach, until the next full sync (E32 S2, the watch reconcile's twin)."""
for group in by_guid.values():
for row in group:
row.title_hu = row.summary_hu = None
if not episodes:
row.tagline_hu = None
row.genres_hu = hu_genres(row.genres)
+11 -2
View File
@@ -245,6 +245,9 @@ def _probe_video_kind(src: Path) -> str:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
except (subprocess.SubprocessError, OSError) as exc:
# The safe kind is a real answer, but it is the answer for an 8-bit SDR source — a blind
@@ -502,6 +505,9 @@ def _probe_audio_langs(src: Path) -> list[str]:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
return [ln.strip() for ln in out.stdout.splitlines()]
except (subprocess.SubprocessError, OSError) as exc:
@@ -615,6 +621,9 @@ def _probe_first_pts(seg: Path) -> float | None:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
for line in out.stdout.splitlines():
tok = line.strip().strip(",")
@@ -736,8 +745,7 @@ def start_session(
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
# whoever is registered now). The INITIAL spawn passes replace_only=None and always supersedes —
# that is the seek-restart's intended last-writer-wins.
# whoever is registered now). The INITIAL spawn never gets None — see its call site.
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
@@ -814,6 +822,7 @@ def start_session(
return s
s = _spawn(*_select_encode(db, item, transcode, src))
assert s is not None # replace_only=None always supersedes: a seek-restart's last-writer-wins
# Wait for the first video segment outside the lock (this blocks on ffmpeg, and we must not hold up
# other sessions). Pass the proc so a HW session that dies at VAAPI init fails fast (not after 25s).
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
+2 -1
View File
@@ -162,7 +162,8 @@ def ffmpeg_to_vtt(src: str, sub_ord: int | None) -> str:
"webvtt",
"pipe:1",
]
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S)
# check=False: the exit status is read on the next line, with ffmpeg's own stderr as the reason.
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S, check=False)
if r.returncode != 0 or not r.stdout:
raise PlexError(
(r.stderr or b"").decode("utf-8", "replace")[:200] or "ffmpeg subtitle convert failed"
+58 -123
View File
@@ -15,7 +15,6 @@ dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are
item-detail/player path (P2), not here.
"""
from collections.abc import Iterable
from datetime import datetime, timezone
from sqlalchemy import func, select
@@ -23,7 +22,7 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
from app.plex import detail, people
from app.plex import detail, overlay, people
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.genres import hu_genres
from app.plex.lookups import by_rating_key
@@ -194,20 +193,6 @@ def _enabled_section_keys(db: Session) -> set[str] | None:
return keys or None
def _overlay_map(db: Session) -> dict[str, str]:
"""The configured HU-overlay pairing: {huSectionKey: enSectionKey} from `plex_overlay_map`
('huKey:enKey,...'). Each HU section is synced as a non-browsable overlay onto its EN twin.
Malformed pairs are skipped."""
raw = sysconfig.get_str(db, "plex_overlay_map")
out: dict[str, str] = {}
for pair in raw.split(","):
hu, sep, en = pair.partition(":")
hu, en = hu.strip(), en.strip()
if sep and hu and en:
out[hu] = en
return out
def sync(db: Session) -> dict:
"""Full reconcile of the enabled movie/show sections. Returns a small stats dict."""
if not sysconfig.get_bool(db, "plex_enabled"):
@@ -223,7 +208,7 @@ def sync(db: Session) -> dict:
"pruned": 0,
}
wanted = _enabled_section_keys(db)
overlay_map = _overlay_map(db) # {huKey: enKey}
overlay_map = overlay.overlay_map(db) # {huKey: enKey}
overlay_keys = set(overlay_map)
try:
with PlexClient(db) as plex:
@@ -267,13 +252,13 @@ def sync(db: Session) -> dict:
lib.metadata_lang = "hu"
lib.overlay_of = en_lib.id
db.flush()
_sync_overlay(db, plex, lib, en_lib.id, stats)
overlay.sync_overlay(db, plex, lib, en_lib.id, stats)
db.commit()
# Rich-fetch a bounded batch of not-yet-enriched titles' FULL cast, so the whole library
# becomes filterable over successive runs (the cheap listing above only stored the top ~3).
try:
stats["people_enriched"] = _enrich_people_batch(db, plex)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
except Exception: # best-effort; never fail the sync over a backfill
# With the traceback: a blind catch over a batch that fetches AND writes can be
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
log.exception("the Plex people-enrich batch failed")
@@ -282,7 +267,7 @@ def sync(db: Session) -> dict:
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
except Exception: # best-effort; never fail the sync over a backfill
log.exception("the Plex media-probe batch failed")
# silent-ok: the return value IS the report. `{"skipped": …}` goes to the scheduler, which
# logs the job result, and to the admin route, which answers it to the caller.
@@ -296,7 +281,7 @@ def sync(db: Session) -> dict:
# appear in ANY item's cast/directors — drop those rows, mirroring how Plex drops orphaned metadata.
try:
stats["people_pruned"] = people.prune_orphan_people(db)
except Exception: # noqa: BLE001 — best-effort housekeeping; never fail the sync over it
except Exception: # best-effort housekeeping; never fail the sync over it
log.exception("the Plex people prune failed")
log.info("Plex sync done: %s", stats)
return stats
@@ -433,7 +418,10 @@ def _paginate(plex: PlexClient, key: str, item_type: int, progress: dict | None
if progress is not None:
# A truthy `total` we reached = a trustworthy complete listing. `total` unknown/zero-with-rows
# (a Plex that doesn't report totalSize) is treated as NOT complete, so prune stays conservative.
# `total` itself is kept for a caller to whom a server that SAID 0 is news it can trust (the
# watch reconcile); an empty mirror or overlay must not read it so (E32 S2).
progress["complete"] = bool(total) and start >= total
progress["total"] = total
# The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this
@@ -661,98 +649,6 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
_prune_stale(db, shows, shows_seen, stats, complete=shows_progress.get("complete", False))
def _sync_overlay(
db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict
) -> None:
"""Write an overlay section's Hungarian metadata onto its EN twin's rows, matched by guid. Creates
NO browsable rows — an overlay library is never listed on its own. Only title/summary/tagline/genres
differ by language; the language-neutral people_text feeds both search vectors. A show section's
episodes are overlaid too (E30): the HU section lists every episode with its Hungarian title and
summary, and the episode rows live in the EN show library.
This is a full RECONCILE: every guid-bearing EN row's HU fields are cleared first, then the HU
section repopulates the matches — so an EN row whose HU twin vanished (or was re-guid'd) falls back
to EN instead of keeping stale Hungarian metadata. HU items with no EN match are skipped."""
if lib.kind == "movie":
passes = [
(
_TYPE_MOVIE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "movie",
PlexItem.guid.is_not(None),
),
)
]
else:
passes = [
(
_TYPE_SHOW,
select(PlexShow).where(
PlexShow.library_id == en_lib_id, PlexShow.guid.is_not(None)
),
),
(
_TYPE_EPISODE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "episode",
PlexItem.guid.is_not(None),
),
),
]
for item_type, rows_q in passes:
_overlay_pass(db, plex, lib, item_type, db.execute(rows_q).scalars(), stats)
db.flush()
def _overlay_pass(
db: Session,
plex: PlexClient,
lib: PlexLibrary,
item_type: int,
rows: Iterable[PlexItem] | Iterable[PlexShow],
stats: dict,
) -> None:
"""One kind of the overlay (movies, shows or episodes) — see `_sync_overlay`."""
# guid -> the EN rows carrying it (a list: two editions of one film can share a guid; both overlay).
by_guid: dict[str, list[PlexItem | PlexShow]] = {}
for r in rows:
if r.guid: # the passes select guid-bearing rows only
by_guid.setdefault(r.guid, []).append(r)
# Reconcile step: clear HU fields on all guid-bearing rows. Only rows that actually held HU data
# become dirty (SQLAlchemy skips a no-net-change UPDATE), so this is cheap; matches are re-set below.
# The genres are the exception: every row gets Hungarian ones, from its HU twin or, lacking one, from
# its own English list — otherwise the HU genre list mixed "Drama" (no twin) with "Dráma".
episodes = item_type == _TYPE_EPISODE
for group in by_guid.values():
for row in group:
row.title_hu = row.summary_hu = None
if not episodes:
row.tagline_hu = None
row.genres_hu = hu_genres(row.genres)
seen: set[str] = (
set()
) # first HU entry for a guid wins; a dup HU item can't overwrite or double-count
for meta in _paginate(plex, lib.plex_key, item_type):
g = _canonical_guid(meta)
targets = by_guid.get(g) if g else None
if not targets or g in seen:
continue
seen.add(g)
title_hu = (meta.get("title") or "").strip() or None
summary_hu = meta.get("summary")
tagline_hu = (meta.get("tagline") or "").strip() or None
genres_hu = hu_genres(_tags(meta, "Genre"))
for row in targets:
row.title_hu = title_hu
row.summary_hu = summary_hu
if not episodes:
row.tagline_hu = tagline_hu
row.genres_hu = genres_hu or row.genres_hu
stats["overlaid"] = stats.get("overlaid", 0) + 1
def _upsert_collection(
db: Session, existing: PlexCollection | None, rk: str, lib_id: int, meta: dict
) -> PlexCollection:
@@ -778,9 +674,11 @@ def resync_collection(db: Session, plex: PlexClient, lib: PlexLibrary, rk: str)
meta = plex.metadata(rk) or {}
col = _upsert_collection(db, by_rating_key(db, PlexCollection, rk), rk, lib.id, meta)
Model = PlexItem if lib.kind == "movie" else PlexShow
new_members = {
str(c.get("ratingKey")) for c in plex.collection_children(rk) if c.get("ratingKey")
}
children = _believed_children(plex, rk, meta)
if children is None:
db.flush()
return col
new_members = {str(c.get("ratingKey")) for c in children if c.get("ratingKey")}
current = db.execute(select(Model).where(Model.collection_keys.contains([rk]))).scalars().all()
for row in current: # drop rk from rows that are no longer members
if row.rating_key not in new_members:
@@ -793,6 +691,22 @@ def resync_collection(db: Session, plex: PlexClient, lib: PlexLibrary, rk: str)
return col
def _believed_children(plex: PlexClient, rk: str, meta: dict) -> list[dict] | None:
"""A collection's members, or None when the answer is not believed: no members from a collection
whose own listing counts some (`childCount`) is a glitch, not a collection emptied — the same
empty-answer rule as the collection listing's (E32 review). Rebuilding on it showed the
collection empty until the next sync."""
children = plex.collection_children(rk)
count = str(meta.get("childCount") or "0")
if children or not count.isdigit() or int(count) == 0:
return children
log.warning(
"Plex listed no members for a non-empty collection; keeping its membership",
extra=logs.event("plex.sync.collection_children_empty", collection=rk),
)
return None
def delete_collection_local(db: Session, lib: PlexLibrary, rk: str) -> None:
"""Local cleanup after a collection is deleted from Plex: strip rk from members + drop the row."""
Model = PlexItem if lib.kind == "movie" else PlexShow
@@ -804,6 +718,19 @@ def delete_collection_local(db: Session, lib: PlexLibrary, rk: str) -> None:
db.flush()
def _rebuild_membership(
db: Session, lib: PlexLibrary, membership: dict[str, list[str]], kept: set[str]
) -> None:
"""Rebuild collection_keys on every member row of this library (clear + set), carrying over the
keys of a collection whose empty answer was not believed (`_believed_children`)."""
for Model in (PlexItem, PlexShow):
for row in db.execute(select(Model).where(Model.library_id == lib.id)).scalars():
held = [k for k in (row.collection_keys or []) if k in kept]
keys = [*membership.get(row.rating_key, []), *held]
row.collection_keys = list(dict.fromkeys(keys)) or None
db.flush()
def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None:
"""Mirror a library's collections + membership. Reads never touch Plex afterwards. Membership is
written as `collection_keys` on member movies (plex_items) / shows (plex_shows). A full reconcile:
@@ -815,7 +742,17 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
select(PlexCollection).where(PlexCollection.library_id == lib.id)
).scalars()
}
if not cols and existing:
# The catalog's own rule (see the module doc): an empty listing while we hold rows is a Plex
# glitch, not a library emptied of collections. Reconciling on it dropped every collection
# (its user-set `editable` flag with it) and every member's keys (E32 S2).
log.warning(
"Plex listed no collections for a library that has some; keeping them this run",
extra=logs.event("plex.sync.collections_empty", library=lib.plex_key),
)
return
seen: set[str] = set()
kept: set[str] = set() # collections whose empty children answer is not believed
membership: dict[str, list[str]] = {} # member rating_key -> [collection rating_key, ...]
for meta in cols:
rk = str(meta.get("ratingKey") or "")
@@ -823,7 +760,10 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
continue
_upsert_collection(db, existing.get(rk), rk, lib.id, meta)
seen.add(rk)
for child in plex.collection_children(rk):
children = _believed_children(plex, rk, meta)
if children is None:
kept.add(rk)
for child in children or []:
crk = str(child.get("ratingKey") or "")
if crk:
membership.setdefault(crk, []).append(rk)
@@ -832,9 +772,4 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
for rk, col in existing.items():
if rk not in seen:
db.delete(col)
# Rebuild collection_keys on every member row of this library (clear + set).
for row in db.execute(select(PlexItem).where(PlexItem.library_id == lib.id)).scalars():
row.collection_keys = membership.get(row.rating_key) or None
for row in db.execute(select(PlexShow).where(PlexShow.library_id == lib.id)).scalars():
row.collection_keys = membership.get(row.rating_key) or None
db.flush()
_rebuild_membership(db, lib, membership, kept)
+69 -20
View File
@@ -29,7 +29,7 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import PlexItem, PlexLink, PlexState
from app.models import PlexItem, PlexLibrary, PlexLink, PlexState
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.sync import (
_TYPE_EPISODE,
@@ -72,13 +72,27 @@ def _rk_to_id(db: Session) -> dict[str, int]:
def _scan_plex_states(
db: Session, plex: PlexClient
) -> tuple[dict[int, tuple[str, int, datetime | None, datetime | None]], int]:
) -> tuple[dict[int, tuple[str, int, datetime | None, datetime | None]], int, set[str], set[str]]:
"""Scan the enabled movie/show sections and return ``{item_id: (status, pos, watched_at,
prog_at)}`` for every mirrored leaf Plex holds a watch signal for, plus the scanned-leaf count.
Shared by the one-time import and the full reconcile (both need Plex's whole current picture)."""
prog_at)}`` for every mirrored leaf Plex holds a watch signal for, the scanned-leaf count, the
keys of the sections SETTLED — walked to the server's total (or said empty by a server we mirror
nothing from) — and the keys of every section walked.
Shared by the one-time import and the full reconcile (both need Plex's whole current picture).
The settled keys matter to the reconcile alone: it reads "absent from the scan" as "Plex
un-watched it", which a walk cut short by a Plex hiccup does not mean (see `_paginate`). Settled
is decided PER SECTION: one section that never settles (a library emptied on Plex while its
mirrored items are kept) must not stop un-watches syncing in all the others (E32 review)."""
item_id_by_rk = _rk_to_id(db)
wanted = _enabled_section_keys(db)
# The sections we mirror something from: a total of 0 from one of those contradicts the mirror.
mirrored = set(
db.execute(
select(PlexLibrary.plex_key).join(PlexItem, PlexItem.library_id == PlexLibrary.id)
).scalars()
)
scanned = 0
settled: set[str] = set()
walked: set[str] = set()
out: dict[int, tuple[str, int, datetime | None, datetime | None]] = {}
for s in plex.sections():
stype = s.get("type")
@@ -88,7 +102,10 @@ def _scan_plex_states(
if wanted is not None and key not in wanted:
continue
item_type = _TYPE_MOVIE if stype == "movie" else _TYPE_EPISODE
for meta in _paginate(plex, key, item_type):
walked.add(key)
progress: dict = {}
before = scanned
for meta in _paginate(plex, key, item_type, progress):
scanned += 1
item_id = item_id_by_rk.get(str(meta.get("ratingKey") or ""))
if item_id is None:
@@ -98,7 +115,15 @@ def _scan_plex_states(
)
if res is not None:
out[item_id] = res
return out, scanned
# A section the server SAYS is empty (total 0, nothing listed) settles too. Believed only for
# a section we mirror nothing from: one whose items we hold answering 0 is a Plex that lost
# the library for a moment, not one that un-watched all of it (E32 review) — so it is left
# unsettled, and its rows are not cleared.
listed = scanned > before
said_empty = progress.get("total") == 0 and not listed and key not in mirrored
if (progress.get("complete") and listed) or said_empty:
settled.add(key)
return out, scanned, settled, walked
def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
@@ -111,7 +136,8 @@ def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
stats = {"watched": 0, "in_progress": 0, "scanned": 0}
try:
with PlexClient(db) as plex:
scanned_states, stats["scanned"] = _scan_plex_states(db, plex)
# A partial scan is still safe here: the import only ever ADDS what Plex reports.
scanned_states, stats["scanned"], _settled, _walked = _scan_plex_states(db, plex)
# One row per item (keyed by item_id → inherently de-duped) then bulk-UPSERT. An upsert is
# idempotent AND concurrency-safe: a repeated/overlapping enable (the toggle firing twice
# while the multi-second import runs) can't raise a duplicate-key error — it re-updates.
@@ -590,6 +616,28 @@ def run_plex_watch_sync(db: Session) -> dict:
return total
def _clear_unwatched(db: Session, user_id: int, plex_states: dict, settled: set[str]) -> int:
"""Clear the user's rows Plex no longer has a record for — the mirrored ones (Plex un-watched
them); hidden and dirty rows stay (a dirty one is re-pushed by `_repush_dirty`). Only rows of the
sections the scan SETTLED: a walk a Plex hiccup cut short leaves out rows it merely failed to
reach, and clearing those un-watched them here until the next complete run (E32 S2); an item of
a library the allowlist leaves out is absent from every scan, and was cleared on every run (E32
review)."""
n = 0
rows = db.execute(
select(PlexState)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.join(PlexLibrary, PlexLibrary.id == PlexItem.library_id)
.where(PlexState.user_id == user_id, PlexLibrary.plex_key.in_(settled))
)
for st in rows.scalars().all():
if st.item_id in plex_states or st.status == "hidden" or not st.synced_to_plex:
continue
db.delete(st)
n += 1
return n
def run_plex_watch_reconcile(db: Session) -> dict:
"""Scheduler job (default daily) — the authoritative full reconcile. Rescans every section and,
per item, uses `synced_to_plex` to settle what the incremental feed can't:
@@ -608,7 +656,7 @@ def run_plex_watch_reconcile(db: Session) -> dict:
try:
with PlexClient(db) as plex:
_resolve_account_id(db, plex, link)
plex_states, scanned = _scan_plex_states(db, plex)
plex_states, scanned, settled, walked = _scan_plex_states(db, plex)
total["scanned"] += scanned
for item_id, (status, pos, watched_at, prog_at) in plex_states.items():
@@ -616,18 +664,19 @@ def run_plex_watch_reconcile(db: Session) -> dict:
if ch:
total["applied"] += 1
# Siftlode rows Plex no longer has a record for: clear the ones that were mirrored
# (Plex un-watched them); leave hidden + dirty (dirty gets re-pushed just below).
for st in (
db.execute(select(PlexState).where(PlexState.user_id == link.user_id))
.scalars()
.all()
):
if st.item_id in plex_states or st.status == "hidden":
continue
if st.synced_to_plex:
db.delete(st)
total["cleared"] += 1
total["cleared"] += _clear_unwatched(db, link.user_id, plex_states, settled)
if unsettled := sorted(walked - settled):
total["incomplete"] = total.get("incomplete", 0) + 1
log.warning(
"Plex sections %s were cut short or answered empty; their rows are not cleared"
" this run",
", ".join(unsettled),
extra=logs.event(
"plex.watch.reconcile_incomplete",
user_id=link.user_id,
sections=unsettled,
),
)
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = datetime.now(timezone.utc)
+17 -2
View File
@@ -8,7 +8,7 @@ its own thread/context) never cross wires. When no sink is bound (e.g. a manual
"""
import contextvars
from typing import Callable, Optional
from typing import Callable, Literal, Optional
Sink = Callable[[int, Optional[int], Optional[str]], None]
@@ -24,6 +24,19 @@ def report(current: int, total: int | None = None, phase: str | None = None) ->
sink(current, total, phase)
class JobFailed(RuntimeError):
"""A job's own failure, said so the reader's language can name it (E31): the client reads
`scheduler.failure.<reason>` and interpolates `params` (keep them to values the reader may see —
numbers, ids, short codes). The message is the English line the log and `last_error` keep. An
upstream's own error (a timeout, a refused connection) is not one of these: it stays as raised,
being diagnostic."""
def __init__(self, reason: str, message: str, **params: object):
super().__init__(message)
self.reason = reason
self.params = params
class bind:
"""Context manager that binds a progress sink for the duration of a job run."""
@@ -35,7 +48,9 @@ class bind:
self._token = _sink.set(self._sink)
return self
def __exit__(self, *exc) -> bool:
# Literal[False], not bool: this never swallows the job's exception, and saying so is what lets
# a type checker know a name assigned inside `with bind(...)` is bound after it.
def __exit__(self, *exc) -> Literal[False]:
if self._token is not None:
_sink.reset(self._token)
return False
+6 -3
View File
@@ -60,14 +60,17 @@ def owned_or_404(
return obj
def youtube_refusal(exc: Exception) -> HTTPException:
def youtube_refusal(exc: Exception, *, playlist: bool = False) -> HTTPException:
"""The answer to an action YouTube refused: a 422 whose reason the UI translates
(`errors.youtube.<reason>`). One definition, because the routes had grown three shapes of it —
a bare English string quoting the upstream message, a playlist-scoped reason, and nothing at
all (a raw 500 from the manual syncs). The upstream failure is already logged where it
happened (`youtube.api.error`). A failure that is not the Data API's (the search scraper's) is
YouTube being unavailable to us — the retryable answer."""
reason = refusal_reason(exc) if isinstance(exc, YouTubeError) else "unavailable"
YouTube being unavailable to us — the retryable answer. `playlist`: the route acts on one
playlist the user picked, so a missing playlist is THAT one, gone (`refusal_reason`)."""
reason = (
refusal_reason(exc, playlist=playlist) if isinstance(exc, YouTubeError) else "unavailable"
)
return HTTPException(status_code=422, detail={"code": "youtube", "reason": reason})
+24 -17
View File
@@ -6,7 +6,7 @@ from datetime import datetime, timezone
from fastapi import APIRouter, BackgroundTasks, Depends
from pydantic import BaseModel
from sqlalchemy import func, select, update
from sqlalchemy import Select, func, select, update
from sqlalchemy.orm import Session
from app.collation import collated, sorted_words
@@ -40,6 +40,22 @@ router = APIRouter(prefix="/api/channels", tags=["channels"])
log = logs.get_logger(__name__)
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
)
@router.get("")
def list_channels(user: User = Depends(current_user), db: Session = Depends(get_db)) -> list[dict]:
rows = db.execute(
@@ -49,24 +65,15 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
.order_by(Subscription.priority.desc(), collated(Channel.title, lang_for_user(user)))
).all()
# Per-channel aggregates over the shared catalog in ONE grouped pass: stored count, last
# upload, total duration. Cheap — a single grouped scan, so computed on read rather than
# denormalised (measure-first: reads aren't frequent/hot enough to warrant cached columns).
# Per-channel aggregates over the shared catalog (channel_aggregates), computed on read rather than
# denormalised: an Index Only Scan over a ~20 MB index, where it used to read every video's heap page
# (681 MB on prod, a 2–6 s first open once something else had pushed it out of memory, E31 S4d).
# (The Normal/Shorts/Live breakdown moved to the channel-detail page in R14, so it is no longer
# computed here — the manager table dropped its N/S/L column.)
channel_ids = [c.id for _, c in rows]
agg: dict[str, dict] = {}
if channel_ids:
for cid, n, last_up, total_dur in db.execute(
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
).all():
for cid, n, last_up, total_dur in db.execute(channel_aggregates(channel_ids)).all():
agg[cid] = {
"stored": n,
"last_video_at": iso(last_up),
@@ -429,7 +436,7 @@ def explore_channel(
channel = db.get(Channel, channel_id)
result = explore_ingest_page(db, yt, channel, page_token)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
return {
"ok": True,
"channel_id": channel_id,
@@ -590,7 +597,7 @@ def subscribe(
if "subscriptionDuplicate" in exc.reason.split(","):
yt_sub_id = ""
else:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
# Enrich a stub channel (title/thumbnail/subscriber count) so it shows up properly right
# away, on BOTH the normal and the desync path — a resurfaced stub still needs metadata to
# render. Keep the client open (we're still inside the `with`); a metadata failure here
@@ -651,7 +658,7 @@ def unsubscribe(
except YouTubeError as exc:
# 422 (a real, explainable error → speaking modal), not 502 which the client treats as
# a transient "connection lost" gateway blip.
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
db.delete(sub)
db.commit()
return {"ok": True, "unsubscribed": channel_id}
+1 -1
View File
@@ -170,7 +170,7 @@ def add_pin(
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
# item), which trips the FK. Semantically that is the same "no such thing" the pre-check
# returns above — re-raising handed the user a 500 they could do nothing with.
raise not_found("item")
raise not_found("item") from None
return {"ok": True, "id": winner.id, "created": False}
return {"ok": True, "id": pin.id, "created": True}
+5 -5
View File
@@ -322,7 +322,7 @@ def set_credential(
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)})
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
audit.record(
db,
@@ -407,7 +407,7 @@ def enqueue_download(
try:
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
@@ -520,7 +520,7 @@ def upload_download(
display_name=upload.display_name,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
@@ -547,9 +547,9 @@ def enqueue_edit(
try:
job = service.enqueue_edit(db, user.id, source_job, spec, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
except service.EditError as e:
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason})
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
return _serialize_job(db, job)
+30 -12
View File
@@ -19,8 +19,11 @@ from app.feedquery import (
filtered_query,
keyset_predicate,
order_by,
rank_window_start,
serialize,
sort_keys,
windowed_rank,
windowed_relevance_rows,
)
from app.models import (
POSITION_MAX_S,
@@ -109,30 +112,45 @@ def get_feed(
db: Session = Depends(get_db),
) -> dict:
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys(sort, seed, rank_expr, lang_for_user(user))
# Expose each key column on the row so we can build the next cursor from the last item.
query = query.add_columns(*[k["expr"].label(f"_k{i}") for i, k in enumerate(keys)])
query = query.order_by(*order_by(keys))
lang = lang_for_user(user)
keys = sort_keys(sort, seed, rank_expr, lang)
vals, cursor_id, window = None, None, None
if cursor:
# The read-model raises a plain ValueError on a malformed cursor; translate it to the
# HTTP 400 here so feedquery stays web-framework-free.
try:
vals, cursor_id = decode_cursor(keys, cursor)
vals, cursor_id, window = decode_cursor(keys, cursor)
except ValueError:
# `from None`: the ValueError is an expected bad-input signal, not an error to chain.
raise refused(400, "invalid_cursor") from None
for k, v in zip(keys, vals):
k["value"] = v
query = query.where(keyset_predicate(keys, cursor_id))
ranked = sort == "relevance" and rank_expr is not None
if ranked and not cursor:
window = rank_window_start(db, query)
if ranked and window is not None:
keys = sort_keys(sort, seed, windowed_rank(rank_expr, window), lang)
rows = db.execute(query.limit(limit + 1)).all()
# Expose each key column on the row so we can build the next cursor from the last item.
query = query.add_columns(*[k["expr"].label(f"_k{i}") for i, k in enumerate(keys)])
if vals is not None:
for k, v in zip(keys, vals, strict=True):
k["value"] = v
if ranked and window is not None:
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
else:
query = query.order_by(*order_by(keys))
if cursor_id is not None:
query = query.where(keyset_predicate(keys, cursor_id))
rows = db.execute(query.limit(limit + 1)).all()
has_more = len(rows) > limit
page = rows[:limit]
return {
"items": [serialize(r) for r in page],
"next_cursor": encode_cursor(keys, page[-1]) if (has_more and page) else None,
"next_cursor": (
encode_cursor(keys, page[-1], window if ranked else None)
if (has_more and page)
else None
),
"has_more": has_more,
"limit": limit,
}
@@ -413,7 +431,7 @@ def get_video_detail(
):
items = yt.get_videos([video_id])
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
if not items:
raise not_found("video")
+23 -5
View File
@@ -94,13 +94,31 @@ def _system_partner() -> dict:
return {"id": SYSTEM_PARTNER_ID, "name": SYSTEM_NAME, "avatar_url": None}
def _serialize_msg(m: Message) -> dict:
# The welcome's opening line in every language — how a stored copy is recognised. It was rendered
# once, in the language of the day it was created, so an account that later switched language kept
# reading it in the old one (E31 S1: the owner's, in English). Recognised here, it is served in the
# reader's language instead. Derived from WELCOME itself, so a change to an OPENING line, or a
# language taken out of WELCOME, orphans the copies already stored: they are served as stored. The
# one retired language (German, 2026-07) left none on prod (checked 2026-10-02).
_WELCOME_OPENINGS = {text.partition("\n")[0] for text in WELCOME.values()}
def _system_body(body: str | None, lang: str | None) -> str | None:
"""A system message's text for a reader of `lang`: the welcome in their language, any other as
stored."""
if body is None or lang is None or body.partition("\n")[0] not in _WELCOME_OPENINGS:
return body
return WELCOME.get(lang, WELCOME["en"])
def _serialize_msg(m: Message, lang: str | None = None) -> dict:
"""`lang`: the reader's language, for a system message's text (a user message is ciphertext)."""
return {
"id": m.id,
"kind": m.kind,
"sender_id": m.sender_id,
"recipient_id": m.recipient_id,
"body": m.body, # only set for system messages
"body": _system_body(m.body, lang) if m.kind == "system" else m.body,
"ciphertext": m.ciphertext,
"iv": m.iv,
"read_at": iso(m.read_at),
@@ -124,7 +142,7 @@ def ensure_welcome(db: Session, user: User) -> None:
db.add(m)
db.commit()
db.refresh(m)
manager.push(user.id, {"type": "message", "message": _serialize_msg(m)})
manager.push(user.id, {"type": "message", "message": _serialize_msg(m, lang_for_user(user))})
# --- E2EE key directory / blob store -----------------------------------------
@@ -313,7 +331,7 @@ def list_conversations(user: User = Depends(require_human), db: Session = Depend
items.append(
{
"partner": partner,
"last_message": _serialize_msg(m),
"last_message": _serialize_msg(m, lang_for_user(user)),
"unread": int(unread.get(pid, 0)),
}
)
@@ -366,7 +384,7 @@ def get_thread(
# refreshes live instead of waiting for the next 20s poll. The client re-fetches the
# count on this signal, so no number is carried here.
manager.push(user.id, {"type": "unread"})
return {"partner": partner, "items": [_serialize_msg(m) for m in msgs]}
return {"partner": partner, "items": [_serialize_msg(m, lang_for_user(user)) for m in msgs]}
@router.post("")
+7 -7
View File
@@ -342,7 +342,7 @@ def sync_youtube(
return {"ok": True, **pull_user_playlists(db, user)}
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
@router.post("/{playlist_id}/revert-youtube")
@@ -369,10 +369,10 @@ def revert_youtube(
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return {"ok": True, **repull_playlist(db, user, pl)}
except PlaylistGone:
raise not_found("playlist")
raise not_found("playlist") from None
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
@@ -398,7 +398,7 @@ def push_plan(
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_merge(db, user, pl)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
plan["remaining_today"] = quota.remaining_today(db)
plan["affordable"] = plan["units_estimate"] <= plan["remaining_today"]
return plan
@@ -436,10 +436,10 @@ def push(
raise _refused(429, "quota_exceeded")
return {"ok": True, **merge_playlist(db, user, pl, live=live)}
except PlaylistGone:
raise not_found("playlist")
raise not_found("playlist") from None
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
@router.get("/{playlist_id}")
@@ -539,7 +539,7 @@ def delete_playlist(
with YouTubeClient(db, user) as yt:
yt.delete_playlist(pl.yt_playlist_id)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
# The playlist first, as every item write: the cascade deletes the items before the list row.
lock_playlists(db, pl.id)
db.delete(pl) # items cascade
+20 -8
View File
@@ -358,6 +358,24 @@ def facets(
uid = user.id
# The shows that pass the watch-state scope, asked ONCE. The per-user episode rollup behind it is a
# full pass over the user's episodes (~10 ms on a prod-sized library), and each of the show groups
# (genres, ratings, studios, years, rating max) asked it again: half of a filtered facets call
# (E31 S4e). The same filter on the same rows, so the counts do not change
# (test_plex_facets_grid_parity).
shows_in_scope: list[int] | None = None
if show_ids and show in ("watched", "in_progress", "unwatched"):
agg = rollup.agg_subquery(db, uid).subquery()
in_scope = select(PlexShow.id).outerjoin(agg, agg.c.show_id == PlexShow.id)
in_scope = apply_show_state_filter(
in_scope,
func.coalesce(agg.c.total, 0),
func.coalesce(agg.c.watched, 0),
func.coalesce(agg.c.inprog, 0),
show,
)
shows_in_scope = list(db.scalars(in_scope.where(PlexShow.library_id.in_(show_ids))))
def wsq(query, model, ids, pp, kind_movie, with_dur):
"""Apply the (self-excluded) metadata filters + library scope + the per-user watch-state to a
facet aggregation query, so the facets reflect exactly the visible result set."""
@@ -376,14 +394,8 @@ def facets(
st = aliased(PlexState)
query = query.outerjoin(st, and_(st.item_id == model.id, st.user_id == uid))
query = apply_movie_state_filter(query, st, show)
else:
if show in ("watched", "in_progress", "unwatched"):
agg = rollup.agg_subquery(db, uid).subquery()
query = query.outerjoin(agg, agg.c.show_id == model.id)
tot = func.coalesce(agg.c.total, 0)
wat = func.coalesce(agg.c.watched, 0)
inp = func.coalesce(agg.c.inprog, 0)
query = apply_show_state_filter(query, tot, wat, inp, show)
elif shows_in_scope is not None:
query = query.where(model.id.in_(shows_in_scope))
return query
genre_counts: dict[str, int] = {}
+4 -4
View File
@@ -428,7 +428,7 @@ def create_collection(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -450,7 +450,7 @@ def collection_add_item(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -472,7 +472,7 @@ def collection_remove_item(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -502,7 +502,7 @@ def collection_rename(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
+12 -4
View File
@@ -124,8 +124,14 @@ def _recently_added(
# share one), so the merge doesn't depend on object identity staying alive.
top_movies = [o for _, kind, o in top if kind == "movie"]
top_shows = [o for _, kind, o in top if kind == "show"]
mcards = {m.rating_key: c for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang))}
scards = {s.rating_key: c for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang))}
mcards = {
m.rating_key: c
for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang), strict=True)
}
return [mcards[o.rating_key] if kind == "movie" else scards[o.rating_key] for _, kind, o in top]
@@ -351,10 +357,12 @@ def _because_you_watched(
ordered_shows = [shows[rk] for rk in rks if rk in shows]
# rating_key is globally unique in Plex, so it keys the serialized cards back safely (no id()).
mcards = {
m.rating_key: c for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang))
m.rating_key: c
for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang), strict=True)
}
scards = {
s.rating_key: c for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang))
s.rating_key: c
for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang), strict=True)
}
items: list[dict] = []
for rk in rks:
+2 -2
View File
@@ -65,7 +65,7 @@ def image(rating_key: str, request: Request, variant: str = "thumb") -> Response
brief session only to resolve the image key + read the Plex config, then RELEASES it before the
fetch. Auth is the signed-session check (no DB user-load)."""
_require_session(request)
v = "art" if variant == "art" else "thumb"
v = variant if variant in imgcache.SIZES else "thumb"
w, h = imgcache.SIZES[v]
# Size is part of the cache key: Plex resizes server-side (see PlexClient.image_bytes), so we cache
# the small variant. The width tag also means changing a target size just re-caches, never serves a
@@ -77,7 +77,7 @@ def image(rating_key: str, request: Request, variant: str = "thumb") -> Response
# Cold miss: short-lived session for the key + Plex client config, released before the slow fetch.
try:
with SessionLocal() as db:
key = imgcache.key_for(db, str(rating_key), v)
key = imgcache.key_for(db, str(rating_key), imgcache.picture(v))
if not key:
raise not_found("image")
plex = PlexClient(db) # reads base URL + token now; the fetch below needs no DB
+24 -3
View File
@@ -1,5 +1,5 @@
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from pydantic import BaseModel, field_validator
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -53,11 +53,30 @@ def list_views(
return [_serialize(v) for v in rows]
# A feed view's tag pick is read by more than the feed: the autotag job keeps every tag a view names
# (sync/autotag.py). Held here to the shape the feed sends — a list of tag ids, at most this many — so
# a hand-made body (`"tags": 5`, or 60k ids) can't break that job for every user (E32 review).
_MAX_VIEW_TAGS = 500
def _check_filters(filters: dict | None) -> dict | None:
tags = (filters or {}).get("tags")
if tags is None:
return filters
if not isinstance(tags, list) or len(tags) > _MAX_VIEW_TAGS:
raise ValueError(f"filters.tags must be a list of at most {_MAX_VIEW_TAGS} tag ids")
if any(type(t) is not int for t in tags): # `type`, not isinstance: True is not a tag id
raise ValueError("filters.tags must hold tag ids")
return filters
class CreateViewIn(BaseModel):
name: str
filters: dict
source: str = "feed"
_filters = field_validator("filters")(_check_filters)
@router.post("")
def create_view(
@@ -87,7 +106,7 @@ def create_view(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "view_name_taken", name=name)
raise refused(409, "view_name_taken", name=name) from None
return _serialize(view)
@@ -98,6 +117,8 @@ class UpdateViewIn(BaseModel):
filters: dict | None = None
is_default: bool | None = None
_filters = field_validator("filters")(_check_filters)
@router.patch("/{view_id}")
def update_view(
@@ -135,7 +156,7 @@ def update_view(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "view_name_taken", name=view.name)
raise refused(409, "view_name_taken", name=view.name) from None
return _serialize(view)
+1 -1
View File
@@ -74,7 +74,7 @@ def run_job_now(
try:
result = trigger_job(job_id, actor_id=user.id)
except KeyError:
raise not_found("job")
raise not_found("job") from None
if result == "already_running":
raise refused(409, "job_running")
return {"ok": True, "id": job_id, "started": True}
+1 -1
View File
@@ -331,7 +331,7 @@ def search_youtube(
# bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
try:
collected.extend(_ingest_candidates(db, yt, user, candidates))
except Exception as exc: # noqa: BLE001 — any ingest failure, not just YouTubeError
except Exception as exc: # any ingest failure, not just YouTubeError
db.rollback()
# The term is NOT in this line. A search string is exactly what the privacy
# policy keeps out of the log unless the user turned Teljes/TRACE on for
+1 -1
View File
@@ -39,7 +39,7 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
# The onboarding wizard's first call: an unguarded failure here was a raw 500 and the
# blocking "Server error" dialog, with nothing the user could act on (E9 F1).
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
result["quota_used_estimate"] = m.used
result["quota_remaining_today"] = m.remaining
return {"ok": True, **result}
+2 -2
View File
@@ -75,7 +75,7 @@ def create_tag(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "tag_name_taken", name=name)
raise refused(409, "tag_name_taken", name=name) from None
return {
"ok": True,
"id": tag.id,
@@ -113,7 +113,7 @@ def update_tag(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "tag_name_taken", name=tag.name)
raise refused(409, "tag_name_taken", name=tag.name) from None
return {
"ok": True,
"id": tag.id,
+8 -3
View File
@@ -231,17 +231,22 @@ def _job(name: str, fn) -> None:
db.rollback()
log.exception("job %s failed", name, extra=logs.event("scheduler.job.failed", job=name))
err = str(exc) or exc.__class__.__name__
# A failure the job named (progress.JobFailed) rides along as data, as a result does, so the
# client says it in the reader's language; `err` stays the English line.
failure = (
{"failure": exc.reason, **exc.params} if isinstance(exc, progress.JobFailed) else None
)
_record(
name,
status="error",
last_finished=_now_iso(),
last_error=err,
result=None,
result=failure,
progress=None,
)
if actor_id is not None:
_notify_done(db, actor_id, name, "error", err)
_audit_run(db, actor_id, name, "error", err, None)
_notify_done(db, actor_id, name, "error", err, failure)
_audit_run(db, actor_id, name, "error", err, failure)
finally:
# ALWAYS release the claim, even if the body raised before its own status update — so a
# failure can never wedge the job as permanently "running".
+29 -12
View File
@@ -12,7 +12,7 @@ from sqlalchemy import and_, exists, func, select
from sqlalchemy.orm import Session
from app import logs, progress, sysconfig
from app.models import Channel, ChannelTag, Tag, Video
from app.models import Channel, ChannelTag, SavedView, Tag, Video
log = logs.get_logger(__name__)
@@ -288,18 +288,35 @@ def run_autotag_all(db: Session, only_missing: bool = False) -> dict:
return {"channels_tagged": tagged, "orphan_tags_removed": removed}
def _view_tag_ids(filters: object) -> list[int]:
"""The tag ids a stored view's filters name; anything else in that slot names none."""
tags = filters.get("tags") if isinstance(filters, dict) else None
if not isinstance(tags, list):
return []
return [t for t in tags if type(t) is int] # `type`, not isinstance: True is not a tag id
def _cleanup_orphan_system_tags(db: Session) -> int:
"""Delete system tags that no longer link to any channel (e.g. after a mapping change)."""
orphans = (
db.execute(
select(Tag).where(
Tag.user_id.is_(None),
~exists().where(ChannelTag.tag_id == Tag.id),
)
)
.scalars()
.all()
)
"""Delete system tags that no longer link to any channel (e.g. after a mapping change) — but not
one a saved feed view filters on. Views hold tag IDS, and a deleted tag comes back under a new id
when a channel earns it again, so the view's filter would point at nothing; the feed drops an
unknown id, and the view showed the whole feed under its old name (E32 S2).
The views are read defensively, and the kept ids are filtered here rather than in SQL: a view
stored before its route checked the shape (or written around it) must not raise for everyone's
job, nor push a `NOT IN` past the driver's bind-parameter limit (E32 review)."""
in_views = {
t
for (filters,) in db.execute(select(SavedView.filters).where(SavedView.source == "feed"))
for t in _view_tag_ids(filters)
}
orphans = [
tag
for tag in db.execute(
select(Tag).where(Tag.user_id.is_(None), ~exists().where(ChannelTag.tag_id == Tag.id))
).scalars()
if tag.id not in in_views
]
for tag in orphans:
db.delete(tag)
db.commit()
+10 -2
View File
@@ -25,7 +25,7 @@ from app.models import (
)
from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_details
from app.utils import now_utc as _now
from app.youtube.client import YouTubeClient
from app.youtube.client import YouTubeClient, YouTubeError, is_playlist_gone
log = logs.get_logger(__name__)
@@ -81,7 +81,15 @@ def explore_ingest_page(
the organic deep backfill) is never touched."""
if not channel.uploads_playlist_id:
return {"ingested": 0, "next_page_token": None}
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
try:
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
except YouTubeError as exc:
# A channel with no uploads has an uploads id that YouTube answers as not found: an empty
# back-catalogue, not a failure (and not a "playlist gone" — the user picked a channel).
if not is_playlist_gone(exc):
raise
log.info("explore ingest channel=%s has no uploads playlist", channel.id)
return {"ingested": 0, "next_page_token": None}
stubs: list[dict] = []
for item in data.get("items", []):
stub = stub_from_playlist_item(item, channel.id)
+15 -9
View File
@@ -20,9 +20,9 @@ Quota: phase 1 costs ~(flagged / 50) units, phase 2 ~(batch / 50). Both respect
backfill reserve as the other jobs so interactive syncs never starve.
"""
from datetime import timedelta
from datetime import datetime, timedelta
from sqlalchemy import select
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app import logs, progress, quota, sysconfig
@@ -129,14 +129,14 @@ def _notify_removed(db: Session, impact: dict[int, list[dict]]) -> int:
return len(impact)
def _recheck_flagged(db: Session, yt: YouTubeClient) -> dict:
"""Phase 1: recover or hard-delete videos already flagged unavailable."""
def _recheck_flagged(db: Session, yt: YouTubeClient, now: datetime) -> dict:
"""Phase 1: recover or hard-delete videos already flagged unavailable. `now` is the run's:
phase 2 skips a video stamped with it, so one recovered here is not fetched again."""
flagged = db.execute(select(Video).where(Video.unavailable_since.is_not(None))).scalars().all()
if not flagged:
return {"checked": 0, "recovered": 0, "deleted": 0, "notified": 0}
progress.report(0, len(flagged), "recheck")
items = {it["id"]: it for it in yt.get_videos([v.id for v in flagged])}
now = _now()
recovered = 0
to_delete: list[Video] = []
for video in flagged:
@@ -173,7 +173,7 @@ def _recheck_flagged(db: Session, yt: YouTubeClient) -> dict:
}
def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
def _revalidate_rolling(db: Session, yt: YouTubeClient, now: datetime) -> dict:
"""Phase 2: re-check the least-recently-checked currently-available videos and flag any
that have become unplayable (hidden immediately; deleted later if they don't recover)."""
batch = sysconfig.get_int(
@@ -181,7 +181,6 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
) # admin override, else config default
flagged_new = 0
checked = 0
now = _now()
# Process in videos.list-sized pages so we commit incrementally and stop on low quota.
while checked < batch:
if quota.backfill_reserve_reached():
@@ -190,6 +189,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
db.execute(
select(Video)
.where(Video.unavailable_since.is_(None))
# Not one this run already checked: with fewer available videos than the batch, the
# loop used to wrap around and re-fetch the same ones until `checked` reached the
# batch, spending quota on answers it already had (E32 S2).
.where(or_(Video.last_checked_at.is_(None), Video.last_checked_at < now))
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
@@ -220,10 +223,13 @@ def run_maintenance(db: Session) -> dict:
user = get_service_user(db)
if user is None:
return {"reason": "no service user"}
# One `now` for the whole run: each phase taking its own let a video phase 1 had just recovered
# (stamped with phase 1's) pass phase 2's "not checked this run" test and be fetched twice.
now = _now()
with YouTubeClient(db, user) as yt:
phase1 = _recheck_flagged(db, yt)
phase1 = _recheck_flagged(db, yt, now)
phase2 = (
_revalidate_rolling(db, yt)
_revalidate_rolling(db, yt, now)
if not quota.backfill_reserve_reached()
else {"checked": 0, "flagged": 0, "skipped": "low quota"}
)
+89
View File
@@ -0,0 +1,89 @@
"""The 3-way merge of a YouTube-linked playlist, as a pure function: what each side gains and loses
so neither loses an item, the merged order, and the reorder moves YouTube needs to reach it. No I/O —
`sync/playlists.py` reads both sides, calls `merge_plan`, and writes the outcome."""
from dataclasses import dataclass
def _reorder_moves(current: list[str], desired: list[str]) -> int:
"""Count the playlistItems.update moves an insertion-sort would need to turn
`current` order into `desired` order (both already the same set of video ids).
Items already in place are skipped; this matches what apply_push actually executes."""
model = list(current)
moves = 0
for i, want in enumerate(desired):
if i < len(model) and model[i] == want:
continue
j = model.index(want, i)
model.pop(j)
model.insert(i, want)
moves += 1
return moves
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
+38 -94
View File
@@ -1,6 +1,7 @@
"""YouTube playlist sync, both directions. Read: mirror each user's own YouTube playlists
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: two-way sync
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope.
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope; the
pure 3-way merge they run is `sync/playlist_merge.py`.
YouTube's special Watch Later / History playlists are not exposed by the Data API and are
never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and
@@ -10,7 +11,7 @@ import hashlib
import json
from collections.abc import Callable, Iterator
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TypeVar
@@ -21,9 +22,11 @@ from sqlalchemy.pool import NullPool
from app import logs, progress, quota
from app.auth import has_read_scope
from app.models import Channel, Playlist, PlaylistItem, User, Video
from app.sync.playlist_merge import merge_plan
from app.sync.runner import token_users
from app.sync.subscriptions import blank_listing, refuse_blank_listing, vanished
from app.sync.videos import apply_video_details, parse_dt
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.client import YouTubeClient, YouTubeError, is_playlist_gone
log = logs.get_logger(__name__)
# Cost of a single YouTube write op (playlists/playlistItems insert/update/delete).
@@ -316,6 +319,24 @@ def user_sync_claim(db: Session, user_id: int) -> AbstractContextManager[bool]:
return _advisory_claim(db, _USER_SYNC_LOCK_NS, user_id)
def _mirror_listing(yt: YouTubeClient, ytid: str, count: object, pl_id: int) -> list[str] | None:
"""A mirror's video ids from YouTube, or None when there is nothing to believe: the list was
deleted between the pull's listing and this read (the next pull drops the mirror as vanished —
not a reason to fail every other list, E32 review round 2), or YouTube listed no items for a
list it counts as non-empty (`blank_listing`)."""
try:
ids = list(yt.iter_my_playlist_video_ids(ytid))
except YouTubeError as exc:
if not is_playlist_gone(exc):
raise
log.info("YouTube playlist of %s vanished mid-pull; skipped", pl_id)
return None
if blank_listing(ids, count):
log.warning("YouTube listed no items for non-empty playlist %s; kept", pl_id)
return None
return ids
def pull_user_playlists(db: Session, user: User) -> dict:
"""The mirror pull itself; the caller holds `user_sync_claim`."""
synced = 0
@@ -337,12 +358,10 @@ def pull_user_playlists(db: Session, user: User) -> dict:
Playlist.source == "local",
Playlist.yt_playlist_id.is_not(None),
)
)
.scalars()
.all()
).scalars()
)
# Drop mirrors whose YouTube playlist no longer exists — locked first, as every item write.
gone = [pl for ytid, pl in existing.items() if ytid not in yt_ids]
# Drop vanished mirrors (none on an empty answer) — locked first, as every item write.
gone = vanished(existing, yt_ids)
if gone:
lock_playlists(db, *(pl.id for pl in gone))
for pl in gone:
@@ -376,7 +395,10 @@ def pull_user_playlists(db: Session, user: User) -> dict:
# The user is pushing or re-pulling this very list: it writes the same rows.
# The next scheduled run refreshes it.
continue
ids = list(yt.iter_my_playlist_video_ids(ytid))
ids = _mirror_listing(yt, ytid, p.get("item_count"), pl.id)
if ids is None:
db.rollback() # a mirror this run just created is not kept empty either
continue
# Ingest first — it commits — so the row lock below is held from the dirty check
# to the commit that replaces the items.
_ensure_videos(db, yt, ids)
@@ -408,90 +430,6 @@ def _local_video_ids(db: Session, pl: Playlist) -> list[str]:
)
def _reorder_moves(current: list[str], desired: list[str]) -> int:
"""Count the playlistItems.update moves an insertion-sort would need to turn
`current` order into `desired` order (both already the same set of video ids).
Items already in place are skipped; this matches what apply_push actually executes."""
model = list(current)
moves = 0
for i, want in enumerate(desired):
if i < len(model) and model[i] == want:
continue
j = model.index(want, i)
model.pop(j)
model.insert(i, want)
moves += 1
return moves
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what a two-way sync of this playlist would do: create vs
update, how many items each side gains/loses, reorders, and the quota estimate (YouTube
@@ -519,6 +457,7 @@ def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
rename = bool(snippet) and (snippet.get("title") or "") != pl.name
yt_ids = [it["video_id"] for it in current]
refuse_blank_listing(yt_ids, (snippet or {}).get("item_count"))
mp = merge_plan(parse_baseline(pl), local, yt_ids, parse_unavailable(pl))
ops = len(mp.add_to_yt) + len(mp.remove_from_yt) + mp.reorders + (1 if rename else 0)
return {
@@ -665,6 +604,8 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
yt_ids = [it["video_id"] for it in current]
# A merge would read "YouTube lists none of them" as "YouTube deleted them all".
refuse_blank_listing(yt_ids, (snippet or {}).get("item_count"))
# Re-attempt ingestion of ids previously marked un-ingestable that are still on YouTube: a
# video that has since become available (e.g. private→public) should re-enter the merge, not
# stay skipped forever. Those that still won't ingest remain filtered (`effective_unavailable`).
@@ -803,8 +744,11 @@ def repull_playlist(db: Session, user: User, pl: Playlist) -> dict:
with YouTubeClient(db, user) as yt:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
if snippet is None:
raise YouTubeError("Playlist no longer exists on YouTube")
raise YouTubeError(
"Playlist no longer exists on YouTube", reason="playlistNotFound", path="playlists"
)
ids = list(yt.iter_my_playlist_video_ids(pl.yt_playlist_id))
refuse_blank_listing(ids, snippet.get("item_count"))
_ensure_videos(db, yt, ids)
_lock_or_gone(db, pl)
pl.name = snippet.get("title") or pl.name
+24 -9
View File
@@ -15,8 +15,6 @@ from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import logs, progress, quota, sysconfig
log = logs.get_logger(__name__)
from app.db import SessionLocal
from app.models import Channel, OAuthToken, Subscription, User
from app.sync.explore import followed_channel
@@ -35,6 +33,8 @@ from app.sync.videos import (
from app.youtube.client import YouTubeClient, YouTubeError, is_quota_refusal
from app.youtube.rss import RssError, fetch_channel_feed
log = logs.get_logger(__name__)
# RSS feeds are free and network-bound, so fetch many at once; DB writes stay single-threaded.
RSS_POLL_WORKERS = 16
# Failed channels named in a poll's summary warning; the rest are counted by reason.
@@ -83,6 +83,8 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# outage from one permanently dead feed id — which is the whole difference on an instance with
# a single subscription, where BOTH render as 100% failed.
first_error: str | None = None
# The same first failure as data, for the reader's language: whose feed, and why (`reasons`' key).
first_failure: tuple[str, str] | None = None
# What failed, as counts per reason (`http_404`, `transport`, `apply`, …) plus a few examples —
# the ONE warning a poll writes. One line per failed channel was what turned YouTube's RSS outage
# of 2026-09-16 into 11k warnings in four hours; each channel's own line is DEBUG now.
@@ -92,18 +94,19 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# hand the same rows to the same broken write) — the feed fallback's candidates.
unread: list[str] = []
def fail(reason: str, detail: str, *, unexpected: str | None = None) -> None:
def fail(reason: str, channel_id: str, detail: str, *, unexpected: str | None = None) -> None:
"""Count a channel that did not get polled. `unexpected` is the message for a failure that
is a bug on our side: its FIRST occurrence per reason gets a traceback at ERROR, the rest
DEBUG — a read-only database fails every channel the same way, and 979 identical tracebacks
tell nobody anything the first one didn't. Call it from inside the `except` block."""
nonlocal failed, first_error
nonlocal failed, first_error, first_failure
if unexpected is not None:
level = logging.ERROR if reasons[reason] == 0 else logging.DEBUG
log.log(level, unexpected, exc_info=True)
failed += 1
reasons[reason] += 1
first_error = first_error or detail
first_failure = first_failure or (channel_id, reason)
if len(examples) < RSS_FAILURE_EXAMPLES:
examples.append(detail)
@@ -125,13 +128,16 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# bug on our side and earns a traceback (the first of its kind per poll).
except RssError as exc:
log.debug("RSS poll skipped for channel %s: %s", channel.id, exc)
fail(exc.reason, f"{channel.id}: {exc}")
fail(exc.reason, channel.id, f"{channel.id}: {exc}")
unread.append(channel.id)
# silent-ok: `fail(unexpected=…)` is the report — it logs at ERROR with the
# traceback, once per reason per poll, which is the flood control this loop needs.
except Exception as exc: # noqa: BLE001 — one channel must not stop the poll
fail(
"error", f"{channel.id}: {exc}", unexpected=f"RSS fetch failed for {channel.id}"
"error",
channel.id,
f"{channel.id}: {exc}",
unexpected=f"RSS fetch failed for {channel.id}",
)
else:
try:
@@ -144,6 +150,7 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
db.rollback()
fail(
"apply",
channel.id,
f"{channel.id} (apply): {exc}",
unexpected=f"RSS apply failed for {channel.id}",
)
@@ -180,9 +187,17 @@ def run_rss_poll(db: Session, channels: list[Channel] | None = None) -> dict[str
# "fix this subscription"; `Connection refused` says "the egress is down".
# The fallback's count rides along: a red card during an RSS outage should still say
# whether the uploads are arriving another way.
raise RuntimeError(
# As data too (E31): the audit log and the inbox say it in the reader's language, the first
# failure as its channel and its reason code (`http_404` vs `transport` is that difference).
first_channel, first_reason = first_failure or ("", "error")
raise progress.JobFailed(
"rss_all_failed",
f"RSS poll failed for all {total} channels — first: {first_error}"
f" (read through the API instead: {fallback['read']})"
f" (read through the API instead: {fallback['read']})",
total=total,
channel=first_channel,
cause=first_reason,
read=fallback["read"],
)
return {"new": new + fallback["new"], "failed": failed, "fallback": fallback["read"]}
@@ -252,7 +267,7 @@ def run_feed_fallback(db: Session, channel_ids: list[str]) -> dict[str, int]:
channel.feed_fallback_at = datetime.now(timezone.utc)
db.add(channel)
db.commit()
except Exception: # noqa: BLE001 — one channel must not stop the rest
except Exception: # one channel must not stop the rest
db.rollback()
log.exception(
"Feed fallback could not store channel %s",
+30 -3
View File
@@ -1,6 +1,6 @@
"""Import the authenticated user's YouTube subscriptions and channel metadata."""
from collections.abc import Iterable
from collections.abc import Collection, Iterable
from datetime import datetime, timedelta, timezone
from sqlalchemy import or_, select
@@ -9,7 +9,7 @@ from sqlalchemy.orm import Session
from app import logs
from app.models import Channel, Subscription, User
from app.utils import normalize_description
from app.youtube.client import YouTubeClient, best_thumbnail
from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail
log = logs.get_logger(__name__)
@@ -115,10 +115,37 @@ def should_prune(fetched_count: int, existing_count: int) -> bool:
has — taking each row's `deep_requested` flag with it — in a single run. Someone who genuinely
unsubscribed from everything simply keeps their rows until a sync that returns something.
Pure so it can be tested without a database; `import_subscriptions` is the only caller."""
Pure so it can be tested without a database; its callers are `import_subscriptions` and, through
`vanished`, every mirror that drops what the remote no longer lists."""
return fetched_count > 0 or existing_count == 0
def vanished[K, V](existing: dict[K, V], listed: Collection[K]) -> list[V]:
"""The local mirrors whose key the remote listing no longer has — none at all when the listing
is an empty answer that contradicts them (`should_prune`). The YouTube playlist mirrors use it:
without it, one blip dropped every mirror (E32 S2)."""
if not should_prune(len(listed), len(existing)):
return []
return [row for key, row in existing.items() if key not in listed]
def blank_listing(listed: Collection, count: object) -> bool:
"""The same rule one level down: no members from a container whose own listing counts some (a
playlist's `itemCount`) is a blip, not a list emptied. Replacing or merging on it deleted every
mirrored item and reset the merge baseline (E32 review)."""
return not listed and isinstance(count, int) and count > 0
def refuse_blank_listing(listed: Collection, count: object) -> None:
"""`blank_listing` for a user-started pull or push: refused as retryable ("try again")."""
if blank_listing(listed, count):
raise YouTubeError(
"YouTube listed no items for a playlist it counts as non-empty",
reason="emptyAnswer",
path="playlistItems",
)
def import_subscriptions(db: Session, user: User) -> dict:
"""Pull all subscriptions for `user`, upsert channels + subscription links, prune
channels the user has unsubscribed from on YouTube, and fetch channel details for
+20 -4
View File
@@ -11,12 +11,14 @@ from sqlalchemy import and_, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import progress, quota, sysconfig
from app import logs, progress, quota, sysconfig
from app.models import LIVE_OR_UPCOMING, Channel, Video
from app.titles import normalize_title
from app.youtube.client import YouTubeClient, best_thumbnail
from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail, is_playlist_gone
from app.youtube.shorts import make_client, probe_is_short
log = logs.get_logger(__name__)
_DURATION_RE = re.compile(r"P(?:(\d+)D)?T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?", re.IGNORECASE)
@@ -104,6 +106,20 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
}
def _uploads_page(yt: YouTubeClient, channel_id: str, uploads: str, page_token: str | None) -> dict:
"""One page of the channel's uploads. YouTube answers the uploads list of a channel with no
uploads at all as not found: an empty back-catalogue, so an empty last page — not a failure
that left `recent_synced_at` unstamped and charged the channel again on every run (E32 review).
Any other failure raises."""
try:
return yt.get_playlist_items(uploads, page_token)
except YouTubeError as exc:
if not is_playlist_gone(exc):
raise
log.info("backfill channel=%s has no uploads playlist", channel_id)
return {}
def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) -> int:
"""First pass: newest videos up to the configured count or age cutoff. Records a
resume cursor for the later deep backfill."""
@@ -121,7 +137,7 @@ def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) ->
while True:
page_start = page_token # token that fetched THIS page (None = first page)
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
data = _uploads_page(yt, channel.id, channel.uploads_playlist_id, page_token)
stopped_mid_page = False
for item in data.get("items", []):
stub = stub_from_playlist_item(item, channel.id)
@@ -175,7 +191,7 @@ def backfill_channel_deep(
for _ in range(max_pages):
if quota.backfill_reserve_reached():
break
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
data = _uploads_page(yt, channel.id, channel.uploads_playlist_id, page_token)
rows = [
s
for s in (stub_from_playlist_item(it, channel.id) for it in data.get("items", []))
+33 -8
View File
@@ -10,6 +10,7 @@ stored in the DB and stay env-only).
from dataclasses import dataclass
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs, security
@@ -45,8 +46,8 @@ class ConfigSpec:
# Registry of DB-overridable keys. Add a key here + wire its read through get_*() to move it
# off env into the admin UI; the admin page renders it automatically from this list. The ORDER is the
# admin page's tab order (first appearance of each group), so keep a group's keys together.
# off env into the admin UI; the admin page renders it automatically from this list. A group's keys
# stay together, in the order its section lists them; a NEW group also needs its place in SECTIONS.
SPECS: tuple[ConfigSpec, ...] = (
# --- Access / registration ---
ConfigSpec("allow_registration", "bool", "access", "allow_registration"),
@@ -219,6 +220,16 @@ SPECS: tuple[ConfigSpec, ...] = (
# is a config.py default — both stay out of SPECS, like plex_sync_interval_min.)
)
# The admin page's sections (E31, the owner's pick): sixteen groups as tabs wrapped to three rows at 133%
# zoom, so they are a vertical list under four headings, in this order. Every group of SPECS sits in
# exactly one section (test-held), so a new group cannot fall off the page.
SECTIONS: tuple[tuple[str, tuple[str, ...]], ...] = (
("access", ("access", "google", "email")),
("youtube", ("youtube", "quota", "backfill", "shorts", "batch", "explore")),
("media", ("downloads", "plex", "tmdb")),
("system", ("maintenance", "audit", "diagnostics", "issues")),
)
_BY_KEY: dict[str, ConfigSpec] = {s.key: s for s in SPECS}
@@ -297,7 +308,11 @@ def _raw_override(db: Session, key: str) -> str | None:
per key so the operator gets a real signal without a per-request traceback. Recovery is documented in
docs/self-hosting.md: the key must never change once secrets are stored. (B-core-2)
"""
row = db.get(SystemConfig, key)
return _raw_of(db.get(SystemConfig, key), key)
def _raw_of(row: SystemConfig | None, key: str) -> str | None:
"""`_raw_override` over a row already read (describe() reads them all in one query)."""
if row is None or row.value is None:
return None
if not row.encrypted:
@@ -317,8 +332,11 @@ def _raw_override(db: Session, key: str) -> str | None:
def get(db: Session, key: str):
"""Effective value: the DB override (typed) if set, else the env/config default."""
s = _BY_KEY[key]
raw = _raw_override(db, key)
return _effective(_BY_KEY[key], _raw_override(db, key))
def _effective(s: ConfigSpec, raw: str | None):
"""`get` over an override already read."""
if raw is None:
return _default(s)
if raw == "" and not s.allow_empty:
@@ -410,8 +428,12 @@ def describe(db: Session) -> dict:
"""Admin-UI metadata + current values, grouped. Secret values are never echoed —
only whether an override is set and whether an env default exists."""
groups: dict[str, list[dict]] = {}
# Every override in one query: a db.get per key was one round trip each (two for a key with no
# row, which the session cannot cache), ~40 ms of the page's ~55 on prod (E31 S4d).
rows = {r.key: r for r in db.scalars(select(SystemConfig))}
for s in SPECS:
is_set = db.get(SystemConfig, s.key) is not None
row = rows.get(s.key)
is_set = row is not None
item: dict = {
"key": s.key,
"type": s.type,
@@ -426,7 +448,10 @@ def describe(db: Session) -> dict:
if s.secret:
item["default_is_set"] = bool(_default(s))
else:
item["value"] = get(db, s.key)
item["value"] = _effective(s, _raw_of(row, s.key))
item["default"] = _default(s)
groups.setdefault(s.group, []).append(item)
return {"groups": groups, "secrets_manageable": secrets_manageable()}
sections = [
{"id": sid, "groups": [g for g in members if g in groups]} for sid, members in SECTIONS
]
return {"groups": groups, "sections": sections, "secrets_manageable": secrets_manageable()}
-2
View File
@@ -60,7 +60,6 @@ _SHORT_LOWER = {
"for",
"or",
"vs",
"the",
"és",
"az",
"egy",
@@ -120,7 +119,6 @@ _ACRONYMS = {
"nfl",
"asmr",
"pov",
"diy",
"wtf",
"lol",
"rtx",
+10 -2
View File
@@ -60,16 +60,24 @@ def now_utc() -> datetime:
return datetime.now(timezone.utc)
# The shortest last word that is prefix-matched. A 1–2 letter prefix matches a large share of the
# library and ts_rank then scores every match: "m" took ~5–8 s on prod against ~100 ms for a whole
# word (E31 S4e). A shorter last word is matched whole instead ("tv", "4k" still find themselves).
MIN_PREFIX_LEN = 3
def to_tsquery_str(q: str) -> str | None:
"""Turn free-text into a to_tsquery() string: word runs ANDed together, the LAST word
prefix-matched (`:*`) for as-you-type search — e.g. "ti amo magyar" → "ti & amo & magyar:*".
prefix-matched (`:*`) for as-you-type search — e.g. "ti amo magyar" → "ti & amo & magyar:*" —
once it has MIN_PREFIX_LEN characters ("ti amo m" → "ti & amo & m").
Only `\\w` runs containing an alphanumeric are kept, so no tsquery operators can leak in
(the string is built from sanitised tokens, never the raw query). Shared by the YouTube feed
search and the Plex library search, hence this neutral home rather than either domain module."""
tokens = [t for t in re.findall(r"\w+", q, flags=re.UNICODE) if any(c.isalnum() for c in t)]
if not tokens:
return None
tokens[-1] = tokens[-1] + ":*"
if len(tokens[-1]) >= MIN_PREFIX_LEN:
tokens[-1] = tokens[-1] + ":*"
return " & ".join(tokens)
+9 -7
View File
@@ -25,6 +25,7 @@ import threading
import time
from collections.abc import Callable
from datetime import datetime, timezone
from functools import partial
from sqlalchemy import text
@@ -72,7 +73,7 @@ def _wait_for_schema(timeout: float = 300.0) -> bool:
with SessionLocal() as db:
db.execute(text("SELECT 1 FROM download_jobs LIMIT 1"))
return True
except Exception: # noqa: BLE001 — table not migrated yet / DB not up yet
except Exception: # table not migrated yet / DB not up yet
if time.monotonic() > deadline:
# The traceback, not `str(exc)[:120]`: this is the last thing the worker says
# before it exits, and which driver call failed is the whole question.
@@ -366,7 +367,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
try:
proc = _spawn_download_child(job_id)
except Exception: # noqa: BLE001 — spawn failed; don't leave the asset stuck 'downloading'
except Exception: # spawn failed; don't leave the asset stuck 'downloading'
log.exception("failed to spawn download child for job %s", job_id)
_reset_asset_pending(asset_id)
_requeue_if_running(job_id, None)
@@ -438,7 +439,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
cause="child_exit",
exit_code=proc.returncode,
)
except Exception: # noqa: BLE001 — boot recovery is the backstop for a stuck 'running' job
except Exception: # boot recovery is the backstop for a stuck 'running' job
log.exception("post-exit reconciliation failed for job %s", job_id)
return
@@ -501,6 +502,7 @@ def _process_job(job_id: int) -> None:
then run in a supervised CHILD PROCESS (`_run_supervised`) so a wedged yt-dlp/ffmpeg can be
hard-killed — a worker THREAD cannot be killed, nor (when it spins in a C call) even
cooperatively cancelled, and one such thread starves the others via the GIL."""
diagnosis = None # set exactly when the asset has already failed (asset_status == "error")
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
if job is None:
@@ -528,7 +530,7 @@ def _process_job(job_id: int) -> None:
if asset_status == "ready":
_finish_from_cache(job_id, asset_id)
return
if asset_status == "error":
if diagnosis is not None:
_fail_running(
job_id,
error=asset_error or "Download failed",
@@ -577,7 +579,7 @@ def _worker_loop(idx: int) -> None:
while not _stop.is_set():
try:
job_id = _claim_job()
except Exception: # noqa: BLE001
except Exception:
log.exception("claim failed")
_stop.wait(settings.download_poll_seconds)
continue
@@ -597,12 +599,12 @@ def _worker_loop(idx: int) -> None:
job_id,
exc,
"crash",
lambda: _fail_running(job_id, error="Worker error", error_code=None),
partial(_fail_running, job_id, error="Worker error", error_code=None),
)
# silent-ok: not silent — `_report_worker_error` logged the crash before this
# re-raise. Failing the job hit the database, and letting that out would end this
# worker thread for good; boot recovery settles a job left `running`.
except Exception: # noqa: BLE001
except Exception:
log.exception(
"could not fail job %s after its crash",
job_id,
+28 -8
View File
@@ -114,18 +114,32 @@ _DEAD_GRANT = "invalid_grant"
# What a refused YouTube action means to the person who asked for it — the three things they can
# act on differently: wait for tomorrow's budget, reconnect the account, or try again later.
YouTubeRefusal = Literal["quota", "reconnect", "unavailable"]
YouTubeRefusal = Literal["quota", "reconnect", "unavailable", "gone"]
def refusal_reason(exc: YouTubeError) -> YouTubeRefusal:
def is_playlist_gone(exc: YouTubeError) -> bool:
"""Whether YouTube answered that the playlist read or written does not exist. Only the caller
knows which playlist that was: a user's list deleted on YouTube, or a channel's uploads list,
which YouTube also answers so for a channel with no uploads at all."""
return "playlistNotFound" in exc.reason.split(",")
def refusal_reason(exc: YouTubeError, *, playlist: bool = False) -> YouTubeRefusal:
"""Sort a failure by what the user can do about it. A dead grant (Google refused the refresh
with `invalid_grant`, which also drops the grant — see `_drop_dead_grant`) or no usable token at
all (raised before any endpoint answered, so with no path and no transport reason) means the
account must be reconnected; retrying cannot help. Any other token-endpoint failure — a 5xx, a
429, a misconfigured client — leaves the grant as it was: reconnecting would not fix it, and
the account would still read as connected, with no connect button to press."""
the account would still read as connected, with no connect button to press.
`playlist`: the action was on ONE playlist the user picked, so a missing playlist is that list,
gone (deleted on YouTube) — neither a retry nor a reconnect helps. Anywhere else the playlist
is one the user never named (a channel's uploads, one list of a bulk pull), and "this playlist
no longer exists" would describe nothing on their screen (E32 review round 2)."""
if is_quota_refusal(exc):
return "quota"
if playlist and is_playlist_gone(exc):
return "gone"
if exc.path == "oauth2/token":
return "reconnect" if exc.reason == _DEAD_GRANT else "unavailable"
if not exc.path and exc.reason != "transport":
@@ -189,7 +203,7 @@ class YouTubeClient:
return cached
try:
refresh = decrypt(tok.refresh_token_enc)
except Exception as exc: # noqa: BLE001 — Fernet InvalidToken (TOKEN_ENCRYPTION_KEY changed)
except Exception as exc: # Fernet InvalidToken (TOKEN_ENCRYPTION_KEY changed)
# A key mismatch must degrade like any other token problem (caller handles YouTubeError),
# not surface as a raw 500 on every scheduler job / connect check. (B-core-2)
raise YouTubeError("Stored OAuth token is unreadable — reconnect your account") from exc
@@ -423,16 +437,22 @@ class YouTubeClient:
return resp.json() if resp.content else {}
def get_playlist_snippet(self, playlist_id: str) -> dict | None:
"""The snippet (title/description) of one playlist, or None if it no longer
exists. 1 unit (OAuth, so private playlists work)."""
"""The snippet (title/description) of one playlist plus its `item_count` (what an empty
item listing is checked against), or None if it no longer exists. 1 unit (OAuth, so
private playlists work)."""
data = self._get(
"playlists",
{"part": "snippet", "id": playlist_id, "maxResults": 1},
{"part": "snippet,contentDetails", "id": playlist_id, "maxResults": 1},
cost=1,
allow_key=False,
)
items = data.get("items", [])
return items[0].get("snippet") if items else None
if not items:
return None
return {
**(items[0].get("snippet") or {}),
"item_count": items[0].get("contentDetails", {}).get("itemCount"),
}
def update_playlist_title(self, playlist_id: str, title: str) -> None:
"""Rename a playlist on YouTube. 50 units."""
+3
View File
@@ -40,6 +40,7 @@ class RssError(Exception):
def _get_feed(channel_id: str) -> bytes:
"""The feed's body, after up to `1 + len(RETRY_DELAYS)` tries; the LAST failure is raised."""
url = RSS_URL.format(channel_id=channel_id)
error: RssError | None = None
for attempt in range(len(RETRY_DELAYS) + 1):
if attempt:
time.sleep(RETRY_DELAYS[attempt - 1])
@@ -60,6 +61,8 @@ def _get_feed(channel_id: str) -> bytes:
)
if resp.status_code not in _RETRYABLE_STATUS:
break
if error is None: # cannot happen: the range above is never empty, and every try records one
raise RssError(f"RSS fetch for {channel_id} made no attempt", "transport")
raise error
+28 -14
View File
@@ -42,6 +42,9 @@ _HEADERS = {
"Accept-Language": "en-US,en;q=0.9",
}
# The top-level keys a first page (contents) and a continuation page (onResponseReceivedCommands)
# put their result list under.
_RESULT_FRAMES = {"contents", "onResponseReceivedCommands"}
_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
@@ -65,29 +68,36 @@ def _runs_text(node: dict | None) -> str | None:
def _get_config(client: httpx.Client) -> tuple[str, str]:
"""The current InnerTube (api_key, client_version), cached after the first fetch."""
"""The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
the life of the process — and when those went stale, search returned nothing, with no error
anywhere (E32 S2). Now the next search simply tries the live pair again."""
global _config
if _config is not None:
return _config
try:
resp = client.get(_RESULTS_URL, params={"search_query": "music"})
html = resp.text
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
_config = (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
except httpx.HTTPError as exc:
# The fallback pair is CACHED in `_config` for the life of the process, so one failed
# fetch at the wrong moment pins every later search to a hard-coded key and version —
# and when those go stale, search returns nothing at all, with no error anywhere.
log.warning(
"the InnerTube config could not be fetched; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
)
_config = (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
return _config
return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
if key and ver:
_config = (key.group(1), ver.group(1))
return _config
log.warning(
"the results page carried no InnerTube config; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
)
return (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
def _video_item(vr: dict) -> dict | None:
@@ -182,6 +192,10 @@ def search(q: str, continuation: str | None = None) -> dict:
except ValueError as exc: # non-JSON body (blocked / consent / captcha)
raise ScrapeError("InnerTube returned a non-JSON response") from exc
# Even a search with no results carries its result list's frame. Without one this is not "no
# results" but a shape we do not know (YouTube moved, or a block page in JSON) — say so.
if not isinstance(data, dict) or not _RESULT_FRAMES & data.keys():
raise ScrapeError("InnerTube returned an unexpected shape")
items: list[dict] = []
_collect_videos(data, items, set())
return {"items": items, "next_page_token": _find_continuation(data)}
+6 -1
View File
@@ -35,4 +35,9 @@ services:
volumes:
- ./:/app
working_dir: /app
command: python -m pytest
# The suite, then the coverage total (E32 D4: reported, no threshold). pytest's exit code is the
# lane's — the report is printed either way and never decides it.
command:
- sh
- -c
- 'python -m pytest --cov=app --cov-report= ; rc=$$?; echo ">> backend coverage (app/) $$(python -m coverage report --format=total)% of lines"; exit $$rc'
+1 -1
View File
@@ -25,7 +25,7 @@
{
"include": ["app", "alembic"],
"extraPaths": ["."],
"typeCheckingMode": "basic",
"typeCheckingMode": "standard",
"pythonVersion": "3.12",
"pythonPlatform": "Linux"
}
+4 -2
View File
@@ -2,7 +2,9 @@
# Run from backend/ (the image's /app). `pythonpath = .` puts the app package on sys.path so tests
# import it as `app.*`, matching runtime. Most of the suite is pure-logic; the DB-backed lane opts in
# via the `db` fixture and runs in the compose test image. `-rs` reports skip REASONS so a silently
# skipped lane is visible (the compose lane sets SIFTLODE_TEST_REQUIRE_DB=1 to make skips a hard error).
# skipped lane is visible (and conftest makes a missing DB/ffmpeg a hard error unless
# SIFTLODE_TEST_REQUIRE_DB=0). `--strict-markers` (E32): a misspelt `@pytest.mark.skipiff` is an error,
# not an unknown marker that quietly does nothing.
testpaths = tests
pythonpath = .
addopts = -q -rs
addopts = -q -rs --strict-markers
+5
View File
@@ -7,7 +7,12 @@
# basedpyright: the backend type-check lane (E27 S3). Pinned like everything else here — a type
# checker whose version floats silently changes what "clean" means, and the ratchet baseline it
# writes is keyed to the diagnostics THIS version produces.
# pytest-cov: the coverage REPORT the gate prints (E32 D4) — a number to read, deliberately with no
# threshold; "every regression test seen red" is the bar, not a percentage.
# pip-audit: the gate's audit of requirements.lock (scripts/check.sh), blocking at any severity.
basedpyright==1.40.1
pytest==8.4.2
pytest-cov==7.1.0
ruff==0.15.21
httpx2==2.13.0
pip-audit==2.10.1
+5 -2
View File
@@ -1,7 +1,10 @@
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI.
# The prod Dockerfile + Dockerfile.test install THIS file; the gate's pip-audit (scripts/check.sh)
# scans it, blocking since E32. A security fix may be ONE pin bumped in place instead of a full
# regeneration, which would move every other package too — then `pip install -r requirements.lock && pip check` must
# pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177, E32).
APScheduler==3.11.3
Authlib==1.7.2
Mako==1.4.1
@@ -53,7 +56,7 @@ starlette==1.6.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzlocal==5.4.4
urllib3==2.7.0
urllib3==2.8.0
uvicorn==0.52.3
uvloop==0.22.1
watchfiles==1.2.0
+27 -8
View File
@@ -1,7 +1,8 @@
# Ruff config for the Siftlode backend. Keeps ruff's default rule set (E4/E7/E9 + F = the
# high-signal pyflakes/pycodestyle checks that catch real dead code + bugs), but ignores E402:
# a handful of modules (e.g. main.py) deliberately run setup — logging config, sys.path — before
# importing route modules, so "module import not at top of file" is intentional there, not cruft.
# high-signal pyflakes/pycodestyle checks that catch real dead code + bugs), all of it on. E402 was
# ignored for main.py's logging setup; the ignore had let 11 incidental late imports in elsewhere
# (E32 review), so it is on now and main.py marks each deliberate one (`# noqa: E402 — <why>`).
# `scripts/e2e_seed.py` edits `sys.path` before its imports, which E402 itself allows.
#
# line-length 100 (ruff's default is 88): matches the frontend prettier printWidth so both halves of
# the repo wrap at the same width, and it's the width `ruff format` enforces (gated via
@@ -17,9 +18,10 @@ target-version = "py312"
# LOG — stdlib logging misuse (`.warn()`, a logger built by hand, `exc_info` on `.exception`)
# G — the call's shape: %-style args, never an f-string the level filter cannot skip, and
# never `str(exc)` where `exc_info` would carry the traceback
# T20 — `print` instead of a log line. Two deliberate ones remain, each with its `# noqa`
# and a why: `downloads/runner.py`'s usage line is for whoever typed it, and
# `logsink.py` reports the SINK's own failure, which cannot be logged into the sink.
# T20 — `print` instead of a log line. Three deliberate ones remain, each with its `# noqa`
# and a why: `downloads/runner.py`'s usage line is for whoever typed it, `logsink.py`
# reports the SINK's own failure, which cannot be logged into the sink, and
# `scripts/e2e_seed.py` prints the seeded ids for the e2e run that reads them.
# TRY400 — `.error()` in an `except` where `.exception()` would keep the traceback
# TRY401 — the exception spelled into the message AND passed as `exc_info` (it is in both)
# S112 — `except: continue`, the silent swallow inside a loop
@@ -29,5 +31,22 @@ target-version = "py312"
# nobody to raise to, a best-effort backfill inside a sync) and always worth a sentence.
# The wider rule these two enforce lives in tests/test_swallowed_exceptions.py: a handler that
# neither logs nor re-raises says `# silent-ok: <why>`. Ruff sees only the two narrow shapes.
extend-select = ["LOG", "G", "T20", "TRY400", "TRY401", "S110", "S112", "BLE001"]
ignore = ["E402"]
#
# E32 (gate integrity) added three more, each measured before it went on and with every hit fixed:
# RUF100 — a `# noqa` that suppresses nothing. A stale one is a claim with no rule behind it, and
# it stays armed for the day the line changes (24 were removed). A comment that merely
# explains a noqa below it names the rule as `# TRY400: …`, never `# noqa TRY400`.
# PLW1510 — `subprocess.run` without an explicit `check=`. Found the real E32 bug: three ffprobe
# probes read the output of a probe that had FAILED and took it for the file's truth.
# B — flake8-bugbear: loop-variable capture, `raise` without `from` inside `except`, a
# `zip` that silently truncates (each one is `strict=True` now — the lengths are an
# invariant, so a mismatch should be loud). B008 (a call in a default argument) stays ON:
# FastAPI's `Depends(...)` / `Query(...)` ARE that shape by design (474 hits, all of
# them), so those two are declared immutable below instead of switching the rule off —
# any other call in a default still fails.
extend-select = [
"LOG", "G", "T20", "TRY400", "TRY401", "S110", "S112", "BLE001", "RUF100", "PLW1510", "B",
]
[lint.flake8-bugbear]
extend-immutable-calls = ["fastapi.Depends", "fastapi.Query"]
+12 -9
View File
@@ -25,7 +25,7 @@ from fastapi.testclient import TestClient
from sqlalchemy import create_engine, event, text
from starlette.requests import HTTPConnection
import app.models # noqa: F401 — register every model on Base.metadata
import app.models # register every model on Base.metadata
import app.state as app_state
from app import security
from app.auth import current_user
@@ -37,9 +37,11 @@ from app.models import User
# Sentinel for "this dependency had no override before the fixture set one" (see override_dep).
_MISSING = object()
# The compose test lane sets this so a would-be silent skip (no DB, or no ffmpeg) becomes a hard error
# — the gate must not print green while the DB-backed / ffmpeg tests were quietly not run.
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB") == "1"
# A would-be silent skip (no DB, or no ffmpeg) is a hard error unless the run says otherwise — the
# suite must not print green while the DB-backed / ffmpeg tests were quietly not run. ON by default
# since E32: as an opt-in only the compose lane set, a hand-run `pytest` skipped ~60 DB tests
# and still passed. `SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only" run.
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB", "1") != "0"
def _db_reachable() -> bool:
@@ -86,19 +88,20 @@ def _truncate_all():
def pytest_configure(config):
"""Fail the whole run — loudly, once — when SIFTLODE_TEST_REQUIRE_DB=1 but the environment can't
actually run the full suite. Without this, an unreachable DATABASE_URL or a missing ffmpeg turns
"""Fail the whole run — loudly, once — when the full suite is required (the default) but the
environment can't actually run it. Without this, an unreachable DATABASE_URL or a missing ffmpeg turns
~60 DB tests + the poster tests into silent skips and the gate still exits green."""
if not _REQUIRE_FULL:
return
if not _DB_REACHABLE:
raise pytest.UsageError(
"SIFTLODE_TEST_REQUIRE_DB=1 but no Postgres is reachable at the test DATABASE_URL — the "
"DB-backed lane would silently skip. Fix the DB wiring or unset the flag."
"no Postgres is reachable at the test DATABASE_URL — the DB-backed lane would silently "
"skip. Fix the DB wiring, or run the pure-logic tests only with SIFTLODE_TEST_REQUIRE_DB=0."
)
if shutil.which("ffmpeg") is None:
raise pytest.UsageError(
"SIFTLODE_TEST_REQUIRE_DB=1 but ffmpeg is missing — the thumbnail/poster tests would skip."
"ffmpeg is missing — the thumbnail/poster tests would skip. Install it, or run the "
"pure-logic tests only with SIFTLODE_TEST_REQUIRE_DB=0."
)
+21 -6
View File
@@ -36,9 +36,14 @@ def test_a_public_url_gets_past_the_guard(db, as_user, make_user, monkeypatch):
"getaddrinfo",
lambda *a, **kw: [(None, None, None, "", ("93.184.216.34", 0))],
)
r = as_user(user).post("/api/downloads", json={"source": "https://example.com/v"})
detail = r.json().get("detail") if r.status_code >= 400 else None
assert not (isinstance(detail, dict) and detail.get("code") == "url_blocked")
# An inline spec, so nothing ELSE refuses it: without one the request stopped at the
# profile lookup (400 no_profile) and "not url_blocked" held for a request the guard never
# had to wave through (E32 S2).
r = as_user(user).post(
"/api/downloads", json={"source": "https://example.com/v", "spec": {"format": "best"}}
)
assert r.status_code == 200, r.text
assert r.json()["status"] == "queued"
def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeypatch):
@@ -50,7 +55,13 @@ def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeyp
user = make_user()
db.commit()
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
as_user(user).post("/api/downloads", json={"source": "dQw4w9WgXcQ"})
# The job must actually be created: a request refused for any other reason would never reach
# the guard either, and the test would pass without proving anything (E32 S2).
r = as_user(user).post(
"/api/downloads", json={"source": "dQw4w9WgXcQ", "spec": {"format": "best"}}
)
assert r.status_code == 200, r.text
assert r.json()["source_kind"] == "youtube"
def test_manual_subscription_sync_goes_on_cooldown(db, as_user, make_user, monkeypatch):
@@ -81,8 +92,12 @@ def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
ca = as_user(a)
for _ in range(4):
ca.post("/api/sync/subscriptions")
assert [ca.post("/api/sync/subscriptions").status_code for _ in range(4)] == [
200,
200,
200,
429,
]
assert ca.post("/api/sync/subscriptions").status_code == 429
assert as_user(b).post("/api/sync/subscriptions").status_code == 200
+64
View File
@@ -0,0 +1,64 @@
"""sync/autotag's orphan cleanup (E32 S2: the module was 32% covered).
A system tag no channel carries any more is deleted — unless a saved feed view filters on it. The
views hold tag ids; a deleted tag returns under a NEW id when a channel earns it again, and the feed
drops an unknown id, so the view silently showed the whole feed under its old name.
"""
from sqlalchemy import select
from app.models import Channel, ChannelTag, SavedView, Tag
from app.sync import autotag
def test_an_orphan_system_tag_goes_but_one_a_saved_view_filters_on_stays(db, make_user):
user = make_user()
used = Tag(user_id=None, name="Gaming", category="topic")
orphan = Tag(user_id=None, name="Cooking", category="topic")
in_view = Tag(user_id=None, name="Music", category="topic")
own = Tag(user_id=user.id, name="Mine", category="other")
ch = Channel(id="UC_tags", title="T")
db.add_all([used, orphan, in_view, own, ch])
db.flush()
db.add(ChannelTag(channel_id=ch.id, tag_id=used.id, user_id=None, source="auto"))
db.add(
SavedView(
user_id=user.id, source="feed", name="Music", filters={"tags": [in_view.id], "q": ""}
)
)
# A Plex view's filters are another vocabulary; they must not break the scan.
db.add(SavedView(user_id=user.id, source="plex", name="P", filters={"genres": ["Drama"]}))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
db.expire_all()
names = set(db.execute(select(Tag.name)).scalars())
assert names == {"Gaming", "Music", "Mine"} # only the unreferenced system orphan went
def test_a_malformed_saved_view_cannot_break_the_cleanup(db, make_user):
"""A view stored around the route's shape check (or before it) must not raise for everyone's job
— `"tags": 5` raised TypeError — and `True` is not tag id 1 (E32 review)."""
user = make_user()
orphan = Tag(user_id=None, name="Cooking", category="topic")
db.add(orphan)
db.flush()
for i, filters in enumerate(
({"tags": 5}, {"tags": [True]}, ["not", "a", "dict"], {"tags": None})
):
db.add(SavedView(user_id=user.id, source="feed", name=f"bad{i}", filters=filters))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
db.expire_all()
assert db.execute(select(Tag).where(Tag.name == "Cooking")).first() is None
def test_a_view_kept_tag_list_larger_than_a_bind_parameter_limit_still_cleans_up(db, make_user):
user = make_user()
orphan = Tag(user_id=None, name="Cooking", category="topic")
db.add(orphan)
db.flush()
huge = list(range(10_000_000, 10_070_000)) # past psycopg's 65535 bind parameters
db.add(SavedView(user_id=user.id, source="feed", name="huge", filters={"tags": huge}))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
+1 -1
View File
@@ -321,7 +321,7 @@ def _body_reading_functions() -> dict[tuple[str, str], dict]:
continue
names = [name for _, name in calls]
limits = []
for node, name in zip([n for n, _ in calls], names):
for node, name in zip([n for n, _ in calls], names, strict=True):
if name.startswith("request."):
limits.append("<no ceiling declared>")
continue
@@ -0,0 +1,83 @@
"""The channel manager's sums and the feed sidebar's counts read the index alone (E31 S4d).
`GET /api/channels` counts each subscribed channel's videos and sums their durations; `/feed/summary`
and `/feed/years` count the reader's videos per channel and per year under the default filters. They
used to read every video's heap page for that (681 MB on prod), a 2–6 s first open once something else
had pushed the table out of memory. Migration 0095 makes `ix_videos_channel_pub_cover` carry every
column those queries read, so Postgres answers them with Index Only Scans. This test holds the queries
and the index together: a column added to one and not the other fails it. DB lane.
"""
import re
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
from app.feedquery import filtered_query
from app.models import Video
from app.routes.channels import channel_aggregates
INDEX = "ix_videos_channel_pub_cover"
# The feed's own defaults (routes/feed.py _feed_params): what the sidebar asks on a plain open.
FEED_DEFAULTS = {
"tags": [],
"tag_modes": {"language": "or", "topic": "or", "other": "or"},
"untagged": False,
"channel_ids": [],
"q": None,
"min_duration": None,
"max_duration": None,
"max_age_days": None,
"published_after": None,
"published_before": None,
"show_normal": True,
"include_shorts": False,
"include_live": False,
"show": "unwatched",
"scope": "my",
"library_source": "organic",
}
def _index_columns(db) -> list[str]:
# Key columns first, then the INCLUDE ones, in the index's own order.
return list(
db.execute(
text(
"SELECT a.attname FROM pg_index i "
"JOIN pg_class c ON c.oid = i.indexrelid "
"CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS k(attnum, n) "
"JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = k.attnum "
"WHERE c.relname = :name ORDER BY k.n"
),
{"name": INDEX},
).scalars()
)
def _video_columns(stmt) -> set[str]:
# The columns of `videos` the statement as SENT reads: the compiled SQL, not the expression tree,
# which still carries the projection a with_only_columns replaced.
sql = str(stmt.compile(dialect=postgresql.dialect()))
return set(re.findall(r"\bvideos\.(\w+)", sql))
def _assert_covered(db, read: set[str]) -> None:
held = _index_columns(db)
assert held, f"{INDEX} is missing"
# It leads with channel_id, so it also serves every per-channel lookup the old index did.
assert held[0] == "channel_id"
assert read <= set(held), f"read but not in {INDEX}: {sorted(read - set(held))}"
def test_the_channel_sums_read_only_columns_the_index_holds(db):
_assert_covered(db, _video_columns(channel_aggregates(["UC_x"])))
def test_the_feed_sidebar_counts_read_only_columns_the_index_holds(db, make_user):
# feed_summary groups the default filter by channel, feed_years by publish date.
base, _rank = filtered_query(db, make_user(), **FEED_DEFAULTS)
_assert_covered(
db, _video_columns(base.with_only_columns(Video.channel_id, Video.published_at))
)
+24 -14
View File
@@ -151,7 +151,7 @@ def test_nobody_reaches_another_users_session(db, make_user, as_user):
def test_the_list_is_the_users_own_newest_first(db, make_user, as_user):
client = as_user(make_user())
first = _start(client)["id"]
client.post(f"/api/debug-sessions/{first}/stop")
assert client.post(f"/api/debug-sessions/{first}/stop").status_code == 200
second = _start(client)["id"]
assert [s["id"] for s in client.get("/api/debug-sessions").json()["items"]] == [second, first]
@@ -199,10 +199,13 @@ def test_an_erased_account_takes_its_sessions_with_it(db, make_user, as_user):
user = make_user()
client = as_user(user)
sid = _start(client)["id"]
client.post(
f"/api/debug-sessions/{sid}/screenshots",
content=_PNG,
headers={"content-type": "image/png"},
assert (
client.post(
f"/api/debug-sessions/{sid}/screenshots",
content=_PNG,
headers={"content-type": "image/png"},
).status_code
== 200
)
purge_user(db, user, BackgroundTasks())
assert db.get(DebugSession, sid) is None
@@ -250,7 +253,7 @@ def test_a_request_naming_the_own_open_session_is_tagged_and_raised(gated, db, c
_signed_in(client, make_user)
sid = _start(client, "DEBUG")["id"]
gated.records.clear()
client.get("/api/debug-sessions", headers={"x-debug-session": sid})
assert client.get("/api/debug-sessions", headers={"x-debug-session": sid}).status_code == 200
(line,) = _request_lines(gated)
assert getattr(line, "debug_session_id", None) == sid
# Raised to the session's level: the line is kept for storage though the user's own is Alap.
@@ -263,7 +266,7 @@ def test_a_header_naming_anything_else_is_ignored_not_refused(gated, db, client,
db.commit()
_signed_in(client, make_user)
own = _start(client)["id"]
client.post(f"/api/debug-sessions/{own}/stop")
assert client.post(f"/api/debug-sessions/{own}/stop").status_code == 200
for value in (foreign, own, "not-a-session", "f" * 16, "A" * 16):
gated.records.clear()
# Not `/api/me`: a polled route logs at DEBUG, and an un-raised request would drop the very
@@ -280,7 +283,7 @@ def test_a_header_naming_the_own_session_past_its_cap_is_ignored(gated, db, clie
sid = _start(client)["id"]
_expire(db, sid) # past its cap, not yet swept: still open in the table
gated.records.clear()
client.get("/api/debug-sessions", headers={"x-debug-session": sid})
assert client.get("/api/debug-sessions", headers={"x-debug-session": sid}).status_code == 200
(line,) = _request_lines(gated)
assert getattr(line, "debug_session_id", None) is None
assert getattr(line, "elevated", False) is False
@@ -616,7 +619,7 @@ def test_a_users_screenshots_are_capped_across_their_sessions(recording, monkeyp
client, sid = recording
monkeypatch.setattr(debugsession, "SCREENSHOT_USER_MAX_BYTES", len(_PNG) * 2)
assert _upload(client, sid, _PNG).status_code == 200
client.post(f"/api/debug-sessions/{sid}/stop")
assert client.post(f"/api/debug-sessions/{sid}/stop").status_code == 200
other = _start(client)["id"]
assert _upload(client, other, _PNG).status_code == 200
r = _upload(client, other, _PNG)
@@ -631,7 +634,7 @@ def test_new_sessions_are_paced_but_joining_one_is_not(db, make_user, as_user, m
client = as_user(make_user())
sid = _start(client)["id"]
assert _start(client)["id"] == sid # another tab joining
client.post(f"/api/debug-sessions/{sid}/stop")
assert client.post(f"/api/debug-sessions/{sid}/stop").status_code == 200
r = client.post("/api/debug-sessions", json={"level": "DEBUG"})
assert r.status_code == 429 and r.json()["detail"]["reason"] == "debug_session_rate_limited"
@@ -745,9 +748,16 @@ def test_the_export_is_the_saved_draft_the_evidence_and_the_screenshots(db, make
db.commit()
_upload(client, sid, _PNG)
_upload(client, sid, _JPEG)
client.put(
f"/api/debug-sessions/{sid}/draft",
json={"title": "Spinner stays", "body": "It spins.", "excluded": [dropped_web.id, 999999]},
assert (
client.put(
f"/api/debug-sessions/{sid}/draft",
json={
"title": "Spinner stays",
"body": "It spins.",
"excluded": [dropped_web.id, 999999],
},
).status_code
== 200
)
r = client.get(f"/api/debug-sessions/{sid}/export")
@@ -1144,7 +1154,7 @@ def test_new_issues_are_paced_per_user(gitea_fake, db, make_user, as_user, monke
first = _start(client)["id"]
_ready(client, first)
assert client.post(f"/api/debug-sessions/{first}/issue").status_code == 200
client.post(f"/api/debug-sessions/{first}/stop")
assert client.post(f"/api/debug-sessions/{first}/stop").status_code == 200
second = _start(client)["id"]
_ready(client, second)
r = client.post(f"/api/debug-sessions/{second}/issue")
+26 -11
View File
@@ -212,7 +212,12 @@ def test_a_timestamp_outside_the_consent_window_is_clamped(elevated):
now = datetime.now(timezone.utc)
future = (now + timedelta(days=365)).isoformat()
past = (now - timedelta(days=30)).isoformat()
client.post("/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]})
assert (
client.post(
"/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]}
).status_code
== 200
)
late, early = rows()
assert late.ts <= datetime.now(timezone.utc)
assert early.ts >= now - timedelta(hours=diagnostics.ELEVATION_HOURS, minutes=1)
@@ -342,15 +347,15 @@ def _app() -> TestClient:
def test_the_account_reaches_the_handlers_own_lines(gated):
_app().get("/api/thing/41")
assert _app().get("/api/thing/41").status_code == 200
line = next(r for r in gated.records if r.getMessage() == "an ordinary line")
assert getattr(line, "user_id", None) == 41
def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG"})
client.get("/api/thing/8")
assert client.get("/api/thing/7", params={"level": "DEBUG"}).status_code == 200
assert client.get("/api/thing/8").status_code == 200
detail = [r for r in gated.records if r.getMessage().startswith("detail only")]
assert [r.user_id for r in detail] == [7]
assert getattr(detail[0], "elevated", False) is True
@@ -363,9 +368,9 @@ def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gate
def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
client = _app()
client.get("/api/thing/7", params={"level": "TRACE"})
assert client.get("/api/thing/7", params={"level": "TRACE"}).status_code == 200
assert logging.getLogger().level == logs.TRACE
client.get("/api/thing/7") # the same user, back at the base
assert client.get("/api/thing/7").status_code == 200 # the same user, back at the base
assert logging.getLogger().level == logging.INFO
logs._track_elevation(9, logging.DEBUG, datetime.now(timezone.utc) - timedelta(seconds=1))
assert logging.getLogger().level == logging.INFO # an already-expired one is pruned at once
@@ -373,15 +378,25 @@ def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
def test_the_query_string_is_logged_only_for_a_traced_request_and_redacted(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"})
client.get("/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"})
assert client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"}).status_code == 200
assert (
client.get(
"/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"}
).status_code
== 200
)
lines = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in lines[0].attrs
assert "q=lofi" in lines[1].attrs["query"] and "SECRET" not in lines[1].attrs["query"]
def test_an_address_typed_into_a_search_is_redacted_though_the_browser_encodes_it(gated):
_app().get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
assert (
_app()
.get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
.status_code
== 200
)
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert line.attrs["query"] == "level=TRACE&q=[email]&new_password=[redacted]"
@@ -400,7 +415,7 @@ def test_a_credential_param_goes_whole_however_its_value_decodes(raw):
def test_an_admin_level_of_trace_does_not_log_other_users_query_strings(gated):
logs.set_level("TRACE")
_app().get("/api/thing/7", params={"q": "something private"})
assert _app().get("/api/thing/7", params={"q": "something private"}).status_code == 200
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in line.attrs
@@ -488,7 +503,7 @@ def test_a_real_sign_in_carries_the_consented_level_into_the_process(gated, clie
user = _signed_in(client, make_user)
assert client.put("/api/me/diagnostics", json={"level": "DEBUG"}).status_code == 200
logs._elevations.clear() # only a request resolving the account through auth may put it back
client.get("/api/me")
assert client.get("/api/me").status_code == 200
assert logs._elevations[user.id][0] == logging.DEBUG
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
assert user.id not in logs._elevations
+3 -3
View File
@@ -13,12 +13,12 @@ hold for Google Workspace custom domains unless their admin enabled it.
import pytest
from sqlalchemy import select, text
_UAT_PW = "correct horse battery"
from app.auth import is_allowed
from app.models import Invite, User
from app.utils import canonical_email
_UAT_PW = "correct horse battery"
# (input, expected canonical) — the matrix the sprint promised, plus the shapes that must NOT fold.
CASES = [
# Gmail: dots and +tags are noise…
@@ -334,7 +334,7 @@ def test_me_reports_the_google_address_only_when_it_differs(client, db, make_use
)
assert client.get("/api/me").json()["google_email"] is None
client.post("/auth/logout")
assert client.post("/auth/logout").status_code == 200
tagged = make_user(email="me+siftlode@gmail.com", password=_UAT_PW)
tagged.google_email = "me@gmail.com"
db.commit()
+259
View File
@@ -0,0 +1,259 @@
"""An empty answer is not an emptied library (E32 S2).
Every mirror here reconciles: what the remote no longer lists is removed locally. The catalog
(`_prune_stale`) and the subscriptions (`should_prune`) had long refused to do that on an empty
listing that contradicts what we hold — a 200 with no items is a blip far more often than a user
who removed everything. Two mirrors did not: the Plex collections (a row carries the user-set
`editable` flag, and every member its keys) and the YouTube playlist mirrors. These hold both — and,
one level down (E32 review), a collection's members and a playlist's items: an empty answer that
contradicts the remote's own count is not believed either.
"""
import pytest
from sqlalchemy import select
from app.models import (
Channel,
Playlist,
PlaylistItem,
PlexCollection,
PlexItem,
PlexLibrary,
Video,
)
from app.plex import sync as plex_sync
from app.sync import playlists as plsync
from app.youtube.client import YouTubeError
class _Plex:
def __init__(self, collections: list[dict], children: dict[str, list[dict]]):
self._collections = collections
self._children = children
def collections(self, section_key):
return self._collections
def collection_children(self, rk):
return self._children.get(rk, [])
def metadata(self, rk):
return next((c for c in self._collections if c.get("ratingKey") == rk), {})
def _library_with_a_collection(db):
lib = PlexLibrary(plex_key="4", title="Movies", kind="movie")
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="c1", library_id=lib.id, title="Bond", editable=True))
movie = PlexItem(
rating_key="m1", library_id=lib.id, kind="movie", title="Dr. No", collection_keys=["c1"]
)
db.add(movie)
db.commit()
return lib, movie
def test_an_empty_collection_listing_keeps_the_collections_and_their_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
plex_sync._sync_collections(db, _Plex([], {}), lib, stats)
db.commit()
db.expire_all()
col = db.execute(select(PlexCollection).where(PlexCollection.rating_key == "c1")).scalar_one()
assert col.editable is True
assert db.get(PlexItem, movie.id).collection_keys == ["c1"]
def test_a_listing_with_something_in_it_still_prunes_what_it_no_longer_has(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
other = {"ratingKey": "c2", "title": "Alien"}
plex_sync._sync_collections(db, _Plex([other], {"c2": []}), lib, stats)
db.commit()
db.expire_all()
keys = set(db.execute(select(PlexCollection.rating_key)).scalars())
assert keys == {"c2"}
assert db.get(PlexItem, movie.id).collection_keys is None
def test_no_members_from_a_collection_plex_counts_as_non_empty_keeps_its_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
bond = {"ratingKey": "c1", "title": "Bond", "childCount": 2}
plex_sync._sync_collections(db, _Plex([bond], {"c1": []}), lib, stats)
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys == ["c1"]
def test_a_collection_plex_counts_as_empty_does_lose_its_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
bond = {"ratingKey": "c1", "title": "Bond", "childCount": 0}
plex_sync._sync_collections(db, _Plex([bond], {"c1": []}), lib, stats)
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys is None
@pytest.mark.parametrize(("count", "kept"), [(2, ["c1"]), (0, None)])
def test_a_collection_resync_believes_an_empty_member_list_only_with_a_zero_count(db, count, kept):
lib, movie = _library_with_a_collection(db)
bond = {"ratingKey": "c1", "title": "Bond", "childCount": count}
plex_sync.resync_collection(db, _Plex([bond], {"c1": []}), lib, "c1")
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys == kept
class _YT:
def __init__(self, playlists: list[dict], snippet: dict | None = None):
self._playlists = playlists
self._snippet = snippet
def __call__(self, *_):
return self
def __enter__(self):
return self
def __exit__(self, *_):
return False
def iter_my_playlists(self):
return self._playlists
def iter_playlist_items_with_ids(self, _ytid):
return []
def iter_my_playlist_video_ids(self, _ytid):
return []
def get_playlist_snippet(self, _ytid):
return self._snippet
def get_videos(self, _ids):
return []
def test_an_empty_playlist_answer_keeps_the_youtube_mirrors(db, make_user, monkeypatch):
user = make_user()
db.add(Playlist(user_id=user.id, name="Mixes", source="youtube", yt_playlist_id="PL1"))
db.commit()
monkeypatch.setattr(plsync, "YouTubeClient", _YT([]))
plsync.pull_user_playlists(db, user)
db.expire_all()
names = db.execute(select(Playlist.name).where(Playlist.user_id == user.id)).scalars().all()
assert "Mixes" in names
def test_a_playlist_answer_with_something_in_it_still_drops_a_vanished_mirror(
db, make_user, monkeypatch
):
user = make_user()
db.add(Playlist(user_id=user.id, name="Mixes", source="youtube", yt_playlist_id="PL1"))
db.commit()
monkeypatch.setattr(plsync, "YouTubeClient", _YT([{"id": "PL2", "title": "New"}]))
plsync.pull_user_playlists(db, user)
db.expire_all()
ytids = set(
db.execute(select(Playlist.yt_playlist_id).where(Playlist.source == "youtube")).scalars()
)
assert "PL1" not in ytids
class _YTOneGone(_YT):
"""A list deleted on YouTube between the listing and the read of its items."""
def iter_my_playlist_video_ids(self, ytid):
if ytid == "PLgone":
raise YouTubeError("m", reason="playlistNotFound", path="playlistItems")
return []
def test_a_list_deleted_mid_pull_is_skipped_and_the_rest_still_sync(db, make_user, monkeypatch):
# Not "this playlist no longer exists" for a bulk pull the user never named a list in, and not
# a failure for every other list either (E32 review round 2): the next pull drops the mirror.
user = make_user()
db.commit()
listing = [
{"id": "PLgone", "title": "Gone", "item_count": 3},
{"id": "PL2", "title": "Kept", "item_count": 0},
]
monkeypatch.setattr(plsync, "YouTubeClient", _YTOneGone(listing))
assert plsync.pull_user_playlists(db, user) == {"synced": 1}
db.expire_all()
ytids = set(
db.execute(select(Playlist.yt_playlist_id).where(Playlist.user_id == user.id)).scalars()
)
assert ytids == {"PL2"}
def _mirror_with_an_item(db, user, **kw):
db.add(Channel(id="UC_pl", title="C"))
db.flush()
db.add(Video(id="v1", channel_id="UC_pl", title="One"))
pl = Playlist(user_id=user.id, name="Mixes", yt_playlist_id="PL1", **kw)
db.add(pl)
db.flush()
db.add(PlaylistItem(playlist_id=pl.id, video_id="v1", position=0))
db.commit()
return pl
def _items(db, pl):
db.expire_all()
return (
db.execute(select(PlaylistItem.video_id).where(PlaylistItem.playlist_id == pl.id))
.scalars()
.all()
)
def test_no_items_from_a_playlist_youtube_counts_as_non_empty_keeps_the_mirror_items(
db, make_user, monkeypatch
):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
listing = [{"id": "PL1", "title": "Mixes", "item_count": 1}]
monkeypatch.setattr(plsync, "YouTubeClient", _YT(listing))
plsync.pull_user_playlists(db, user)
assert _items(db, pl) == ["v1"]
def test_a_playlist_youtube_counts_as_empty_is_emptied(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
listing = [{"id": "PL1", "title": "Mixes", "item_count": 0}]
monkeypatch.setattr(plsync, "YouTubeClient", _YT(listing))
plsync.pull_user_playlists(db, user)
assert _items(db, pl) == []
def test_a_reset_to_youtube_refuses_a_blank_listing_and_keeps_the_items(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
monkeypatch.setattr(plsync, "YouTubeClient", _YT([], {"title": "Mixes", "item_count": 1}))
with pytest.raises(YouTubeError) as err:
plsync.repull_playlist(db, user, pl)
db.rollback()
assert err.value.reason == "emptyAnswer"
assert _items(db, pl) == ["v1"]
def test_a_reset_of_a_deleted_playlist_says_it_is_gone(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
monkeypatch.setattr(plsync, "YouTubeClient", _YT([], None))
with pytest.raises(YouTubeError) as err:
plsync.repull_playlist(db, user, pl)
assert err.value.reason == "playlistNotFound"
def test_a_push_plan_refuses_a_blank_listing_rather_than_reading_it_as_all_deleted(db, make_user):
user = make_user()
pl = _mirror_with_an_item(db, user, source="local")
with pytest.raises(YouTubeError) as err:
plsync.plan_merge(db, user, pl, live=([], {"title": "Mixes", "item_count": 1}))
assert err.value.reason == "emptyAnswer"
+133
View File
@@ -0,0 +1,133 @@
"""Relevance ranks only the newest RANK_WINDOW matches (E31), DB-lane.
A broad word ("a") matched most of the prod library and ts_rank read every match's tsvector: 6.8 s a page.
The ranking now covers the newest RANK_WINDOW matches and the rest follow newest first. The window's start
rides the cursor, so a video ingested mid-scroll cannot push a row behind the cursor and repeat it.
"""
import base64
import json
from datetime import datetime, timezone
import pytest
from app import feedquery
from app.models import Channel, Subscription, User, Video
def _video(vid: str, day: int, title: str) -> Video:
return Video(
id=vid,
channel_id="UCw",
published_at=datetime(2024, 1, day, tzinfo=timezone.utc),
title=title,
is_short=False,
live_status="none",
via_search=False,
via_explore=False,
)
def _seed(db) -> User:
db.add(Channel(id="UCw", title="Window"))
user = User(email="window@example.com", email_verified=True)
db.add(user)
db.flush()
db.add(Subscription(user_id=user.id, channel_id="UCw", priority=0, hidden=False))
# m1 is the oldest and the strongest match, m4 the strongest of the newest three; the rest tie on one
# title "foo" each (ts_rank counts occurrences, it does not normalise by length).
db.add_all(
[
_video("m1", 1, "foo foo foo foo foo"),
_video("m2", 2, "foo and some other words here"),
_video("m3", 3, "foo plus some filler words"),
_video("m4", 4, "foo foo foo"),
_video("m5", 5, "foo with a longer title"),
_video("m6", 6, "foo with yet another title"),
]
)
db.commit()
return user
def _ids(client, cursor=None, limit=50):
url = f"/api/feed?q=foo&sort=relevance&limit={limit}"
if cursor:
url += f"&cursor={cursor}"
resp = client.get(url)
assert resp.status_code == 200, resp.text
body = resp.json()
return [it["id"] for it in body["items"]], body["next_cursor"]
def test_a_specific_word_ranks_every_match(db, as_user):
client = as_user(_seed(db))
ids, _ = _ids(client)
assert ids[:2] == ["m1", "m4"], ids
def test_a_broad_word_ranks_the_newest_window_and_the_rest_follow_newest_first(
db, as_user, monkeypatch
):
monkeypatch.setattr(feedquery, "RANK_WINDOW", 3)
client = as_user(_seed(db))
ids, _ = _ids(client)
assert ids[0] == "m4", ids # the strongest of the newest three
assert sorted(ids[:3]) == ["m4", "m5", "m6"], ids
assert ids[3:] == ["m3", "m2", "m1"], ids # past the window: newest first, m1 not ranked
def test_paging_keeps_the_window_page_one_ranked_against(db, as_user, monkeypatch):
monkeypatch.setattr(feedquery, "RANK_WINDOW", 3)
client = as_user(_seed(db))
full, _ = _ids(client)
first, cursor = _ids(client, limit=2)
assert first == full[:2]
# A newer match lands mid-scroll. Ranked against a recomputed window, m4 (shown on page one) would
# fall out of it to rank 0, sort behind the cursor and come back on page two.
db.add(_video("m7", 7, "foo arriving while the reader scrolls"))
db.commit()
walked = list(first)
while cursor:
page, cursor = _ids(client, cursor, limit=2)
walked.extend(page)
assert len(walked) == len(set(walked)), walked
assert set(full) <= set(walked), walked
@pytest.mark.parametrize(
("window", "limit"),
# 3: the page is read as its two halves (the window ranked, the rest newest first), and pages of 2
# and 3 cross the window's edge mid-page and exactly at it. 2000: one ranked query.
[(3, 2), (3, 3), (2000, 2)],
)
def test_cursor_walk_matches_the_unpaged_order(db, as_user, monkeypatch, window, limit):
monkeypatch.setattr(feedquery, "RANK_WINDOW", window)
client = as_user(_seed(db))
full, _ = _ids(client)
if window == 3:
assert full == ["m4", "m6", "m5", "m3", "m2", "m1"]
walked, cursor = _ids(client, limit=limit)
while cursor:
page, cursor = _ids(client, cursor, limit=limit)
walked.extend(page)
assert walked == full
@pytest.mark.parametrize(
"payload",
[
{"k": [5, None], "id": "m1", "w": "2024-01-01T00:00:00+00:00"}, # null published_at
{"k": [5, "2024-01-03T00:00:00+00:00"], "id": "m1", "w": "2024-01-01T00:00:00"}, # naive w
{"k": [5, "2024-01-03T00:00:00"], "id": "m1", "w": "2024-01-01T00:00:00+00:00"}, # naive k
],
)
def test_a_hand_made_relevance_cursor_is_refused_not_a_crash(db, as_user, payload):
# The page halves compare the cursor's timestamp with the window's in Python: a null or a naive
# one raised there, past the route's 400 — a 500 (E31 review 2).
client = as_user(_seed(db))
cursor = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode()
resp = client.get(f"/api/feed?q=foo&sort=relevance&limit=2&cursor={cursor}")
assert resp.status_code == 400, resp.text
+100
View File
@@ -0,0 +1,100 @@
"""Every feed sort on a videos column rides an index, and an ICU upgrade rebuilds the title ones (E31 S4e).
Without an index in the exact ORDER BY, each feed page read the whole videos heap and kept a top-N
(prod: 350 ms a page for views/duration, 550–780 ms for title). Migration 0096 adds one per direction
and per reader language. The planner is told it may not sort or scan the table, so a plan that still
sorts it whole means no index matches the ORDER BY `feedquery.order_by` builds: a key, direction, NULLS
rule or collation changed on one side only. An Incremental Sort is fine: the index gives the key's order
and only its ties are sorted by id (`newest`/`oldest` ride the published_at indexes that way). DB lane.
"""
import re
import pytest
from sqlalchemy import select, text
from sqlalchemy.dialects import postgresql
from app import collation
from app.db import engine
from app.feedquery import order_by, sort_keys
from app.models import Video
VIDEO_SORTS = [
"newest",
"oldest",
"views",
"views_asc",
"duration_desc",
"duration_asc",
"title",
"title_desc",
]
def _plan(db, sort: str, lang: str) -> str:
stmt = select(Video.id).order_by(*order_by(sort_keys(sort, 0, None, lang))).limit(60)
sql = str(stmt.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
db.execute(text("SET LOCAL enable_seqscan = off"))
db.execute(text("SET LOCAL enable_sort = off"))
db.execute(text("SET LOCAL enable_bitmapscan = off"))
return "\n".join(db.execute(text(f"EXPLAIN {sql}")).scalars())
@pytest.mark.parametrize("lang", sorted(collation.ICU))
@pytest.mark.parametrize("sort", VIDEO_SORTS)
def test_the_sort_walks_an_index(db, sort, lang):
plan = _plan(db, sort, lang)
db.rollback()
full_sort = re.search(r"(?<!Incremental )Sort \(", plan)
assert not full_sort, f"{sort} ({lang}) sorts the table:\n{plan}"
assert "Index Scan" in plan or "Index Only Scan" in plan, plan
def _drifted(db) -> list[str]:
return list(
db.execute(
text(
"SELECT collname FROM pg_collation WHERE collname = ANY(:names) "
"AND collversion IS DISTINCT FROM pg_collation_actual_version(oid)"
),
{"names": list(collation.ICU.values())},
).scalars()
)
def _files(db) -> dict[str, int]:
return dict(
db.execute(
text("SELECT relname, relfilenode FROM pg_class WHERE relname LIKE 'ix_videos_title_%'")
).all()
)
def test_an_icu_upgrade_rebuilds_the_title_indexes(db):
# What an image update with a newer ICU looks like from inside: the recorded version no longer
# matches the library's (the test database runs as the bootstrap superuser, as the app's does).
db.execute(text("UPDATE pg_collation SET collversion = '0.0' WHERE collname = 'hu-x-icu'"))
db.commit()
try:
assert _drifted(db) == ["hu-x-icu"]
before = _files(db)
# A concurrent rebuild waits for every open transaction, this session's too.
db.rollback()
rebuilt = collation.heal_drift(engine)
assert sorted(rebuilt) == ["ix_videos_title_hu", "ix_videos_title_hu_desc"]
after = _files(db)
# Rebuilt = written to new files; the English pair, on an unchanged ICU, is left alone.
assert {n for n in before if before[n] != after[n]} == set(rebuilt)
assert _drifted(db) == []
finally:
db.execute(
text(
"UPDATE pg_collation SET collversion = pg_collation_actual_version(oid) "
"WHERE collname = 'hu-x-icu'"
)
)
db.commit()
def test_no_drift_rebuilds_nothing(db):
assert collation.heal_drift(engine) == []
+43
View File
@@ -12,6 +12,8 @@ from datetime import datetime, timezone
import pytest
from app import feedquery
from app.models import Channel, ChannelTag, Subscription, Tag, Video, VideoState
@@ -168,9 +170,50 @@ _SUMMARY_CASES = [
"show=all&untagged=true&tags={L1}",
"show=all&q=clip",
"show=unwatched",
# A deleted tag's id (a saved view or the feed state still holds it) matches nothing.
"show=all&tags=987654",
"show=all&tags={L1}&tags=987654",
]
def test_a_deleted_tags_id_filters_to_nothing_not_to_everything(db, as_user, make_user):
"""A saved view or the feed's filter state keeps a tag id after the tag is deleted. The feed used
to drop an unknown id, so that view silently showed the WHOLE feed under its old name (E32 S2).
A filter on a tag that no longer exists matches nothing; the chip says the tag is gone."""
user, tag_ids = _seed_tagged(db, make_user)
client = as_user(user)
everything = client.get("/api/feed/count?show=all")
assert everything.status_code == 200
assert everything.json()["count"] == 8
for qs in ("show=all&tags=987654", f"show=all&tags={tag_ids['L1']}&tags=987654"):
r = client.get(f"/api/feed/count?{qs}")
assert r.status_code == 200
assert r.json()["count"] == 0, qs
r = client.get(f"/api/feed?{qs}")
assert r.status_code == 200
assert r.json()["items"] == [], qs
def test_another_users_private_tag_answers_like_one_that_does_not_exist(db, as_user, make_user):
"""The category split resolved any tag id, so a foreign PRIVATE tag landed in its real category
and the summary's facets differed from a non-existent id's: an oracle for which foreign ids exist
(E32 review). Both must now read as "missing"."""
user, tag_ids = _seed_tagged(db, make_user)
stranger = make_user(email="stranger@example.com")
private = Tag(user_id=stranger.id, name="Secret", category="language")
db.add(private)
db.commit()
assert feedquery.tags_by_category(db, user, [tag_ids["L1"], private.id]) == {
"language": [tag_ids["L1"]],
feedquery.MISSING_TAG_CATEGORY: [private.id],
}
client = as_user(user)
foreign = client.get(f"/api/feed/summary?show=all&tags={tag_ids['T1']}&tags={private.id}")
absent = client.get(f"/api/feed/summary?show=all&tags={tag_ids['T1']}&tags=987654")
assert foreign.status_code == absent.status_code == 200
assert foreign.json() == absent.json()
def test_untagged_filter_keeps_only_channels_the_user_has_not_tagged(db, as_user, make_user):
"""The "No tag" filter (item B): `untagged=true` keeps only channels with NO tag the USER personally
attached (no user-owned ChannelTag). System/auto tags (user_id NULL) don't count, and another user's
+6 -1
View File
@@ -100,7 +100,12 @@ def test_the_admin_user_list_carries_last_login(as_user, db, make_user):
def test_the_account_sees_its_own_last_sign_in(client, db, make_user):
# Settings → Account shows it (prod UAT 2026-09-19: only the admin's user list did).
make_user(email="me@example.com", password=_PW)
client.post("/auth/password-login", json={"email": "me@example.com", "password": _PW})
assert (
client.post(
"/auth/password-login", json={"email": "me@example.com", "password": _PW}
).status_code
== 200
)
stamp = client.get("/api/me").json()["last_login_at"]
+5 -5
View File
@@ -277,7 +277,7 @@ def test_a_signed_in_request_is_never_capped_and_names_its_account(sink):
client = TestClient(app)
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
client.get("/api/me")
assert client.get("/api/me").status_code == 200
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
assert all(getattr(r, logs.PERSIST, True) for r in records)
assert sink.objects()[-1]["user_id"] == 42 and "user_id" not in sink.objects()[-1]["attrs"]
@@ -325,9 +325,9 @@ def test_the_log_consoles_own_reads_are_debug_but_its_failures_are_not(sink):
http = logging.getLogger("siftlode.http")
http.setLevel(logs.TRACE)
client = _client()
client.get("/api/admin/logs/facets")
assert client.get("/api/admin/logs/facets").status_code == 200
assert sink.objects()[-1]["level"] == "DEBUG"
client.get("/api/admin/logs/facets?fail=1")
assert client.get("/api/admin/logs/facets?fail=1").status_code == 400
assert sink.objects()[-1]["level"] == "WARNING"
@@ -344,7 +344,7 @@ def test_a_polled_read_is_debug_a_write_to_the_same_route_is_not(sink):
r = client.post("/api/messages/unread_count")
assert sink.objects()[-1]["level"] == "INFO"
assert "X-Log-Quiet" not in r.headers
client.get("/api/videos/abc")
assert client.get("/api/videos/abc").status_code == 200
assert sink.objects()[-1]["level"] == "INFO"
@@ -360,7 +360,7 @@ def test_every_polled_route_is_a_route_the_app_has():
def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
client = _client()
client.get("/api/videos/missing/deeper") # 404 from the router
assert client.get("/api/videos/missing/deeper").status_code == 404 # 404 from the router
assert sink.objects()[-1]["level"] == "WARNING"
client.get("/boom")
payload = sink.objects()[-1]
+1 -1
View File
@@ -259,7 +259,7 @@ def test_the_writer_drains_what_was_queued_when_it_is_stopped(db):
log.addHandler(sink)
try:
for i in range(logsink.BATCH_MAX + 5):
# noqa G004: the f-string is the POINT — a distinct rendered message per line is
# G004: the f-string is the POINT — a distinct rendered message per line is
# what keeps the flood limiter (which groups by template) out of this case.
log.warning(f"line {i}") # noqa: G004
finally:
+37 -7
View File
@@ -154,8 +154,18 @@ def test_an_access_request_records_the_language_it_was_made_in(db, client):
def test_asking_again_from_another_language_updates_the_row(db, client):
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "en"})
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "en"}
).status_code
== 200
)
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
db.expire_all()
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
@@ -165,7 +175,12 @@ def test_asking_again_from_another_language_updates_the_row(db, client):
def test_an_unsupported_language_is_ignored_rather_than_stored(db, client):
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "../../etc"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "../../etc"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
assert inv.lang is None
@@ -194,7 +209,12 @@ def test_the_confirmation_mail_is_written_in_the_accounts_language(db, client, m
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
client.post("/auth/register", json={"email": "hu@example.com", "password": _PW, "lang": "hu"})
assert (
client.post(
"/auth/register", json={"email": "hu@example.com", "password": _PW, "lang": "hu"}
).status_code
== 200
)
(_, subject, body) = sent[0]
assert subject == mailtext.MAIL_TEXT["verify_email.subject"]["hu"]
@@ -209,7 +229,12 @@ def test_the_approval_mail_answers_in_the_language_the_request_was_made_in(
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
admin = make_user(email="admin@example.com", role="admin")
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
sent.clear() # drop the admin notification; this test is about the reply to the requester
@@ -227,11 +252,16 @@ def test_a_denial_answers_in_that_language_too(db, client, make_user, as_user, m
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
admin = make_user(email="admin@example.com", role="admin")
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
sent.clear()
as_user(admin).post(f"/api/admin/invites/{inv.id}/deny")
assert as_user(admin).post(f"/api/admin/invites/{inv.id}/deny").status_code == 200
assert sent[0][1] == mailtext.MAIL_TEXT["access_denied.subject"]["hu"]
+182
View File
@@ -0,0 +1,182 @@
"""sync/maintenance — the job that HARD-DELETES videos (E32 S2: 19% covered).
Its lifecycle: a video missing from videos.list is flagged and hidden; after the grace period a
still-missing one is deleted (cascading to every playlist that held it) and each affected user
gets one notification; a video that answers again is unhidden. A live broadcast is never a delete
trigger, and an upcoming premiere that never went live is. YouTube is a fake that answers
videos.list from a dict.
"""
from datetime import timedelta
import pytest
from sqlalchemy import select
from app.models import Channel, Notification, Playlist, PlaylistItem, Video
from app.sync import maintenance
from app.utils import now_utc
class _YT:
def __init__(self, items: dict[str, dict]):
self.items = items
self.asked: list[list[str]] = []
def get_videos(self, ids):
self.asked.append(list(ids))
return [self.items[i] for i in ids if i in self.items]
def _item(vid: str, **snippet) -> dict:
return {"id": vid, "snippet": {"title": vid, **snippet}}
@pytest.fixture
def channel(db):
ch = Channel(id="UC_maint", title="Maint")
db.add(ch)
db.flush()
return ch
def test_a_flagged_video_that_answers_again_is_unhidden(db, channel):
v = Video(
id="v1", channel_id=channel.id, unavailable_since=now_utc(), unavailable_reason="removed"
)
db.add(v)
db.commit()
out = maintenance._recheck_flagged(db, _YT({"v1": _item("v1")}), now_utc())
db.refresh(v)
assert v.unavailable_since is None and v.unavailable_reason is None
assert out == {"checked": 1, "recovered": 1, "deleted": 0, "notified": 0}
def test_a_video_still_missing_inside_its_grace_period_is_kept_hidden(db, channel):
v = Video(
id="v1",
channel_id=channel.id,
unavailable_since=now_utc() - timedelta(days=maintenance.DEFAULT_GRACE_DAYS - 1),
unavailable_reason="removed",
)
db.add(v)
db.commit()
out = maintenance._recheck_flagged(db, _YT({}), now_utc())
assert db.get(Video, "v1") is not None
assert out["deleted"] == 0
def test_past_the_grace_period_it_is_deleted_and_each_user_who_kept_it_told_once(
db, channel, make_user
):
a, b = make_user(), make_user()
gone_since = now_utc() - timedelta(days=maintenance.DEFAULT_GRACE_DAYS + 1)
for vid in ("v1", "v2"):
db.add(
Video(
id=vid,
channel_id=channel.id,
title=f"T {vid}",
unavailable_since=gone_since,
unavailable_reason="removed",
)
)
pa = Playlist(user_id=a.id, name="A")
pb = Playlist(user_id=b.id, name="B")
db.add_all([pa, pb])
db.flush()
db.add_all(
[
PlaylistItem(playlist_id=pa.id, video_id="v1", position=0),
PlaylistItem(playlist_id=pa.id, video_id="v2", position=1),
PlaylistItem(playlist_id=pb.id, video_id="v2", position=0),
]
)
db.commit()
out = maintenance._recheck_flagged(db, _YT({}), now_utc())
db.expire_all()
assert db.get(Video, "v1") is None and db.get(Video, "v2") is None
assert db.execute(select(PlaylistItem)).scalars().all() == [] # the cascade
notes = {
n.user_id: n.data["count"]
for n in db.execute(select(Notification).where(Notification.type == "maintenance"))
.scalars()
.all()
}
assert notes == {a.id: 2, b.id: 1} # one per user, not one per video
assert out == {"checked": 2, "recovered": 0, "deleted": 2, "notified": 2}
def test_rolling_revalidation_flags_the_missing_and_keeps_a_live_stream(db, channel, monkeypatch):
monkeypatch.setattr(maintenance.sysconfig, "get_int", lambda db, key: 10)
monkeypatch.setattr(maintenance.quota, "backfill_reserve_reached", lambda: False)
db.add_all(
[
Video(id="ok", channel_id=channel.id),
Video(id="gone", channel_id=channel.id),
Video(id="live", channel_id=channel.id),
]
)
db.commit()
yt = _YT({"ok": _item("ok"), "live": _item("live", liveBroadcastContent="live")})
out = maintenance._revalidate_rolling(db, yt, now_utc())
db.expire_all()
assert db.get(Video, "gone").unavailable_reason == "removed"
assert db.get(Video, "ok").unavailable_since is None
assert db.get(Video, "live").unavailable_since is None
assert out == {"checked": 3, "flagged": 1}
assert all(db.get(Video, v).last_checked_at is not None for v in ("ok", "gone", "live"))
def test_a_run_asks_youtube_about_each_video_once(db, channel, monkeypatch, make_user):
# Phase 1 recovers v1; phase 2 must not take it as "not checked this run" and fetch it again.
monkeypatch.setattr(maintenance.sysconfig, "get_int", lambda db, key: 10)
monkeypatch.setattr(maintenance.quota, "backfill_reserve_reached", lambda: False)
yt = _YT({"v1": _item("v1"), "v2": _item("v2")})
class _Client:
def __init__(self, *_):
pass
def __enter__(self):
return yt
def __exit__(self, *_):
return False
monkeypatch.setattr(maintenance, "YouTubeClient", _Client)
monkeypatch.setattr(maintenance, "get_service_user", lambda db: make_user())
db.add_all(
[
Video(
id="v1",
channel_id=channel.id,
unavailable_since=now_utc(),
unavailable_reason="removed",
),
Video(id="v2", channel_id=channel.id),
]
)
db.commit()
maintenance.run_maintenance(db)
asked = [vid for batch in yt.asked for vid in batch]
assert sorted(asked) == ["v1", "v2"]
def test_an_upcoming_premiere_counts_as_abandoned_only_when_it_never_went_live():
past = (now_utc() - timedelta(days=3)).isoformat()
soon = (now_utc() - timedelta(days=1)).isoformat()
def item(**live):
return {"snippet": {"liveBroadcastContent": "upcoming"}, "liveStreamingDetails": live}
assert maintenance._is_abandoned_upcoming(item(scheduledStartTime=past)) is True
assert maintenance._is_abandoned_upcoming(item(scheduledStartTime=soon)) is False
assert (
maintenance._is_abandoned_upcoming(item(scheduledStartTime=past, actualStartTime=past))
is False
)
assert (
maintenance._is_abandoned_upcoming({"snippet": {"liveBroadcastContent": "none"}}) is False
)
# …and an abandoned premiere has no grace period: the confirming sweep deletes it.
assert maintenance._grace_days("abandoned") == 0
+53
View File
@@ -0,0 +1,53 @@
"""E31 S1: the welcome message is rendered once, when the account is created, in the language of that day.
An account that switched language afterwards kept reading it in the old one (the owner's: English under a
Hungarian UI). It is served in the reader's language now; any other system message stays as stored.
Needs the DB."""
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import Message, User
from app.routes.messages import WELCOME
def _client(db, user, monkeypatch, override_dep) -> TestClient:
monkeypatch.setattr(state, "_configured_cache", True)
override_dep(get_db, db.info["request"])
override_dep(current_user, lambda: user)
return TestClient(app)
def _user_with_english_welcome(db) -> User:
"""An account whose welcome was written in English, then switched to Hungarian."""
user = User(email="welcome-hu@example.com", preferences={"language": "hu"})
db.add(user)
db.flush()
db.add(Message(kind="system", sender_id=None, recipient_id=user.id, body=WELCOME["en"]))
db.commit()
return user
def test_the_welcome_is_read_in_the_readers_language(db, monkeypatch, override_dep):
user = _user_with_english_welcome(db)
client = _client(db, user, monkeypatch, override_dep)
thread = client.get("/api/messages/conversations/0").json()["items"]
assert [m["body"] for m in thread] == [WELCOME["hu"]]
convs = client.get("/api/messages/conversations").json()["items"]
assert [c["last_message"]["body"] for c in convs if c["partner"]["id"] == 0] == [WELCOME["hu"]]
def test_another_system_message_is_served_as_stored(db, monkeypatch, override_dep):
user = _user_with_english_welcome(db)
db.add(
Message(kind="system", sender_id=None, recipient_id=user.id, body="Maintenance tonight.")
)
db.commit()
client = _client(db, user, monkeypatch, override_dep)
bodies = [m["body"] for m in client.get("/api/messages/conversations/0").json()["items"]]
assert bodies == [WELCOME["hu"], "Maintenance tonight."]
@@ -0,0 +1,53 @@
"""A concurrent index build that failed part-way, then re-run (E31 S5 review), DB-lane.
CREATE INDEX CONCURRENTLY that fails leaves an INVALID index under its name, and alembic_version is not
stamped, so the next start runs the migration again. `IF NOT EXISTS` matched the name and kept the
leftover, which the planner never uses; 0095 then dropped the old channel_id index, leaving `videos`
with no usable one. A re-run now rebuilds the leftover and drops the old index only once the new one is
valid.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def _index(name: str) -> tuple[bool, str] | None:
with engine.connect() as conn:
row = conn.execute(
text(
"SELECT i.indisvalid, pg_get_indexdef(i.indexrelid) FROM pg_index i "
"JOIN pg_class c ON c.oid = i.indexrelid WHERE c.relname = :name"
),
{"name": name},
).first()
return None if row is None else (bool(row[0]), row[1])
def _leave_invalid(name: str) -> None:
"""What a build that failed part-way leaves behind: the name taken, the index INVALID."""
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
conn.execute(text(f"CREATE INDEX {name} ON videos (id)"))
conn.execute(
text(f"UPDATE pg_index SET indisvalid = false WHERE indexrelid = '{name}'::regclass")
)
def test_a_rerun_rebuilds_an_invalid_leftover_before_dropping_the_old_index(db):
db.rollback() # CONCURRENTLY waits for every open transaction, this session's too
cfg = Config("alembic.ini")
command.downgrade(cfg, "0094_plex_genres_hu_everywhere")
try:
_leave_invalid("ix_videos_channel_pub_cover")
_leave_invalid("ix_videos_views_desc")
finally:
command.upgrade(cfg, "head")
cover = _index("ix_videos_channel_pub_cover")
assert cover is not None and cover[0], cover
assert "INCLUDE (duration_seconds" in cover[1]
assert _index("ix_videos_channel_pub_desc") is None
views = _index("ix_videos_views_desc")
assert views is not None and views[0] and "view_count DESC NULLS LAST" in views[1], views
+1 -1
View File
@@ -97,7 +97,7 @@ def test_putting_a_removed_video_back_leaves_a_linked_list_clean(db, make_user,
pl.synced_fingerprint = current_fingerprint(db, pl)
pl.dirty = False
db.commit()
client.delete(f"/api/playlists/{pl.id}/items/b")
assert client.delete(f"/api/playlists/{pl.id}/items/b").status_code == 200
db.expire_all()
assert db.get(Playlist, pl.id).dirty is True
_add(client, pl.id, {"video_id": "b", "position": 1})
+1 -1
View File
@@ -12,8 +12,8 @@ import pytest
from app.models import Channel, Playlist, PlaylistItem, Video
from app.sync import playlists as pl_mod
from app.sync.playlist_merge import merge_plan
from app.sync.playlists import (
merge_plan,
merge_playlist,
parse_baseline,
parse_unavailable,
+154
View File
@@ -0,0 +1,154 @@
"""routes/plex/admin — the connection test and the owner's watch-sync link (E32 S2: 37% covered).
The connection test is the admin's one early warning that the media mount or path map is wrong, and
it must not quote the server's URL back in a toast. The watch link runs the one-time "Plex is master"
import exactly once. Plex itself is a stub.
"""
from sqlalchemy import select
import app.routes.plex.admin as admin_routes
from app import sysconfig
from app.models import PlexLink
from app.plex.client import PlexError, PlexNotConfigured
class _Plex:
def __init__(self, *, sections=None, leaf=None, fail=None):
self._sections = sections or []
self._leaf = leaf
self._fail = fail
self.section_items_calls: list[str] = []
def __enter__(self):
if self._fail:
raise self._fail
return self
def __exit__(self, *exc):
return False
def server_info(self):
return {"friendlyName": "Basement", "version": "1.40"}
def sections(self):
return self._sections
def section_items(self, key, *_args):
self.section_items_calls.append(key)
return ([self._leaf] if self._leaf else []), 1
SECTIONS = [
{"key": 1, "title": "Movies", "type": "movie", "count": 10},
{"key": 2, "title": "Shows", "type": "show", "count": 3},
{"key": 3, "title": "Music", "type": "artist", "count": 99},
]
LEAF = {"Media": [{"Part": [{"file": "/data/movies/a.mkv"}]}]}
def _use(monkeypatch, plex):
monkeypatch.setattr(admin_routes, "PlexClient", lambda db: plex)
def test_the_connection_test_lists_movie_and_show_sections_and_probes_one_file(
db, as_user, make_user, monkeypatch, tmp_path
):
(tmp_path / "a.mkv").write_bytes(b"x")
sysconfig.set_value(db, "plex_path_map", f"/data/movies={tmp_path}")
sysconfig.set_value(db, "plex_libraries", "4")
db.commit()
# Only section 4 is enabled; the movie section listed first (1) must not be the one sampled.
other = {"key": 4, "title": "Kids", "type": "movie", "count": 2}
plex = _Plex(sections=[*SECTIONS, other], leaf=LEAF)
_use(monkeypatch, plex)
r = as_user(make_user(role="admin")).post("/api/plex/test")
assert r.status_code == 200, r.text
body = r.json()
assert body["server_name"] == "Basement"
assert [s["key"] for s in body["sections"]] == ["1", "2", "4"] # music is not ours
assert body["media_check"] == {
"checked": True,
"ok": True,
"plex_path": "/data/movies/a.mkv",
"local_path": str(tmp_path / "a.mkv"),
}
assert plex.section_items_calls == ["4"] # only an enabled movie section is sampled
def test_a_wrong_mount_is_reported_and_an_empty_library_is_not_checked(
db, as_user, make_user, monkeypatch, tmp_path
):
sysconfig.set_value(db, "plex_path_map", f"/data/movies={tmp_path}")
db.commit()
client = as_user(make_user(role="admin"))
_use(monkeypatch, _Plex(sections=SECTIONS, leaf=LEAF)) # the file is not on the mount
check = client.post("/api/plex/test").json()["media_check"]
assert check["checked"] is True and check["ok"] is False
_use(monkeypatch, _Plex(sections=SECTIONS))
assert client.post("/api/plex/test").json()["media_check"] == {"checked": False}
def test_a_failed_connection_says_so_without_quoting_the_server(
db, as_user, make_user, monkeypatch
):
client = as_user(make_user(role="admin"))
_use(monkeypatch, _Plex(fail=PlexNotConfigured()))
r = client.post("/api/plex/test")
assert r.status_code == 400
assert r.json()["detail"]["reason"] == "plex_not_configured"
_use(monkeypatch, _Plex(fail=PlexError("GET http://10.0.0.5:32400/secret refused")))
r = client.post("/api/plex/test")
assert r.status_code == 422
assert r.json()["detail"]["reason"] == "plex_unreachable"
assert "10.0.0.5" not in r.text
def test_plex_admin_routes_refuse_a_regular_user(db, as_user, make_user, monkeypatch):
_use(monkeypatch, _Plex(sections=SECTIONS))
client = as_user(make_user())
for path in ("/api/plex/test", "/api/plex/sync", "/api/plex/watch/import"):
assert client.post(path).status_code == 403, path
assert client.post("/api/plex/watch/link", json={"enabled": True}).status_code == 403
def test_the_watch_link_imports_once_and_needs_plex_on(db, as_user, make_user, monkeypatch):
imports: list[int] = []
def fake_import(db_, link):
imports.append(link.user_id)
link.initial_import_done = True
db_.commit()
return {"imported": 7}
monkeypatch.setattr(admin_routes.plex_watch, "import_owner_watch_state", fake_import)
owner = make_user(role="admin")
client = as_user(owner)
r = client.post("/api/plex/watch/link", json={"enabled": True})
assert r.status_code == 409 and r.json()["detail"]["reason"] == "plex_disabled"
assert client.post("/api/plex/watch/import").json()["detail"]["reason"] == "watch_sync_off"
sysconfig.set_value(db, "plex_enabled", True)
db.commit()
first = client.post("/api/plex/watch/link", json={"enabled": True})
assert first.status_code == 200
assert first.json()["import"] == {"imported": 7}
assert first.json()["linked"] and first.json()["sync_enabled"] and first.json()["uses_admin"]
assert client.post("/api/plex/watch/link", json={"enabled": False}).json()["import"] is None
again = client.post("/api/plex/watch/link", json={"enabled": True})
assert again.json()["import"] is None # the one-time import does not run twice
assert imports == [owner.id]
status = client.get("/api/plex/watch/link")
assert status.status_code == 200
assert status.json()["initial_import_done"] is True
links = db.execute(select(PlexLink).where(PlexLink.user_id == owner.id)).scalars().all()
assert len(links) == 1 # toggling never creates a second link
rerun = client.post("/api/plex/watch/import")
assert rerun.status_code == 200 and rerun.json()["import"] == {"imported": 7}
assert imports == [owner.id, owner.id]
+204
View File
@@ -0,0 +1,204 @@
"""plex/client — the one door to the Plex server (E32 S2: 31% covered).
What matters here is what the callers cannot see: a failure must surface as PlexError (never as an
empty answer), the paged reads must stop where they should, and the write URIs must be well formed.
The network is an httpx MockTransport; the configuration is stubbed.
"""
import httpx
import pytest
from app.plex import client as plex_client
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
@pytest.fixture
def plex(monkeypatch):
"""A PlexClient on a stub server; set `routes[path] = handler-or-json` per test."""
routes: dict = {}
seen: list[httpx.Request] = []
def transport(request: httpx.Request) -> httpx.Response:
seen.append(request)
answer = routes.get(request.url.path)
if answer is None:
return httpx.Response(404)
if callable(answer):
return answer(request)
return httpx.Response(200, json={"MediaContainer": answer})
real = httpx.Client
monkeypatch.setattr(
plex_client.httpx,
"Client",
lambda **kw: real(transport=httpx.MockTransport(transport), **kw),
)
conf = {"plex_server_url": "http://plex.test:32400/", "plex_server_token": "tok"}
monkeypatch.setattr(plex_client.sysconfig, "get_str", lambda db, key: conf[key])
c = PlexClient(None) # type: ignore[arg-type] # the stubbed config never reads the session
c.routes, c.seen = routes, seen # type: ignore[attr-defined]
yield c
c.close()
def test_an_unconfigured_server_is_refused_up_front(monkeypatch):
monkeypatch.setattr(plex_client.sysconfig, "get_str", lambda db, key: "")
with pytest.raises(PlexNotConfigured):
PlexClient(None) # type: ignore[arg-type]
def test_reads_carry_the_token_and_unwrap_the_container(plex):
plex.routes["/library/sections"] = {"Directory": [{"key": "1", "type": "movie"}]}
assert plex.sections() == [{"key": "1", "type": "movie"}]
req = plex.seen[-1]
assert str(req.url) == "http://plex.test:32400/library/sections"
assert req.headers["x-plex-token"] == "tok"
assert req.headers["accept"] == "application/json"
@pytest.mark.parametrize(
"answer",
[
lambda r: httpx.Response(500),
lambda r: httpx.Response(200, text="<html>login</html>"),
lambda r: (_ for _ in ()).throw(httpx.ConnectError("down", request=r)),
],
ids=["http-500", "not-json", "unreachable"],
)
def test_a_failed_read_is_a_plex_error_never_an_empty_answer(plex, answer):
plex.routes["/library/sections"] = answer
with pytest.raises(PlexError):
plex.sections()
def test_a_section_page_asks_for_guids_and_reports_the_server_total(plex):
plex.routes["/library/sections/7/all"] = {"Metadata": [{"ratingKey": "1"}], "totalSize": 250}
items, total = plex.section_items("7", 1, 200, 50)
assert items == [{"ratingKey": "1"}] and total == 250
params = plex.seen[-1].url.params
assert params["includeGuids"] == "1" and params["type"] == "1"
assert params["X-Plex-Container-Start"] == "200" and params["X-Plex-Container-Size"] == "50"
# Measured on the real server (E32 review): a paged request carries `totalSize` even when the
# page is empty, past the end, or the section holds nothing — `size: 0, totalSize: 0`.
plex.routes["/library/sections/7/all"] = {"Metadata": [], "size": 0, "totalSize": 0}
assert plex.section_items("7", 2) == ([], 0)
# No total at all is not "0": the caller must not read it as a complete, empty section.
plex.routes["/library/sections/7/all"] = {"Metadata": []}
assert plex.section_items("7", 2) == ([], None)
# …and `size` is THIS page's count, never the section's: one full page is not the whole section.
plex.routes["/library/sections/7/all"] = {"Metadata": [{"ratingKey": "1"}], "size": 1}
assert plex.section_items("7", 2) == ([{"ratingKey": "1"}], None)
def test_related_flattens_the_hubs_and_degrades_to_empty(plex):
plex.routes["/library/metadata/5/related"] = {
"Hub": [{"Metadata": [{"ratingKey": "a"}]}, {"Metadata": [{"ratingKey": "b"}]}, {}]
}
assert [m["ratingKey"] for m in plex.related("5")] == ["a", "b"]
plex.routes["/library/metadata/5/related"] = lambda r: httpx.Response(500)
assert plex.related("5") == []
def test_metadata_is_the_first_item_or_none(plex):
plex.routes["/library/metadata/9"] = {"Metadata": [{"ratingKey": "9", "title": "X"}]}
assert plex.metadata("9", markers=True) == {"ratingKey": "9", "title": "X"}
assert plex.seen[-1].url.params["includeMarkers"] == "1"
plex.routes["/library/metadata/9"] = {}
assert plex.metadata("9") is None
def _history(rows_by_start: dict[int, list[dict]]):
def answer(request: httpx.Request) -> httpx.Response:
start = int(request.url.params["X-Plex-Container-Start"])
return httpx.Response(
200, json={"MediaContainer": {"Metadata": rows_by_start.get(start, [])}}
)
return answer
def test_watch_history_pages_until_the_cutoff_and_keeps_only_the_owner(plex):
plex.routes["/status/sessions/history/all"] = _history(
{
0: [
{"ratingKey": "1", "viewedAt": 500, "accountID": 2},
{"ratingKey": "2", "viewedAt": 400, "accountID": 1},
],
2: [
{"ratingKey": "3", "viewedAt": 300, "accountID": 1},
{"ratingKey": "4", "viewedAt": 100, "accountID": 1},
],
4: [{"ratingKey": "never", "viewedAt": 50, "accountID": 1}],
}
)
rows = plex.watch_history(min_viewed_at=200, account_id=1, size=2)
assert [r["ratingKey"] for r in rows] == ["2", "3"] # page 2 reached, the cutoff stops it
assert [int(r.url.params["X-Plex-Container-Start"]) for r in plex.seen] == [0, 2]
def test_watch_history_stops_on_a_short_page_and_at_the_page_cap(plex):
plex.routes["/status/sessions/history/all"] = _history({0: [{"ratingKey": "1", "viewedAt": 9}]})
assert [r["ratingKey"] for r in plex.watch_history(size=2)] == ["1"]
assert len(plex.seen) == 1
full = [{"ratingKey": "r", "viewedAt": 9}, {"ratingKey": "r", "viewedAt": 9}]
plex.routes["/status/sessions/history/all"] = lambda r: httpx.Response(
200, json={"MediaContainer": {"Metadata": full}}
)
plex.seen.clear()
assert len(plex.watch_history(size=2, max_pages=3)) == 6
assert len(plex.seen) == 3
def test_a_collection_is_created_with_a_well_formed_uri(plex):
plex.routes["/identity"] = {"machineIdentifier": "abc"}
plex.routes["/library/collections"] = {"Metadata": [{"ratingKey": "c1"}]}
assert plex.create_collection("3", "Best of: 90s", "42") == {"ratingKey": "c1"}
params = plex.seen[-1].url.params
assert plex.seen[-1].method == "POST"
assert params["title"] == "Best of: 90s" and params["sectionId"] == "3"
assert params["uri"] == "server://abc/com.plexapp.plugins.library/library/metadata/42"
plex.routes["/library/collections"] = {}
with pytest.raises(PlexError):
plex.create_collection("3", "Empty", "42")
assert [r.url.path for r in plex.seen].count("/identity") == 1 # cached
def test_a_server_without_a_machine_id_refuses_the_write(plex):
"""No machineIdentifier used to become `server:///…` in the write URI — a malformed request
Plex answers in its own way. It is refused here, named (E32 S2)."""
plex.routes["/identity"] = {}
plex.routes["/library/collections"] = {"Metadata": [{"ratingKey": "c1"}]}
with pytest.raises(PlexError, match="machine"):
plex.create_collection("3", "T", "42")
assert all(r.url.path == "/identity" for r in plex.seen)
def test_watch_state_writes_target_the_library_item(plex):
for path in ("/:/scrobble", "/:/unscrobble", "/:/timeline"):
plex.routes[path] = lambda r: httpx.Response(200) # Plex answers with an empty body
plex.scrobble("42")
assert plex.seen[-1].url.params["key"] == "42"
plex.unscrobble("42")
assert plex.seen[-1].url.path == "/:/unscrobble"
plex.set_timeline("42", 61_500.7, 120_000) # type: ignore[arg-type]
p = plex.seen[-1].url.params
assert p["key"] == "/library/metadata/42" and p["time"] == "61500" and p["state"] == "stopped"
plex.routes["/:/scrobble"] = lambda r: httpx.Response(401)
with pytest.raises(PlexError):
plex.scrobble("42")
def test_an_image_is_resized_by_plex_when_a_box_is_given(plex):
plex.routes["/photo/:/transcode"] = lambda r: httpx.Response(
200, content=b"small", headers={"Content-Type": "image/webp"}
)
plex.routes["/library/metadata/1/thumb/2"] = lambda r: httpx.Response(200, content=b"full")
assert plex.image_bytes("/library/metadata/1/thumb/2", 300, 450) == (b"small", "image/webp")
assert plex.seen[-1].url.params["url"] == "/library/metadata/1/thumb/2"
body, ctype = plex.image_bytes("/library/metadata/1/thumb/2")
assert body == b"full" and ctype == "image/jpeg"
with pytest.raises(PlexError):
plex.raw_get("/library/streams/404")
@@ -214,3 +214,28 @@ def test_hidden_movie_absent_from_grid_and_facets(db, as_user):
facets = client.get("/api/plex/facets?scope=movie&show=all")
crs = {c["value"]: c["count"] for c in facets.json()["content_ratings"]}
assert crs.get("R") == 1, crs
def test_a_watch_state_scope_rolls_the_episodes_up_once(db, as_user):
"""The per-user episode rollup behind a show's watch state is a pass over every episode; each show
facet group asked it again (E31 S4e: half of a filtered facets call). It is asked once now."""
from sqlalchemy import event
from app.db import engine
seed = _seed(db)
client = as_user(seed["user"])
rollups: list[str] = []
def count(conn, cursor, statement, *_):
if "GROUP BY plex_items.show_id" in statement:
rollups.append(statement)
event.listen(engine, "before_cursor_execute", count)
try:
for show in ("watched", "in_progress", "unwatched"):
rollups.clear()
assert client.get(f"/api/plex/facets?scope=both&show={show}").status_code == 200
assert len(rollups) == 1, show
finally:
event.remove(engine, "before_cursor_execute", count)
+77
View File
@@ -0,0 +1,77 @@
"""A card asks the Plex image proxy for a small cut; a detail page keeps the full one (E31 S4e).
Cards fetched the 400×600 poster for a ~144px picture and the 1280×720 backdrop for a ~212px shelf
picture. The `_sm` variants are cut by Plex at a card's size and cached apart from the full ones; an
unknown variant still gets the full poster. DB lane (the proxy opens its own short session).
"""
from fastapi.testclient import TestClient
import app.state as app_state
from app.main import app
from app.models import PlexItem, PlexLibrary
from app.plex import imgcache
from app.routes.plex import playback
class _FakePlex:
calls: list[tuple[str, int | None, int | None]] = []
def __init__(self, db) -> None:
pass
def __enter__(self):
return self
def __exit__(self, *exc) -> None:
return None
def image_bytes(self, key, width=None, height=None):
_FakePlex.calls.append((key, width, height))
return b"\xff\xd8jpeg", "image/jpeg"
def _setup(db, make_user, monkeypatch, tmp_path) -> TestClient:
user = make_user()
lib = PlexLibrary(plex_key="90", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
db.add(
PlexItem(
rating_key="pm1",
library_id=lib.id,
kind="movie",
title="A Movie",
thumb_key="/library/metadata/pm1/thumb/1",
art_key="/library/metadata/pm1/art/1",
)
)
db.commit()
monkeypatch.setattr(app_state, "_configured_cache", True)
monkeypatch.setattr(playback, "resolved_user_id", lambda request: (user.id, True))
monkeypatch.setattr(playback, "PlexClient", _FakePlex)
monkeypatch.setattr(imgcache, "CACHE_DIR", tmp_path)
_FakePlex.calls = []
return TestClient(app)
def test_each_variant_is_cut_at_its_own_size(db, make_user, monkeypatch, tmp_path):
client = _setup(db, make_user, monkeypatch, tmp_path)
for variant in ("thumb", "thumb_sm", "art", "art_sm"):
assert client.get(f"/api/plex/image/pm1?variant={variant}").status_code == 200
assert _FakePlex.calls == [
("/library/metadata/pm1/thumb/1", 400, 600),
("/library/metadata/pm1/thumb/1", 240, 360),
("/library/metadata/pm1/art/1", 1280, 720),
("/library/metadata/pm1/art/1", 480, 270),
]
# Each cut is cached under its own name: asking again reaches Plex no more.
for variant in ("thumb", "thumb_sm", "art", "art_sm"):
assert client.get(f"/api/plex/image/pm1?variant={variant}").status_code == 200
assert len(_FakePlex.calls) == 4
def test_an_unknown_variant_gets_the_full_poster(db, make_user, monkeypatch, tmp_path):
client = _setup(db, make_user, monkeypatch, tmp_path)
assert client.get("/api/plex/image/pm1?variant=huge").status_code == 200
assert _FakePlex.calls == [("/library/metadata/pm1/thumb/1", 400, 600)]
+52 -11
View File
@@ -16,18 +16,21 @@ from app import sysconfig
from app.db import engine
from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex.genres import hu_genre, hu_genres
from app.plex.sync import _apply_item, _canonical_guid, _sync_overlay
from app.plex.overlay import sync_overlay
from app.plex.sync import _apply_item, _canonical_guid
from app.plex.query import content_libs
class _StubPlex:
"""Minimal PlexClient stand-in: _sync_overlay only calls section_items via _paginate."""
"""Minimal PlexClient stand-in: sync_overlay only calls section_items via _paginate. `total`
over the item count is a listing cut short (the server said more than it handed over)."""
def __init__(self, items: list[dict]):
def __init__(self, items: list[dict], total: int | None = None):
self._items = items
self._total = len(items) if total is None else total
def section_items(self, key, item_type, start, size):
return self._items[start : start + size], len(self._items)
return self._items[start : start + size], self._total
def _hu_movie(rk: str, title: str, summary: str, genres: list[str], guid: str | None) -> dict:
@@ -71,7 +74,7 @@ def test_sync_overlay_writes_hu_onto_matched_en_rows(db):
]
)
stats = {"overlaid": 0}
_sync_overlay(db, stub, hu_lib, en_lib.id, stats)
sync_overlay(db, stub, hu_lib, en_lib.id, stats)
db.commit()
db.refresh(matched)
@@ -185,13 +188,51 @@ def test_overlay_reconcile_clears_stale_hu_when_twin_vanishes(db):
)
db.add(row)
db.commit()
# The HU section no longer lists this film (empty overlay).
_sync_overlay(db, _StubPlex([]), hu_lib, en_lib.id, {"overlaid": 0})
# The HU section, read in full, no longer lists this film (only another one).
other = _hu_movie("h9", "Más", "más", [], "tmdb://99")
sync_overlay(db, _StubPlex([other]), hu_lib, en_lib.id, {"overlaid": 0})
db.commit()
db.refresh(row)
assert row.title_hu is None and row.summary_hu is None and row.genres_hu is None
def test_a_listing_cut_short_keeps_the_hungarian_text_it_did_not_reach(db):
"""E32 S2: the reconcile clears HU text only after a COMPLETE listing. A walk a Plex hiccup cut
short (an empty page before the server's total), or an empty section, says nothing about the
titles it did not reach — clearing them dropped their Hungarian text until the next full sync."""
en_lib = PlexLibrary(plex_key="4", title="Movies", kind="movie")
hu_lib = PlexLibrary(plex_key="9", title="Filmek", kind="movie", metadata_lang="hu")
db.add_all([en_lib, hu_lib])
db.flush()
hu_lib.overlay_of = en_lib.id
kept = PlexItem(
rating_key="m1",
library_id=en_lib.id,
kind="movie",
title="Heat",
guid="tmdb:1",
title_hu="Szemtől szemben",
summary_hu="régi",
genres_hu=["Akció"],
)
fresh = PlexItem(rating_key="m2", library_id=en_lib.id, kind="movie", title="Up", guid="tmdb:2")
db.add_all([kept, fresh])
db.commit()
stats: dict = {}
reached = _hu_movie("h2", "Fel", "fel", [], "tmdb://2")
sync_overlay(db, _StubPlex([reached], total=2), hu_lib, en_lib.id, stats)
db.commit()
db.refresh(kept)
db.refresh(fresh)
assert kept.title_hu == "Szemtől szemben" and kept.summary_hu == "régi"
assert fresh.title_hu == "Fel" # what the listing did reach is still applied
assert stats["overlay_incomplete"] == 1
sync_overlay(db, _StubPlex([]), hu_lib, en_lib.id, {}) # an empty section: the same answer
db.commit()
db.refresh(kept)
assert kept.title_hu == "Szemtől szemben"
def test_apply_item_keeps_prior_guid_when_listing_lacks_guid():
row = PlexItem(rating_key="m1", kind="movie", guid="tmdb:1")
_apply_item(row, lib_id=7, kind="movie", meta={"title": "Heat"}) # no Guid array
@@ -225,7 +266,7 @@ def test_overlay_applies_to_all_editions_sharing_a_guid(db):
]
)
stats = {"overlaid": 0}
_sync_overlay(db, stub, hu_lib, en_lib.id, stats)
sync_overlay(db, stub, hu_lib, en_lib.id, stats)
db.commit()
db.refresh(ed1)
db.refresh(ed2)
@@ -291,7 +332,7 @@ def test_a_show_overlay_writes_hungarian_episode_titles_and_summaries(db):
],
}
)
_sync_overlay(db, stub, hu_lib, en_lib.id, {})
sync_overlay(db, stub, hu_lib, en_lib.id, {})
db.commit()
db.refresh(ep)
db.refresh(show)
@@ -328,7 +369,7 @@ def test_the_overlay_gives_every_row_one_hungarian_word_per_genre(db):
"h1", "Szemtől szemben", "HU.", ["Action", "Bűnügyi", "Drama", "Dráma"], "tmdb://1"
)
meta["tagline"] = "Magyar mottó."
_sync_overlay(db, _StubPlex([meta]), hu_lib, en_lib.id, {})
sync_overlay(db, _StubPlex([meta]), hu_lib, en_lib.id, {})
db.commit()
db.refresh(twin)
db.refresh(lone)
@@ -353,7 +394,7 @@ def test_a_twin_without_genres_keeps_the_rows_own_hungarian_list(db):
)
db.add(row)
db.commit()
_sync_overlay(
sync_overlay(
db,
_StubPlex([_hu_movie("h1", "Szemtől szemben", "HU.", [], "tmdb://1")]),
hu_lib,
+82
View File
@@ -0,0 +1,82 @@
"""plex/paths — the Plex-side path → local mount translation (E32 S2: 23% covered, and a wrong answer
here reads as "media file not found" for a whole library at once, with nothing else failing).
Pure logic: the admin map is read through sysconfig, stubbed per test.
"""
from pathlib import Path
import pytest
from app.plex import paths
@pytest.fixture
def path_map(monkeypatch):
def use(raw: str) -> None:
monkeypatch.setattr(paths.sysconfig, "get_str", lambda db, key: raw)
return use
def test_the_map_parses_both_separators_drops_junk_and_puts_the_longest_prefix_first():
raw = " /data=/mnt/a ,\n/data/movies = /mnt/m\nnot-a-pair,=/x,/y=, /data/tv=/mnt/tv "
assert paths.parse_map(raw) == [
("/data/movies", "/mnt/m"),
("/data/tv", "/mnt/tv"),
("/data", "/mnt/a"),
]
assert paths.parse_map("") == []
def test_the_longest_matching_prefix_wins(path_map):
path_map("/data=/mnt/a,/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies/Foo.mkv") == str(Path("/mnt/m") / "Foo.mkv")
assert paths.map_path(None, "/data/tv/Bar.mkv") == str(Path("/mnt/a") / "tv/Bar.mkv")
def test_a_prefix_matches_only_at_a_path_boundary(path_map):
# "/data/movies2" shares the characters of "/data/movies" but is another folder.
path_map("/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies2/Foo.mkv") == "/data/movies2/Foo.mkv"
def test_the_prefix_itself_and_a_trailing_slash_in_the_map(path_map):
path_map("/data/movies/=/mnt/m")
assert paths.map_path(None, "/data/movies/Foo.mkv") == str(Path("/mnt/m") / "Foo.mkv")
path_map("/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies") == "/mnt/m"
def test_no_map_leaves_the_path_as_plex_sees_it(path_map):
path_map("")
assert paths.map_path(None, "/data/movies/Foo.mkv") == "/data/movies/Foo.mkv"
def test_the_local_file_must_exist(path_map, tmp_path):
(tmp_path / "movies").mkdir()
(tmp_path / "movies" / "Foo.mkv").write_bytes(b"x")
path_map(f"/data={tmp_path}")
assert (
paths.local_media_path(None, "/data/movies/Foo.mkv")
== (tmp_path / "movies" / "Foo.mkv").resolve()
)
assert paths.local_media_path(None, "/data/movies/Missing.mkv") is None
assert paths.local_media_path(None, "/data/movies") is None # a folder is not a file
assert paths.local_media_path(None, None) is None
assert paths.local_media_path(None, "") is None
def test_a_path_that_cannot_be_resolved_is_logged_and_answered_none(path_map, monkeypatch, caplog):
path_map("")
def broken(self, strict=False):
raise OSError("dead mount")
monkeypatch.setattr(paths.Path, "resolve", broken)
assert paths.local_media_path(None, "/data/movies/Foo.mkv") is None
rec = next(
r for r in caplog.records if getattr(r, "event", None) == "plex.paths.resolve_failed"
)
# The title stays out of the log line: the path carries it.
assert "Foo" not in rec.getMessage() and rec.attrs == {"reason": "OSError"}
@@ -105,8 +105,11 @@ def test_remove_then_put_back_is_no_change(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b", "c", "d"], extra=())
client = as_user(user)
client.post(
f"/api/plex/playlists/{pl.id}/items/remove-bulk", json={"item_rating_keys": ["b", "d"]}
assert (
client.post(
f"/api/plex/playlists/{pl.id}/items/remove-bulk", json={"item_rating_keys": ["b", "d"]}
).status_code
== 200
)
assert _order(db, pl.id)[0] == ["a", "c"]
_add(client, pl.id, ["b", "d"], [1, 3])
+41
View File
@@ -8,6 +8,7 @@ Since the sweep stopped being awaited at startup it also runs CONCURRENTLY with
second half of this file pins what keeps it off a LIVE session's segments."""
import os
import sys
from pathlib import Path
import pytest
@@ -830,6 +831,46 @@ class TestProbeVideoKind:
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
class TestProbeExitStatus:
"""A probe whose ffprobe EXITS NON-ZERO is a failed probe, not an answer (E32). Each probe used to
read stdout without looking at the exit status, so a refusing ffprobe silently picked the fallback
— HW-safe, single audio, the requested offset — and skipped the warning its except branch promises.
Drives the REAL subprocess.run with the probe's own kwargs against a stand-in command that prints a
plausible answer and then fails, so the test fails whenever the exit status is ignored."""
@staticmethod
def _failing_ffprobe(monkeypatch, stdout: str):
real_run = stream.subprocess.run
script = f"import sys; sys.stdout.write({stdout!r}); sys.exit(1)"
monkeypatch.setattr(
stream.subprocess,
"run",
lambda _cmd, **kw: real_run([sys.executable, "-c", script], **kw),
)
@staticmethod
def _events(caplog) -> list[str]:
return [getattr(r, "event", "") for r in caplog.records]
def test_video_kind(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "pix_fmt=yuv420p10le\ncolor_transfer=smpte2084\n")
with caplog.at_level("INFO"):
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
assert "plex.stream.probe_kind_failed" in self._events(caplog)
def test_audio_langs(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "hun\neng\n")
with caplog.at_level("INFO"):
assert stream._probe_audio_langs(Path("/x")) == []
assert "plex.stream.probe_audio_failed" in self._events(caplog)
def test_first_pts(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "12.5\n")
with caplog.at_level("INFO"):
assert stream._probe_first_pts(Path("/x")) is None
assert "plex.stream.first_pts_probe_failed" in self._events(caplog)
class TestOpenclPresent:
"""The OpenCL-runtime probe (_opencl_present) — the precondition for the HW HDR tonemap graph. Drives
the REAL glob against a temp ICD dir instead of monkeypatching the function, so a regression to the
+1 -1
View File
@@ -208,7 +208,7 @@ def test_an_unshowable_track_says_why_and_maps_to_a_status(client, monkeypatch):
raise subtitles.SubtitleUnavailable("bitmap")
monkeypatch.setattr(subtitles, "_extract_and_publish", _bitmap)
client.get("/api/plex/subtitle/98614/0/status")
assert client.get("/api/plex/subtitle/98614/0/status").status_code == 200
pool.run_all()
st = client.get("/api/plex/subtitle/98614/0/status").json()
assert st["state"] == "failed" and st["code"] == "bitmap"
+277
View File
@@ -0,0 +1,277 @@
"""The daily full watch reconcile (`run_plex_watch_reconcile`) — the one pass that DELETES watch state:
a mirrored row Plex no longer has a record for is read as "Plex un-watched it" and cleared. E32 S2
found it trusting a scan a Plex hiccup had cut short; these hold what it may and may not clear.
"""
import pytest
from sqlalchemy import select
from app import sysconfig
from app.models import PlexItem, PlexLibrary, PlexLink, PlexState, User
from app.plex import watch_sync
class _FakePlex:
"""A PlexClient stand-in: one movie section whose listing reports `total` (more than it hands
over = a walk cut short), plus any `more` sections as `(type, leaves, total)` — listed AFTER
the movies — and a record of every write the reconcile sends back to Plex."""
def __init__(self, leaves: list[dict], total: int | None = None, more=()):
self.listings = {"1": ("movie", leaves, len(leaves) if total is None else total)}
for i, (stype, extra, extra_total) in enumerate(more, start=2):
self.listings[str(i)] = (stype, extra, extra_total)
self.calls: list[tuple] = []
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def sections(self):
return [{"type": stype, "key": key} for key, (stype, _l, _t) in self.listings.items()]
def section_items(self, key, item_type, start, size):
_stype, leaves, total = self.listings[key]
return leaves[start : start + size], total
def scrobble(self, rk):
self.calls.append(("scrobble", rk))
def unscrobble(self, rk):
self.calls.append(("unscrobble", rk))
def set_timeline(self, rk, time_ms, duration_ms):
self.calls.append(("timeline", rk, time_ms))
@pytest.fixture
def world(db):
"""Five mirrored movies and the owner's link; each test sets the local states it needs."""
sysconfig.set_value(db, "plex_enabled", True)
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
user = User(email="reconcile@example.com")
db.add_all([lib, user])
db.flush()
items = {}
for rk in ("m1", "m2", "m3", "m4", "m5"):
items[rk] = PlexItem(
rating_key=rk, library_id=lib.id, kind="movie", title=rk, duration_s=6000
)
db.add(items[rk])
db.add(
PlexLink(
user_id=user.id,
uses_admin=True,
sync_enabled=True,
initial_import_done=True,
plex_account_id=1,
)
)
db.flush()
def state(rk, **kw):
db.add(PlexState(user_id=user.id, item_id=items[rk].id, **kw))
def rows():
db.expire_all()
by_id = {i.id: rk for rk, i in items.items()}
return {
by_id[s.item_id]: (s.status, s.position_seconds, s.synced_to_plex)
for s in db.execute(select(PlexState).where(PlexState.user_id == user.id)).scalars()
if s.item_id in by_id # the five movies; a test's own extra items it reads itself
}
return state, rows
def _run(db, monkeypatch, plex):
monkeypatch.setattr(watch_sync, "PlexClient", lambda _db: plex)
return watch_sync.run_plex_watch_reconcile(db)
def _local_states(state):
state("m1", status="watched", synced_to_plex=True) # mirrored; Plex has no record now
state("m2", status="watched", synced_to_plex=False) # local change that never reached Plex
state("m3", status="hidden", synced_to_plex=True) # Siftlode-only, never cleared
state("m5", status="new", position_seconds=300, synced_to_plex=True) # Plex says otherwise
def test_a_complete_scan_clears_what_plex_unwatched_and_keeps_the_rest(db, monkeypatch, world):
state, rows = world
_local_states(state)
db.commit()
plex = _FakePlex(
[
{"ratingKey": "m3"}, # listed, no watch signal
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
]
)
out = _run(db, monkeypatch, plex)
assert rows() == {
"m2": ("watched", 0, True), # re-pushed, then settled
"m3": ("hidden", 0, True),
"m4": ("watched", 0, True), # pulled in
"m5": ("new", 120, True), # Plex's position wins
}
assert ("scrobble", "m2") in plex.calls
assert out["cleared"] == 1 and out["repushed"] == 1 and "incomplete" not in out
def test_a_scan_cut_short_clears_nothing(db, monkeypatch, world):
state, rows = world
_local_states(state)
db.commit()
# The server said 5, the walk got 3: m1 may simply not have been reached.
plex = _FakePlex(
[
{"ratingKey": "m3"},
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
],
total=5,
)
out = _run(db, monkeypatch, plex)
after = rows()
assert after["m1"] == ("watched", 0, True) # NOT un-watched on a partial picture
assert after["m4"] == ("watched", 0, True) and after["m5"] == ("new", 120, True) # pulls stand
assert after["m2"] == ("watched", 0, True) # the dirty row is still pushed up
assert out["cleared"] == 0 and out["incomplete"] == 1
_LISTED = [
{"ratingKey": "m3"},
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
]
def test_a_genuinely_empty_second_section_still_lets_the_reconcile_clear(db, monkeypatch, world):
# The server SAYS the shows section holds 0: that is a complete listing, not a hiccup — before
# the E32 review it read as "incomplete" forever and un-watches never synced again.
state, rows = world
_local_states(state)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
assert "m1" not in rows()
assert out["cleared"] == 1 and "incomplete" not in out
def _watched_episode_in_a_shows_section(db) -> int:
"""A second mirrored section (key "2") holding one episode the owner has watched, synced."""
shows = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(shows)
db.flush()
ep = PlexItem(rating_key="e1", library_id=shows.id, kind="episode", title="e1")
db.add(ep)
db.flush()
user_id = db.execute(select(PlexLink.user_id)).scalar_one()
db.add(PlexState(user_id=user_id, item_id=ep.id, status="watched", synced_to_plex=True))
return ep.id
def _state_of(db, item_id: int) -> str | None:
db.expire_all()
st = db.execute(select(PlexState).where(PlexState.item_id == item_id)).scalar_one_or_none()
return st.status if st else None
def test_a_section_we_mirror_that_says_0_keeps_its_rows_and_does_not_hold_up_the_rest(
db, monkeypatch, world
):
# A total of 0 from a section our items come from is a Plex that lost the library for a moment,
# not one that un-watched all of it (E32 review round 2): its rows stay. And it is decided per
# section: before round 3 one such section (a library emptied on Plex, its mirrored items kept)
# made EVERY run incomplete, and un-watches in every other section never synced again.
state, rows = world
_local_states(state)
ep_id = _watched_episode_in_a_shows_section(db)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
assert _state_of(db, ep_id) == "watched"
assert "m1" not in rows() # the movies section was walked whole: Plex un-watched m1
assert out["cleared"] == 1 and out["incomplete"] == 1
def test_a_library_the_allowlist_leaves_out_keeps_its_watch_states(db, monkeypatch, world):
# Its items are absent from every scan because the scan never walks it — not because Plex
# un-watched them. It cleared them on every run (E32 review round 2).
state, _rows = world
other = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(other)
db.flush()
ep = PlexItem(rating_key="e1", library_id=other.id, kind="episode", title="e1")
db.add(ep)
db.flush()
user_id = db.execute(select(PlexLink.user_id)).scalar_one()
db.add(PlexState(user_id=user_id, item_id=ep.id, status="watched", synced_to_plex=True))
_local_states(state)
sysconfig.set_value(db, "plex_libraries", "1")
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
db.expire_all()
left = {
rk
for rk in db.execute(
select(PlexItem.rating_key).join(PlexState, PlexState.item_id == PlexItem.id)
).scalars()
}
assert "m1" not in left # the walked section still clears
kept = db.execute(select(PlexState).where(PlexState.item_id == ep.id)).scalar_one_or_none()
assert kept is not None and kept.status == "watched"
assert out["cleared"] == 1
def test_a_cut_short_section_is_not_rescued_by_a_complete_one_after_it(db, monkeypatch, world):
# Each section settles on its own walk: movies cut short (5 said, 3 handed), then a complete
# shows listing — m1 may simply not have been reached.
state, rows = world
_local_states(state)
db.commit()
shows = [{"ratingKey": "s1"}]
out = _run(db, monkeypatch, _FakePlex(_LISTED, total=5, more=[("show", shows, 1)]))
assert rows()["m1"] == ("watched", 0, True)
assert out["cleared"] == 0 and out["incomplete"] == 1
def test_a_listing_without_a_total_is_never_complete(db, monkeypatch, world):
state, rows = world
_local_states(state)
ep_id = _watched_episode_in_a_shows_section(db)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], None)]))
assert _state_of(db, ep_id) == "watched" # its section never said how much it holds
assert "m1" not in rows()
assert out["cleared"] == 1 and out["incomplete"] == 1
def test_an_empty_listing_is_not_a_library_of_unwatched_titles(db, monkeypatch, world):
state, rows = world
state("m1", status="watched", synced_to_plex=True)
db.commit()
out = _run(db, monkeypatch, _FakePlex([]))
assert rows() == {"m1": ("watched", 0, True)}
assert out["cleared"] == 0
def test_a_dirty_resume_is_re_pushed_as_a_timeline_and_a_dirty_reset_as_an_unscrobble(
db, monkeypatch, world
):
state, rows = world
state("m1", status="new", position_seconds=300, synced_to_plex=False)
state("m2", status="new", position_seconds=2, synced_to_plex=False) # under the 5 s floor
db.commit()
plex = _FakePlex([{"ratingKey": rk} for rk in ("m1", "m2", "m3", "m4", "m5")])
_run(db, monkeypatch, plex)
assert ("timeline", "m1", 300_000) in plex.calls
assert ("unscrobble", "m2") in plex.calls
assert {rk: synced for rk, (_s, _p, synced) in rows().items()} == {"m1": True, "m2": True}
def test_watch_fields_map_to_a_state_only_when_plex_holds_a_signal():
to = watch_sync._plex_watch_to_state
assert to(None, None, None) is None
assert to(0, 4_000, None) is None # a stray few seconds is not "in progress"
assert to(0, 90_000, 1_700_000_000)[:2] == ("new", 90)
assert to("2", 90_000, 1_700_000_000)[:2] == ("watched", 0) # watched wins over an offset
+2 -2
View File
@@ -9,7 +9,7 @@ independent literal, there was no `.python-version`, no `requires-python`, and r
**The contract is a chain, and it is split across two test files because no single runner can see
every link.** This suite runs inside the test image, which mounts `backend/` and nothing else, so it
cannot read the repo-root Dockerfile or the workflow. It therefore holds the link it CAN see, and the
cannot read the repo-root Dockerfile. It therefore holds the link it CAN see, and the
one that matters most here:
the interpreter actually running == backend/pyrightconfig.json == backend/ruff.toml
@@ -17,7 +17,7 @@ one that matters most here:
and `frontend/src/lib/pythonEngine.contract.test.ts` — which runs on the host, with the whole tree —
holds the rest of the chain:
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test == check.yml
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test
Together they close the loop. To move the version: edit `/.python-version`, then follow the failures
in both files. Every place they name is a place that would otherwise have been forgotten.
+6 -1
View File
@@ -51,7 +51,12 @@ def test_a_link_whose_account_was_deleted_is_reported_dead(client, db, make_user
def test_a_spent_link_is_reported_dead(client, db, make_user):
raw = _issue(db, make_user(email="spent@example.com", password=_PW))
client.post("/auth/password-reset/confirm", json={"token": raw, "password": _NEW})
assert (
client.post(
"/auth/password-reset/confirm", json={"token": raw, "password": _NEW}
).status_code
== 200
)
assert client.post("/auth/password-reset/check", json={"token": raw}).json() == {"ok": False}

Some files were not shown because too many files have changed in this diff Show More