Compare commits

..
64 Commits
Author SHA1 Message Date
peter 0dfb4a984b release: 0.99.2 (bug/b1-e34-defects) 2026-10-04 18:28:53 +02:00
peter 5bcd1d8d8a chore(release): 0.99.2 2026-10-04 18:19:37 +02:00
peter f49789665e fix(i18n): label the history cleanup's quota_reaped result
The scheduler row read 'quota reaped 0' — the raw key, humanised — because the new result key had no
label (found in the browser check; nothing tests result-key labels).
2026-10-04 18:03:19 +02:00
peter 0e077b84c8 fix(search): the live-search tooltip says what a search costs
It always said a live search spends the shared API quota, while the default source (the website's
own endpoint, scrape) spends none — the owner avoided a free search because of it. /api/me now names
the running search_source, and the tooltip names the cost under it: none for scrape (the enrich step
costs a unit or two), 100 units a page for the official API.
2026-10-04 17:53:19 +02:00
peter ce8ccfc3c0 fix(quota): prune quota history past a retention window
quota_events had no retention: one row per API spend plus a zero-unit row per live search, kept
forever. The daily history cleanup (audit_gc) now also prunes it past quota_retention_days (a new
Quota setting, default 365 so the admin usage view's longest window stays whole; 0 = forever). The
floor of two days keeps today's rows, which the per-user search cap counts.
2026-10-04 17:53:19 +02:00
peter 33dec56e45 fix(worker): obey the stored concurrency and log level, live
The admin page's download_worker_concurrency and log_level were stored but never reached the worker:
it read the env once at boot (worker.py) and never looked again. The worker now re-reads both every
10 s (_sync_knobs): extra loop threads start at once, surplus ones retire after their current job, and
a changed level is applied to the process. An unreadable database keeps what already runs.
2026-10-04 17:53:19 +02:00
peter 2a699e26ba release: 0.99.1 (improvement/drop-ci) 2026-10-04 06:45:48 +02:00
peter 13c8c05661 chore(release): 0.99.1 2026-10-04 06:36:06 +02:00
peter cc414de952 build: the gate is scripts/check.sh, and CI goes
The Gitea Actions gate repeated, after the merge, what a release already runs
before it, and its runner shared the git host's container: one run there
starved Gitea of memory until it stopped answering. The lanes now live in
scripts/check.sh, run the same by the release dispatcher and by hand, and the
gate-contract test holds its lines (the ruff and pip-audit checks read it
instead of the workflow). The two audits only CI ran join the gate: pip-audit
over the lock (pinned in the test image, blocking at any severity) and npm
audit of the production dependencies. The engine contracts drop the CI
containers; the register records that pushes have no server-side gate.
2026-10-04 06:20:58 +02:00
peter 0528b35f2d release: 0.99.0 (feature/e32-gate-integrity)
gate / frontend (push) Successful in 8m32s
gate / backend (push) Successful in 8m5s
2026-10-04 05:19:05 +02:00
peter cb46774851 chore(release): 0.99.0 2026-10-04 05:09:33 +02:00
peter f1cc6ce5dc fix: the reconcile scan's return type names its settled sections; the composer stays at its size ceiling 2026-10-04 05:04:14 +02:00
peter 4efe799eda fix(ui): the issue composer opens on its title, not on the Edit switch
A form dialog opens on the body's first control, and the composer's first
control is the Edit / Preview switch above the form: the focus landed there,
at open and when a retried load brought the form, and typing went nowhere. A
form can now mark the field to open on (data-initial-focus); every other
dialog keeps its first control. The Modal's comment on the re-armed watch now
says where the focus goes when the focused control is removed but others stay.
2026-10-04 04:56:57 +02:00
peter 40b27f2982 fix(sync): a channel with no uploads backfills as done, not as a failure
YouTube answers the uploads list of a channel with no uploads as
playlistNotFound. The recent and deep backfills raised on it, so the channel
was never stamped and was picked again, charged a quota unit and logged an
ERROR on every run. Both now read it as an empty last page, as explore already
does; any other failure still raises.
2026-10-04 04:51:25 +02:00
peter 0314c4cce2 fix(plex): the watch reconcile settles each library on its own walk
A mirrored library that answers "0 items" is not believed (its rows stay), but
that verdict stopped the whole reconcile: one library emptied on Plex with its
mirrored items kept made every run incomplete, and un-watches in every other
library never synced again. Each section now settles on its own walk, and only
the settled sections' rows are cleared; the unsettled ones are named in the
warning.
2026-10-04 04:49:55 +02:00
peter 182ec31120 test(gates): hold the lint and lane configs to a committed snapshot
The register test detected narrowings by modelling config shapes, and each
review round found one more shape it did not model (a basePath global ignore,
a rule src never declares, a widened glob filed under a narrow row, a substring
check on a CI line, knip.json beside knip.jsonc). It now compares instead:
gates.snapshot.json holds ESLint's effective config for every code file and the
lines that run each lane, and any difference fails until it is regenerated.
The register is read off the snapshot's configs, which surfaced the ui/ and
lib/ test scopes the sampling never reached.

The complexity ratchet also names a wrapped function by its binding
(const onSave = useCallback(() => ...)), not as "useCallback callback".
2026-10-04 04:47:10 +02:00
peter 12a39f5291 fix(plex): the watch reconcile clears only what it walked, and doubts a mirrored 0
A library left out of the allowlist is never scanned, so its watch states were cleared on
every reconcile; clearing is now limited to the sections walked. A section we mirror items
from that says it holds 0 is not believed complete. The page size is no longer read as the
section total when totalSize is missing (measured: the server always sends totalSize on a
paged request, 0 for an empty one).
2026-10-04 04:16:52 +02:00
peter 79704b0001 fix(youtube): "this playlist no longer exists" only where the user picked that list
playlistNotFound was mapped to "gone" everywhere: exploring a channel without uploads (its
uploads list answers not found) and a list deleted mid bulk pull both said a playlist no
longer exists. The reason is now "gone" only on single-playlist routes; explore reads it
as an empty back-catalogue, and the bulk pull skips that list and syncs the rest.
2026-10-04 04:16:52 +02:00
peter 4191189b4c fix(ui): a late form takes a dialog's focus once, even if it arrived elsewhere
The watch stayed armed when the form arrived while the focus was off the close button, so a
later change in the body pulled the focus off a close button the user went back to. It now
disarms on arrival and re-arms only when the focus is taken away.
2026-10-04 04:16:52 +02:00
peter 7dbda60815 fix(ui): an undo in the same tick as a rebase steps back to a rebased snapshot
rebase updated the state only, so the mirror the callbacks read still held the removed
row and an undo right after it wrote that row back for a moment.
2026-10-04 04:16:52 +02:00
peter d1df76e8e5 test(gates): the complexity ratchet names a function by what it is bound to
ESLint leaves an arrow bound to a const or a property unnamed, so three ratchet keys were
(anonymous) and ranked by position again. A tiny rule reports each unnamed function's
binding (const, property, assignment, else <callee> callback) at the head complexity
reports, and the keys line up by location.
2026-10-04 04:16:52 +02:00
peter 3f6a9efd57 test(gates): the register test parses ruff's TOML and holds pytest.ini line by line
Round 2 of the E32 review found the key whitelists blind: a line regex saw neither a
[lint.per-file-ignores] table nor a quoted key, and pytest.ini's continuation lines and
key: value form slipped past. Ruff's config is now parsed (smol-toml) and held to its
key paths; pytest.ini to its whole lines; the CI ruff steps to --config backend/ruff.toml,
which is what makes that file the whole configuration.

ESLint: every glob of a block must reach a sample (reactPlugin.ts and devOnlyGuard.ts
rode along unseen), a block's own severity is held to src, a no-restricted-* list is held
to the src block's declaration rather than Feed.tsx's effective config, named global
ignores count, an ignored sample is named, and linterOptions are checked. tsconfig.node
must keep allowJs + checkJs while it lists a .js file; knip's positive glob is pinned.
2026-10-04 04:16:52 +02:00
peter 64e8f1da3d fix(lint): E402 is on — the ignore meant for main.py had let 11 late imports in elsewhere
ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
2026-10-04 03:14:24 +02:00
peter a1224e9a8c test(gates): the repo-root scripts are linted and format-checked
scripts/check-node-modules.mjs and gen-changelog.mjs run in check and release but sat outside
every lane: the eslint and prettier lanes run from frontend/, and an ESLint config cannot reach
past its own directory. A contract test now holds them to the js-configs bar (recommended rules
under Node, the frontend's Prettier style); an unused variable and the old formatting are each
red. 0 lint hits; both files reformatted. Registered in docs/gates.md.
2026-10-04 03:00:49 +02:00
peter da4b19a39a docs(lint): the config comments say which rules can report, and count what is suppressed
eslint.config.js called every zero-hit React Compiler diagnostic a tripwire. Under
eslint-plugin-react-hooks 7.1.1 three cannot report anything: exhaustive-effect-dependencies (its
validation is compiled off), memoized-effect-dependencies (nothing emits its category) and
rule-suppression (lint mode passes no suppression rules). Probed: a missing effect dependency is
reported by exhaustive-deps only, and an inline disable is named by nothing. The header's
"33 suppressions" of exhaustive-deps is 0. reactPlugin.ts said one react-hooks rule was still
off; none is (three run at warn, which --max-warnings 0 fails).
2026-10-04 03:00:49 +02:00
peter 5a74cf86b7 test(api): the wire-key checks send every filter field, and tsc says when one is new
The Plex and feed filter fixtures were typed as the filter type itself, whose scalar filters are
optional: a new optional field compiled without a test value, so its wire key went unchecked.
Both are Required<…> now (a probe field fails typecheck on each). The feed set gained the fields
it omitted (year, librarySource, the scope and show switches) and goes out with and without year,
which wins over the other date filters; a misspelt year key is now red in the wire check, which
passed it before.
2026-10-04 02:57:09 +02:00
peter e432dc2780 test(gates): the complexity ratchet holds each function by name, and its scope is registered
The ceilings were a per-file ranked list, so one function could grow by what another in the same
file lost: Home.tsx's [34, 32] passed with Home at 32 and Hero at 34. Both twins now key every
offender as module::function (an unnamed one is (anonymous); a repeated name gets 2, 3 in source
order) and re-baseline at today's measured values. Swapping two ceilings is red on both sides
(Home 34 > 32; feedquery filtered_query 28 > 14).

The ratchet measures shipped code only. docs/gates.md said C901 covered backend + scripts; it
covers backend/app. The exclusion of tests, test kits, e2e and scripts/ is now a row with its
measured hits, and the stale counts are replaced (9 BE modules over 800, not 11).
2026-10-04 02:55:07 +02:00
peter 3741d1a46d fix(playlists): a list reopened from the cache keeps its rows
Leaving the Playlists page and coming back painted the selected list empty ("0 videos") while
the rail counted its rows. The page re-seeds its undoable order from the cached list in an
effect; StrictMode runs that effect twice on the remount, so a reset was followed in the same
tick by a rebase. Since E32 S3 a reset is state only and the rebase read the mirror ref, which
still held the empty initial history, and wrote it back over the reset.

rebase is now a state updater, so it builds on whatever an earlier call in the tick installed.
Both tests seen RED (hook: [] for [10, 20]; page: no rows), and the bug reproduced in the
browser on the old hook.
2026-10-04 02:48:10 +02:00
peter 98ec1abf25 test(gates): the register test reads every config block, option and key, not eight samples and severities
E32 review round 1 (T1–T5, s1). The test that holds docs/gates.md to the effective configuration
could itself be passed by a narrowing it did not look at:

- ESLint: besides each scope's sample, every config block is read on its own. A block that targets
  files no sample reaches fails as an "unsampled block"; a no-restricted-* list re-declared without
  an entry src has, an option that lets something through (allow…/ignore…/except…), a block-level
  `ignores` and EVERY global ignores object (not only the first) count. A scope switching a src rule
  at warn off now counts too (it skipped them). Two live narrowings surfaced and are registered:
  no-floating-promises' allowForKnownSafeCalls (src and e2e) and the three network owners'
  no-restricted-globals.
- TypeScript: a strict sub-flag set false beside `strict: true`, an extends/exclude/files, a changed
  include, and a frontend-root config file no project checks. eslint.config.js is that file: listed,
  with the 43 errors strict checkJs reports in it (measured, none a defect).
- Ruff, basedpyright, pytest and knip are read as key whitelists (an unknown key fails until someone
  decides whether it narrows); pyright's information/hint count as weak; ruff's extend-select is
  pinned; pytest's three keys are pinned exactly; knip's script must run both modes.

Each new check was probed with a planted narrowing (an unsampled block, a block ignores, a second
global ignores, a warn rule off in a scope, strictNullChecks false, a ruff extend-exclude, a pytest
-m) and failed on it.
2026-10-04 02:37:33 +02:00
peter e5b155ada9 refactor(ui): reuse the store, media-query and deleted-tag rules the review found rebuilt, and name a loading pick
E32 review round 1 (F3–F6).

- Channels' open-row store and the chat dock use lib/store's createStore instead of a hand-rolled
  listener set (the chat dock gets a test of its own).
- The Plex backdrop reads prefers-reduced-motion through useMediaQuery; the query string is one
  constant (REDUCED_MOTION) for the hook and the carousel's one-off read.
- A link-restored collection filter showed its raw rating_key in the picker while the titles
  loaded: usePlexCollectionTitles now says it is pending, the picker draws a placeholder
  (labelsPending, as the channel picker does) and so does the chip row.
- The "deleted tag" rule was written twice with two i18n keys and had drifted: the channel manager
  showed NO chip for a pick while the tag list loaded, or ever after a failed load, while the pick
  still narrowed the table. One rule now (lib/tagPicks), one key (common.deletedTag); the manager's
  row shows every pick, as a placeholder while loading and as its id after a failure.
2026-10-04 02:29:03 +02:00
peter e09d613b04 fix(ui): a reviewed log takes the focus back once, and a dialog's focus survives a Retry
E32 review round 1 (F1, F2).

- Issue composer: the review's focus-return token became state in S4 and was never spent, so every
  Preview → Edit round trip remounted the attachments and moved the focus from the toggle to the
  log's Review button. The token is cleared once it has been used, as the ref was.
- Modal (initialFocus "field"): the late-form watch moved the focus to the first control once and
  stopped. A failed load's Retry then gave way to the spinner, its focus fell to <body> inside the
  aria-modal dialog, and the form's first field never got it. The watch now re-arms when the focus
  falls to <body>: it waits on the ✕ and moves to the first field when the form arrives. Focus in a
  dialog stacked on top is left alone.
2026-10-04 02:29:03 +02:00
peter 38943fa93e fix(maintenance): one clock per run, so a video phase 1 recovers is not fetched again in phase 2
E32 review round 1 (B7). Each phase took its own now; phase 2's "not checked this run" test then
let a just-recovered video through and spent another videos.list call on it.
2026-10-04 02:14:04 +02:00
peter 887b4e546c fix(feed): a malformed saved view cannot break the autotag cleanup, and a foreign private tag reads as missing
E32 review round 1 (B5, B6).

- The autotag orphan cleanup reads every user's feed views. A view whose tag pick was not a list
  ("tags": 5) raised for everyone's job, True passed as tag id 1, and ~66k ids pushed the NOT IN
  past psycopg's 65535 bind parameters. The views are now read defensively and the kept ids
  filtered in Python; the saved-views route holds filters.tags to a list of at most 500 ids, on
  create and on update.
- tags_by_category resolved any tag id, so another user's private tag landed in its real category
  and the feed summary answered differently than for an id that does not exist, which told the
  caller which foreign ids exist. It is owner-scoped now (pre-existing; S2's missing-tag branch
  made the difference visible).
2026-10-04 02:14:04 +02:00
peter df4a79a1f1 fix(sync): an empty answer the remote's own count contradicts is not believed, one level down
E32 review round 1 (B1, B2, B3, B8).

- Plex watch reconcile: a section the server SAYS is empty (totalSize 0, nothing listed) is a
  complete listing. It read as "cut short" forever, so one empty library stopped un-watches from
  ever syncing. A listing with no total at all stays incomplete (the client now returns None for
  it instead of 0), and a whole scan that listed nothing still clears nothing. Tests now cover
  completeness across two sections, both orders.
- Plex collections: no members from a collection whose listing counts some (childCount) keeps
  the membership we hold, in the full sync and in the targeted resync after a write.
- YouTube playlists: no items for a playlist whose listing counts some (itemCount) keeps the
  mirror in the bulk pull, and is refused as retryable in "reset to YouTube" and in the push
  plan/merge, where it read as "YouTube deleted them all". The rule sits beside vanished().
- A playlist deleted on YouTube answered "reconnect your account"; it now says it is gone
  (errors.youtube.gone, HU+EN).
- The pure 3-way merge moved to sync/playlist_merge.py; playlists.py is back under 800 lines and
  off the size list.
2026-10-04 02:13:53 +02:00
peter 5d10a521a8 docs(gates): react-hooks/refs is on, the compiler skips one function, and the async-cleanup idiom that keeps it so 2026-10-04 00:05:37 +02:00
peter d7fb7075b8 refactor(ui): every function compiles under the React Compiler but VirtualGrid, and react-hooks/refs is on
E32 S4 — the burn-down from 29 skipped functions to 1:

- try/finally (26 sites): the cleanup runs after a try/catch; steps that return early moved into
  their own function (push, the setup wizard's next, the share page's unlock).
- A conditional, loop or optional call inside a try skips the function too, and react-hooks/todo
  does not see it: the value is computed before the try, or the try lives in a module-level helper
  (config writes, clip queueing, clipboard, frame counts, seekable end, YouTube player times).
- The two exhaustive-deps suppressions are gone. useYtProgress takes its per-item snapshot through
  one effect event, read once when the player is built; PlexPlayer's unmount relies on the
  leaving-the-item cleanup for the generation bump it duplicated.
- What the skips hid: PlexPlayer's 17 render-phase ref writes (useLatest), its tracks popover alias,
  the loadSession self-reference (declared early, set after commit); PlayerModal's session-position
  read during render (now where the player starts), navRef/modeRef writes, and a description helper
  that ran a ref-carrying callback during render (now PlayerDescription); SavedViewsProvider's
  latest-cfg ref, and its setState in the seed effect (preferSticky now prefers the starred default
  among same-filter views, so the seed needs no sticky write — test seen red on the old helper).
- A store's hook was called `use()`, which the compiler does not know as a hook: inside the newly
  compiled usePlaylistActions it was memoized, ran on the first render alone, and the playlist rail
  threw "rendered fewer hooks" on its second render. Renamed to `useValue()` everywhere; the old
  contract test claimed the call was safe inside any `use…` function, which was false — replaced by
  a render test that is red under the old name.
- PlexPlayer's palette ref was written and never read since the Backspace guard moved to the layer
  stack: removed.

react-hooks/refs is back at error with no exception. The ratchet lists only VirtualGrid
(IncompatibleLibrary) and pins both players as compiled. No size or complexity ceiling was raised.
2026-10-04 00:05:37 +02:00
peter a486c7521a docs(gates): the compiler burn-down after E32 S3, and the two idioms that keep it
react-hooks/refs 245 -> 31 hits and 93 -> 29 skipped functions in the register and the eslint
comment. The canonical-patterns table gets the latest-value row (useEffectEvent; a handler or timer:
useLatest / useStableCallback, never a render-phase ref write) and the popover / scroll-fade row
(destructure the ref setter).
2026-10-03 23:18:33 +02:00
peter 76358a0d77 refactor(ui): the popover engine and every render-phase ref write compile under the React Compiler
E32 S3. The compiler skipped 93 functions; it now skips 29 (try/finally shapes, two suppressions,
VirtualGrid). react-hooks/refs hits 245 -> 31.

- Popover engine: `ref={pop.refs.setReference}` made the compiler treat the whole popover handle as
  a ref, so every component holding one was skipped (21 call sites, the engine and its surface
  among them). The setter is destructured now; the same for useScrollFade and useReturnFocus.
- Render-phase `ref.current = x` writes moved to `useEffectEvent` where only effects read them,
  else to the new `useLatest` (written in a layout effect) / `useStableCallback`.
- FocusSafeButton (under every Button) read the focused element during render; it reads it through
  useSyncExternalStore, which React asks on every render instead of a cached memo.

Bugs this surfaced, fixed:
- Combobox: a chip could keep showing its raw id. The label cache was a ref an effect filled after
  the render that needed it; it is state merged during render now (test seen RED).
- useUndoable rendered from a ref plus a forced re-render; compiled, that memoizes a stale value.
  The history is state now, mirrored in a ref for the stable callbacks.
- ConfigPanel and the saved-views editor re-seeded their drafts in an effect, painting the stale
  draft for a frame; both adjust during render against the remembered key.
- Toaster's handover is written after commit instead of during render; a new test pins that the
  inherited set is taken once, at the takeover (seen RED on a per-render re-read).
- IssueComposer mutated a ref prop as a one-shot focus token; it is plain state.
- PlexBrowse wrote its info-scroll memory from an effect declared before the ref, past its own
  "one writer" rule; the writer comes first now and the effect goes through it.

The compiler ratchet drops to the measured 29 and now also requires FocusSafeButton and the popover
engine to compile.
2026-10-03 23:18:26 +02:00
peter dc00b55feb test(auth): warm the Welcome import once so the first case cannot time out under load 2026-10-03 21:53:06 +02:00
peter fb6b56f121 test(api): every filter key the client sends is one the backend reads
FastAPI ignores an unknown query parameter, so a misspelt or renamed key silently stops a
filter while its chip still shows. Contract tests read routes/feed.py _feed_params and
plex/query.py LibraryFilters. The superseded feedCount/facets calls and keys are removed.
2026-10-03 21:53:06 +02:00
peter 2e2d6f5746 fix(downloads): the quota editor opens at once and says it is loading or failed
It rendered nothing until its quota arrived and nothing at all when the load failed, so the
click looked dead. The first field still takes focus when the form arrives.
2026-10-03 21:53:06 +02:00
peter 2ab9ff6a42 fix(plex): a server without a machine id refuses the collection write by name
The missing identifier became server:/// in the write URI. Client tests over a MockTransport
(errors, paging, cutoff, write URIs, image transcode) and admin-route tests (connection test,
enabled-section sampling, no server URL in a 422, the one-time watch import).
2026-10-03 21:53:06 +02:00
peter 378c4daf0f fix(youtube): the scrape search never pins a stale key, and an unknown shape is an error
A failed config fetch or a keyless consent page cached the fallback InnerTube key for the life
of the process; once it went stale, search returned nothing with no error. The fallback is now
used for that call only. A 200 JSON without a result frame returned an empty page that read as
no results; it is a ScrapeError now. Parser, cursor and Shorts-probe tests added.
2026-10-03 21:53:05 +02:00
peter 8f1208fdfe fix(feed): a filter on a deleted tag matches nothing and its chip says so
A saved view or the stored filter outlives its tag. feedquery dropped the unknown id, so the
view showed the whole feed under its old name. The id now lands in a MISSING_TAG_CATEGORY that
filters to nothing in the SQL and in feed_summary's mirror alike; the feed chip reads "Deleted
tag" (a placeholder while the tag list loads, the id if it failed). Channels' persisted tag
filter gets the same chip, first in the row so the +N overflow cannot fold it away.
2026-10-03 21:53:05 +02:00
peter 52f12e5331 test(e2e): click only a channel link that is the element hit at its centre
A card inside the viewport can still sit under the chrome above the scroller (the update
banner); the covered click made Playwright scroll and retry, moving the feed the scroll-restore
test then measured. webkit failed 3 in 24 before, 0 in 48 after.
2026-10-03 21:10:27 +02:00
peter c4abbe48c1 refactor(sync): the empty-answer rule for a mirror is one function, vanished()
The YouTube playlist guard from the previous commit put sync/playlists.py over its size ceiling;
the rule now lives beside should_prune as vanished(existing, listed), which the mirror calls in
place of its own list. The pyright baseline also drops by one (187), from the plex/overlay split.
2026-10-03 19:45:14 +02:00
peter 4fc307c426 fix(autotag): the orphan cleanup keeps a system tag a saved view filters on
Feed views store tag ids. A system tag no channel carried for a while was deleted and, once a
channel earned it again, recreated under a new id — the view's filter then pointed at nothing, the
feed dropped the unknown id, and the view showed the whole feed under its old name. The cleanup now
skips tags a saved feed view references (test seen red without it).
2026-10-03 19:37:24 +02:00
peter 328d415cfe test(e2ee): the DM crypto round-trips, refuses the wrong key, and keeps its nonces apart
lib/e2ee was 14% covered and fails silently in every way it can break. Real WebCrypto and an
in-memory IndexedDB: a message decrypts for its partner and nobody else, every message gets its own
IV, a new device unlocks with the passphrase and refuses a wrong one, the key survives a reload
until reset, a partner's rotated key is used only after forgetConv, and the key check never shares
the wrap IV (GCM nonce reuse). Mutations seen red: the shared IV, a no-op forgetConv, no persist.
2026-10-03 19:35:36 +02:00
peter 43e4929b45 fix(sync): an empty answer no longer empties a mirror; maintenance checks each video once a run
The catalog prune and the subscription import have long refused to reconcile against an empty
listing that contradicts what we hold (should_prune: a 200 with no items is a blip far more often
than a user who removed everything). Two mirrors did not: the YouTube playlist mirrors (dropped
every mirror) and the Plex collections (dropped every collection with its user-set `editable`
flag, and every member's keys — the guard landed in plex/sync.py with the previous commit). Both
now hold the same rule; test_empty_listing_guards covers each way (seen red).

sync/maintenance (19% covered) gets a lifecycle test: recover, grace, delete + one notification
per user, live stream kept, abandoned premiere. It found the rolling revalidation re-fetching the
same videos until `checked` reached the batch (15000 by default) whenever fewer videos were
available — quota spent on answers it already had; a run now skips what it has checked.
2026-10-03 19:33:38 +02:00
peter 5806adda7e fix(plex): a listing cut short no longer clears watch state or Hungarian text
The daily watch reconcile reads "absent from Plex's scan" as "Plex un-watched it" and deletes the
row — but it walked the sections without asking _paginate whether the walk reached the server's
total, so a Plex hiccup mid-walk (an empty page early) un-watched every title it failed to reach
until the next complete run. The scan now reports completeness; an incomplete one still pulls in
what it saw, re-pushes dirty rows, and clears nothing (logged as plex.watch.reconcile_incomplete).

The HU overlay had the same shape: clear every row's HU text, then refill from the listing. It now
reads the listing first and clears only after a complete one. The overlay moved to plex/overlay.py
(plex/sync.py drops under the 800-line limit and off the size ratchet's list).

Tests: plex/paths (23% -> 100%: prefix boundary, longest prefix, dead mount), the reconcile's
clear/keep/re-push rules, the cut-short scan and listing (each seen red with the guard removed).
2026-10-03 19:33:30 +02:00
peter c73761ae39 refactor(plex): one cast row for a show's page and a title's info
CastStrip (a show's page) and PlexInfo (a title's page and the player's overlay) each drew the
cast carousel by hand, and the copies had drifted: the gap, lazy pictures, the name's tooltip, the
initial's size. Both now render CastRow, which takes the overlay's white-on-art palette as a tone;
each caller keeps its own frame and heading. A DOM test holds the palette per variant (seen red
with the tone wired to "panel"). jscpd over the shipped code: 1 clone >= 10 lines -> 0.
2026-10-03 19:12:17 +02:00
peter ca9ca85b18 test: no test throws a response away; the stream session is narrowed, not baselined
The 7 baselined Optional accesses in plex/stream.py were one fact the type did not carry: the
initial spawn (replace_only=None) always supersedes, so only the fallback respawn can get None.
An assert at the call site says so; the pyright baseline drops 195 -> 188.

test_abuse_limits: the public-URL and YouTube enqueue tests stopped at the profile lookup
(400 no_profile) and passed on that unrelated refusal. They now send an inline spec and assert
the job is queued. The same shape — a request whose response is discarded — sat at 53 more
places across the suite (all returned what they should, measured with a TestClient probe);
each now asserts its status, and test_response_checked keeps it that way (seen red).
2026-10-03 19:06:47 +02:00
peter 614d728561 test(e2e): the nav spec opens the admin pages from their rail panel
E31 put the admin pages behind one rail row ("Administration", nav-admin)
that opens a panel. The spec still clicked nav-scheduler, nav-config,
nav-users and nav-audit as rows of their own, and timed out on every engine.
That went unseen from E31 to now because e2e ran in no required gate, the
exact hole E32 closes: the first run of the new release e2e step found it.

The rail test keeps the rail rows. A new test opens each admin page from the
panel and checks that the group row is current and that the reopened panel
marks the page. Seen red before the fix; the whole suite then passes, 52/52.
2026-10-03 17:41:26 +02:00
peter 783f6408f9 docs: the gate register, held to the effective config by a test
docs/gates.md lists every place a gate deliberately checks less than it
could, each with its reason and the count measured when the row was
written:
- an ESLint rule off, at warn, or not applied, by scope;
- a weak compiler flag, a ruff ignore, the basedpyright mode and baseline;
- a knip exclusion, a lane that does not run everywhere;
- an accepted CVE.
gates.contract.test.ts reads the EFFECTIVE configuration and fails when a
narrowing has no row:
- ESLint's computed config per scope;
- the three tsconfigs, ruff.toml, pyrightconfig, pytest.ini and knip.jsonc;
- .trivyignore.yaml, where an entry also needs a statement and an expiry;
- CI's pip-audit ignores.
It also pins `--max-warnings 0`, which is what makes a warn-level rule fail.
Seen red on a missing row, a new `off` rule and an allowlist entry without
an expiry.

CLAUDE.md gains the rule E32 exists for: a gate with any real hit is never
switched off or narrowed, and an "enforced by" comment is checked by running
the check. The gate descriptions there, in docs/development.md and in
backend/app/README.md name the lanes as they now are, and the "directly"
recipe points at the full list.
2026-10-03 17:26:42 +02:00
peter 751e1ae5ed test: a size and complexity ratchet; coverage reported on both sides
The "class size" item from the quality checklist, measured:
- A shipped file over 800 lines may not grow past its line count today
  (13 frontend, 11 backend). No other file may cross 800.
- A function over the complexity limit may not rise, and no new one may
  appear. The limit is eslint `complexity` 20 on the frontend (64 functions
  over it today) and ruff C901 10 on the backend (32).
- Each is a per-file ceiling. Shrinking is free; an entry back under the
  limit must be deleted, so the lists only shrink.

C901 is measured in a test rather than switched on in ruff.toml: ruff has no
per-function baseline, and a noqa would let a function grow without limit.

Coverage (owner decision D4) is printed with no threshold:
`vitest --coverage` (v8) gives 80% of lines, and the pytest lane ends with
the total for app/, 78%. The bar stays "every regression test seen red".
.coverage is ignored by git and kept out of the image's build context.
2026-10-03 17:18:53 +02:00
peter da554df7c7 build(image): the image passes a Trivy scan: OS security updates, no runtime pip
The first Trivy scan of the shipped image (0.98.0) found four fixable HIGH
advisories. One was pcre2: a Debian fix newer than the base snapshot, now
taken by `apt-get upgrade` at build time. The others were urllib3 (bumped in
the lock), and msgpack, setuptools and urllib3 vendored inside pip itself.
pip is removed once it has installed the lock, since nothing runs it at
runtime.

A local build of this Dockerfile with --pull scans clean: 0 HIGH/CRITICAL
with a fix, no secrets. .trivyignore.yaml is the publish gate's allowlist,
empty today. Each entry must carry a statement and an expiry date.
2026-10-03 17:06:28 +02:00
peter b1ff4058ac build: CI builds the bundle and audits what ships; the toolchain advisories are cleared
The server gate adds `vite build` (and with it devOnlyGuard, which until now
first ran inside the image build, after main was pushed) and
`npm audit --omit=dev --audit-level=high`. pip-audit is blocking: when it was
finally read it held three urllib3 advisories with a fix available.
urllib3 goes 2.7.0 -> 2.8.0 as a single in-place pin, passing pip check,
rather than a regeneration that would move every package.

The dev toolchain's 18 advisories are gone:
- vite 5 -> 6.4.3 (path traversal, fs.deny bypass, and esbuild <= 0.24.2
  through it);
- vitest 3 -> 4.1.11 (mocker path traversal);
- typescript-eslint 8.46 -> 8.71;
- `npm audit fix` for the rest.
tsc, eslint, knip, all 2269 tests and the build pass unchanged.

scripts/check-node-modules.mjs checks that frontend/node_modules holds
exactly what package-lock.json pins. The local gate's frontend lanes judge
the host tree, and nothing tied it to the lock the image installs.
2026-10-03 16:50:56 +02:00
peter 7ed2ac4e95 build(frontend): knip also checks what ships; strict indexing in every project
`npm run knip` now runs a second, --production pass, which asks whether the
code that ships uses an export, not just whether anything does. Of the 47
exports only tests reached:
- 34 are exported for their test and say so with @internal, naming the test.
- 11 were module-private names with a stray `export`. Two of them were not
  used at all and are deleted.
- The unused alreadyReported re-export is gone.
Test kits, the e2e suite and the two build-tooling modules are excluded from
the production project, each named in knip.jsonc.

noUncheckedIndexedAccess is now on for the node and e2e tsconfigs too (0
errors), matching the app project. tsconfig.node.json listed a
tailwind.config.js that has not existed since Tailwind 4. That entry is now
playwright.config.ts, which the typed lint finds through this project.
2026-10-03 16:44:28 +02:00
peter cff8fe9a6a test(frontend): a React Compiler ratchet over everything the build ships
compilerCoverage.contract.test.ts used to check two files. It now runs the
real compiler over all of src (minus tests) and holds the exact set it skips
today, file -> function -> category: 93 functions. A new skip or a category
change is red, and so is a listed skip that has gone, so the list only
shrinks. Channels and DataTable must also show up as compiled, so a harness
that compiled nothing cannot pass.

Measured against the audit's scan: no function this sprint touched started
to skip. PlexPersonView and useOverlayOrder now compile.

reactPlugin.ts claimed the compiler's rules were linted, which was false
while react-hooks/refs was off. That comment and the refs block in
eslint.config.js now say what holds the line and why refs is still off:
245 real hits, pinned by the ratchet, burned down from S3.
2026-10-03 16:38:46 +02:00
peter 471846d784 build(frontend): close the lint gaps the E32 audit measured
eslint now also enforces:
- the React Compiler diagnostics that `recommended` leaves out (todo,
  memo-dependencies and ten zero-hit tripwires);
- a description on every suppression, and no blanket disables;
- no focused or skipped tests (vitest and playwright);
- the typed pair no-floating-promises and no-deprecated.

Lint also covers e2e/ and the TypeScript build configs, which `eslint .`
had been skipping in silence.

Every hit is fixed or suppressed inline with a named reason:
- The 31 try statements the compiler cannot lower yet carry a suppression
  until S3 rewrites them. Five other compiler skips were plain fixes:
  a module counter bumped inside a hook, a ref read in a default
  parameter, and a local named like the <section> it renders.
- The real defects behind floating promises are fixed. ChatDock threw at
  nobody when the device key store could not be opened. VideoEditor's
  play() rejection went unhandled. loadYouTubeApi never rejected, so a
  blocked IFrame API left the player on an empty stage forever; it now
  fails on a script error or after 15 s, shows its own error with the
  way out, and lets the next open try again.
- MutableRefObject, FormEvent and lucide's deprecated brand icon are
  replaced. The YouTube glyph now lives in lib/brandIcons.
- The lint-probe harness lets its virtual file names into TypeScript's
  default project, so the contract tests still run the real config.
2026-10-03 16:34:20 +02:00
peter 3c16012524 build(backend): close the lint gaps the E32 audit measured
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.

PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
2026-10-03 16:04:06 +02:00
peter cdfe7a8986 perf(channels): the row menu opens and closes without re-rendering the table
The React Compiler silently skipped Channels and DataTable (a ref written
during render; the refs lint is off), and even once compiled it dropped the
memo of the rows and columns (they reach filterRows/optionsFromColumns past
the page's hooks) and the columns read the open row. Every open, close and
reposition of the right-click menu rebuilt the whole table: on "All" rows
~0.9 s to open and ~0.6 s to close (dev). Now: effect events instead of the
render-written refs, explicit memos for rows, columns and their inputs, a
module-scope keyOf, and the open row reaches its "…" through a small store.
Measured on 397 rows (dev): ~75 ms to open, ~35 ms to close.

A contract test runs the real compiler over both files and fails on any
skipped function. The dropped memos have no automated guard: they need
the full page rendered with its providers; the measurement above is the
evidence.
2026-10-03 05:10:52 +02:00
peter 3e96211db0 fix(plex): the backdrop crossfade toggle wins over reduced motion once set
prefers-reduced-motion hard-disabled the rotation whatever the toggle said,
and Windows reports it whenever its animation effects are off — so the
owner never saw the crossfade. Untouched, the setting now follows the
system (and the toggle shows that state); a pick made on the toggle is
stored and wins on every device.
2026-10-03 04:38:50 +02:00
peter e9c15baae0 fix(feed): a channel filter's chips name their channels from the first paint
After a reload the chips (filter panel and the row above the feed) read as
raw UC… ids until the subscription list landed. The names seen last time
are now kept per account (useChannelNames) and name the chips at once; the
list then replaces the copy, so a renamed or dropped channel heals. A
channel never named on this browser draws a placeholder bone while the
list loads; a failed list still falls back to the id.
2026-10-03 03:29:54 +02:00
333 changed files with 11672 additions and 3313 deletions
+1
View File
@@ -4,6 +4,7 @@
*.env.local
**/__pycache__/
**/*.pyc
**/.coverage
.venv/
venv/
# Docker patterns are anchored at the context root with NO implicit `**/` (unlike .gitignore, where
-119
View File
@@ -1,119 +0,0 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# They are declared side by side but they DO NOT run in parallel: the lab has a single act_runner
# with one slot, so they queue. Measured across runs #86-#91 — the backend job starts at the exact
# second the frontend one ends, every time, and the pair costs ~4 + ~6 = **~10 minutes from the
# push**. Anything that waits on this gate has to budget for the sum, not the max; the ops
# `release` read-back timed out on its first real run for believing otherwise.
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip.
#
# The frontend container's node major is NOT free: it must equal the prod build image's and the dev
# machine's, because vitest/tsc read the engine's own Intl and web-API implementations and those split
# between majors. `.nvmrc` is the single source of truth and
# `frontend/src/lib/nodeEngine.contract.test.ts` fails this gate if any of them drift apart.
name: gate
# Branch pushes only. A release pushes `main` and then the `v<ver>` tag, and a bare `on: [push]`
# treated the tag as a second push of the IDENTICAL commit — every release ran this whole gate
# twice, to the same verdict, for nothing (runs #81-#90 are five such pairs). `branches: ['**']`
# keeps every branch covered and drops tag refs.
#
# It has to be `branches`, NOT the `tags-ignore` this obviously wants to be. Gitea's matchPushEvent
# requires EVERY named filter to match, and each one bails on the wrong ref kind: `tags-ignore`
# starts with `if !refName.IsTag() { break }`, so a filter meant to skip tags rejects every BRANCH
# push and the gate stops running entirely. `branches` bails the same way on a tag, which is the
# direction we want. (Measured in go-gitea v1.27.3 modules/actions/workflows.go, not assumed — it
# differs from GitHub, where `tags-ignore` alone leaves branches unfiltered.)
#
# If that filter were ever wrong the gate would not run at all — which is why `siftlode release`
# now ends by reading the verdict back for the sha it pushed, and says so out loud when there is
# no run for it. A gate that silently stops running is the failure this repo already lived through.
on:
push:
branches: ["**"]
pull_request:
jobs:
frontend:
runs-on: ubuntu-latest
container: node:24-bookworm
defaults:
run:
working-directory: frontend
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run typecheck
- run: npm run lint
- run: npm run format:check
- run: npm run knip
- run: npm run test
backend:
runs-on: ubuntu-latest
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
#
# This node runs the checkout action, never our code, so it is deliberately OUTSIDE the engine
# contract the frontend job is held to — it tracks `.nvmrc` as hygiene (node 20 went EOL in April
# 2026), but a missing upstream tag here must never be able to block the gate.
container: nikolaik/python-nodejs:python3.12-nodejs24
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: siftlode
POSTGRES_PASSWORD: siftlode
POSTGRES_DB: siftlode_test
options: >-
--health-cmd "pg_isready -U siftlode -d siftlode_test"
--health-interval 2s
--health-timeout 3s
--health-retries 30
env:
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
SIFTLODE_TEST_REQUIRE_DB: "1"
steps:
- uses: actions/checkout@v4
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
# The ninth lane (E27 S3). Run from backend/ so it resolves exactly as the local lane does —
# pyrightconfig.json's include/extraPaths are relative to the config, but the baseline is
# found relative to the working directory. New errors fail; the ~200 pre-existing ones are in
# .basedpyright/baseline.json and burn down epic by epic. Matching is per (file, rule, column
# span) plus a count, so a new error of the SAME shape in a file where a sibling was just
# fixed is absorbed — burn the hot files down first.
#
# The second step is not redundant. basedpyright TIGHTENS the baseline in place when the count
# drops, with no --writebaseline, and CI throws its workspace away — so without this a
# burn-down is green here and red on the developer's `siftlode check`, which has the same
# guard. Two gates over one tree must not answer differently.
- name: basedpyright
run: python -m basedpyright --project pyrightconfig.json
working-directory: backend
- name: basedpyright baseline is committed
run: git diff --exit-code -- backend/.basedpyright/baseline.json
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
- name: pip-audit (advisory)
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
continue-on-error: true
+1
View File
@@ -9,6 +9,7 @@ __pycache__/
venv/
.mypy_cache/
.pytest_cache/
.coverage
.ruff_cache/
# Node / frontend
+15
View File
@@ -0,0 +1,15 @@
# The CVE gate's allowlist (E32). The publish gate scans the built image and the committed tree with
# Trivy and fails on any HIGH or CRITICAL finding that has a fix available. A finding that cannot be
# fixed yet is let through ONLY by an entry here, and every entry carries both:
# statement — why it is acceptable for now (what is exposed, what is not)
# expired_at — the date it stops being accepted; after it, the gate fails again on its own
# A contract test (frontend/src/lib/gates.contract.test.ts) refuses an entry without either, and
# docs/gates.md lists them.
#
# Example:
# vulnerabilities:
# - id: CVE-2026-00000
# statement: in libfoo's XML path, which nothing in the image calls; no fixed Debian build yet
# expired_at: 2026-12-31
vulnerabilities: []
secrets: []
+33
View File
@@ -7,6 +7,39 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.99.2 — 2026-10-04
### Fixed
- Settings: the download worker's concurrency and the log level now take effect within seconds — before, the worker ignored both until it was rebuilt.
- The YouTube quota history no longer grows forever: the daily cleanup prunes it after a year (Configuration → Quota → Quota history kept).
- The live-search button no longer says a search spends the API quota: with the default website source the search itself is free. It names the cost when the official API is selected.
## 0.99.1 — 2026-10-04
### Fixed
- Behind the scenes: every release is now checked by one script on the build machine, including the security audit of every dependency.
## 0.99.0 — 2026-10-04
### New
- Nearly every screen is now compiled by the React Compiler, so lists and menus re-render less.
- The channel list's row menu opens and closes without redrawing the table.
### Fixed
- A failed or empty answer from YouTube or Plex no longer empties a playlist, a mirror or your Plex watch history: Siftlode keeps what it had until a complete answer arrives.
- Plex watch states un-watched on Plex now sync for each library on its own, even if another library is briefly unreachable.
- A channel with no uploads no longer fails its backfill on every run.
- “This playlist no longer exists” appears only for a list you picked yourself.
- A filter on a deleted tag matches nothing and its chip says so, and the tag cleanup keeps tags a saved view filters on.
- A channel filter names its channels from the first moment, and a playlist reopened from the cache keeps its rows.
- The bug report opens on its title, and a dialog whose form loads late gives the form the focus once.
- The Plex backdrop crossfade setting is respected even with reduced motion on.
- Smaller fixes to keyboard focus, undo and loading states.
## 0.98.0 — 2026-10-03
### New
+41 -36
View File
@@ -12,12 +12,15 @@ router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
The local gate is **ruff (`check` + `format --check`) + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip (both modes) + vitest + vite build + pytest + pip-audit + npm audit**, then a coverage report (no threshold). It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every file and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: a new code file or any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` and burn down epic by epic. Fixed some? Regenerate with `--writebaseline`.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
@@ -77,37 +80,39 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE button | `ui/Button` (a `SIZES` rung) · `IconButton` · `Pressable` (a clickable card/row) · `Chip` · `DragHandle` | a raw `<button>` or `FocusSafeButton` outside the control primitives (lint-enforced) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE text field / checkbox / slider | `ui/Input` · `Textarea` · `InnerInput` in a `fieldClass` frame · `Checkbox`/`CheckRow` · `Slider` | a raw `<input>`/`<textarea>` anywhere but those primitives (lint-enforced) |
| FE glass surface | a role: `SURFACE.panel` / `card` / `menu` / `popover` / `tooltip` (`ui/surfaces.ts`) | a bare `glass`/`glass-card`/`glass-menu` beside a hand-picked radius (lint-enforced) |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+), else a latest-value ref | an excused dep list asserting the same thing in prose |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| Area | Use | Not |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; helpers `flush` (test-held list) | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE button | `ui/Button` (a `SIZES` rung) · `IconButton` · `Pressable` (a clickable card/row) · `Chip` · `DragHandle` | a raw `<button>` or `FocusSafeButton` outside the control primitives (lint-enforced) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE text field / checkbox / slider | `ui/Input` · `Textarea` · `InnerInput` in a `fieldClass` frame · `Checkbox`/`CheckRow` · `Slider` | a raw `<input>`/`<textarea>` anywhere but those primitives (lint-enforced) |
| FE glass surface | a role: `SURFACE.panel` / `card` / `menu` / `popover` / `tooltip` (`ui/surfaces.ts`) | a bare `glass`/`glass-card`/`glass-menu` beside a hand-picked radius (lint-enforced) |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE control | `SegmentedToggle`; a long or user's list: `FilterSelect` | a native `<select>` (lint-enforced) |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| FE network call | `req`/`uploadReq`/`beacon`, or `lib/http.ts` `send` (public pages) | a bare `fetch`/`XMLHttpRequest`/`WebSocket` (lint-enforced) |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
## House rules that bite (learned from real bugs)
+10 -4
View File
@@ -11,7 +11,7 @@ ARG BUILD_DATE=
# Stage 1: build the frontend SPA
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image, the CI gate's image and the dev machine must be ONE engine (see that test for why).
# this image and the dev machine the gate runs on must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
@@ -53,8 +53,11 @@ RUN adduser --disabled-password --gecos "" appuser \
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade` (E32): the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
@@ -76,11 +79,14 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock (E32): nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls.
COPY backend/requirements.lock .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && pip install -r requirements.lock
pip install --upgrade pip && pip install -r requirements.lock && pip uninstall -y pip
COPY --chown=appuser backend/ .
# Read at runtime by config.py as the app_version (a committed single source of truth).
+1 -1
View File
@@ -1 +1 @@
0.98.0
0.99.2
-84
View File
@@ -548,64 +548,6 @@
}
}
],
"./app/plex/stream.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 51,
"endColumn": 55,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 23,
"endColumn": 25,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 18,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 75,
"endColumn": 79,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 23,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 12,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 10,
"endColumn": 17,
"lineCount": 1
}
}
],
"./app/plex/sync.py": [
{
"code": "reportArgumentType",
@@ -638,14 +580,6 @@
"endColumn": 89,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 17,
"endColumn": 18,
"lineCount": 1
}
}
],
"./app/plex/tmdb.py": [
@@ -1650,24 +1584,6 @@
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 46,
"endColumn": 52,
"lineCount": 1
}
}
],
"./app/youtube/rss.py": [
{
"code": "reportGeneralTypeIssues",
"range": {
"startColumn": 10,
"endColumn": 15,
"lineCount": 1
}
}
]
}
+6 -4
View File
@@ -72,7 +72,9 @@ mutation it records, so an audit entry can never exist without its change (or vi
## The gate
Everything below must be green before a change is done (run via `siftlode check`):
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
`ruff` + `basedpyright` (backend) · the lock check + `tsc` ×3 + `eslint` + `prettier` + `knip` +
`vite build` (frontend) · `vitest` · `pytest`. The pytest lane runs in `backend/docker-compose.test.yml`
— an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the `db`
fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks
for pure-logic only with `SIFTLODE_TEST_REQUIRE_DB=0`. See `docs/development.md`, and `docs/gates.md`
for what each gate deliberately leaves out.
+1 -1
View File
@@ -219,7 +219,7 @@ class BodyLimitMiddleware:
try:
await self.app(scope, counting_receive, guarded_send)
except Exception: # noqa: BLE001 — only swallowed when WE caused it, see below
except Exception: # only swallowed when WE caused it, see below
# ClientDisconnect (or whatever a handler raises on a truncated read) is the expected
# consequence of the disconnect we just faked, and the caller must get the 413 rather
# than a 500. Anything raised when we did NOT refuse is a real failure and is re-raised
+3
View File
@@ -260,6 +260,9 @@ class Settings(BaseSettings):
# Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs.
audit_retention_days: int = 180
audit_gc_minutes: int = 1440
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: DownloadLayout = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
+2 -2
View File
@@ -400,7 +400,7 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
try:
proc.wait(timeout=grace_s)
except subprocess.TimeoutExpired:
# noqa TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# `wait` call and nothing else — it would add three frames and no fact the line lacks.
log.error("%s pid %s survived SIGKILL", label, proc.pid) # noqa: TRY400
@@ -456,7 +456,7 @@ def run_ffmpeg(
try:
ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
except subprocess.TimeoutExpired: # pipes closed but the process lingers
raise StatedFailure("ffmpeg did not exit after closing its output")
raise StatedFailure("ffmpeg did not exit after closing its output") from None
except BaseException:
# ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
# function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
+2 -2
View File
@@ -57,7 +57,7 @@ def main() -> None:
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
# noqa T201: a usage line is for the person who typed the command. Logging is already
# T201: a usage line is for the person who typed the command. Logging is already
# configured here, and in a deployment it writes JSON records — the one reader of this
# line wants the bare sentence, not a record with a request id they do not have.
print( # noqa: T201
@@ -104,7 +104,7 @@ def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) ->
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
# noqa TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
# TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
log.error( # noqa: TRY400
"the download child for job %s failed",
job_id,
+24 -9
View File
@@ -106,15 +106,30 @@ def _tag_visible(user: User):
return or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
def tags_by_category(db: Session, tag_ids: list[int]) -> dict[str, list[int]]:
# The pseudo-category of a selected tag id no Tag row answers (a saved view or the feed state outlives
# a deleted tag). It filters like a category whose tags no channel carries — to nothing, in the SQL and
# in feed_summary's mirror alike — instead of the id being dropped, which showed the WHOLE feed under
# the view's old name (E32 S2). No real category is called this (language/topic/other).
MISSING_TAG_CATEGORY = "missing"
def tags_by_category(db: Session, user: User, tag_ids: list[int]) -> dict[str, list[int]]:
"""Group tag ids by their category (language/topic/other). The single source for the category split
shared by the feed's tag WHERE (filtered_query) and the facet/count summary (feed_summary), so the
two can't disagree on which selected tag belongs to which category."""
two can't disagree on which selected tag belongs to which category. An unknown id lands in
MISSING_TAG_CATEGORY — and so does another user's private tag: resolving it to its real category
told the caller which foreign ids exist (E32 review)."""
if not tag_ids:
return {} # skip the pointless `WHERE id IN ()` round-trip on the no-tag-filter hot path
out: dict[str, list[int]] = {}
for tid, cat in db.execute(select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids))).all():
visible = or_(Tag.user_id.is_(None), Tag.user_id == user.id)
for tid, cat in db.execute(
select(Tag.id, Tag.category).where(Tag.id.in_(tag_ids), visible)
).all():
out.setdefault(cat, []).append(tid)
known = {tid for ids in out.values() for tid in ids}
if missing := [tid for tid in tag_ids if tid not in known]:
out[MISSING_TAG_CATEGORY] = missing
return out
@@ -304,7 +319,7 @@ def filtered_query(
# ALL its selected tags. ⚠️ feed_summary.passes() reimplements this exact rule in Python for the
# sidebar count/facets — keep the two in lockstep (equivalence-tested by
# test_feed_summary_matches_count_plus_facets).
by_category = tags_by_category(db, tags)
by_category = tags_by_category(db, user, tags)
# Facet counting drops the category being counted so its own chips don't zero each
# other out (standard drill-down faceting: a category's count ignores its own
# selections but still honours the other categories' filters).
@@ -395,7 +410,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
return {"count": 0, "facets": {}}
# channel -> its visible tag ids, plus each tag's category (for grouping the facet output).
visible = or_(ChannelTag.user_id.is_(None), ChannelTag.user_id == user.id)
visible = _tag_visible(user)
channel_tags: dict[str, set[int]] = {}
tag_category: dict[int, str] = {}
for cid, tid, cat in db.execute(
@@ -410,7 +425,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
# base channel carries still filters everything out for its category — matching the SQL `IN (empty)`
# behaviour).
selected_by_cat: dict[str, set[int]] = {
cat: set(ids) for cat, ids in tags_by_category(db, params.get("tags") or []).items()
cat: set(ids) for cat, ids in tags_by_category(db, user, params.get("tags") or []).items()
}
def passes(ch_tags: set[int], exclude_cat: str | None) -> bool:
@@ -438,7 +453,7 @@ def feed_summary(db: Session, user: User, **params) -> dict:
# Drop this category's own selections so its chips keep independent counts; a conjunctive
# topic-AND keeps them (each remaining chip narrows to channels that ALSO have the selected).
exclude = None if tag_modes.get(category, "or") == "and" else category
for cid, ch_tags in channel_tags.items():
for ch_tags in channel_tags.values():
if not passes(ch_tags, exclude):
continue
for tid in ch_tags:
@@ -671,8 +686,8 @@ def decode_cursor(keys: list[dict], cursor: str) -> tuple[list, str, datetime |
raw = payload["k"]
if len(raw) != len(keys):
raise ValueError("key arity mismatch")
vals = [_cursor_dt(v) if k["kind"] == "dt" else v for k, v in zip(keys, raw)]
for k, v in zip(keys, vals):
vals = [_cursor_dt(v) if k["kind"] == "dt" else v for k, v in zip(keys, raw, strict=True)]
for k, v in zip(keys, vals, strict=True):
if v is None and not k["nullable"]:
raise ValueError("null value for a non-nullable key")
window = payload.get("w")
+1 -1
View File
@@ -239,7 +239,7 @@ def deliver(
# already open on the tracker and this is what would have recorded its link. We
# still must not write — another send owns the row and is opening its own issue —
# so say which issue was left behind, because nothing else will.
# noqa TRY400: the traceback is one SendClaimLost carrying the session id and
# TRY400: the traceback is one SendClaimLost carrying the session id and
# nothing else; the issue number below is the whole news.
log.error( # noqa: TRY400
"an issue was created by a send that had already lost its claim",
+1 -1
View File
@@ -166,7 +166,7 @@ def bind(**fields: Any):
try:
yield
finally:
for key, token in zip(keys, tokens):
for key, token in zip(keys, tokens, strict=True):
_CONTEXT_VARS[key].reset(token)
+21 -25
View File
@@ -1,38 +1,36 @@
import asyncio
import math
import mimetypes
from contextlib import asynccontextmanager, suppress
from pathlib import Path
from fastapi import FastAPI, HTTPException, Request
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
# Logging is configured HERE, before the route modules are imported, so every logger they create at
# import time is already covered. ONE format for the api, the worker, the download child and alembic
# alike (app/logs.py) — this replaces the old uvicorn --log-config + the hand-rolled fallback that
# used to live here, and uvicorn's access log is replaced by the `http.request` middleware below.
from app import logs
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logs.get_logger(__name__)
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from sqlalchemy.exc import DataError
from starlette.middleware.sessions import SessionMiddleware
from app import bodylimit
from app import collation
from app import logsink
from app import state
from app import sysconfig
from app.config import settings
from app.db import SessionLocal, engine
from app.plex import stream as plex_stream
from app.plex import subtitles as plex_subtitles
from app.routes import (
# Logging is configured HERE, before the app's own modules are imported, so a line one of them logs
# WHILE importing already has the format (a logger is configured whenever it is created —
# app/logs.py — but a line emitted before `configure_logging` goes out raw). ONE format for the api,
# the worker, the download child and alembic alike — this replaces the old uvicorn --log-config +
# the hand-rolled fallback that used to live here, and uvicorn's access log is replaced by the
# `http.request` middleware below. The imports after it are E402 on purpose, each one marked.
from app import logs
from app.logs import configure_logging, current_level, log_requests, set_level
configure_logging("api")
log = logs.get_logger(__name__)
from app import bodylimit, collation, logsink, state, sysconfig # noqa: E402 — logging is configured first
from app.config import settings # noqa: E402 — logging is configured first
from app.db import SessionLocal, engine # noqa: E402 — logging is configured first
from app.plex import stream as plex_stream # noqa: E402 — logging is configured first
from app.plex import subtitles as plex_subtitles # noqa: E402 — logging is configured first
from app.routes import ( # noqa: E402 — logging is configured first
admin,
admin_logs as admin_logs_routes,
audit as audit_routes,
@@ -60,7 +58,7 @@ from app.routes import (
tags,
version,
)
from app.scheduler import shutdown_scheduler, start_scheduler
from app.scheduler import shutdown_scheduler, start_scheduler # noqa: E402 — logging is configured first
@asynccontextmanager
@@ -77,7 +75,7 @@ async def lifespan(app: FastAPI):
# booting — an unreachable or not-yet-migrated database has to start into the env level.
try:
set_level(sysconfig.get_str(db, "log_level"))
except Exception: # noqa: BLE001 — any DB/config failure keeps the bootstrap level
except Exception: # any DB/config failure keeps the bootstrap level
# With the traceback: this catch spans a connection, a query and a coercion at boot,
# and `%s` of the exception does not say which of them raised.
log.exception("could not read the stored log level; staying at %s", current_level())
@@ -332,8 +330,6 @@ app.include_router(setup_routes.router)
# Ensure modern image types resolve to the right Content-Type when FileResponse guesses from the
# filename (the runtime's mimetypes db doesn't always know .webp → it'd fall back to octet-stream).
import mimetypes
mimetypes.add_type("image/webp", ".webp")
mimetypes.add_type("image/avif", ".avif")
+1 -3
View File
@@ -23,6 +23,7 @@ from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from app.db import Base
from app.utils import canonical_email
# Every user-supplied display string in this schema shares one width, and it lives HERE so the
# columns below are declared with it rather than with a literal: a name a person typed goes into
@@ -51,9 +52,6 @@ def display_text(raw: str | None) -> str | None:
return (raw or "").strip()[:DISPLAY_TEXT_MAX] or None
from app.utils import canonical_email
# The resume-position columns (VideoState / PlexState.position_seconds) are int4. The checkpoint
# routes bound their bodies by this, so an out-of-range position is a 422 rather than a 500.
POSITION_MAX_S = 2**31 - 1
+14 -5
View File
@@ -78,9 +78,10 @@ class PlexClient:
# --- Catalog mirror (P1) ---
def section_items(
self, key: str, item_type: int, start: int = 0, size: int = 200
) -> tuple[list[dict], int]:
) -> tuple[list[dict], int | None]:
"""A page of a section's top-level items. type 1 = movie, 2 = show. Returns
(items, total_size)."""
(items, total_size); the total is None when the answer carries none, so a container that
says "0" (a genuinely empty section) stays distinguishable from one that says nothing."""
mc = self._get(
f"/library/sections/{key}/all",
params={
@@ -93,7 +94,10 @@ class PlexClient:
"X-Plex-Container-Size": size,
},
)
return mc.get("Metadata", []) or [], int(mc.get("totalSize", mc.get("size", 0)) or 0)
# `totalSize` only: `size` is THIS page's count, and read as the total it made a single
# page look like the whole section — "complete" to a prune or a reconcile (E32 review).
total = mc.get("totalSize")
return mc.get("Metadata", []) or [], None if total is None else int(total)
def children(self, rating_key: str) -> list[dict]:
"""Direct children (show → seasons, season → episodes)."""
@@ -136,9 +140,14 @@ class PlexClient:
# --- Collection editing (P2 — write-back to Plex; all shared/global, admin-gated in the routes) ---
@property
def machine_id(self) -> str:
"""Server machineIdentifier — needed in the `uri` of collection add/create calls. Cached."""
"""Server machineIdentifier — needed in the `uri` of collection add/create calls. Cached.
Without one the uri would read `server:///…`, a malformed write Plex answers in its own way;
it is refused here, by name (E32 S2)."""
if not getattr(self, "_machine_id", None):
self._machine_id = self._get("/identity").get("machineIdentifier") or ""
mid = self._get("/identity").get("machineIdentifier")
if not mid:
raise PlexError("Plex did not report its machine identifier")
self._machine_id = mid
return self._machine_id
def _write(self, method: str, path: str) -> dict:
+150
View File
@@ -0,0 +1,150 @@
"""The HU overlay: a Hungarian Plex section written onto its English twin's rows (E30), split out of
`plex/sync.py` (E32 S2), which runs it once per paired section during the full sync.
The pairing is admin-configured (`plex_overlay_map`, ``huKey:enKey,...``). An overlay section creates
no browsable rows of its own; it only fills the `*_hu` columns of the EN rows it matches by guid.
`plex/sync.py` and this module import each other as MODULES (attribute access at call time), which is
what lets either be imported first.
"""
from collections.abc import Iterable
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex import sync as plex_sync
from app.plex.client import PlexClient
from app.plex.genres import hu_genres
log = logs.get_logger(__name__)
def overlay_map(db: Session) -> dict[str, str]:
"""The configured HU-overlay pairing: {huSectionKey: enSectionKey} from `plex_overlay_map`
('huKey:enKey,...'). Each HU section is synced as a non-browsable overlay onto its EN twin.
Malformed pairs are skipped."""
raw = sysconfig.get_str(db, "plex_overlay_map")
out: dict[str, str] = {}
for pair in raw.split(","):
hu, sep, en = pair.partition(":")
hu, en = hu.strip(), en.strip()
if sep and hu and en:
out[hu] = en
return out
def sync_overlay(
db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict
) -> None:
"""Write an overlay section's Hungarian metadata onto its EN twin's rows, matched by guid. Creates
NO browsable rows — an overlay library is never listed on its own. Only title/summary/tagline/genres
differ by language; the language-neutral people_text feeds both search vectors. A show section's
episodes are overlaid too (E30): the HU section lists every episode with its Hungarian title and
summary, and the episode rows live in the EN show library.
This is a full RECONCILE: after a COMPLETE HU listing every guid-bearing EN row's HU fields are
cleared, then the matches are repopulated — so an EN row whose HU twin vanished (or was re-guid'd)
falls back to EN instead of keeping stale Hungarian metadata. A listing cut short only adds. HU
items with no EN match are skipped."""
if lib.kind == "movie":
passes = [
(
plex_sync._TYPE_MOVIE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "movie",
PlexItem.guid.is_not(None),
),
)
]
else:
passes = [
(
plex_sync._TYPE_SHOW,
select(PlexShow).where(
PlexShow.library_id == en_lib_id, PlexShow.guid.is_not(None)
),
),
(
plex_sync._TYPE_EPISODE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "episode",
PlexItem.guid.is_not(None),
),
),
]
for item_type, rows_q in passes:
_overlay_pass(db, plex, lib, item_type, db.execute(rows_q).scalars(), stats)
db.flush()
def _overlay_pass(
db: Session,
plex: PlexClient,
lib: PlexLibrary,
item_type: int,
rows: Iterable[PlexItem] | Iterable[PlexShow],
stats: dict,
) -> None:
"""One kind of the overlay (movies, shows or episodes) — see `sync_overlay`."""
# guid -> the EN rows carrying it (a list: two editions of one film can share a guid; both overlay).
by_guid: dict[str, list[PlexItem | PlexShow]] = {}
for r in rows:
if r.guid: # the passes select guid-bearing rows only
by_guid.setdefault(r.guid, []).append(r)
# The HU listing is read first and applied after, so the clear can wait for its verdict.
hu_by_guid, complete = _read_hu_listing(plex, lib, item_type, by_guid)
episodes = item_type == plex_sync._TYPE_EPISODE
if complete:
_clear_hu(by_guid, episodes)
else:
stats["overlay_incomplete"] = stats.get("overlay_incomplete", 0) + 1
log.warning(
"the HU listing ended before the server's total; existing HU text is kept",
extra=logs.event("plex.sync.overlay_incomplete", library=lib.plex_key),
)
for g, meta in hu_by_guid.items():
targets = by_guid[g]
title_hu = (meta.get("title") or "").strip() or None
summary_hu = meta.get("summary")
tagline_hu = (meta.get("tagline") or "").strip() or None
genres_hu = hu_genres(plex_sync._tags(meta, "Genre"))
for row in targets:
row.title_hu = title_hu
row.summary_hu = summary_hu
if not episodes:
row.tagline_hu = tagline_hu
row.genres_hu = genres_hu or row.genres_hu
stats["overlaid"] = stats.get("overlaid", 0) + 1
def _read_hu_listing(
plex: PlexClient, lib: PlexLibrary, item_type: int, by_guid: dict
) -> tuple[dict[str, dict], bool]:
"""The HU section's entries for the guids the EN side carries, and whether the walk reached the
server's total. The first HU entry for a guid wins; a dup HU item can't overwrite or double-count."""
progress: dict = {}
hu_by_guid: dict[str, dict] = {}
for meta in plex_sync._paginate(plex, lib.plex_key, item_type, progress):
g = plex_sync._canonical_guid(meta)
if g and g in by_guid and g not in hu_by_guid:
hu_by_guid[g] = meta
return hu_by_guid, bool(progress.get("complete"))
def _clear_hu(by_guid: dict, episodes: bool) -> None:
"""Reconcile step: clear HU fields on all guid-bearing rows; the matches are re-set after. Only
rows that actually held HU data become dirty (SQLAlchemy skips a no-net-change UPDATE), so this is
cheap. The genres are the exception: every row gets Hungarian ones, from its HU twin or, lacking
one, from its own English list — otherwise the HU genre list mixed "Drama" (no twin) with "Dráma".
Only after a COMPLETE listing: one a Plex hiccup cut short would clear the HU text of every title
it failed to reach, until the next full sync (E32 S2, the watch reconcile's twin)."""
for group in by_guid.values():
for row in group:
row.title_hu = row.summary_hu = None
if not episodes:
row.tagline_hu = None
row.genres_hu = hu_genres(row.genres)
+11 -2
View File
@@ -245,6 +245,9 @@ def _probe_video_kind(src: Path) -> str:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
except (subprocess.SubprocessError, OSError) as exc:
# The safe kind is a real answer, but it is the answer for an 8-bit SDR source — a blind
@@ -502,6 +505,9 @@ def _probe_audio_langs(src: Path) -> list[str]:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
return [ln.strip() for ln in out.stdout.splitlines()]
except (subprocess.SubprocessError, OSError) as exc:
@@ -615,6 +621,9 @@ def _probe_first_pts(seg: Path) -> float | None:
capture_output=True,
text=True,
timeout=10,
# A non-zero exit is a failed probe (CalledProcessError, a SubprocessError → the branch
# below), not an answer: whatever it printed before refusing is not the file's truth.
check=True,
)
for line in out.stdout.splitlines():
tok = line.strip().strip(",")
@@ -736,8 +745,7 @@ def start_session(
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
# seek-restart) installed between the fallback's decision and here is never clobbered. When the
# registered session is no longer `replace_only`, we abort and return None (the caller yields to
# whoever is registered now). The INITIAL spawn passes replace_only=None and always supersedes —
# that is the seek-restart's intended last-writer-wins.
# whoever is registered now). The INITIAL spawn never gets None — see its call site.
# Software-fallback encode tuning — read only on the SW transcode path (the VAAPI path and
# remux ignore preset/crf/max_height, and remux must not touch the DB for them).
preset, crf, max_height = "veryfast", 20, 0
@@ -814,6 +822,7 @@ def start_session(
return s
s = _spawn(*_select_encode(db, item, transcode, src))
assert s is not None # replace_only=None always supersedes: a seek-restart's last-writer-wins
# Wait for the first video segment outside the lock (this blocks on ffmpeg, and we must not hold up
# other sessions). Pass the proc so a HW session that dies at VAAPI init fails fast (not after 25s).
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
+2 -1
View File
@@ -162,7 +162,8 @@ def ffmpeg_to_vtt(src: str, sub_ord: int | None) -> str:
"webvtt",
"pipe:1",
]
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S)
# check=False: the exit status is read on the next line, with ffmpeg's own stderr as the reason.
r = subprocess.run(cmd, capture_output=True, timeout=EXTRACT_TIMEOUT_S, check=False)
if r.returncode != 0 or not r.stdout:
raise PlexError(
(r.stderr or b"").decode("utf-8", "replace")[:200] or "ffmpeg subtitle convert failed"
+58 -123
View File
@@ -15,7 +15,6 @@ dead `enabled` column) is a separate follow-up. Cast + intro/credit markers are
item-detail/player path (P2), not here.
"""
from collections.abc import Iterable
from datetime import datetime, timezone
from sqlalchemy import func, select
@@ -23,7 +22,7 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.models import PlexCollection, PlexItem, PlexLibrary, PlexSeason, PlexShow
from app.plex import detail, people
from app.plex import detail, overlay, people
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.genres import hu_genres
from app.plex.lookups import by_rating_key
@@ -194,20 +193,6 @@ def _enabled_section_keys(db: Session) -> set[str] | None:
return keys or None
def _overlay_map(db: Session) -> dict[str, str]:
"""The configured HU-overlay pairing: {huSectionKey: enSectionKey} from `plex_overlay_map`
('huKey:enKey,...'). Each HU section is synced as a non-browsable overlay onto its EN twin.
Malformed pairs are skipped."""
raw = sysconfig.get_str(db, "plex_overlay_map")
out: dict[str, str] = {}
for pair in raw.split(","):
hu, sep, en = pair.partition(":")
hu, en = hu.strip(), en.strip()
if sep and hu and en:
out[hu] = en
return out
def sync(db: Session) -> dict:
"""Full reconcile of the enabled movie/show sections. Returns a small stats dict."""
if not sysconfig.get_bool(db, "plex_enabled"):
@@ -223,7 +208,7 @@ def sync(db: Session) -> dict:
"pruned": 0,
}
wanted = _enabled_section_keys(db)
overlay_map = _overlay_map(db) # {huKey: enKey}
overlay_map = overlay.overlay_map(db) # {huKey: enKey}
overlay_keys = set(overlay_map)
try:
with PlexClient(db) as plex:
@@ -267,13 +252,13 @@ def sync(db: Session) -> dict:
lib.metadata_lang = "hu"
lib.overlay_of = en_lib.id
db.flush()
_sync_overlay(db, plex, lib, en_lib.id, stats)
overlay.sync_overlay(db, plex, lib, en_lib.id, stats)
db.commit()
# Rich-fetch a bounded batch of not-yet-enriched titles' FULL cast, so the whole library
# becomes filterable over successive runs (the cheap listing above only stored the top ~3).
try:
stats["people_enriched"] = _enrich_people_batch(db, plex)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
except Exception: # best-effort; never fail the sync over a backfill
# With the traceback: a blind catch over a batch that fetches AND writes can be
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
log.exception("the Plex people-enrich batch failed")
@@ -282,7 +267,7 @@ def sync(db: Session) -> dict:
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception: # noqa: BLE001 — best-effort; never fail the sync over a backfill
except Exception: # best-effort; never fail the sync over a backfill
log.exception("the Plex media-probe batch failed")
# silent-ok: the return value IS the report. `{"skipped": …}` goes to the scheduler, which
# logs the job result, and to the admin route, which answers it to the caller.
@@ -296,7 +281,7 @@ def sync(db: Session) -> dict:
# appear in ANY item's cast/directors — drop those rows, mirroring how Plex drops orphaned metadata.
try:
stats["people_pruned"] = people.prune_orphan_people(db)
except Exception: # noqa: BLE001 — best-effort housekeeping; never fail the sync over it
except Exception: # best-effort housekeeping; never fail the sync over it
log.exception("the Plex people prune failed")
log.info("Plex sync done: %s", stats)
return stats
@@ -433,7 +418,10 @@ def _paginate(plex: PlexClient, key: str, item_type: int, progress: dict | None
if progress is not None:
# A truthy `total` we reached = a trustworthy complete listing. `total` unknown/zero-with-rows
# (a Plex that doesn't report totalSize) is treated as NOT complete, so prune stays conservative.
# `total` itself is kept for a caller to whom a server that SAID 0 is news it can trust (the
# watch reconcile); an empty mirror or overlay must not read it so (E32 S2).
progress["complete"] = bool(total) and start >= total
progress["total"] = total
# The section listing exposes at most the top ~20 cast (Plex's own cap); a stored cast longer than this
@@ -661,98 +649,6 @@ def _sync_shows(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) ->
_prune_stale(db, shows, shows_seen, stats, complete=shows_progress.get("complete", False))
def _sync_overlay(
db: Session, plex: PlexClient, lib: PlexLibrary, en_lib_id: int, stats: dict
) -> None:
"""Write an overlay section's Hungarian metadata onto its EN twin's rows, matched by guid. Creates
NO browsable rows — an overlay library is never listed on its own. Only title/summary/tagline/genres
differ by language; the language-neutral people_text feeds both search vectors. A show section's
episodes are overlaid too (E30): the HU section lists every episode with its Hungarian title and
summary, and the episode rows live in the EN show library.
This is a full RECONCILE: every guid-bearing EN row's HU fields are cleared first, then the HU
section repopulates the matches — so an EN row whose HU twin vanished (or was re-guid'd) falls back
to EN instead of keeping stale Hungarian metadata. HU items with no EN match are skipped."""
if lib.kind == "movie":
passes = [
(
_TYPE_MOVIE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "movie",
PlexItem.guid.is_not(None),
),
)
]
else:
passes = [
(
_TYPE_SHOW,
select(PlexShow).where(
PlexShow.library_id == en_lib_id, PlexShow.guid.is_not(None)
),
),
(
_TYPE_EPISODE,
select(PlexItem).where(
PlexItem.library_id == en_lib_id,
PlexItem.kind == "episode",
PlexItem.guid.is_not(None),
),
),
]
for item_type, rows_q in passes:
_overlay_pass(db, plex, lib, item_type, db.execute(rows_q).scalars(), stats)
db.flush()
def _overlay_pass(
db: Session,
plex: PlexClient,
lib: PlexLibrary,
item_type: int,
rows: Iterable[PlexItem] | Iterable[PlexShow],
stats: dict,
) -> None:
"""One kind of the overlay (movies, shows or episodes) — see `_sync_overlay`."""
# guid -> the EN rows carrying it (a list: two editions of one film can share a guid; both overlay).
by_guid: dict[str, list[PlexItem | PlexShow]] = {}
for r in rows:
if r.guid: # the passes select guid-bearing rows only
by_guid.setdefault(r.guid, []).append(r)
# Reconcile step: clear HU fields on all guid-bearing rows. Only rows that actually held HU data
# become dirty (SQLAlchemy skips a no-net-change UPDATE), so this is cheap; matches are re-set below.
# The genres are the exception: every row gets Hungarian ones, from its HU twin or, lacking one, from
# its own English list — otherwise the HU genre list mixed "Drama" (no twin) with "Dráma".
episodes = item_type == _TYPE_EPISODE
for group in by_guid.values():
for row in group:
row.title_hu = row.summary_hu = None
if not episodes:
row.tagline_hu = None
row.genres_hu = hu_genres(row.genres)
seen: set[str] = (
set()
) # first HU entry for a guid wins; a dup HU item can't overwrite or double-count
for meta in _paginate(plex, lib.plex_key, item_type):
g = _canonical_guid(meta)
targets = by_guid.get(g) if g else None
if not targets or g in seen:
continue
seen.add(g)
title_hu = (meta.get("title") or "").strip() or None
summary_hu = meta.get("summary")
tagline_hu = (meta.get("tagline") or "").strip() or None
genres_hu = hu_genres(_tags(meta, "Genre"))
for row in targets:
row.title_hu = title_hu
row.summary_hu = summary_hu
if not episodes:
row.tagline_hu = tagline_hu
row.genres_hu = genres_hu or row.genres_hu
stats["overlaid"] = stats.get("overlaid", 0) + 1
def _upsert_collection(
db: Session, existing: PlexCollection | None, rk: str, lib_id: int, meta: dict
) -> PlexCollection:
@@ -778,9 +674,11 @@ def resync_collection(db: Session, plex: PlexClient, lib: PlexLibrary, rk: str)
meta = plex.metadata(rk) or {}
col = _upsert_collection(db, by_rating_key(db, PlexCollection, rk), rk, lib.id, meta)
Model = PlexItem if lib.kind == "movie" else PlexShow
new_members = {
str(c.get("ratingKey")) for c in plex.collection_children(rk) if c.get("ratingKey")
}
children = _believed_children(plex, rk, meta)
if children is None:
db.flush()
return col
new_members = {str(c.get("ratingKey")) for c in children if c.get("ratingKey")}
current = db.execute(select(Model).where(Model.collection_keys.contains([rk]))).scalars().all()
for row in current: # drop rk from rows that are no longer members
if row.rating_key not in new_members:
@@ -793,6 +691,22 @@ def resync_collection(db: Session, plex: PlexClient, lib: PlexLibrary, rk: str)
return col
def _believed_children(plex: PlexClient, rk: str, meta: dict) -> list[dict] | None:
"""A collection's members, or None when the answer is not believed: no members from a collection
whose own listing counts some (`childCount`) is a glitch, not a collection emptied — the same
empty-answer rule as the collection listing's (E32 review). Rebuilding on it showed the
collection empty until the next sync."""
children = plex.collection_children(rk)
count = str(meta.get("childCount") or "0")
if children or not count.isdigit() or int(count) == 0:
return children
log.warning(
"Plex listed no members for a non-empty collection; keeping its membership",
extra=logs.event("plex.sync.collection_children_empty", collection=rk),
)
return None
def delete_collection_local(db: Session, lib: PlexLibrary, rk: str) -> None:
"""Local cleanup after a collection is deleted from Plex: strip rk from members + drop the row."""
Model = PlexItem if lib.kind == "movie" else PlexShow
@@ -804,6 +718,19 @@ def delete_collection_local(db: Session, lib: PlexLibrary, rk: str) -> None:
db.flush()
def _rebuild_membership(
db: Session, lib: PlexLibrary, membership: dict[str, list[str]], kept: set[str]
) -> None:
"""Rebuild collection_keys on every member row of this library (clear + set), carrying over the
keys of a collection whose empty answer was not believed (`_believed_children`)."""
for Model in (PlexItem, PlexShow):
for row in db.execute(select(Model).where(Model.library_id == lib.id)).scalars():
held = [k for k in (row.collection_keys or []) if k in kept]
keys = [*membership.get(row.rating_key, []), *held]
row.collection_keys = list(dict.fromkeys(keys)) or None
db.flush()
def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: dict) -> None:
"""Mirror a library's collections + membership. Reads never touch Plex afterwards. Membership is
written as `collection_keys` on member movies (plex_items) / shows (plex_shows). A full reconcile:
@@ -815,7 +742,17 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
select(PlexCollection).where(PlexCollection.library_id == lib.id)
).scalars()
}
if not cols and existing:
# The catalog's own rule (see the module doc): an empty listing while we hold rows is a Plex
# glitch, not a library emptied of collections. Reconciling on it dropped every collection
# (its user-set `editable` flag with it) and every member's keys (E32 S2).
log.warning(
"Plex listed no collections for a library that has some; keeping them this run",
extra=logs.event("plex.sync.collections_empty", library=lib.plex_key),
)
return
seen: set[str] = set()
kept: set[str] = set() # collections whose empty children answer is not believed
membership: dict[str, list[str]] = {} # member rating_key -> [collection rating_key, ...]
for meta in cols:
rk = str(meta.get("ratingKey") or "")
@@ -823,7 +760,10 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
continue
_upsert_collection(db, existing.get(rk), rk, lib.id, meta)
seen.add(rk)
for child in plex.collection_children(rk):
children = _believed_children(plex, rk, meta)
if children is None:
kept.add(rk)
for child in children or []:
crk = str(child.get("ratingKey") or "")
if crk:
membership.setdefault(crk, []).append(rk)
@@ -832,9 +772,4 @@ def _sync_collections(db: Session, plex: PlexClient, lib: PlexLibrary, stats: di
for rk, col in existing.items():
if rk not in seen:
db.delete(col)
# Rebuild collection_keys on every member row of this library (clear + set).
for row in db.execute(select(PlexItem).where(PlexItem.library_id == lib.id)).scalars():
row.collection_keys = membership.get(row.rating_key) or None
for row in db.execute(select(PlexShow).where(PlexShow.library_id == lib.id)).scalars():
row.collection_keys = membership.get(row.rating_key) or None
db.flush()
_rebuild_membership(db, lib, membership, kept)
+69 -20
View File
@@ -29,7 +29,7 @@ from sqlalchemy.orm import Session
from app import logs, sysconfig
from app.db import SessionLocal
from app.models import PlexItem, PlexLink, PlexState
from app.models import PlexItem, PlexLibrary, PlexLink, PlexState
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
from app.plex.sync import (
_TYPE_EPISODE,
@@ -72,13 +72,27 @@ def _rk_to_id(db: Session) -> dict[str, int]:
def _scan_plex_states(
db: Session, plex: PlexClient
) -> tuple[dict[int, tuple[str, int, datetime | None, datetime | None]], int]:
) -> tuple[dict[int, tuple[str, int, datetime | None, datetime | None]], int, set[str], set[str]]:
"""Scan the enabled movie/show sections and return ``{item_id: (status, pos, watched_at,
prog_at)}`` for every mirrored leaf Plex holds a watch signal for, plus the scanned-leaf count.
Shared by the one-time import and the full reconcile (both need Plex's whole current picture)."""
prog_at)}`` for every mirrored leaf Plex holds a watch signal for, the scanned-leaf count, the
keys of the sections SETTLED — walked to the server's total (or said empty by a server we mirror
nothing from) — and the keys of every section walked.
Shared by the one-time import and the full reconcile (both need Plex's whole current picture).
The settled keys matter to the reconcile alone: it reads "absent from the scan" as "Plex
un-watched it", which a walk cut short by a Plex hiccup does not mean (see `_paginate`). Settled
is decided PER SECTION: one section that never settles (a library emptied on Plex while its
mirrored items are kept) must not stop un-watches syncing in all the others (E32 review)."""
item_id_by_rk = _rk_to_id(db)
wanted = _enabled_section_keys(db)
# The sections we mirror something from: a total of 0 from one of those contradicts the mirror.
mirrored = set(
db.execute(
select(PlexLibrary.plex_key).join(PlexItem, PlexItem.library_id == PlexLibrary.id)
).scalars()
)
scanned = 0
settled: set[str] = set()
walked: set[str] = set()
out: dict[int, tuple[str, int, datetime | None, datetime | None]] = {}
for s in plex.sections():
stype = s.get("type")
@@ -88,7 +102,10 @@ def _scan_plex_states(
if wanted is not None and key not in wanted:
continue
item_type = _TYPE_MOVIE if stype == "movie" else _TYPE_EPISODE
for meta in _paginate(plex, key, item_type):
walked.add(key)
progress: dict = {}
before = scanned
for meta in _paginate(plex, key, item_type, progress):
scanned += 1
item_id = item_id_by_rk.get(str(meta.get("ratingKey") or ""))
if item_id is None:
@@ -98,7 +115,15 @@ def _scan_plex_states(
)
if res is not None:
out[item_id] = res
return out, scanned
# A section the server SAYS is empty (total 0, nothing listed) settles too. Believed only for
# a section we mirror nothing from: one whose items we hold answering 0 is a Plex that lost
# the library for a moment, not one that un-watched all of it (E32 review) — so it is left
# unsettled, and its rows are not cleared.
listed = scanned > before
said_empty = progress.get("total") == 0 and not listed and key not in mirrored
if (progress.get("complete") and listed) or said_empty:
settled.add(key)
return out, scanned, settled, walked
def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
@@ -111,7 +136,8 @@ def import_owner_watch_state(db: Session, link: PlexLink) -> dict:
stats = {"watched": 0, "in_progress": 0, "scanned": 0}
try:
with PlexClient(db) as plex:
scanned_states, stats["scanned"] = _scan_plex_states(db, plex)
# A partial scan is still safe here: the import only ever ADDS what Plex reports.
scanned_states, stats["scanned"], _settled, _walked = _scan_plex_states(db, plex)
# One row per item (keyed by item_id → inherently de-duped) then bulk-UPSERT. An upsert is
# idempotent AND concurrency-safe: a repeated/overlapping enable (the toggle firing twice
# while the multi-second import runs) can't raise a duplicate-key error — it re-updates.
@@ -590,6 +616,28 @@ def run_plex_watch_sync(db: Session) -> dict:
return total
def _clear_unwatched(db: Session, user_id: int, plex_states: dict, settled: set[str]) -> int:
"""Clear the user's rows Plex no longer has a record for — the mirrored ones (Plex un-watched
them); hidden and dirty rows stay (a dirty one is re-pushed by `_repush_dirty`). Only rows of the
sections the scan SETTLED: a walk a Plex hiccup cut short leaves out rows it merely failed to
reach, and clearing those un-watched them here until the next complete run (E32 S2); an item of
a library the allowlist leaves out is absent from every scan, and was cleared on every run (E32
review)."""
n = 0
rows = db.execute(
select(PlexState)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.join(PlexLibrary, PlexLibrary.id == PlexItem.library_id)
.where(PlexState.user_id == user_id, PlexLibrary.plex_key.in_(settled))
)
for st in rows.scalars().all():
if st.item_id in plex_states or st.status == "hidden" or not st.synced_to_plex:
continue
db.delete(st)
n += 1
return n
def run_plex_watch_reconcile(db: Session) -> dict:
"""Scheduler job (default daily) — the authoritative full reconcile. Rescans every section and,
per item, uses `synced_to_plex` to settle what the incremental feed can't:
@@ -608,7 +656,7 @@ def run_plex_watch_reconcile(db: Session) -> dict:
try:
with PlexClient(db) as plex:
_resolve_account_id(db, plex, link)
plex_states, scanned = _scan_plex_states(db, plex)
plex_states, scanned, settled, walked = _scan_plex_states(db, plex)
total["scanned"] += scanned
for item_id, (status, pos, watched_at, prog_at) in plex_states.items():
@@ -616,18 +664,19 @@ def run_plex_watch_reconcile(db: Session) -> dict:
if ch:
total["applied"] += 1
# Siftlode rows Plex no longer has a record for: clear the ones that were mirrored
# (Plex un-watched them); leave hidden + dirty (dirty gets re-pushed just below).
for st in (
db.execute(select(PlexState).where(PlexState.user_id == link.user_id))
.scalars()
.all()
):
if st.item_id in plex_states or st.status == "hidden":
continue
if st.synced_to_plex:
db.delete(st)
total["cleared"] += 1
total["cleared"] += _clear_unwatched(db, link.user_id, plex_states, settled)
if unsettled := sorted(walked - settled):
total["incomplete"] = total.get("incomplete", 0) + 1
log.warning(
"Plex sections %s were cut short or answered empty; their rows are not cleared"
" this run",
", ".join(unsettled),
extra=logs.event(
"plex.watch.reconcile_incomplete",
user_id=link.user_id,
sections=unsettled,
),
)
total["repushed"] += _repush_dirty(db, plex, link)
link.last_watch_sync_at = datetime.now(timezone.utc)
+4 -2
View File
@@ -8,7 +8,7 @@ its own thread/context) never cross wires. When no sink is bound (e.g. a manual
"""
import contextvars
from typing import Callable, Optional
from typing import Callable, Literal, Optional
Sink = Callable[[int, Optional[int], Optional[str]], None]
@@ -48,7 +48,9 @@ class bind:
self._token = _sink.set(self._sink)
return self
def __exit__(self, *exc) -> bool:
# Literal[False], not bool: this never swallows the job's exception, and saying so is what lets
# a type checker know a name assigned inside `with bind(...)` is bound after it.
def __exit__(self, *exc) -> Literal[False]:
if self._token is not None:
_sink.reset(self._token)
return False
+18 -2
View File
@@ -7,11 +7,13 @@ always be allowed; expensive backfill must yield to the remaining budget.
import contextlib
import contextvars
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from typing import Any, cast
from zoneinfo import ZoneInfo
from sqlalchemy import func, select
from sqlalchemy import delete, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import sysconfig
@@ -96,6 +98,20 @@ def pacific_today():
return datetime.now(_PACIFIC).date()
def gc(db: Session, retention_days: int) -> int:
"""Delete quota-history rows older than `retention_days` (<= 0 keeps them forever) and return how
many went. Flushes only; the cleanup job commits. Today's rows are never in reach: the per-user
search cap counts them, and the floor of two days outlasts any Pacific day (25 h at most)."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=max(2, retention_days))
# A DELETE answers with a CursorResult; `execute` is typed as the general Result, without rowcount.
result = cast(
CursorResult[Any], db.execute(delete(QuotaEvent).where(QuotaEvent.created_at < cutoff))
)
return result.rowcount or 0
def pacific_day_start_utc() -> datetime:
"""UTC instant of the current Pacific day's midnight — aligns 'today' with the budget reset."""
start = datetime.now(_PACIFIC).replace(hour=0, minute=0, second=0, microsecond=0)
+6 -3
View File
@@ -60,14 +60,17 @@ def owned_or_404(
return obj
def youtube_refusal(exc: Exception) -> HTTPException:
def youtube_refusal(exc: Exception, *, playlist: bool = False) -> HTTPException:
"""The answer to an action YouTube refused: a 422 whose reason the UI translates
(`errors.youtube.<reason>`). One definition, because the routes had grown three shapes of it —
a bare English string quoting the upstream message, a playlist-scoped reason, and nothing at
all (a raw 500 from the manual syncs). The upstream failure is already logged where it
happened (`youtube.api.error`). A failure that is not the Data API's (the search scraper's) is
YouTube being unavailable to us — the retryable answer."""
reason = refusal_reason(exc) if isinstance(exc, YouTubeError) else "unavailable"
YouTube being unavailable to us — the retryable answer. `playlist`: the route acts on one
playlist the user picked, so a missing playlist is THAT one, gone (`refusal_reason`)."""
reason = (
refusal_reason(exc, playlist=playlist) if isinstance(exc, YouTubeError) else "unavailable"
)
return HTTPException(status_code=422, detail={"code": "youtube", "reason": reason})
+3 -3
View File
@@ -436,7 +436,7 @@ def explore_channel(
channel = db.get(Channel, channel_id)
result = explore_ingest_page(db, yt, channel, page_token)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
return {
"ok": True,
"channel_id": channel_id,
@@ -597,7 +597,7 @@ def subscribe(
if "subscriptionDuplicate" in exc.reason.split(","):
yt_sub_id = ""
else:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
# Enrich a stub channel (title/thumbnail/subscriber count) so it shows up properly right
# away, on BOTH the normal and the desync path — a resurfaced stub still needs metadata to
# render. Keep the client open (we're still inside the `with`); a metadata failure here
@@ -658,7 +658,7 @@ def unsubscribe(
except YouTubeError as exc:
# 422 (a real, explainable error → speaking modal), not 502 which the client treats as
# a transient "connection lost" gateway blip.
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
db.delete(sub)
db.commit()
return {"ok": True, "unsubscribed": channel_id}
+1 -1
View File
@@ -170,7 +170,7 @@ def add_pin(
# between `resolve_ref` confirming it and this INSERT (a Plex sync dropping a removed
# item), which trips the FK. Semantically that is the same "no such thing" the pre-check
# returns above — re-raising handed the user a 500 they could do nothing with.
raise not_found("item")
raise not_found("item") from None
return {"ok": True, "id": winner.id, "created": False}
return {"ok": True, "id": pin.id, "created": True}
+5 -5
View File
@@ -322,7 +322,7 @@ def set_credential(
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)})
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
audit.record(
db,
@@ -407,7 +407,7 @@ def enqueue_download(
try:
job = service.enqueue(db, user.id, source_kind, source_ref, spec, profile_id, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
@@ -520,7 +520,7 @@ def upload_download(
display_name=upload.display_name,
)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
return _serialize_job(db, job)
@@ -547,9 +547,9 @@ def enqueue_edit(
try:
job = service.enqueue_edit(db, user.id, source_job, spec, display_name)
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=_quota_detail(e))
raise HTTPException(status_code=422, detail=_quota_detail(e)) from e
except service.EditError as e:
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason})
raise HTTPException(status_code=400, detail={"code": "edit", "reason": e.reason}) from e
return _serialize_job(db, job)
+2 -2
View File
@@ -133,7 +133,7 @@ def get_feed(
# Expose each key column on the row so we can build the next cursor from the last item.
query = query.add_columns(*[k["expr"].label(f"_k{i}") for i, k in enumerate(keys)])
if vals is not None:
for k, v in zip(keys, vals):
for k, v in zip(keys, vals, strict=True):
k["value"] = v
if ranked and window is not None:
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
@@ -431,7 +431,7 @@ def get_video_detail(
):
items = yt.get_videos([video_id])
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
if not items:
raise not_found("video")
+3
View File
@@ -148,6 +148,9 @@ def get_me(
"google_enabled": google_enabled(db),
# Whether the optional Plex module is enabled (UI shows the Plex feed source when so).
"plex_enabled": sysconfig.get_bool(db, "plex_enabled"),
# Which live-search source runs, so the search button can say what a search costs: "scrape"
# spends no API quota, "api" spends 100 units a page.
"search_source": sysconfig.get_str(db, "search_source"),
"can_read": has_read_scope(user),
"can_write": has_write_scope(user),
"pending_invites": pending_invites,
+7 -7
View File
@@ -342,7 +342,7 @@ def sync_youtube(
return {"ok": True, **pull_user_playlists(db, user)}
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
@router.post("/{playlist_id}/revert-youtube")
@@ -369,10 +369,10 @@ def revert_youtube(
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PULL):
return {"ok": True, **repull_playlist(db, user, pl)}
except PlaylistGone:
raise not_found("playlist")
raise not_found("playlist") from None
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
def _pushable(db: Session, user: User, playlist_id: int) -> Playlist:
@@ -398,7 +398,7 @@ def push_plan(
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_merge(db, user, pl)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
plan["remaining_today"] = quota.remaining_today(db)
plan["affordable"] = plan["units_estimate"] <= plan["remaining_today"]
return plan
@@ -436,10 +436,10 @@ def push(
raise _refused(429, "quota_exceeded")
return {"ok": True, **merge_playlist(db, user, pl, live=live)}
except PlaylistGone:
raise not_found("playlist")
raise not_found("playlist") from None
except YouTubeError as exc:
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
@router.get("/{playlist_id}")
@@ -539,7 +539,7 @@ def delete_playlist(
with YouTubeClient(db, user) as yt:
yt.delete_playlist(pl.yt_playlist_id)
except YouTubeError as exc:
raise youtube_refusal(exc)
raise youtube_refusal(exc, playlist=True) from exc
# The playlist first, as every item write: the cascade deletes the items before the list row.
lock_playlists(db, pl.id)
db.delete(pl) # items cascade
+4 -4
View File
@@ -428,7 +428,7 @@ def create_collection(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -450,7 +450,7 @@ def collection_add_item(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -472,7 +472,7 @@ def collection_remove_item(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
@@ -502,7 +502,7 @@ def collection_rename(
db.commit()
except (PlexError, PlexNotConfigured) as e:
db.rollback()
raise _write_or_pending_error(wrote, e)
raise _write_or_pending_error(wrote, e) from e
return _collection_card(col)
+12 -4
View File
@@ -124,8 +124,14 @@ def _recently_added(
# share one), so the merge doesn't depend on object identity staying alive.
top_movies = [o for _, kind, o in top if kind == "movie"]
top_shows = [o for _, kind, o in top if kind == "show"]
mcards = {m.rating_key: c for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang))}
scards = {s.rating_key: c for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang))}
mcards = {
m.rating_key: c
for m, c in zip(top_movies, _movie_cards(db, uid, top_movies, lang), strict=True)
}
scards = {
s.rating_key: c
for s, c in zip(top_shows, _show_cards(db, uid, top_shows, lang), strict=True)
}
return [mcards[o.rating_key] if kind == "movie" else scards[o.rating_key] for _, kind, o in top]
@@ -351,10 +357,12 @@ def _because_you_watched(
ordered_shows = [shows[rk] for rk in rks if rk in shows]
# rating_key is globally unique in Plex, so it keys the serialized cards back safely (no id()).
mcards = {
m.rating_key: c for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang))
m.rating_key: c
for m, c in zip(ordered_movies, _movie_cards(db, uid, ordered_movies, lang), strict=True)
}
scards = {
s.rating_key: c for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang))
s.rating_key: c
for s, c in zip(ordered_shows, _show_cards(db, uid, ordered_shows, lang), strict=True)
}
items: list[dict] = []
for rk in rks:
+24 -3
View File
@@ -1,5 +1,5 @@
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from pydantic import BaseModel, field_validator
from sqlalchemy import func, select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -53,11 +53,30 @@ def list_views(
return [_serialize(v) for v in rows]
# A feed view's tag pick is read by more than the feed: the autotag job keeps every tag a view names
# (sync/autotag.py). Held here to the shape the feed sends — a list of tag ids, at most this many — so
# a hand-made body (`"tags": 5`, or 60k ids) can't break that job for every user (E32 review).
_MAX_VIEW_TAGS = 500
def _check_filters(filters: dict | None) -> dict | None:
tags = (filters or {}).get("tags")
if tags is None:
return filters
if not isinstance(tags, list) or len(tags) > _MAX_VIEW_TAGS:
raise ValueError(f"filters.tags must be a list of at most {_MAX_VIEW_TAGS} tag ids")
if any(type(t) is not int for t in tags): # `type`, not isinstance: True is not a tag id
raise ValueError("filters.tags must hold tag ids")
return filters
class CreateViewIn(BaseModel):
name: str
filters: dict
source: str = "feed"
_filters = field_validator("filters")(_check_filters)
@router.post("")
def create_view(
@@ -87,7 +106,7 @@ def create_view(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "view_name_taken", name=name)
raise refused(409, "view_name_taken", name=name) from None
return _serialize(view)
@@ -98,6 +117,8 @@ class UpdateViewIn(BaseModel):
filters: dict | None = None
is_default: bool | None = None
_filters = field_validator("filters")(_check_filters)
@router.patch("/{view_id}")
def update_view(
@@ -135,7 +156,7 @@ def update_view(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "view_name_taken", name=view.name)
raise refused(409, "view_name_taken", name=view.name) from None
return _serialize(view)
+1 -1
View File
@@ -74,7 +74,7 @@ def run_job_now(
try:
result = trigger_job(job_id, actor_id=user.id)
except KeyError:
raise not_found("job")
raise not_found("job") from None
if result == "already_running":
raise refused(409, "job_running")
return {"ok": True, "id": job_id, "started": True}
+1 -1
View File
@@ -331,7 +331,7 @@ def search_youtube(
# bare 500 hidden behind the FE's generic "search failed" (same rationale as B-core-1).
try:
collected.extend(_ingest_candidates(db, yt, user, candidates))
except Exception as exc: # noqa: BLE001 — any ingest failure, not just YouTubeError
except Exception as exc: # any ingest failure, not just YouTubeError
db.rollback()
# The term is NOT in this line. A search string is exactly what the privacy
# policy keeps out of the log unless the user turned Teljes/TRACE on for
+1 -1
View File
@@ -39,7 +39,7 @@ def sync_subscriptions(user: User = Depends(require_human), db: Session = Depend
# The onboarding wizard's first call: an unguarded failure here was a raw 500 and the
# blocking "Server error" dialog, with nothing the user could act on (E9 F1).
db.rollback()
raise youtube_refusal(exc)
raise youtube_refusal(exc) from exc
result["quota_used_estimate"] = m.used
result["quota_remaining_today"] = m.remaining
return {"ok": True, **result}
+2 -2
View File
@@ -75,7 +75,7 @@ def create_tag(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "tag_name_taken", name=name)
raise refused(409, "tag_name_taken", name=name) from None
return {
"ok": True,
"id": tag.id,
@@ -113,7 +113,7 @@ def update_tag(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "tag_name_taken", name=tag.name)
raise refused(409, "tag_name_taken", name=tag.name) from None
return {
"ok": True,
"id": tag.id,
+6 -3
View File
@@ -425,12 +425,15 @@ def _people_enrich_job() -> None:
def _audit_gc_job() -> None:
# Prune audit-log rows older than the admin-set retention (audit_retention_days; 0 = keep
# forever). Pure DB, no quota.
# The history cleanup: audit-log rows past audit_retention_days and YouTube quota history past
# quota_retention_days (0 = keep forever, each). Pure DB, no quota.
def work(db):
from app import sysconfig
return {"reaped": audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))}
reaped = audit.gc(db, sysconfig.get_int(db, "audit_retention_days"))
quota_reaped = quota.gc(db, sysconfig.get_int(db, "quota_retention_days"))
db.commit()
return {"reaped": reaped, "quota_reaped": quota_reaped}
_job("audit_gc", work)
+29 -12
View File
@@ -12,7 +12,7 @@ from sqlalchemy import and_, exists, func, select
from sqlalchemy.orm import Session
from app import logs, progress, sysconfig
from app.models import Channel, ChannelTag, Tag, Video
from app.models import Channel, ChannelTag, SavedView, Tag, Video
log = logs.get_logger(__name__)
@@ -288,18 +288,35 @@ def run_autotag_all(db: Session, only_missing: bool = False) -> dict:
return {"channels_tagged": tagged, "orphan_tags_removed": removed}
def _view_tag_ids(filters: object) -> list[int]:
"""The tag ids a stored view's filters name; anything else in that slot names none."""
tags = filters.get("tags") if isinstance(filters, dict) else None
if not isinstance(tags, list):
return []
return [t for t in tags if type(t) is int] # `type`, not isinstance: True is not a tag id
def _cleanup_orphan_system_tags(db: Session) -> int:
"""Delete system tags that no longer link to any channel (e.g. after a mapping change)."""
orphans = (
db.execute(
select(Tag).where(
Tag.user_id.is_(None),
~exists().where(ChannelTag.tag_id == Tag.id),
)
)
.scalars()
.all()
)
"""Delete system tags that no longer link to any channel (e.g. after a mapping change) — but not
one a saved feed view filters on. Views hold tag IDS, and a deleted tag comes back under a new id
when a channel earns it again, so the view's filter would point at nothing; the feed drops an
unknown id, and the view showed the whole feed under its old name (E32 S2).
The views are read defensively, and the kept ids are filtered here rather than in SQL: a view
stored before its route checked the shape (or written around it) must not raise for everyone's
job, nor push a `NOT IN` past the driver's bind-parameter limit (E32 review)."""
in_views = {
t
for (filters,) in db.execute(select(SavedView.filters).where(SavedView.source == "feed"))
for t in _view_tag_ids(filters)
}
orphans = [
tag
for tag in db.execute(
select(Tag).where(Tag.user_id.is_(None), ~exists().where(ChannelTag.tag_id == Tag.id))
).scalars()
if tag.id not in in_views
]
for tag in orphans:
db.delete(tag)
db.commit()
+10 -2
View File
@@ -25,7 +25,7 @@ from app.models import (
)
from app.sync.videos import insert_stubs, stub_from_playlist_item, apply_video_details
from app.utils import now_utc as _now
from app.youtube.client import YouTubeClient
from app.youtube.client import YouTubeClient, YouTubeError, is_playlist_gone
log = logs.get_logger(__name__)
@@ -81,7 +81,15 @@ def explore_ingest_page(
the organic deep backfill) is never touched."""
if not channel.uploads_playlist_id:
return {"ingested": 0, "next_page_token": None}
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
try:
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
except YouTubeError as exc:
# A channel with no uploads has an uploads id that YouTube answers as not found: an empty
# back-catalogue, not a failure (and not a "playlist gone" — the user picked a channel).
if not is_playlist_gone(exc):
raise
log.info("explore ingest channel=%s has no uploads playlist", channel.id)
return {"ingested": 0, "next_page_token": None}
stubs: list[dict] = []
for item in data.get("items", []):
stub = stub_from_playlist_item(item, channel.id)
+15 -9
View File
@@ -20,9 +20,9 @@ Quota: phase 1 costs ~(flagged / 50) units, phase 2 ~(batch / 50). Both respect
backfill reserve as the other jobs so interactive syncs never starve.
"""
from datetime import timedelta
from datetime import datetime, timedelta
from sqlalchemy import select
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app import logs, progress, quota, sysconfig
@@ -129,14 +129,14 @@ def _notify_removed(db: Session, impact: dict[int, list[dict]]) -> int:
return len(impact)
def _recheck_flagged(db: Session, yt: YouTubeClient) -> dict:
"""Phase 1: recover or hard-delete videos already flagged unavailable."""
def _recheck_flagged(db: Session, yt: YouTubeClient, now: datetime) -> dict:
"""Phase 1: recover or hard-delete videos already flagged unavailable. `now` is the run's:
phase 2 skips a video stamped with it, so one recovered here is not fetched again."""
flagged = db.execute(select(Video).where(Video.unavailable_since.is_not(None))).scalars().all()
if not flagged:
return {"checked": 0, "recovered": 0, "deleted": 0, "notified": 0}
progress.report(0, len(flagged), "recheck")
items = {it["id"]: it for it in yt.get_videos([v.id for v in flagged])}
now = _now()
recovered = 0
to_delete: list[Video] = []
for video in flagged:
@@ -173,7 +173,7 @@ def _recheck_flagged(db: Session, yt: YouTubeClient) -> dict:
}
def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
def _revalidate_rolling(db: Session, yt: YouTubeClient, now: datetime) -> dict:
"""Phase 2: re-check the least-recently-checked currently-available videos and flag any
that have become unplayable (hidden immediately; deleted later if they don't recover)."""
batch = sysconfig.get_int(
@@ -181,7 +181,6 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
) # admin override, else config default
flagged_new = 0
checked = 0
now = _now()
# Process in videos.list-sized pages so we commit incrementally and stop on low quota.
while checked < batch:
if quota.backfill_reserve_reached():
@@ -190,6 +189,10 @@ def _revalidate_rolling(db: Session, yt: YouTubeClient) -> dict:
db.execute(
select(Video)
.where(Video.unavailable_since.is_(None))
# Not one this run already checked: with fewer available videos than the batch, the
# loop used to wrap around and re-fetch the same ones until `checked` reached the
# batch, spending quota on answers it already had (E32 S2).
.where(or_(Video.last_checked_at.is_(None), Video.last_checked_at < now))
# NULLS FIRST = never-checked first, then least-recently-checked; equivalent to the old
# `(last_checked_at IS NULL) DESC, last_checked_at ASC` but in the exact shape of the
# partial ix_videos_recheck_order (migration 0069, PERF-18) so it rides that index.
@@ -220,10 +223,13 @@ def run_maintenance(db: Session) -> dict:
user = get_service_user(db)
if user is None:
return {"reason": "no service user"}
# One `now` for the whole run: each phase taking its own let a video phase 1 had just recovered
# (stamped with phase 1's) pass phase 2's "not checked this run" test and be fetched twice.
now = _now()
with YouTubeClient(db, user) as yt:
phase1 = _recheck_flagged(db, yt)
phase1 = _recheck_flagged(db, yt, now)
phase2 = (
_revalidate_rolling(db, yt)
_revalidate_rolling(db, yt, now)
if not quota.backfill_reserve_reached()
else {"checked": 0, "flagged": 0, "skipped": "low quota"}
)
+89
View File
@@ -0,0 +1,89 @@
"""The 3-way merge of a YouTube-linked playlist, as a pure function: what each side gains and loses
so neither loses an item, the merged order, and the reorder moves YouTube needs to reach it. No I/O —
`sync/playlists.py` reads both sides, calls `merge_plan`, and writes the outcome."""
from dataclasses import dataclass
def _reorder_moves(current: list[str], desired: list[str]) -> int:
"""Count the playlistItems.update moves an insertion-sort would need to turn
`current` order into `desired` order (both already the same set of video ids).
Items already in place are skipped; this matches what apply_push actually executes."""
model = list(current)
moves = 0
for i, want in enumerate(desired):
if i < len(model) and model[i] == want:
continue
j = model.index(want, i)
model.pop(j)
model.insert(i, want)
moves += 1
return moves
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
+38 -94
View File
@@ -1,6 +1,7 @@
"""YouTube playlist sync, both directions. Read: mirror each user's own YouTube playlists
into local `source='youtube'` playlists (kept fresh by the scheduler). Write-back: two-way sync
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope.
local edits with YouTube (`plan_merge` / `merge_playlist`), behind the optional write scope; the
pure 3-way merge they run is `sync/playlist_merge.py`.
YouTube's special Watch Later / History playlists are not exposed by the Data API and are
never synced. Videos in a playlist that aren't in our shared catalog yet are fetched and
@@ -10,7 +11,7 @@ import hashlib
import json
from collections.abc import Callable, Iterator
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TypeVar
@@ -21,9 +22,11 @@ from sqlalchemy.pool import NullPool
from app import logs, progress, quota
from app.auth import has_read_scope
from app.models import Channel, Playlist, PlaylistItem, User, Video
from app.sync.playlist_merge import merge_plan
from app.sync.runner import token_users
from app.sync.subscriptions import blank_listing, refuse_blank_listing, vanished
from app.sync.videos import apply_video_details, parse_dt
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.client import YouTubeClient, YouTubeError, is_playlist_gone
log = logs.get_logger(__name__)
# Cost of a single YouTube write op (playlists/playlistItems insert/update/delete).
@@ -316,6 +319,24 @@ def user_sync_claim(db: Session, user_id: int) -> AbstractContextManager[bool]:
return _advisory_claim(db, _USER_SYNC_LOCK_NS, user_id)
def _mirror_listing(yt: YouTubeClient, ytid: str, count: object, pl_id: int) -> list[str] | None:
"""A mirror's video ids from YouTube, or None when there is nothing to believe: the list was
deleted between the pull's listing and this read (the next pull drops the mirror as vanished —
not a reason to fail every other list, E32 review round 2), or YouTube listed no items for a
list it counts as non-empty (`blank_listing`)."""
try:
ids = list(yt.iter_my_playlist_video_ids(ytid))
except YouTubeError as exc:
if not is_playlist_gone(exc):
raise
log.info("YouTube playlist of %s vanished mid-pull; skipped", pl_id)
return None
if blank_listing(ids, count):
log.warning("YouTube listed no items for non-empty playlist %s; kept", pl_id)
return None
return ids
def pull_user_playlists(db: Session, user: User) -> dict:
"""The mirror pull itself; the caller holds `user_sync_claim`."""
synced = 0
@@ -337,12 +358,10 @@ def pull_user_playlists(db: Session, user: User) -> dict:
Playlist.source == "local",
Playlist.yt_playlist_id.is_not(None),
)
)
.scalars()
.all()
).scalars()
)
# Drop mirrors whose YouTube playlist no longer exists — locked first, as every item write.
gone = [pl for ytid, pl in existing.items() if ytid not in yt_ids]
# Drop vanished mirrors (none on an empty answer) — locked first, as every item write.
gone = vanished(existing, yt_ids)
if gone:
lock_playlists(db, *(pl.id for pl in gone))
for pl in gone:
@@ -376,7 +395,10 @@ def pull_user_playlists(db: Session, user: User) -> dict:
# The user is pushing or re-pulling this very list: it writes the same rows.
# The next scheduled run refreshes it.
continue
ids = list(yt.iter_my_playlist_video_ids(ytid))
ids = _mirror_listing(yt, ytid, p.get("item_count"), pl.id)
if ids is None:
db.rollback() # a mirror this run just created is not kept empty either
continue
# Ingest first — it commits — so the row lock below is held from the dirty check
# to the commit that replaces the items.
_ensure_videos(db, yt, ids)
@@ -408,90 +430,6 @@ def _local_video_ids(db: Session, pl: Playlist) -> list[str]:
)
def _reorder_moves(current: list[str], desired: list[str]) -> int:
"""Count the playlistItems.update moves an insertion-sort would need to turn
`current` order into `desired` order (both already the same set of video ids).
Items already in place are skipped; this matches what apply_push actually executes."""
model = list(current)
moves = 0
for i, want in enumerate(desired):
if i < len(model) and model[i] == want:
continue
j = model.index(want, i)
model.pop(j)
model.insert(i, want)
moves += 1
return moves
@dataclass
class MergePlan:
"""The 3-way merge of a linked playlist: what each side gains and loses so neither side
loses an item, plus the merged order. `final` is the id set both sides end up with."""
final: list[str] # merged order (local order, YouTube-only adds appended)
add_to_yt: list[str] # local-adds to push out to YouTube
add_to_local: list[str] # YouTube-adds to pull into Siftlode
remove_from_yt: list[str] # items deleted locally that are still on YouTube
remove_from_local: list[str] # items deleted on YouTube that are still local
reorders: int # estimated YouTube playlistItems.update moves to reach `final`
baseline_known: bool
def merge_plan(
baseline: list[str] | None,
local: list[str],
youtube: list[str],
unavailable: set[str] | None = None,
) -> MergePlan:
"""Pure 3-way merge. Classify each id against the last-synced `baseline`:
- in local but not youtube, not deleted → add to YouTube (a local-add)
- in youtube but not local, not deleted → add to Siftlode (a YouTube-add)
- in baseline but gone from local → a genuine local delete → propagate to YouTube
- in baseline but gone from youtube → a genuine YouTube delete → propagate to Siftlode
With NO baseline (first-ever merge) we can't tell an add from a delete, so we UNION both
sides (never delete) — the lossless default. Merged order: local order first, YouTube-only
additions appended in their YouTube order.
`unavailable` = known un-ingestable-but-live YouTube ids: dropped from the YouTube side up
front so they're never re-classified as a YouTube-add (they can't ingest) nor as a local
delete (they were never local) — otherwise such a video would make the playlist never
converge. They stay in place on YouTube, untouched by this merge."""
un = set(unavailable or ())
L = list(dict.fromkeys(local))
Y = [v for v in dict.fromkeys(youtube) if v not in un]
Lset, Yset = set(L), set(Y)
if baseline is None:
add_to_yt = [v for v in L if v not in Yset]
add_to_local = [v for v in Y if v not in Lset]
remove_from_yt: list[str] = []
remove_from_local: list[str] = []
final = L + [v for v in Y if v not in Lset]
else:
B = set(baseline)
deleted_local = B - Lset # was synced, removed on our side
deleted_yt = B - Yset # was synced, removed on YouTube's side
gone = deleted_local | deleted_yt
remove_from_yt = [v for v in Y if v in deleted_local]
remove_from_local = [v for v in L if v in deleted_yt]
add_to_yt = [v for v in L if v not in Yset and v not in gone]
add_to_local = [v for v in Y if v not in Lset and v not in gone]
final = [v for v in L if v not in gone] + [v for v in Y if v not in Lset and v not in gone]
final_set = set(final)
# On YouTube, after membership reconciliation the order is current-kept + inserts appended;
# the reorder pass then moves items into `final`.
after_membership = [v for v in Y if v in final_set] + add_to_yt
reorders = _reorder_moves(after_membership, final)
return MergePlan(
final=final,
add_to_yt=add_to_yt,
add_to_local=add_to_local,
remove_from_yt=remove_from_yt,
remove_from_local=remove_from_local,
reorders=reorders,
baseline_known=baseline is not None,
)
def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
"""Compute (without writing) what a two-way sync of this playlist would do: create vs
update, how many items each side gains/loses, reorders, and the quota estimate (YouTube
@@ -519,6 +457,7 @@ def plan_merge(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
rename = bool(snippet) and (snippet.get("title") or "") != pl.name
yt_ids = [it["video_id"] for it in current]
refuse_blank_listing(yt_ids, (snippet or {}).get("item_count"))
mp = merge_plan(parse_baseline(pl), local, yt_ids, parse_unavailable(pl))
ops = len(mp.add_to_yt) + len(mp.remove_from_yt) + mp.reorders + (1 if rename else 0)
return {
@@ -665,6 +604,8 @@ def merge_playlist(db: Session, user: User, pl: Playlist, *, live=None) -> dict:
current = list(yt.iter_playlist_items_with_ids(pl.yt_playlist_id))
item_id_by_vid = {it["video_id"]: it["item_id"] for it in current}
yt_ids = [it["video_id"] for it in current]
# A merge would read "YouTube lists none of them" as "YouTube deleted them all".
refuse_blank_listing(yt_ids, (snippet or {}).get("item_count"))
# Re-attempt ingestion of ids previously marked un-ingestable that are still on YouTube: a
# video that has since become available (e.g. private→public) should re-enter the merge, not
# stay skipped forever. Those that still won't ingest remain filtered (`effective_unavailable`).
@@ -803,8 +744,11 @@ def repull_playlist(db: Session, user: User, pl: Playlist) -> dict:
with YouTubeClient(db, user) as yt:
snippet = yt.get_playlist_snippet(pl.yt_playlist_id)
if snippet is None:
raise YouTubeError("Playlist no longer exists on YouTube")
raise YouTubeError(
"Playlist no longer exists on YouTube", reason="playlistNotFound", path="playlists"
)
ids = list(yt.iter_my_playlist_video_ids(pl.yt_playlist_id))
refuse_blank_listing(ids, snippet.get("item_count"))
_ensure_videos(db, yt, ids)
_lock_or_gone(db, pl)
pl.name = snippet.get("title") or pl.name
+3 -3
View File
@@ -15,8 +15,6 @@ from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import logs, progress, quota, sysconfig
log = logs.get_logger(__name__)
from app.db import SessionLocal
from app.models import Channel, OAuthToken, Subscription, User
from app.sync.explore import followed_channel
@@ -35,6 +33,8 @@ from app.sync.videos import (
from app.youtube.client import YouTubeClient, YouTubeError, is_quota_refusal
from app.youtube.rss import RssError, fetch_channel_feed
log = logs.get_logger(__name__)
# RSS feeds are free and network-bound, so fetch many at once; DB writes stay single-threaded.
RSS_POLL_WORKERS = 16
# Failed channels named in a poll's summary warning; the rest are counted by reason.
@@ -267,7 +267,7 @@ def run_feed_fallback(db: Session, channel_ids: list[str]) -> dict[str, int]:
channel.feed_fallback_at = datetime.now(timezone.utc)
db.add(channel)
db.commit()
except Exception: # noqa: BLE001 — one channel must not stop the rest
except Exception: # one channel must not stop the rest
db.rollback()
log.exception(
"Feed fallback could not store channel %s",
+30 -3
View File
@@ -1,6 +1,6 @@
"""Import the authenticated user's YouTube subscriptions and channel metadata."""
from collections.abc import Iterable
from collections.abc import Collection, Iterable
from datetime import datetime, timedelta, timezone
from sqlalchemy import or_, select
@@ -9,7 +9,7 @@ from sqlalchemy.orm import Session
from app import logs
from app.models import Channel, Subscription, User
from app.utils import normalize_description
from app.youtube.client import YouTubeClient, best_thumbnail
from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail
log = logs.get_logger(__name__)
@@ -115,10 +115,37 @@ def should_prune(fetched_count: int, existing_count: int) -> bool:
has — taking each row's `deep_requested` flag with it — in a single run. Someone who genuinely
unsubscribed from everything simply keeps their rows until a sync that returns something.
Pure so it can be tested without a database; `import_subscriptions` is the only caller."""
Pure so it can be tested without a database; its callers are `import_subscriptions` and, through
`vanished`, every mirror that drops what the remote no longer lists."""
return fetched_count > 0 or existing_count == 0
def vanished[K, V](existing: dict[K, V], listed: Collection[K]) -> list[V]:
"""The local mirrors whose key the remote listing no longer has — none at all when the listing
is an empty answer that contradicts them (`should_prune`). The YouTube playlist mirrors use it:
without it, one blip dropped every mirror (E32 S2)."""
if not should_prune(len(listed), len(existing)):
return []
return [row for key, row in existing.items() if key not in listed]
def blank_listing(listed: Collection, count: object) -> bool:
"""The same rule one level down: no members from a container whose own listing counts some (a
playlist's `itemCount`) is a blip, not a list emptied. Replacing or merging on it deleted every
mirrored item and reset the merge baseline (E32 review)."""
return not listed and isinstance(count, int) and count > 0
def refuse_blank_listing(listed: Collection, count: object) -> None:
"""`blank_listing` for a user-started pull or push: refused as retryable ("try again")."""
if blank_listing(listed, count):
raise YouTubeError(
"YouTube listed no items for a playlist it counts as non-empty",
reason="emptyAnswer",
path="playlistItems",
)
def import_subscriptions(db: Session, user: User) -> dict:
"""Pull all subscriptions for `user`, upsert channels + subscription links, prune
channels the user has unsubscribed from on YouTube, and fetch channel details for
+20 -4
View File
@@ -11,12 +11,14 @@ from sqlalchemy import and_, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app import progress, quota, sysconfig
from app import logs, progress, quota, sysconfig
from app.models import LIVE_OR_UPCOMING, Channel, Video
from app.titles import normalize_title
from app.youtube.client import YouTubeClient, best_thumbnail
from app.youtube.client import YouTubeClient, YouTubeError, best_thumbnail, is_playlist_gone
from app.youtube.shorts import make_client, probe_is_short
log = logs.get_logger(__name__)
_DURATION_RE = re.compile(r"P(?:(\d+)D)?T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?", re.IGNORECASE)
@@ -104,6 +106,20 @@ def stub_from_playlist_item(item: dict, channel_id: str) -> dict | None:
}
def _uploads_page(yt: YouTubeClient, channel_id: str, uploads: str, page_token: str | None) -> dict:
"""One page of the channel's uploads. YouTube answers the uploads list of a channel with no
uploads at all as not found: an empty back-catalogue, so an empty last page — not a failure
that left `recent_synced_at` unstamped and charged the channel again on every run (E32 review).
Any other failure raises."""
try:
return yt.get_playlist_items(uploads, page_token)
except YouTubeError as exc:
if not is_playlist_gone(exc):
raise
log.info("backfill channel=%s has no uploads playlist", channel_id)
return {}
def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) -> int:
"""First pass: newest videos up to the configured count or age cutoff. Records a
resume cursor for the later deep backfill."""
@@ -121,7 +137,7 @@ def backfill_channel_recent(db: Session, yt: YouTubeClient, channel: Channel) ->
while True:
page_start = page_token # token that fetched THIS page (None = first page)
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
data = _uploads_page(yt, channel.id, channel.uploads_playlist_id, page_token)
stopped_mid_page = False
for item in data.get("items", []):
stub = stub_from_playlist_item(item, channel.id)
@@ -175,7 +191,7 @@ def backfill_channel_deep(
for _ in range(max_pages):
if quota.backfill_reserve_reached():
break
data = yt.get_playlist_items(channel.uploads_playlist_id, page_token)
data = _uploads_page(yt, channel.id, channel.uploads_playlist_id, page_token)
rows = [
s
for s in (stub_from_playlist_item(it, channel.id) for it in data.get("items", []))
+1
View File
@@ -80,6 +80,7 @@ SPECS: tuple[ConfigSpec, ...] = (
),
# Live-search source: "scrape" (InnerTube, zero API quota) or "api" (search.list, 100u/page).
ConfigSpec("search_source", "choice", "quota", "search_source"),
ConfigSpec("quota_retention_days", "int", "quota", "quota_retention_days", min=0, max=3650),
# --- Backfill (recent-first first pass per channel) ---
ConfigSpec(
"backfill_recent_max_videos",
-2
View File
@@ -60,7 +60,6 @@ _SHORT_LOWER = {
"for",
"or",
"vs",
"the",
"és",
"az",
"egy",
@@ -120,7 +119,6 @@ _ACRONYMS = {
"nfl",
"asmr",
"pov",
"diy",
"wtf",
"lol",
"rtx",
+68 -17
View File
@@ -5,7 +5,8 @@ from the API so long yt-dlp downloads + ffmpeg postprocessing never block web re
Design:
* N loop threads (download_worker_concurrency) each claim a `queued` DownloadJob with
FOR UPDATE SKIP LOCKED and process it.
FOR UPDATE SKIP LOCKED and process it. N and the log level follow the settings page while the
worker runs (`_sync_knobs`), not only the env it booted with.
* The shared MediaAsset is the dedup unit. A job whose asset is already `ready` finishes
instantly (no re-download); a fresh asset is claimed pending→downloading by exactly one
worker (a DB compare-and-set), so two jobs for the same video+format never double-download.
@@ -25,6 +26,7 @@ import threading
import time
from collections.abc import Callable
from datetime import datetime, timezone
from functools import partial
from sqlalchemy import text
@@ -47,12 +49,60 @@ from app.models import DISPLAY_TEXT_MAX, DownloadJob, MediaAsset
log = logs.get_logger(__name__)
_stop = threading.Event()
# How many loop threads may claim jobs. A thread whose index is at or above it stops after its
# current job, so lowering the knob never abandons a download halfway.
_target = 1
# How often the worker re-reads the knobs it obeys; the settings page says nothing about a restart.
_KNOB_SECONDS = 10.0
def _handle_signal(signum, frame):
_stop.set()
def _read_knobs() -> tuple[int, str] | None:
"""The stored concurrency and log level, or None when the database cannot answer — the caller
then keeps what already runs."""
try:
with SessionLocal() as db:
return (
max(1, sysconfig.get_int(db, "download_worker_concurrency")),
sysconfig.get_str(db, "log_level"),
)
except Exception: # noqa: BLE001 — any DB or config failure keeps the knobs already in force
log.warning(
"could not read the worker's settings; keeping the ones in force",
exc_info=True,
extra=logs.event("download.worker.knobs_unreadable"),
)
return None
def _sync_knobs(threads: dict[int, threading.Thread], level: str | None) -> str | None:
"""Apply the stored knobs: run loop threads up to the concurrency (surplus ones retire after
their job, see `_target`) and re-apply the log level when it changed. Returns the level in force."""
global _target
knobs = _read_knobs()
if knobs is not None:
target, stored_level = knobs
if target != _target:
log.info(
"download concurrency is now %d",
target,
extra=logs.event("download.worker.concurrency", before=_target, after=target),
)
_target = target
if stored_level != level:
logs.set_level(stored_level)
level = stored_level
for i in range(_target):
t = threads.get(i)
if t is None or not t.is_alive():
threads[i] = threading.Thread(target=_worker_loop, args=(i,), daemon=True)
threads[i].start()
return level
# --- claim + recovery -----------------------------------------------------------------------
@@ -72,7 +122,7 @@ def _wait_for_schema(timeout: float = 300.0) -> bool:
with SessionLocal() as db:
db.execute(text("SELECT 1 FROM download_jobs LIMIT 1"))
return True
except Exception: # noqa: BLE001 — table not migrated yet / DB not up yet
except Exception: # table not migrated yet / DB not up yet
if time.monotonic() > deadline:
# The traceback, not `str(exc)[:120]`: this is the last thing the worker says
# before it exits, and which driver call failed is the whole question.
@@ -366,7 +416,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
try:
proc = _spawn_download_child(job_id)
except Exception: # noqa: BLE001 — spawn failed; don't leave the asset stuck 'downloading'
except Exception: # spawn failed; don't leave the asset stuck 'downloading'
log.exception("failed to spawn download child for job %s", job_id)
_reset_asset_pending(asset_id)
_requeue_if_running(job_id, None)
@@ -438,7 +488,7 @@ def _run_supervised(job_id: int, asset_id: int) -> None:
cause="child_exit",
exit_code=proc.returncode,
)
except Exception: # noqa: BLE001 — boot recovery is the backstop for a stuck 'running' job
except Exception: # boot recovery is the backstop for a stuck 'running' job
log.exception("post-exit reconciliation failed for job %s", job_id)
return
@@ -501,6 +551,7 @@ def _process_job(job_id: int) -> None:
then run in a supervised CHILD PROCESS (`_run_supervised`) so a wedged yt-dlp/ffmpeg can be
hard-killed — a worker THREAD cannot be killed, nor (when it spins in a C call) even
cooperatively cancelled, and one such thread starves the others via the GIL."""
diagnosis = None # set exactly when the asset has already failed (asset_status == "error")
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
if job is None:
@@ -528,7 +579,7 @@ def _process_job(job_id: int) -> None:
if asset_status == "ready":
_finish_from_cache(job_id, asset_id)
return
if asset_status == "error":
if diagnosis is not None:
_fail_running(
job_id,
error=asset_error or "Download failed",
@@ -574,10 +625,10 @@ def _finish_from_cache(job_id: int, asset_id: int) -> None:
def _worker_loop(idx: int) -> None:
while not _stop.is_set():
while not _stop.is_set() and idx < _target:
try:
job_id = _claim_job()
except Exception: # noqa: BLE001
except Exception:
log.exception("claim failed")
_stop.wait(settings.download_poll_seconds)
continue
@@ -597,12 +648,12 @@ def _worker_loop(idx: int) -> None:
job_id,
exc,
"crash",
lambda: _fail_running(job_id, error="Worker error", error_code=None),
partial(_fail_running, job_id, error="Worker error", error_code=None),
)
# silent-ok: not silent — `_report_worker_error` logged the crash before this
# re-raise. Failing the job hit the database, and letting that out would end this
# worker thread for good; boot recovery settles a job left `running`.
except Exception: # noqa: BLE001
except Exception:
log.exception(
"could not fail job %s after its crash",
job_id,
@@ -632,15 +683,15 @@ def main() -> None:
# has migrated — those few seconds of batches fail and say so on stderr; nothing else breaks.
logsink.start()
_recover_orphans()
n = max(1, settings.download_worker_concurrency)
log.info("Download worker started (concurrency=%d, root=%s).", n, settings.download_root)
threads = [threading.Thread(target=_worker_loop, args=(i,), daemon=True) for i in range(n)]
for t in threads:
t.start()
while not _stop.is_set():
_stop.wait(1.0)
global _target
_target = max(1, settings.download_worker_concurrency) # used only if the DB can't answer
threads: dict[int, threading.Thread] = {}
level = _sync_knobs(threads, None)
log.info("Download worker started (concurrency=%d, root=%s).", _target, settings.download_root)
while not _stop.wait(_KNOB_SECONDS):
level = _sync_knobs(threads, level)
log.info("Download worker stopping…")
for t in threads:
for t in threads.values():
t.join(timeout=5)
+28 -8
View File
@@ -114,18 +114,32 @@ _DEAD_GRANT = "invalid_grant"
# What a refused YouTube action means to the person who asked for it — the three things they can
# act on differently: wait for tomorrow's budget, reconnect the account, or try again later.
YouTubeRefusal = Literal["quota", "reconnect", "unavailable"]
YouTubeRefusal = Literal["quota", "reconnect", "unavailable", "gone"]
def refusal_reason(exc: YouTubeError) -> YouTubeRefusal:
def is_playlist_gone(exc: YouTubeError) -> bool:
"""Whether YouTube answered that the playlist read or written does not exist. Only the caller
knows which playlist that was: a user's list deleted on YouTube, or a channel's uploads list,
which YouTube also answers so for a channel with no uploads at all."""
return "playlistNotFound" in exc.reason.split(",")
def refusal_reason(exc: YouTubeError, *, playlist: bool = False) -> YouTubeRefusal:
"""Sort a failure by what the user can do about it. A dead grant (Google refused the refresh
with `invalid_grant`, which also drops the grant — see `_drop_dead_grant`) or no usable token at
all (raised before any endpoint answered, so with no path and no transport reason) means the
account must be reconnected; retrying cannot help. Any other token-endpoint failure — a 5xx, a
429, a misconfigured client — leaves the grant as it was: reconnecting would not fix it, and
the account would still read as connected, with no connect button to press."""
the account would still read as connected, with no connect button to press.
`playlist`: the action was on ONE playlist the user picked, so a missing playlist is that list,
gone (deleted on YouTube) — neither a retry nor a reconnect helps. Anywhere else the playlist
is one the user never named (a channel's uploads, one list of a bulk pull), and "this playlist
no longer exists" would describe nothing on their screen (E32 review round 2)."""
if is_quota_refusal(exc):
return "quota"
if playlist and is_playlist_gone(exc):
return "gone"
if exc.path == "oauth2/token":
return "reconnect" if exc.reason == _DEAD_GRANT else "unavailable"
if not exc.path and exc.reason != "transport":
@@ -189,7 +203,7 @@ class YouTubeClient:
return cached
try:
refresh = decrypt(tok.refresh_token_enc)
except Exception as exc: # noqa: BLE001 — Fernet InvalidToken (TOKEN_ENCRYPTION_KEY changed)
except Exception as exc: # Fernet InvalidToken (TOKEN_ENCRYPTION_KEY changed)
# A key mismatch must degrade like any other token problem (caller handles YouTubeError),
# not surface as a raw 500 on every scheduler job / connect check. (B-core-2)
raise YouTubeError("Stored OAuth token is unreadable — reconnect your account") from exc
@@ -423,16 +437,22 @@ class YouTubeClient:
return resp.json() if resp.content else {}
def get_playlist_snippet(self, playlist_id: str) -> dict | None:
"""The snippet (title/description) of one playlist, or None if it no longer
exists. 1 unit (OAuth, so private playlists work)."""
"""The snippet (title/description) of one playlist plus its `item_count` (what an empty
item listing is checked against), or None if it no longer exists. 1 unit (OAuth, so
private playlists work)."""
data = self._get(
"playlists",
{"part": "snippet", "id": playlist_id, "maxResults": 1},
{"part": "snippet,contentDetails", "id": playlist_id, "maxResults": 1},
cost=1,
allow_key=False,
)
items = data.get("items", [])
return items[0].get("snippet") if items else None
if not items:
return None
return {
**(items[0].get("snippet") or {}),
"item_count": items[0].get("contentDetails", {}).get("itemCount"),
}
def update_playlist_title(self, playlist_id: str, title: str) -> None:
"""Rename a playlist on YouTube. 50 units."""
+3
View File
@@ -40,6 +40,7 @@ class RssError(Exception):
def _get_feed(channel_id: str) -> bytes:
"""The feed's body, after up to `1 + len(RETRY_DELAYS)` tries; the LAST failure is raised."""
url = RSS_URL.format(channel_id=channel_id)
error: RssError | None = None
for attempt in range(len(RETRY_DELAYS) + 1):
if attempt:
time.sleep(RETRY_DELAYS[attempt - 1])
@@ -60,6 +61,8 @@ def _get_feed(channel_id: str) -> bytes:
)
if resp.status_code not in _RETRYABLE_STATUS:
break
if error is None: # cannot happen: the range above is never empty, and every try records one
raise RssError(f"RSS fetch for {channel_id} made no attempt", "transport")
raise error
+28 -14
View File
@@ -42,6 +42,9 @@ _HEADERS = {
"Accept-Language": "en-US,en;q=0.9",
}
# The top-level keys a first page (contents) and a continuation page (onResponseReceivedCommands)
# put their result list under.
_RESULT_FRAMES = {"contents", "onResponseReceivedCommands"}
_KEY_RE = re.compile(r'"INNERTUBE_API_KEY":"([^"]+)"')
_VER_RE = re.compile(r'"INNERTUBE_CLIENT_VERSION":"([^"]+)"')
@@ -65,29 +68,36 @@ def _runs_text(node: dict | None) -> str | None:
def _get_config(client: httpx.Client) -> tuple[str, str]:
"""The current InnerTube (api_key, client_version), cached after the first fetch."""
"""The current InnerTube (api_key, client_version), cached after the first fetch that FOUND them.
The fallback pair is never cached: it used to be, so one failed fetch (or a consent page with no
keys on it) at the wrong moment pinned every later search to a hard-coded key and version for
the life of the process — and when those went stale, search returned nothing, with no error
anywhere (E32 S2). Now the next search simply tries the live pair again."""
global _config
if _config is not None:
return _config
try:
resp = client.get(_RESULTS_URL, params={"search_query": "music"})
html = resp.text
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
_config = (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
html = client.get(_RESULTS_URL, params={"search_query": "music"}).text
except httpx.HTTPError as exc:
# The fallback pair is CACHED in `_config` for the life of the process, so one failed
# fetch at the wrong moment pins every later search to a hard-coded key and version —
# and when those go stale, search returns nothing at all, with no error anywhere.
log.warning(
"the InnerTube config could not be fetched; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason=type(exc).__name__),
)
_config = (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
return _config
return (_FALLBACK_KEY, _FALLBACK_CLIENT_VERSION)
key = _KEY_RE.search(html)
ver = _VER_RE.search(html)
if key and ver:
_config = (key.group(1), ver.group(1))
return _config
log.warning(
"the results page carried no InnerTube config; falling back to the pinned key",
extra=logs.event("youtube.scrape.config_fallback", reason="not_on_page"),
)
return (
key.group(1) if key else _FALLBACK_KEY,
ver.group(1) if ver else _FALLBACK_CLIENT_VERSION,
)
def _video_item(vr: dict) -> dict | None:
@@ -182,6 +192,10 @@ def search(q: str, continuation: str | None = None) -> dict:
except ValueError as exc: # non-JSON body (blocked / consent / captcha)
raise ScrapeError("InnerTube returned a non-JSON response") from exc
# Even a search with no results carries its result list's frame. Without one this is not "no
# results" but a shape we do not know (YouTube moved, or a block page in JSON) — say so.
if not isinstance(data, dict) or not _RESULT_FRAMES & data.keys():
raise ScrapeError("InnerTube returned an unexpected shape")
items: list[dict] = []
_collect_videos(data, items, set())
return {"items": items, "next_page_token": _find_continuation(data)}
+6 -1
View File
@@ -35,4 +35,9 @@ services:
volumes:
- ./:/app
working_dir: /app
command: python -m pytest
# The suite, then the coverage total (E32 D4: reported, no threshold). pytest's exit code is the
# lane's — the report is printed either way and never decides it.
command:
- sh
- -c
- 'python -m pytest --cov=app --cov-report= ; rc=$$?; echo ">> backend coverage (app/) $$(python -m coverage report --format=total)% of lines"; exit $$rc'
+1 -1
View File
@@ -25,7 +25,7 @@
{
"include": ["app", "alembic"],
"extraPaths": ["."],
"typeCheckingMode": "basic",
"typeCheckingMode": "standard",
"pythonVersion": "3.12",
"pythonPlatform": "Linux"
}
+4 -2
View File
@@ -2,7 +2,9 @@
# Run from backend/ (the image's /app). `pythonpath = .` puts the app package on sys.path so tests
# import it as `app.*`, matching runtime. Most of the suite is pure-logic; the DB-backed lane opts in
# via the `db` fixture and runs in the compose test image. `-rs` reports skip REASONS so a silently
# skipped lane is visible (the compose lane sets SIFTLODE_TEST_REQUIRE_DB=1 to make skips a hard error).
# skipped lane is visible (and conftest makes a missing DB/ffmpeg a hard error unless
# SIFTLODE_TEST_REQUIRE_DB=0). `--strict-markers` (E32): a misspelt `@pytest.mark.skipiff` is an error,
# not an unknown marker that quietly does nothing.
testpaths = tests
pythonpath = .
addopts = -q -rs
addopts = -q -rs --strict-markers
+5
View File
@@ -7,7 +7,12 @@
# basedpyright: the backend type-check lane (E27 S3). Pinned like everything else here — a type
# checker whose version floats silently changes what "clean" means, and the ratchet baseline it
# writes is keyed to the diagnostics THIS version produces.
# pytest-cov: the coverage REPORT the gate prints (E32 D4) — a number to read, deliberately with no
# threshold; "every regression test seen red" is the bar, not a percentage.
# pip-audit: the gate's audit of requirements.lock (scripts/check.sh), blocking at any severity.
basedpyright==1.40.1
pytest==8.4.2
pytest-cov==7.1.0
ruff==0.15.21
httpx2==2.13.0
pip-audit==2.10.1
+5 -2
View File
@@ -1,7 +1,10 @@
# Pinned dependency lock for reproducible builds (SEC-5). GENERATED — do not hand-edit.
# Source of truth is requirements.txt (the floor ranges); regenerate after editing it with:
# docker run --rm -v "$PWD":/b python:3.12-slim sh -c 'pip install -q -r /b/requirements.txt && pip freeze --exclude pip --exclude setuptools --exclude wheel' | sort
# The prod Dockerfile + Dockerfile.test + CI install THIS file; pip-audit scans it in CI.
# The prod Dockerfile + Dockerfile.test install THIS file; the gate's pip-audit (scripts/check.sh)
# scans it, blocking since E32. A security fix may be ONE pin bumped in place instead of a full
# regeneration, which would move every other package too — then `pip install -r requirements.lock && pip check` must
# pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177, E32).
APScheduler==3.11.3
Authlib==1.7.2
Mako==1.4.1
@@ -53,7 +56,7 @@ starlette==1.6.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzlocal==5.4.4
urllib3==2.7.0
urllib3==2.8.0
uvicorn==0.52.3
uvloop==0.22.1
watchfiles==1.2.0
+27 -8
View File
@@ -1,7 +1,8 @@
# Ruff config for the Siftlode backend. Keeps ruff's default rule set (E4/E7/E9 + F = the
# high-signal pyflakes/pycodestyle checks that catch real dead code + bugs), but ignores E402:
# a handful of modules (e.g. main.py) deliberately run setup — logging config, sys.path — before
# importing route modules, so "module import not at top of file" is intentional there, not cruft.
# high-signal pyflakes/pycodestyle checks that catch real dead code + bugs), all of it on. E402 was
# ignored for main.py's logging setup; the ignore had let 11 incidental late imports in elsewhere
# (E32 review), so it is on now and main.py marks each deliberate one (`# noqa: E402 — <why>`).
# `scripts/e2e_seed.py` edits `sys.path` before its imports, which E402 itself allows.
#
# line-length 100 (ruff's default is 88): matches the frontend prettier printWidth so both halves of
# the repo wrap at the same width, and it's the width `ruff format` enforces (gated via
@@ -17,9 +18,10 @@ target-version = "py312"
# LOG — stdlib logging misuse (`.warn()`, a logger built by hand, `exc_info` on `.exception`)
# G — the call's shape: %-style args, never an f-string the level filter cannot skip, and
# never `str(exc)` where `exc_info` would carry the traceback
# T20 — `print` instead of a log line. Two deliberate ones remain, each with its `# noqa`
# and a why: `downloads/runner.py`'s usage line is for whoever typed it, and
# `logsink.py` reports the SINK's own failure, which cannot be logged into the sink.
# T20 — `print` instead of a log line. Three deliberate ones remain, each with its `# noqa`
# and a why: `downloads/runner.py`'s usage line is for whoever typed it, `logsink.py`
# reports the SINK's own failure, which cannot be logged into the sink, and
# `scripts/e2e_seed.py` prints the seeded ids for the e2e run that reads them.
# TRY400 — `.error()` in an `except` where `.exception()` would keep the traceback
# TRY401 — the exception spelled into the message AND passed as `exc_info` (it is in both)
# S112 — `except: continue`, the silent swallow inside a loop
@@ -29,5 +31,22 @@ target-version = "py312"
# nobody to raise to, a best-effort backfill inside a sync) and always worth a sentence.
# The wider rule these two enforce lives in tests/test_swallowed_exceptions.py: a handler that
# neither logs nor re-raises says `# silent-ok: <why>`. Ruff sees only the two narrow shapes.
extend-select = ["LOG", "G", "T20", "TRY400", "TRY401", "S110", "S112", "BLE001"]
ignore = ["E402"]
#
# E32 (gate integrity) added three more, each measured before it went on and with every hit fixed:
# RUF100 — a `# noqa` that suppresses nothing. A stale one is a claim with no rule behind it, and
# it stays armed for the day the line changes (24 were removed). A comment that merely
# explains a noqa below it names the rule as `# TRY400: …`, never `# noqa TRY400`.
# PLW1510 — `subprocess.run` without an explicit `check=`. Found the real E32 bug: three ffprobe
# probes read the output of a probe that had FAILED and took it for the file's truth.
# B — flake8-bugbear: loop-variable capture, `raise` without `from` inside `except`, a
# `zip` that silently truncates (each one is `strict=True` now — the lengths are an
# invariant, so a mismatch should be loud). B008 (a call in a default argument) stays ON:
# FastAPI's `Depends(...)` / `Query(...)` ARE that shape by design (474 hits, all of
# them), so those two are declared immutable below instead of switching the rule off —
# any other call in a default still fails.
extend-select = [
"LOG", "G", "T20", "TRY400", "TRY401", "S110", "S112", "BLE001", "RUF100", "PLW1510", "B",
]
[lint.flake8-bugbear]
extend-immutable-calls = ["fastapi.Depends", "fastapi.Query"]
+12 -9
View File
@@ -25,7 +25,7 @@ from fastapi.testclient import TestClient
from sqlalchemy import create_engine, event, text
from starlette.requests import HTTPConnection
import app.models # noqa: F401 — register every model on Base.metadata
import app.models # register every model on Base.metadata
import app.state as app_state
from app import security
from app.auth import current_user
@@ -37,9 +37,11 @@ from app.models import User
# Sentinel for "this dependency had no override before the fixture set one" (see override_dep).
_MISSING = object()
# The compose test lane sets this so a would-be silent skip (no DB, or no ffmpeg) becomes a hard error
# — the gate must not print green while the DB-backed / ffmpeg tests were quietly not run.
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB") == "1"
# A would-be silent skip (no DB, or no ffmpeg) is a hard error unless the run says otherwise — the
# suite must not print green while the DB-backed / ffmpeg tests were quietly not run. ON by default
# since E32: as an opt-in only the compose lane set, a hand-run `pytest` skipped ~60 DB tests
# and still passed. `SIFTLODE_TEST_REQUIRE_DB=0` is the explicit "pure-logic only" run.
_REQUIRE_FULL = os.getenv("SIFTLODE_TEST_REQUIRE_DB", "1") != "0"
def _db_reachable() -> bool:
@@ -86,19 +88,20 @@ def _truncate_all():
def pytest_configure(config):
"""Fail the whole run — loudly, once — when SIFTLODE_TEST_REQUIRE_DB=1 but the environment can't
actually run the full suite. Without this, an unreachable DATABASE_URL or a missing ffmpeg turns
"""Fail the whole run — loudly, once — when the full suite is required (the default) but the
environment can't actually run it. Without this, an unreachable DATABASE_URL or a missing ffmpeg turns
~60 DB tests + the poster tests into silent skips and the gate still exits green."""
if not _REQUIRE_FULL:
return
if not _DB_REACHABLE:
raise pytest.UsageError(
"SIFTLODE_TEST_REQUIRE_DB=1 but no Postgres is reachable at the test DATABASE_URL — the "
"DB-backed lane would silently skip. Fix the DB wiring or unset the flag."
"no Postgres is reachable at the test DATABASE_URL — the DB-backed lane would silently "
"skip. Fix the DB wiring, or run the pure-logic tests only with SIFTLODE_TEST_REQUIRE_DB=0."
)
if shutil.which("ffmpeg") is None:
raise pytest.UsageError(
"SIFTLODE_TEST_REQUIRE_DB=1 but ffmpeg is missing — the thumbnail/poster tests would skip."
"ffmpeg is missing — the thumbnail/poster tests would skip. Install it, or run the "
"pure-logic tests only with SIFTLODE_TEST_REQUIRE_DB=0."
)
+21 -6
View File
@@ -36,9 +36,14 @@ def test_a_public_url_gets_past_the_guard(db, as_user, make_user, monkeypatch):
"getaddrinfo",
lambda *a, **kw: [(None, None, None, "", ("93.184.216.34", 0))],
)
r = as_user(user).post("/api/downloads", json={"source": "https://example.com/v"})
detail = r.json().get("detail") if r.status_code >= 400 else None
assert not (isinstance(detail, dict) and detail.get("code") == "url_blocked")
# An inline spec, so nothing ELSE refuses it: without one the request stopped at the
# profile lookup (400 no_profile) and "not url_blocked" held for a request the guard never
# had to wave through (E32 S2).
r = as_user(user).post(
"/api/downloads", json={"source": "https://example.com/v", "spec": {"format": "best"}}
)
assert r.status_code == 200, r.text
assert r.json()["status"] == "queued"
def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeypatch):
@@ -50,7 +55,13 @@ def test_youtube_sources_never_consult_the_guard(db, as_user, make_user, monkeyp
user = make_user()
db.commit()
monkeypatch.setattr(urlguard.socket, "getaddrinfo", boom)
as_user(user).post("/api/downloads", json={"source": "dQw4w9WgXcQ"})
# The job must actually be created: a request refused for any other reason would never reach
# the guard either, and the test would pass without proving anything (E32 S2).
r = as_user(user).post(
"/api/downloads", json={"source": "dQw4w9WgXcQ", "spec": {"format": "best"}}
)
assert r.status_code == 200, r.text
assert r.json()["source_kind"] == "youtube"
def test_manual_subscription_sync_goes_on_cooldown(db, as_user, make_user, monkeypatch):
@@ -81,8 +92,12 @@ def test_the_cooldown_is_per_user_not_global(db, as_user, make_user, monkeypatch
monkeypatch.setattr(syncroutes, "import_subscriptions", lambda d, u: {"imported": 0})
ca = as_user(a)
for _ in range(4):
ca.post("/api/sync/subscriptions")
assert [ca.post("/api/sync/subscriptions").status_code for _ in range(4)] == [
200,
200,
200,
429,
]
assert ca.post("/api/sync/subscriptions").status_code == 429
assert as_user(b).post("/api/sync/subscriptions").status_code == 200
+64
View File
@@ -0,0 +1,64 @@
"""sync/autotag's orphan cleanup (E32 S2: the module was 32% covered).
A system tag no channel carries any more is deleted — unless a saved feed view filters on it. The
views hold tag ids; a deleted tag returns under a NEW id when a channel earns it again, and the feed
drops an unknown id, so the view silently showed the whole feed under its old name.
"""
from sqlalchemy import select
from app.models import Channel, ChannelTag, SavedView, Tag
from app.sync import autotag
def test_an_orphan_system_tag_goes_but_one_a_saved_view_filters_on_stays(db, make_user):
user = make_user()
used = Tag(user_id=None, name="Gaming", category="topic")
orphan = Tag(user_id=None, name="Cooking", category="topic")
in_view = Tag(user_id=None, name="Music", category="topic")
own = Tag(user_id=user.id, name="Mine", category="other")
ch = Channel(id="UC_tags", title="T")
db.add_all([used, orphan, in_view, own, ch])
db.flush()
db.add(ChannelTag(channel_id=ch.id, tag_id=used.id, user_id=None, source="auto"))
db.add(
SavedView(
user_id=user.id, source="feed", name="Music", filters={"tags": [in_view.id], "q": ""}
)
)
# A Plex view's filters are another vocabulary; they must not break the scan.
db.add(SavedView(user_id=user.id, source="plex", name="P", filters={"genres": ["Drama"]}))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
db.expire_all()
names = set(db.execute(select(Tag.name)).scalars())
assert names == {"Gaming", "Music", "Mine"} # only the unreferenced system orphan went
def test_a_malformed_saved_view_cannot_break_the_cleanup(db, make_user):
"""A view stored around the route's shape check (or before it) must not raise for everyone's job
— `"tags": 5` raised TypeError — and `True` is not tag id 1 (E32 review)."""
user = make_user()
orphan = Tag(user_id=None, name="Cooking", category="topic")
db.add(orphan)
db.flush()
for i, filters in enumerate(
({"tags": 5}, {"tags": [True]}, ["not", "a", "dict"], {"tags": None})
):
db.add(SavedView(user_id=user.id, source="feed", name=f"bad{i}", filters=filters))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
db.expire_all()
assert db.execute(select(Tag).where(Tag.name == "Cooking")).first() is None
def test_a_view_kept_tag_list_larger_than_a_bind_parameter_limit_still_cleans_up(db, make_user):
user = make_user()
orphan = Tag(user_id=None, name="Cooking", category="topic")
db.add(orphan)
db.flush()
huge = list(range(10_000_000, 10_070_000)) # past psycopg's 65535 bind parameters
db.add(SavedView(user_id=user.id, source="feed", name="huge", filters={"tags": huge}))
db.commit()
assert autotag._cleanup_orphan_system_tags(db) == 1
+1 -1
View File
@@ -321,7 +321,7 @@ def _body_reading_functions() -> dict[tuple[str, str], dict]:
continue
names = [name for _, name in calls]
limits = []
for node, name in zip([n for n, _ in calls], names):
for node, name in zip([n for n, _ in calls], names, strict=True):
if name.startswith("request."):
limits.append("<no ceiling declared>")
continue
+24 -14
View File
@@ -151,7 +151,7 @@ def test_nobody_reaches_another_users_session(db, make_user, as_user):
def test_the_list_is_the_users_own_newest_first(db, make_user, as_user):
client = as_user(make_user())
first = _start(client)["id"]
client.post(f"/api/debug-sessions/{first}/stop")
assert client.post(f"/api/debug-sessions/{first}/stop").status_code == 200
second = _start(client)["id"]
assert [s["id"] for s in client.get("/api/debug-sessions").json()["items"]] == [second, first]
@@ -199,10 +199,13 @@ def test_an_erased_account_takes_its_sessions_with_it(db, make_user, as_user):
user = make_user()
client = as_user(user)
sid = _start(client)["id"]
client.post(
f"/api/debug-sessions/{sid}/screenshots",
content=_PNG,
headers={"content-type": "image/png"},
assert (
client.post(
f"/api/debug-sessions/{sid}/screenshots",
content=_PNG,
headers={"content-type": "image/png"},
).status_code
== 200
)
purge_user(db, user, BackgroundTasks())
assert db.get(DebugSession, sid) is None
@@ -250,7 +253,7 @@ def test_a_request_naming_the_own_open_session_is_tagged_and_raised(gated, db, c
_signed_in(client, make_user)
sid = _start(client, "DEBUG")["id"]
gated.records.clear()
client.get("/api/debug-sessions", headers={"x-debug-session": sid})
assert client.get("/api/debug-sessions", headers={"x-debug-session": sid}).status_code == 200
(line,) = _request_lines(gated)
assert getattr(line, "debug_session_id", None) == sid
# Raised to the session's level: the line is kept for storage though the user's own is Alap.
@@ -263,7 +266,7 @@ def test_a_header_naming_anything_else_is_ignored_not_refused(gated, db, client,
db.commit()
_signed_in(client, make_user)
own = _start(client)["id"]
client.post(f"/api/debug-sessions/{own}/stop")
assert client.post(f"/api/debug-sessions/{own}/stop").status_code == 200
for value in (foreign, own, "not-a-session", "f" * 16, "A" * 16):
gated.records.clear()
# Not `/api/me`: a polled route logs at DEBUG, and an un-raised request would drop the very
@@ -280,7 +283,7 @@ def test_a_header_naming_the_own_session_past_its_cap_is_ignored(gated, db, clie
sid = _start(client)["id"]
_expire(db, sid) # past its cap, not yet swept: still open in the table
gated.records.clear()
client.get("/api/debug-sessions", headers={"x-debug-session": sid})
assert client.get("/api/debug-sessions", headers={"x-debug-session": sid}).status_code == 200
(line,) = _request_lines(gated)
assert getattr(line, "debug_session_id", None) is None
assert getattr(line, "elevated", False) is False
@@ -616,7 +619,7 @@ def test_a_users_screenshots_are_capped_across_their_sessions(recording, monkeyp
client, sid = recording
monkeypatch.setattr(debugsession, "SCREENSHOT_USER_MAX_BYTES", len(_PNG) * 2)
assert _upload(client, sid, _PNG).status_code == 200
client.post(f"/api/debug-sessions/{sid}/stop")
assert client.post(f"/api/debug-sessions/{sid}/stop").status_code == 200
other = _start(client)["id"]
assert _upload(client, other, _PNG).status_code == 200
r = _upload(client, other, _PNG)
@@ -631,7 +634,7 @@ def test_new_sessions_are_paced_but_joining_one_is_not(db, make_user, as_user, m
client = as_user(make_user())
sid = _start(client)["id"]
assert _start(client)["id"] == sid # another tab joining
client.post(f"/api/debug-sessions/{sid}/stop")
assert client.post(f"/api/debug-sessions/{sid}/stop").status_code == 200
r = client.post("/api/debug-sessions", json={"level": "DEBUG"})
assert r.status_code == 429 and r.json()["detail"]["reason"] == "debug_session_rate_limited"
@@ -745,9 +748,16 @@ def test_the_export_is_the_saved_draft_the_evidence_and_the_screenshots(db, make
db.commit()
_upload(client, sid, _PNG)
_upload(client, sid, _JPEG)
client.put(
f"/api/debug-sessions/{sid}/draft",
json={"title": "Spinner stays", "body": "It spins.", "excluded": [dropped_web.id, 999999]},
assert (
client.put(
f"/api/debug-sessions/{sid}/draft",
json={
"title": "Spinner stays",
"body": "It spins.",
"excluded": [dropped_web.id, 999999],
},
).status_code
== 200
)
r = client.get(f"/api/debug-sessions/{sid}/export")
@@ -1144,7 +1154,7 @@ def test_new_issues_are_paced_per_user(gitea_fake, db, make_user, as_user, monke
first = _start(client)["id"]
_ready(client, first)
assert client.post(f"/api/debug-sessions/{first}/issue").status_code == 200
client.post(f"/api/debug-sessions/{first}/stop")
assert client.post(f"/api/debug-sessions/{first}/stop").status_code == 200
second = _start(client)["id"]
_ready(client, second)
r = client.post(f"/api/debug-sessions/{second}/issue")
+26 -11
View File
@@ -212,7 +212,12 @@ def test_a_timestamp_outside_the_consent_window_is_clamped(elevated):
now = datetime.now(timezone.utc)
future = (now + timedelta(days=365)).isoformat()
past = (now - timedelta(days=30)).isoformat()
client.post("/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]})
assert (
client.post(
"/api/diagnostics/events", json={"events": [_event(ts=future), _event(ts=past)]}
).status_code
== 200
)
late, early = rows()
assert late.ts <= datetime.now(timezone.utc)
assert early.ts >= now - timedelta(hours=diagnostics.ELEVATION_HOURS, minutes=1)
@@ -342,15 +347,15 @@ def _app() -> TestClient:
def test_the_account_reaches_the_handlers_own_lines(gated):
_app().get("/api/thing/41")
assert _app().get("/api/thing/41").status_code == 200
line = next(r for r in gated.records if r.getMessage() == "an ordinary line")
assert getattr(line, "user_id", None) == 41
def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG"})
client.get("/api/thing/8")
assert client.get("/api/thing/7", params={"level": "DEBUG"}).status_code == 200
assert client.get("/api/thing/8").status_code == 200
detail = [r for r in gated.records if r.getMessage().startswith("detail only")]
assert [r.user_id for r in detail] == [7]
assert getattr(detail[0], "elevated", False) is True
@@ -363,9 +368,9 @@ def test_one_users_elevation_logs_their_requests_in_detail_and_nobody_elses(gate
def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
client = _app()
client.get("/api/thing/7", params={"level": "TRACE"})
assert client.get("/api/thing/7", params={"level": "TRACE"}).status_code == 200
assert logging.getLogger().level == logs.TRACE
client.get("/api/thing/7") # the same user, back at the base
assert client.get("/api/thing/7").status_code == 200 # the same user, back at the base
assert logging.getLogger().level == logging.INFO
logs._track_elevation(9, logging.DEBUG, datetime.now(timezone.utc) - timedelta(seconds=1))
assert logging.getLogger().level == logging.INFO # an already-expired one is pruned at once
@@ -373,15 +378,25 @@ def test_the_root_level_returns_to_the_base_when_the_elevation_ends(gated):
def test_the_query_string_is_logged_only_for_a_traced_request_and_redacted(gated):
client = _app()
client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"})
client.get("/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"})
assert client.get("/api/thing/7", params={"level": "DEBUG", "q": "lofi"}).status_code == 200
assert (
client.get(
"/api/thing/7", params={"level": "TRACE", "q": "lofi", "token": "SECRET"}
).status_code
== 200
)
lines = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in lines[0].attrs
assert "q=lofi" in lines[1].attrs["query"] and "SECRET" not in lines[1].attrs["query"]
def test_an_address_typed_into_a_search_is_redacted_though_the_browser_encodes_it(gated):
_app().get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
assert (
_app()
.get("/api/thing/7?level=TRACE&q=someone%40example.com&new_password=hunter2")
.status_code
== 200
)
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert line.attrs["query"] == "level=TRACE&q=[email]&new_password=[redacted]"
@@ -400,7 +415,7 @@ def test_a_credential_param_goes_whole_however_its_value_decodes(raw):
def test_an_admin_level_of_trace_does_not_log_other_users_query_strings(gated):
logs.set_level("TRACE")
_app().get("/api/thing/7", params={"q": "something private"})
assert _app().get("/api/thing/7", params={"q": "something private"}).status_code == 200
(line,) = [r for r in gated.records if getattr(r, "event", None) == "http.request"]
assert "query" not in line.attrs
@@ -488,7 +503,7 @@ def test_a_real_sign_in_carries_the_consented_level_into_the_process(gated, clie
user = _signed_in(client, make_user)
assert client.put("/api/me/diagnostics", json={"level": "DEBUG"}).status_code == 200
logs._elevations.clear() # only a request resolving the account through auth may put it back
client.get("/api/me")
assert client.get("/api/me").status_code == 200
assert logs._elevations[user.id][0] == logging.DEBUG
assert client.put("/api/me/diagnostics", json={"level": "INFO"}).status_code == 200
assert user.id not in logs._elevations
+3 -3
View File
@@ -13,12 +13,12 @@ hold for Google Workspace custom domains unless their admin enabled it.
import pytest
from sqlalchemy import select, text
_UAT_PW = "correct horse battery"
from app.auth import is_allowed
from app.models import Invite, User
from app.utils import canonical_email
_UAT_PW = "correct horse battery"
# (input, expected canonical) — the matrix the sprint promised, plus the shapes that must NOT fold.
CASES = [
# Gmail: dots and +tags are noise…
@@ -334,7 +334,7 @@ def test_me_reports_the_google_address_only_when_it_differs(client, db, make_use
)
assert client.get("/api/me").json()["google_email"] is None
client.post("/auth/logout")
assert client.post("/auth/logout").status_code == 200
tagged = make_user(email="me+siftlode@gmail.com", password=_UAT_PW)
tagged.google_email = "me@gmail.com"
db.commit()
+259
View File
@@ -0,0 +1,259 @@
"""An empty answer is not an emptied library (E32 S2).
Every mirror here reconciles: what the remote no longer lists is removed locally. The catalog
(`_prune_stale`) and the subscriptions (`should_prune`) had long refused to do that on an empty
listing that contradicts what we hold — a 200 with no items is a blip far more often than a user
who removed everything. Two mirrors did not: the Plex collections (a row carries the user-set
`editable` flag, and every member its keys) and the YouTube playlist mirrors. These hold both — and,
one level down (E32 review), a collection's members and a playlist's items: an empty answer that
contradicts the remote's own count is not believed either.
"""
import pytest
from sqlalchemy import select
from app.models import (
Channel,
Playlist,
PlaylistItem,
PlexCollection,
PlexItem,
PlexLibrary,
Video,
)
from app.plex import sync as plex_sync
from app.sync import playlists as plsync
from app.youtube.client import YouTubeError
class _Plex:
def __init__(self, collections: list[dict], children: dict[str, list[dict]]):
self._collections = collections
self._children = children
def collections(self, section_key):
return self._collections
def collection_children(self, rk):
return self._children.get(rk, [])
def metadata(self, rk):
return next((c for c in self._collections if c.get("ratingKey") == rk), {})
def _library_with_a_collection(db):
lib = PlexLibrary(plex_key="4", title="Movies", kind="movie")
db.add(lib)
db.flush()
db.add(PlexCollection(rating_key="c1", library_id=lib.id, title="Bond", editable=True))
movie = PlexItem(
rating_key="m1", library_id=lib.id, kind="movie", title="Dr. No", collection_keys=["c1"]
)
db.add(movie)
db.commit()
return lib, movie
def test_an_empty_collection_listing_keeps_the_collections_and_their_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
plex_sync._sync_collections(db, _Plex([], {}), lib, stats)
db.commit()
db.expire_all()
col = db.execute(select(PlexCollection).where(PlexCollection.rating_key == "c1")).scalar_one()
assert col.editable is True
assert db.get(PlexItem, movie.id).collection_keys == ["c1"]
def test_a_listing_with_something_in_it_still_prunes_what_it_no_longer_has(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
other = {"ratingKey": "c2", "title": "Alien"}
plex_sync._sync_collections(db, _Plex([other], {"c2": []}), lib, stats)
db.commit()
db.expire_all()
keys = set(db.execute(select(PlexCollection.rating_key)).scalars())
assert keys == {"c2"}
assert db.get(PlexItem, movie.id).collection_keys is None
def test_no_members_from_a_collection_plex_counts_as_non_empty_keeps_its_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
bond = {"ratingKey": "c1", "title": "Bond", "childCount": 2}
plex_sync._sync_collections(db, _Plex([bond], {"c1": []}), lib, stats)
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys == ["c1"]
def test_a_collection_plex_counts_as_empty_does_lose_its_members(db):
lib, movie = _library_with_a_collection(db)
stats = {"collections": 0}
bond = {"ratingKey": "c1", "title": "Bond", "childCount": 0}
plex_sync._sync_collections(db, _Plex([bond], {"c1": []}), lib, stats)
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys is None
@pytest.mark.parametrize(("count", "kept"), [(2, ["c1"]), (0, None)])
def test_a_collection_resync_believes_an_empty_member_list_only_with_a_zero_count(db, count, kept):
lib, movie = _library_with_a_collection(db)
bond = {"ratingKey": "c1", "title": "Bond", "childCount": count}
plex_sync.resync_collection(db, _Plex([bond], {"c1": []}), lib, "c1")
db.commit()
db.expire_all()
assert db.get(PlexItem, movie.id).collection_keys == kept
class _YT:
def __init__(self, playlists: list[dict], snippet: dict | None = None):
self._playlists = playlists
self._snippet = snippet
def __call__(self, *_):
return self
def __enter__(self):
return self
def __exit__(self, *_):
return False
def iter_my_playlists(self):
return self._playlists
def iter_playlist_items_with_ids(self, _ytid):
return []
def iter_my_playlist_video_ids(self, _ytid):
return []
def get_playlist_snippet(self, _ytid):
return self._snippet
def get_videos(self, _ids):
return []
def test_an_empty_playlist_answer_keeps_the_youtube_mirrors(db, make_user, monkeypatch):
user = make_user()
db.add(Playlist(user_id=user.id, name="Mixes", source="youtube", yt_playlist_id="PL1"))
db.commit()
monkeypatch.setattr(plsync, "YouTubeClient", _YT([]))
plsync.pull_user_playlists(db, user)
db.expire_all()
names = db.execute(select(Playlist.name).where(Playlist.user_id == user.id)).scalars().all()
assert "Mixes" in names
def test_a_playlist_answer_with_something_in_it_still_drops_a_vanished_mirror(
db, make_user, monkeypatch
):
user = make_user()
db.add(Playlist(user_id=user.id, name="Mixes", source="youtube", yt_playlist_id="PL1"))
db.commit()
monkeypatch.setattr(plsync, "YouTubeClient", _YT([{"id": "PL2", "title": "New"}]))
plsync.pull_user_playlists(db, user)
db.expire_all()
ytids = set(
db.execute(select(Playlist.yt_playlist_id).where(Playlist.source == "youtube")).scalars()
)
assert "PL1" not in ytids
class _YTOneGone(_YT):
"""A list deleted on YouTube between the listing and the read of its items."""
def iter_my_playlist_video_ids(self, ytid):
if ytid == "PLgone":
raise YouTubeError("m", reason="playlistNotFound", path="playlistItems")
return []
def test_a_list_deleted_mid_pull_is_skipped_and_the_rest_still_sync(db, make_user, monkeypatch):
# Not "this playlist no longer exists" for a bulk pull the user never named a list in, and not
# a failure for every other list either (E32 review round 2): the next pull drops the mirror.
user = make_user()
db.commit()
listing = [
{"id": "PLgone", "title": "Gone", "item_count": 3},
{"id": "PL2", "title": "Kept", "item_count": 0},
]
monkeypatch.setattr(plsync, "YouTubeClient", _YTOneGone(listing))
assert plsync.pull_user_playlists(db, user) == {"synced": 1}
db.expire_all()
ytids = set(
db.execute(select(Playlist.yt_playlist_id).where(Playlist.user_id == user.id)).scalars()
)
assert ytids == {"PL2"}
def _mirror_with_an_item(db, user, **kw):
db.add(Channel(id="UC_pl", title="C"))
db.flush()
db.add(Video(id="v1", channel_id="UC_pl", title="One"))
pl = Playlist(user_id=user.id, name="Mixes", yt_playlist_id="PL1", **kw)
db.add(pl)
db.flush()
db.add(PlaylistItem(playlist_id=pl.id, video_id="v1", position=0))
db.commit()
return pl
def _items(db, pl):
db.expire_all()
return (
db.execute(select(PlaylistItem.video_id).where(PlaylistItem.playlist_id == pl.id))
.scalars()
.all()
)
def test_no_items_from_a_playlist_youtube_counts_as_non_empty_keeps_the_mirror_items(
db, make_user, monkeypatch
):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
listing = [{"id": "PL1", "title": "Mixes", "item_count": 1}]
monkeypatch.setattr(plsync, "YouTubeClient", _YT(listing))
plsync.pull_user_playlists(db, user)
assert _items(db, pl) == ["v1"]
def test_a_playlist_youtube_counts_as_empty_is_emptied(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
listing = [{"id": "PL1", "title": "Mixes", "item_count": 0}]
monkeypatch.setattr(plsync, "YouTubeClient", _YT(listing))
plsync.pull_user_playlists(db, user)
assert _items(db, pl) == []
def test_a_reset_to_youtube_refuses_a_blank_listing_and_keeps_the_items(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
monkeypatch.setattr(plsync, "YouTubeClient", _YT([], {"title": "Mixes", "item_count": 1}))
with pytest.raises(YouTubeError) as err:
plsync.repull_playlist(db, user, pl)
db.rollback()
assert err.value.reason == "emptyAnswer"
assert _items(db, pl) == ["v1"]
def test_a_reset_of_a_deleted_playlist_says_it_is_gone(db, make_user, monkeypatch):
user = make_user()
pl = _mirror_with_an_item(db, user, source="youtube")
monkeypatch.setattr(plsync, "YouTubeClient", _YT([], None))
with pytest.raises(YouTubeError) as err:
plsync.repull_playlist(db, user, pl)
assert err.value.reason == "playlistNotFound"
def test_a_push_plan_refuses_a_blank_listing_rather_than_reading_it_as_all_deleted(db, make_user):
user = make_user()
pl = _mirror_with_an_item(db, user, source="local")
with pytest.raises(YouTubeError) as err:
plsync.plan_merge(db, user, pl, live=([], {"title": "Mixes", "item_count": 1}))
assert err.value.reason == "emptyAnswer"
+43
View File
@@ -12,6 +12,8 @@ from datetime import datetime, timezone
import pytest
from app import feedquery
from app.models import Channel, ChannelTag, Subscription, Tag, Video, VideoState
@@ -168,9 +170,50 @@ _SUMMARY_CASES = [
"show=all&untagged=true&tags={L1}",
"show=all&q=clip",
"show=unwatched",
# A deleted tag's id (a saved view or the feed state still holds it) matches nothing.
"show=all&tags=987654",
"show=all&tags={L1}&tags=987654",
]
def test_a_deleted_tags_id_filters_to_nothing_not_to_everything(db, as_user, make_user):
"""A saved view or the feed's filter state keeps a tag id after the tag is deleted. The feed used
to drop an unknown id, so that view silently showed the WHOLE feed under its old name (E32 S2).
A filter on a tag that no longer exists matches nothing; the chip says the tag is gone."""
user, tag_ids = _seed_tagged(db, make_user)
client = as_user(user)
everything = client.get("/api/feed/count?show=all")
assert everything.status_code == 200
assert everything.json()["count"] == 8
for qs in ("show=all&tags=987654", f"show=all&tags={tag_ids['L1']}&tags=987654"):
r = client.get(f"/api/feed/count?{qs}")
assert r.status_code == 200
assert r.json()["count"] == 0, qs
r = client.get(f"/api/feed?{qs}")
assert r.status_code == 200
assert r.json()["items"] == [], qs
def test_another_users_private_tag_answers_like_one_that_does_not_exist(db, as_user, make_user):
"""The category split resolved any tag id, so a foreign PRIVATE tag landed in its real category
and the summary's facets differed from a non-existent id's: an oracle for which foreign ids exist
(E32 review). Both must now read as "missing"."""
user, tag_ids = _seed_tagged(db, make_user)
stranger = make_user(email="stranger@example.com")
private = Tag(user_id=stranger.id, name="Secret", category="language")
db.add(private)
db.commit()
assert feedquery.tags_by_category(db, user, [tag_ids["L1"], private.id]) == {
"language": [tag_ids["L1"]],
feedquery.MISSING_TAG_CATEGORY: [private.id],
}
client = as_user(user)
foreign = client.get(f"/api/feed/summary?show=all&tags={tag_ids['T1']}&tags={private.id}")
absent = client.get(f"/api/feed/summary?show=all&tags={tag_ids['T1']}&tags=987654")
assert foreign.status_code == absent.status_code == 200
assert foreign.json() == absent.json()
def test_untagged_filter_keeps_only_channels_the_user_has_not_tagged(db, as_user, make_user):
"""The "No tag" filter (item B): `untagged=true` keeps only channels with NO tag the USER personally
attached (no user-owned ChannelTag). System/auto tags (user_id NULL) don't count, and another user's
+6 -1
View File
@@ -100,7 +100,12 @@ def test_the_admin_user_list_carries_last_login(as_user, db, make_user):
def test_the_account_sees_its_own_last_sign_in(client, db, make_user):
# Settings → Account shows it (prod UAT 2026-09-19: only the admin's user list did).
make_user(email="me@example.com", password=_PW)
client.post("/auth/password-login", json={"email": "me@example.com", "password": _PW})
assert (
client.post(
"/auth/password-login", json={"email": "me@example.com", "password": _PW}
).status_code
== 200
)
stamp = client.get("/api/me").json()["last_login_at"]
+5 -5
View File
@@ -277,7 +277,7 @@ def test_a_signed_in_request_is_never_capped_and_names_its_account(sink):
client = TestClient(app)
for _ in range(logs.ANONYMOUS_LINES_PER_MINUTE + 3):
client.get("/api/me")
assert client.get("/api/me").status_code == 200
records = sink.records[-(logs.ANONYMOUS_LINES_PER_MINUTE + 3) :]
assert all(getattr(r, logs.PERSIST, True) for r in records)
assert sink.objects()[-1]["user_id"] == 42 and "user_id" not in sink.objects()[-1]["attrs"]
@@ -325,9 +325,9 @@ def test_the_log_consoles_own_reads_are_debug_but_its_failures_are_not(sink):
http = logging.getLogger("siftlode.http")
http.setLevel(logs.TRACE)
client = _client()
client.get("/api/admin/logs/facets")
assert client.get("/api/admin/logs/facets").status_code == 200
assert sink.objects()[-1]["level"] == "DEBUG"
client.get("/api/admin/logs/facets?fail=1")
assert client.get("/api/admin/logs/facets?fail=1").status_code == 400
assert sink.objects()[-1]["level"] == "WARNING"
@@ -344,7 +344,7 @@ def test_a_polled_read_is_debug_a_write_to_the_same_route_is_not(sink):
r = client.post("/api/messages/unread_count")
assert sink.objects()[-1]["level"] == "INFO"
assert "X-Log-Quiet" not in r.headers
client.get("/api/videos/abc")
assert client.get("/api/videos/abc").status_code == 200
assert sink.objects()[-1]["level"] == "INFO"
@@ -360,7 +360,7 @@ def test_every_polled_route_is_a_route_the_app_has():
def test_a_failing_request_is_logged_by_status_and_an_unhandled_error_is_logged_at_all(sink):
client = _client()
client.get("/api/videos/missing/deeper") # 404 from the router
assert client.get("/api/videos/missing/deeper").status_code == 404 # 404 from the router
assert sink.objects()[-1]["level"] == "WARNING"
client.get("/boom")
payload = sink.objects()[-1]
+1 -1
View File
@@ -259,7 +259,7 @@ def test_the_writer_drains_what_was_queued_when_it_is_stopped(db):
log.addHandler(sink)
try:
for i in range(logsink.BATCH_MAX + 5):
# noqa G004: the f-string is the POINT — a distinct rendered message per line is
# G004: the f-string is the POINT — a distinct rendered message per line is
# what keeps the flood limiter (which groups by template) out of this case.
log.warning(f"line {i}") # noqa: G004
finally:
+37 -7
View File
@@ -154,8 +154,18 @@ def test_an_access_request_records_the_language_it_was_made_in(db, client):
def test_asking_again_from_another_language_updates_the_row(db, client):
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "en"})
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "en"}
).status_code
== 200
)
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
db.expire_all()
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
@@ -165,7 +175,12 @@ def test_asking_again_from_another_language_updates_the_row(db, client):
def test_an_unsupported_language_is_ignored_rather_than_stored(db, client):
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "../../etc"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "../../etc"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
assert inv.lang is None
@@ -194,7 +209,12 @@ def test_the_confirmation_mail_is_written_in_the_accounts_language(db, client, m
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
client.post("/auth/register", json={"email": "hu@example.com", "password": _PW, "lang": "hu"})
assert (
client.post(
"/auth/register", json={"email": "hu@example.com", "password": _PW, "lang": "hu"}
).status_code
== 200
)
(_, subject, body) = sent[0]
assert subject == mailtext.MAIL_TEXT["verify_email.subject"]["hu"]
@@ -209,7 +229,12 @@ def test_the_approval_mail_answers_in_the_language_the_request_was_made_in(
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
admin = make_user(email="admin@example.com", role="admin")
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
sent.clear() # drop the admin notification; this test is about the reply to the requester
@@ -227,11 +252,16 @@ def test_a_denial_answers_in_that_language_too(db, client, make_user, as_user, m
monkeypatch.setattr(email_mod, "email_enabled", lambda db=None: True)
sent = _bodies(monkeypatch)
admin = make_user(email="admin@example.com", role="admin")
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "hu"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "hu"}
).status_code
== 200
)
inv = db.execute(select(Invite).where(Invite.email == "asking@example.com")).scalar_one()
sent.clear()
as_user(admin).post(f"/api/admin/invites/{inv.id}/deny")
assert as_user(admin).post(f"/api/admin/invites/{inv.id}/deny").status_code == 200
assert sent[0][1] == mailtext.MAIL_TEXT["access_denied.subject"]["hu"]
+182
View File
@@ -0,0 +1,182 @@
"""sync/maintenance — the job that HARD-DELETES videos (E32 S2: 19% covered).
Its lifecycle: a video missing from videos.list is flagged and hidden; after the grace period a
still-missing one is deleted (cascading to every playlist that held it) and each affected user
gets one notification; a video that answers again is unhidden. A live broadcast is never a delete
trigger, and an upcoming premiere that never went live is. YouTube is a fake that answers
videos.list from a dict.
"""
from datetime import timedelta
import pytest
from sqlalchemy import select
from app.models import Channel, Notification, Playlist, PlaylistItem, Video
from app.sync import maintenance
from app.utils import now_utc
class _YT:
def __init__(self, items: dict[str, dict]):
self.items = items
self.asked: list[list[str]] = []
def get_videos(self, ids):
self.asked.append(list(ids))
return [self.items[i] for i in ids if i in self.items]
def _item(vid: str, **snippet) -> dict:
return {"id": vid, "snippet": {"title": vid, **snippet}}
@pytest.fixture
def channel(db):
ch = Channel(id="UC_maint", title="Maint")
db.add(ch)
db.flush()
return ch
def test_a_flagged_video_that_answers_again_is_unhidden(db, channel):
v = Video(
id="v1", channel_id=channel.id, unavailable_since=now_utc(), unavailable_reason="removed"
)
db.add(v)
db.commit()
out = maintenance._recheck_flagged(db, _YT({"v1": _item("v1")}), now_utc())
db.refresh(v)
assert v.unavailable_since is None and v.unavailable_reason is None
assert out == {"checked": 1, "recovered": 1, "deleted": 0, "notified": 0}
def test_a_video_still_missing_inside_its_grace_period_is_kept_hidden(db, channel):
v = Video(
id="v1",
channel_id=channel.id,
unavailable_since=now_utc() - timedelta(days=maintenance.DEFAULT_GRACE_DAYS - 1),
unavailable_reason="removed",
)
db.add(v)
db.commit()
out = maintenance._recheck_flagged(db, _YT({}), now_utc())
assert db.get(Video, "v1") is not None
assert out["deleted"] == 0
def test_past_the_grace_period_it_is_deleted_and_each_user_who_kept_it_told_once(
db, channel, make_user
):
a, b = make_user(), make_user()
gone_since = now_utc() - timedelta(days=maintenance.DEFAULT_GRACE_DAYS + 1)
for vid in ("v1", "v2"):
db.add(
Video(
id=vid,
channel_id=channel.id,
title=f"T {vid}",
unavailable_since=gone_since,
unavailable_reason="removed",
)
)
pa = Playlist(user_id=a.id, name="A")
pb = Playlist(user_id=b.id, name="B")
db.add_all([pa, pb])
db.flush()
db.add_all(
[
PlaylistItem(playlist_id=pa.id, video_id="v1", position=0),
PlaylistItem(playlist_id=pa.id, video_id="v2", position=1),
PlaylistItem(playlist_id=pb.id, video_id="v2", position=0),
]
)
db.commit()
out = maintenance._recheck_flagged(db, _YT({}), now_utc())
db.expire_all()
assert db.get(Video, "v1") is None and db.get(Video, "v2") is None
assert db.execute(select(PlaylistItem)).scalars().all() == [] # the cascade
notes = {
n.user_id: n.data["count"]
for n in db.execute(select(Notification).where(Notification.type == "maintenance"))
.scalars()
.all()
}
assert notes == {a.id: 2, b.id: 1} # one per user, not one per video
assert out == {"checked": 2, "recovered": 0, "deleted": 2, "notified": 2}
def test_rolling_revalidation_flags_the_missing_and_keeps_a_live_stream(db, channel, monkeypatch):
monkeypatch.setattr(maintenance.sysconfig, "get_int", lambda db, key: 10)
monkeypatch.setattr(maintenance.quota, "backfill_reserve_reached", lambda: False)
db.add_all(
[
Video(id="ok", channel_id=channel.id),
Video(id="gone", channel_id=channel.id),
Video(id="live", channel_id=channel.id),
]
)
db.commit()
yt = _YT({"ok": _item("ok"), "live": _item("live", liveBroadcastContent="live")})
out = maintenance._revalidate_rolling(db, yt, now_utc())
db.expire_all()
assert db.get(Video, "gone").unavailable_reason == "removed"
assert db.get(Video, "ok").unavailable_since is None
assert db.get(Video, "live").unavailable_since is None
assert out == {"checked": 3, "flagged": 1}
assert all(db.get(Video, v).last_checked_at is not None for v in ("ok", "gone", "live"))
def test_a_run_asks_youtube_about_each_video_once(db, channel, monkeypatch, make_user):
# Phase 1 recovers v1; phase 2 must not take it as "not checked this run" and fetch it again.
monkeypatch.setattr(maintenance.sysconfig, "get_int", lambda db, key: 10)
monkeypatch.setattr(maintenance.quota, "backfill_reserve_reached", lambda: False)
yt = _YT({"v1": _item("v1"), "v2": _item("v2")})
class _Client:
def __init__(self, *_):
pass
def __enter__(self):
return yt
def __exit__(self, *_):
return False
monkeypatch.setattr(maintenance, "YouTubeClient", _Client)
monkeypatch.setattr(maintenance, "get_service_user", lambda db: make_user())
db.add_all(
[
Video(
id="v1",
channel_id=channel.id,
unavailable_since=now_utc(),
unavailable_reason="removed",
),
Video(id="v2", channel_id=channel.id),
]
)
db.commit()
maintenance.run_maintenance(db)
asked = [vid for batch in yt.asked for vid in batch]
assert sorted(asked) == ["v1", "v2"]
def test_an_upcoming_premiere_counts_as_abandoned_only_when_it_never_went_live():
past = (now_utc() - timedelta(days=3)).isoformat()
soon = (now_utc() - timedelta(days=1)).isoformat()
def item(**live):
return {"snippet": {"liveBroadcastContent": "upcoming"}, "liveStreamingDetails": live}
assert maintenance._is_abandoned_upcoming(item(scheduledStartTime=past)) is True
assert maintenance._is_abandoned_upcoming(item(scheduledStartTime=soon)) is False
assert (
maintenance._is_abandoned_upcoming(item(scheduledStartTime=past, actualStartTime=past))
is False
)
assert (
maintenance._is_abandoned_upcoming({"snippet": {"liveBroadcastContent": "none"}}) is False
)
# …and an abandoned premiere has no grace period: the confirming sweep deletes it.
assert maintenance._grace_days("abandoned") == 0
+26
View File
@@ -0,0 +1,26 @@
"""`/api/me` names the live-search source, so the search button's tooltip can say what a search costs
under the source that actually runs (the default website source spends no API quota)."""
from app import sysconfig
_PW = "correct horse battery"
def _me(client):
r = client.get("/api/me")
assert r.status_code == 200 and r.json()["authenticated"] is True
return r.json()
def test_me_names_the_running_search_source(client, db, make_user):
# /api/me resolves its user through optional_current_user, which the as_user override does not
# cover — so sign in for real (see test_me_email_flag).
user = make_user(password=_PW)
login = client.post("/auth/password-login", json={"email": user.email, "password": _PW})
assert login.status_code == 200
assert _me(client)["search_source"] == "scrape" # the default
sysconfig.set_value(db, "search_source", "api")
db.commit()
assert _me(client)["search_source"] == "api"
+1 -1
View File
@@ -97,7 +97,7 @@ def test_putting_a_removed_video_back_leaves_a_linked_list_clean(db, make_user,
pl.synced_fingerprint = current_fingerprint(db, pl)
pl.dirty = False
db.commit()
client.delete(f"/api/playlists/{pl.id}/items/b")
assert client.delete(f"/api/playlists/{pl.id}/items/b").status_code == 200
db.expire_all()
assert db.get(Playlist, pl.id).dirty is True
_add(client, pl.id, {"video_id": "b", "position": 1})
+1 -1
View File
@@ -12,8 +12,8 @@ import pytest
from app.models import Channel, Playlist, PlaylistItem, Video
from app.sync import playlists as pl_mod
from app.sync.playlist_merge import merge_plan
from app.sync.playlists import (
merge_plan,
merge_playlist,
parse_baseline,
parse_unavailable,
+154
View File
@@ -0,0 +1,154 @@
"""routes/plex/admin — the connection test and the owner's watch-sync link (E32 S2: 37% covered).
The connection test is the admin's one early warning that the media mount or path map is wrong, and
it must not quote the server's URL back in a toast. The watch link runs the one-time "Plex is master"
import exactly once. Plex itself is a stub.
"""
from sqlalchemy import select
import app.routes.plex.admin as admin_routes
from app import sysconfig
from app.models import PlexLink
from app.plex.client import PlexError, PlexNotConfigured
class _Plex:
def __init__(self, *, sections=None, leaf=None, fail=None):
self._sections = sections or []
self._leaf = leaf
self._fail = fail
self.section_items_calls: list[str] = []
def __enter__(self):
if self._fail:
raise self._fail
return self
def __exit__(self, *exc):
return False
def server_info(self):
return {"friendlyName": "Basement", "version": "1.40"}
def sections(self):
return self._sections
def section_items(self, key, *_args):
self.section_items_calls.append(key)
return ([self._leaf] if self._leaf else []), 1
SECTIONS = [
{"key": 1, "title": "Movies", "type": "movie", "count": 10},
{"key": 2, "title": "Shows", "type": "show", "count": 3},
{"key": 3, "title": "Music", "type": "artist", "count": 99},
]
LEAF = {"Media": [{"Part": [{"file": "/data/movies/a.mkv"}]}]}
def _use(monkeypatch, plex):
monkeypatch.setattr(admin_routes, "PlexClient", lambda db: plex)
def test_the_connection_test_lists_movie_and_show_sections_and_probes_one_file(
db, as_user, make_user, monkeypatch, tmp_path
):
(tmp_path / "a.mkv").write_bytes(b"x")
sysconfig.set_value(db, "plex_path_map", f"/data/movies={tmp_path}")
sysconfig.set_value(db, "plex_libraries", "4")
db.commit()
# Only section 4 is enabled; the movie section listed first (1) must not be the one sampled.
other = {"key": 4, "title": "Kids", "type": "movie", "count": 2}
plex = _Plex(sections=[*SECTIONS, other], leaf=LEAF)
_use(monkeypatch, plex)
r = as_user(make_user(role="admin")).post("/api/plex/test")
assert r.status_code == 200, r.text
body = r.json()
assert body["server_name"] == "Basement"
assert [s["key"] for s in body["sections"]] == ["1", "2", "4"] # music is not ours
assert body["media_check"] == {
"checked": True,
"ok": True,
"plex_path": "/data/movies/a.mkv",
"local_path": str(tmp_path / "a.mkv"),
}
assert plex.section_items_calls == ["4"] # only an enabled movie section is sampled
def test_a_wrong_mount_is_reported_and_an_empty_library_is_not_checked(
db, as_user, make_user, monkeypatch, tmp_path
):
sysconfig.set_value(db, "plex_path_map", f"/data/movies={tmp_path}")
db.commit()
client = as_user(make_user(role="admin"))
_use(monkeypatch, _Plex(sections=SECTIONS, leaf=LEAF)) # the file is not on the mount
check = client.post("/api/plex/test").json()["media_check"]
assert check["checked"] is True and check["ok"] is False
_use(monkeypatch, _Plex(sections=SECTIONS))
assert client.post("/api/plex/test").json()["media_check"] == {"checked": False}
def test_a_failed_connection_says_so_without_quoting_the_server(
db, as_user, make_user, monkeypatch
):
client = as_user(make_user(role="admin"))
_use(monkeypatch, _Plex(fail=PlexNotConfigured()))
r = client.post("/api/plex/test")
assert r.status_code == 400
assert r.json()["detail"]["reason"] == "plex_not_configured"
_use(monkeypatch, _Plex(fail=PlexError("GET http://10.0.0.5:32400/secret refused")))
r = client.post("/api/plex/test")
assert r.status_code == 422
assert r.json()["detail"]["reason"] == "plex_unreachable"
assert "10.0.0.5" not in r.text
def test_plex_admin_routes_refuse_a_regular_user(db, as_user, make_user, monkeypatch):
_use(monkeypatch, _Plex(sections=SECTIONS))
client = as_user(make_user())
for path in ("/api/plex/test", "/api/plex/sync", "/api/plex/watch/import"):
assert client.post(path).status_code == 403, path
assert client.post("/api/plex/watch/link", json={"enabled": True}).status_code == 403
def test_the_watch_link_imports_once_and_needs_plex_on(db, as_user, make_user, monkeypatch):
imports: list[int] = []
def fake_import(db_, link):
imports.append(link.user_id)
link.initial_import_done = True
db_.commit()
return {"imported": 7}
monkeypatch.setattr(admin_routes.plex_watch, "import_owner_watch_state", fake_import)
owner = make_user(role="admin")
client = as_user(owner)
r = client.post("/api/plex/watch/link", json={"enabled": True})
assert r.status_code == 409 and r.json()["detail"]["reason"] == "plex_disabled"
assert client.post("/api/plex/watch/import").json()["detail"]["reason"] == "watch_sync_off"
sysconfig.set_value(db, "plex_enabled", True)
db.commit()
first = client.post("/api/plex/watch/link", json={"enabled": True})
assert first.status_code == 200
assert first.json()["import"] == {"imported": 7}
assert first.json()["linked"] and first.json()["sync_enabled"] and first.json()["uses_admin"]
assert client.post("/api/plex/watch/link", json={"enabled": False}).json()["import"] is None
again = client.post("/api/plex/watch/link", json={"enabled": True})
assert again.json()["import"] is None # the one-time import does not run twice
assert imports == [owner.id]
status = client.get("/api/plex/watch/link")
assert status.status_code == 200
assert status.json()["initial_import_done"] is True
links = db.execute(select(PlexLink).where(PlexLink.user_id == owner.id)).scalars().all()
assert len(links) == 1 # toggling never creates a second link
rerun = client.post("/api/plex/watch/import")
assert rerun.status_code == 200 and rerun.json()["import"] == {"imported": 7}
assert imports == [owner.id, owner.id]
+204
View File
@@ -0,0 +1,204 @@
"""plex/client — the one door to the Plex server (E32 S2: 31% covered).
What matters here is what the callers cannot see: a failure must surface as PlexError (never as an
empty answer), the paged reads must stop where they should, and the write URIs must be well formed.
The network is an httpx MockTransport; the configuration is stubbed.
"""
import httpx
import pytest
from app.plex import client as plex_client
from app.plex.client import PlexClient, PlexError, PlexNotConfigured
@pytest.fixture
def plex(monkeypatch):
"""A PlexClient on a stub server; set `routes[path] = handler-or-json` per test."""
routes: dict = {}
seen: list[httpx.Request] = []
def transport(request: httpx.Request) -> httpx.Response:
seen.append(request)
answer = routes.get(request.url.path)
if answer is None:
return httpx.Response(404)
if callable(answer):
return answer(request)
return httpx.Response(200, json={"MediaContainer": answer})
real = httpx.Client
monkeypatch.setattr(
plex_client.httpx,
"Client",
lambda **kw: real(transport=httpx.MockTransport(transport), **kw),
)
conf = {"plex_server_url": "http://plex.test:32400/", "plex_server_token": "tok"}
monkeypatch.setattr(plex_client.sysconfig, "get_str", lambda db, key: conf[key])
c = PlexClient(None) # type: ignore[arg-type] # the stubbed config never reads the session
c.routes, c.seen = routes, seen # type: ignore[attr-defined]
yield c
c.close()
def test_an_unconfigured_server_is_refused_up_front(monkeypatch):
monkeypatch.setattr(plex_client.sysconfig, "get_str", lambda db, key: "")
with pytest.raises(PlexNotConfigured):
PlexClient(None) # type: ignore[arg-type]
def test_reads_carry_the_token_and_unwrap_the_container(plex):
plex.routes["/library/sections"] = {"Directory": [{"key": "1", "type": "movie"}]}
assert plex.sections() == [{"key": "1", "type": "movie"}]
req = plex.seen[-1]
assert str(req.url) == "http://plex.test:32400/library/sections"
assert req.headers["x-plex-token"] == "tok"
assert req.headers["accept"] == "application/json"
@pytest.mark.parametrize(
"answer",
[
lambda r: httpx.Response(500),
lambda r: httpx.Response(200, text="<html>login</html>"),
lambda r: (_ for _ in ()).throw(httpx.ConnectError("down", request=r)),
],
ids=["http-500", "not-json", "unreachable"],
)
def test_a_failed_read_is_a_plex_error_never_an_empty_answer(plex, answer):
plex.routes["/library/sections"] = answer
with pytest.raises(PlexError):
plex.sections()
def test_a_section_page_asks_for_guids_and_reports_the_server_total(plex):
plex.routes["/library/sections/7/all"] = {"Metadata": [{"ratingKey": "1"}], "totalSize": 250}
items, total = plex.section_items("7", 1, 200, 50)
assert items == [{"ratingKey": "1"}] and total == 250
params = plex.seen[-1].url.params
assert params["includeGuids"] == "1" and params["type"] == "1"
assert params["X-Plex-Container-Start"] == "200" and params["X-Plex-Container-Size"] == "50"
# Measured on the real server (E32 review): a paged request carries `totalSize` even when the
# page is empty, past the end, or the section holds nothing — `size: 0, totalSize: 0`.
plex.routes["/library/sections/7/all"] = {"Metadata": [], "size": 0, "totalSize": 0}
assert plex.section_items("7", 2) == ([], 0)
# No total at all is not "0": the caller must not read it as a complete, empty section.
plex.routes["/library/sections/7/all"] = {"Metadata": []}
assert plex.section_items("7", 2) == ([], None)
# …and `size` is THIS page's count, never the section's: one full page is not the whole section.
plex.routes["/library/sections/7/all"] = {"Metadata": [{"ratingKey": "1"}], "size": 1}
assert plex.section_items("7", 2) == ([{"ratingKey": "1"}], None)
def test_related_flattens_the_hubs_and_degrades_to_empty(plex):
plex.routes["/library/metadata/5/related"] = {
"Hub": [{"Metadata": [{"ratingKey": "a"}]}, {"Metadata": [{"ratingKey": "b"}]}, {}]
}
assert [m["ratingKey"] for m in plex.related("5")] == ["a", "b"]
plex.routes["/library/metadata/5/related"] = lambda r: httpx.Response(500)
assert plex.related("5") == []
def test_metadata_is_the_first_item_or_none(plex):
plex.routes["/library/metadata/9"] = {"Metadata": [{"ratingKey": "9", "title": "X"}]}
assert plex.metadata("9", markers=True) == {"ratingKey": "9", "title": "X"}
assert plex.seen[-1].url.params["includeMarkers"] == "1"
plex.routes["/library/metadata/9"] = {}
assert plex.metadata("9") is None
def _history(rows_by_start: dict[int, list[dict]]):
def answer(request: httpx.Request) -> httpx.Response:
start = int(request.url.params["X-Plex-Container-Start"])
return httpx.Response(
200, json={"MediaContainer": {"Metadata": rows_by_start.get(start, [])}}
)
return answer
def test_watch_history_pages_until_the_cutoff_and_keeps_only_the_owner(plex):
plex.routes["/status/sessions/history/all"] = _history(
{
0: [
{"ratingKey": "1", "viewedAt": 500, "accountID": 2},
{"ratingKey": "2", "viewedAt": 400, "accountID": 1},
],
2: [
{"ratingKey": "3", "viewedAt": 300, "accountID": 1},
{"ratingKey": "4", "viewedAt": 100, "accountID": 1},
],
4: [{"ratingKey": "never", "viewedAt": 50, "accountID": 1}],
}
)
rows = plex.watch_history(min_viewed_at=200, account_id=1, size=2)
assert [r["ratingKey"] for r in rows] == ["2", "3"] # page 2 reached, the cutoff stops it
assert [int(r.url.params["X-Plex-Container-Start"]) for r in plex.seen] == [0, 2]
def test_watch_history_stops_on_a_short_page_and_at_the_page_cap(plex):
plex.routes["/status/sessions/history/all"] = _history({0: [{"ratingKey": "1", "viewedAt": 9}]})
assert [r["ratingKey"] for r in plex.watch_history(size=2)] == ["1"]
assert len(plex.seen) == 1
full = [{"ratingKey": "r", "viewedAt": 9}, {"ratingKey": "r", "viewedAt": 9}]
plex.routes["/status/sessions/history/all"] = lambda r: httpx.Response(
200, json={"MediaContainer": {"Metadata": full}}
)
plex.seen.clear()
assert len(plex.watch_history(size=2, max_pages=3)) == 6
assert len(plex.seen) == 3
def test_a_collection_is_created_with_a_well_formed_uri(plex):
plex.routes["/identity"] = {"machineIdentifier": "abc"}
plex.routes["/library/collections"] = {"Metadata": [{"ratingKey": "c1"}]}
assert plex.create_collection("3", "Best of: 90s", "42") == {"ratingKey": "c1"}
params = plex.seen[-1].url.params
assert plex.seen[-1].method == "POST"
assert params["title"] == "Best of: 90s" and params["sectionId"] == "3"
assert params["uri"] == "server://abc/com.plexapp.plugins.library/library/metadata/42"
plex.routes["/library/collections"] = {}
with pytest.raises(PlexError):
plex.create_collection("3", "Empty", "42")
assert [r.url.path for r in plex.seen].count("/identity") == 1 # cached
def test_a_server_without_a_machine_id_refuses_the_write(plex):
"""No machineIdentifier used to become `server:///…` in the write URI — a malformed request
Plex answers in its own way. It is refused here, named (E32 S2)."""
plex.routes["/identity"] = {}
plex.routes["/library/collections"] = {"Metadata": [{"ratingKey": "c1"}]}
with pytest.raises(PlexError, match="machine"):
plex.create_collection("3", "T", "42")
assert all(r.url.path == "/identity" for r in plex.seen)
def test_watch_state_writes_target_the_library_item(plex):
for path in ("/:/scrobble", "/:/unscrobble", "/:/timeline"):
plex.routes[path] = lambda r: httpx.Response(200) # Plex answers with an empty body
plex.scrobble("42")
assert plex.seen[-1].url.params["key"] == "42"
plex.unscrobble("42")
assert plex.seen[-1].url.path == "/:/unscrobble"
plex.set_timeline("42", 61_500.7, 120_000) # type: ignore[arg-type]
p = plex.seen[-1].url.params
assert p["key"] == "/library/metadata/42" and p["time"] == "61500" and p["state"] == "stopped"
plex.routes["/:/scrobble"] = lambda r: httpx.Response(401)
with pytest.raises(PlexError):
plex.scrobble("42")
def test_an_image_is_resized_by_plex_when_a_box_is_given(plex):
plex.routes["/photo/:/transcode"] = lambda r: httpx.Response(
200, content=b"small", headers={"Content-Type": "image/webp"}
)
plex.routes["/library/metadata/1/thumb/2"] = lambda r: httpx.Response(200, content=b"full")
assert plex.image_bytes("/library/metadata/1/thumb/2", 300, 450) == (b"small", "image/webp")
assert plex.seen[-1].url.params["url"] == "/library/metadata/1/thumb/2"
body, ctype = plex.image_bytes("/library/metadata/1/thumb/2")
assert body == b"full" and ctype == "image/jpeg"
with pytest.raises(PlexError):
plex.raw_get("/library/streams/404")
+1 -1
View File
@@ -67,7 +67,7 @@ def test_each_variant_is_cut_at_its_own_size(db, make_user, monkeypatch, tmp_pat
]
# Each cut is cached under its own name: asking again reaches Plex no more.
for variant in ("thumb", "thumb_sm", "art", "art_sm"):
client.get(f"/api/plex/image/pm1?variant={variant}")
assert client.get(f"/api/plex/image/pm1?variant={variant}").status_code == 200
assert len(_FakePlex.calls) == 4
+52 -11
View File
@@ -16,18 +16,21 @@ from app import sysconfig
from app.db import engine
from app.models import PlexItem, PlexLibrary, PlexShow
from app.plex.genres import hu_genre, hu_genres
from app.plex.sync import _apply_item, _canonical_guid, _sync_overlay
from app.plex.overlay import sync_overlay
from app.plex.sync import _apply_item, _canonical_guid
from app.plex.query import content_libs
class _StubPlex:
"""Minimal PlexClient stand-in: _sync_overlay only calls section_items via _paginate."""
"""Minimal PlexClient stand-in: sync_overlay only calls section_items via _paginate. `total`
over the item count is a listing cut short (the server said more than it handed over)."""
def __init__(self, items: list[dict]):
def __init__(self, items: list[dict], total: int | None = None):
self._items = items
self._total = len(items) if total is None else total
def section_items(self, key, item_type, start, size):
return self._items[start : start + size], len(self._items)
return self._items[start : start + size], self._total
def _hu_movie(rk: str, title: str, summary: str, genres: list[str], guid: str | None) -> dict:
@@ -71,7 +74,7 @@ def test_sync_overlay_writes_hu_onto_matched_en_rows(db):
]
)
stats = {"overlaid": 0}
_sync_overlay(db, stub, hu_lib, en_lib.id, stats)
sync_overlay(db, stub, hu_lib, en_lib.id, stats)
db.commit()
db.refresh(matched)
@@ -185,13 +188,51 @@ def test_overlay_reconcile_clears_stale_hu_when_twin_vanishes(db):
)
db.add(row)
db.commit()
# The HU section no longer lists this film (empty overlay).
_sync_overlay(db, _StubPlex([]), hu_lib, en_lib.id, {"overlaid": 0})
# The HU section, read in full, no longer lists this film (only another one).
other = _hu_movie("h9", "Más", "más", [], "tmdb://99")
sync_overlay(db, _StubPlex([other]), hu_lib, en_lib.id, {"overlaid": 0})
db.commit()
db.refresh(row)
assert row.title_hu is None and row.summary_hu is None and row.genres_hu is None
def test_a_listing_cut_short_keeps_the_hungarian_text_it_did_not_reach(db):
"""E32 S2: the reconcile clears HU text only after a COMPLETE listing. A walk a Plex hiccup cut
short (an empty page before the server's total), or an empty section, says nothing about the
titles it did not reach — clearing them dropped their Hungarian text until the next full sync."""
en_lib = PlexLibrary(plex_key="4", title="Movies", kind="movie")
hu_lib = PlexLibrary(plex_key="9", title="Filmek", kind="movie", metadata_lang="hu")
db.add_all([en_lib, hu_lib])
db.flush()
hu_lib.overlay_of = en_lib.id
kept = PlexItem(
rating_key="m1",
library_id=en_lib.id,
kind="movie",
title="Heat",
guid="tmdb:1",
title_hu="Szemtől szemben",
summary_hu="régi",
genres_hu=["Akció"],
)
fresh = PlexItem(rating_key="m2", library_id=en_lib.id, kind="movie", title="Up", guid="tmdb:2")
db.add_all([kept, fresh])
db.commit()
stats: dict = {}
reached = _hu_movie("h2", "Fel", "fel", [], "tmdb://2")
sync_overlay(db, _StubPlex([reached], total=2), hu_lib, en_lib.id, stats)
db.commit()
db.refresh(kept)
db.refresh(fresh)
assert kept.title_hu == "Szemtől szemben" and kept.summary_hu == "régi"
assert fresh.title_hu == "Fel" # what the listing did reach is still applied
assert stats["overlay_incomplete"] == 1
sync_overlay(db, _StubPlex([]), hu_lib, en_lib.id, {}) # an empty section: the same answer
db.commit()
db.refresh(kept)
assert kept.title_hu == "Szemtől szemben"
def test_apply_item_keeps_prior_guid_when_listing_lacks_guid():
row = PlexItem(rating_key="m1", kind="movie", guid="tmdb:1")
_apply_item(row, lib_id=7, kind="movie", meta={"title": "Heat"}) # no Guid array
@@ -225,7 +266,7 @@ def test_overlay_applies_to_all_editions_sharing_a_guid(db):
]
)
stats = {"overlaid": 0}
_sync_overlay(db, stub, hu_lib, en_lib.id, stats)
sync_overlay(db, stub, hu_lib, en_lib.id, stats)
db.commit()
db.refresh(ed1)
db.refresh(ed2)
@@ -291,7 +332,7 @@ def test_a_show_overlay_writes_hungarian_episode_titles_and_summaries(db):
],
}
)
_sync_overlay(db, stub, hu_lib, en_lib.id, {})
sync_overlay(db, stub, hu_lib, en_lib.id, {})
db.commit()
db.refresh(ep)
db.refresh(show)
@@ -328,7 +369,7 @@ def test_the_overlay_gives_every_row_one_hungarian_word_per_genre(db):
"h1", "Szemtől szemben", "HU.", ["Action", "Bűnügyi", "Drama", "Dráma"], "tmdb://1"
)
meta["tagline"] = "Magyar mottó."
_sync_overlay(db, _StubPlex([meta]), hu_lib, en_lib.id, {})
sync_overlay(db, _StubPlex([meta]), hu_lib, en_lib.id, {})
db.commit()
db.refresh(twin)
db.refresh(lone)
@@ -353,7 +394,7 @@ def test_a_twin_without_genres_keeps_the_rows_own_hungarian_list(db):
)
db.add(row)
db.commit()
_sync_overlay(
sync_overlay(
db,
_StubPlex([_hu_movie("h1", "Szemtől szemben", "HU.", [], "tmdb://1")]),
hu_lib,
+82
View File
@@ -0,0 +1,82 @@
"""plex/paths — the Plex-side path → local mount translation (E32 S2: 23% covered, and a wrong answer
here reads as "media file not found" for a whole library at once, with nothing else failing).
Pure logic: the admin map is read through sysconfig, stubbed per test.
"""
from pathlib import Path
import pytest
from app.plex import paths
@pytest.fixture
def path_map(monkeypatch):
def use(raw: str) -> None:
monkeypatch.setattr(paths.sysconfig, "get_str", lambda db, key: raw)
return use
def test_the_map_parses_both_separators_drops_junk_and_puts_the_longest_prefix_first():
raw = " /data=/mnt/a ,\n/data/movies = /mnt/m\nnot-a-pair,=/x,/y=, /data/tv=/mnt/tv "
assert paths.parse_map(raw) == [
("/data/movies", "/mnt/m"),
("/data/tv", "/mnt/tv"),
("/data", "/mnt/a"),
]
assert paths.parse_map("") == []
def test_the_longest_matching_prefix_wins(path_map):
path_map("/data=/mnt/a,/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies/Foo.mkv") == str(Path("/mnt/m") / "Foo.mkv")
assert paths.map_path(None, "/data/tv/Bar.mkv") == str(Path("/mnt/a") / "tv/Bar.mkv")
def test_a_prefix_matches_only_at_a_path_boundary(path_map):
# "/data/movies2" shares the characters of "/data/movies" but is another folder.
path_map("/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies2/Foo.mkv") == "/data/movies2/Foo.mkv"
def test_the_prefix_itself_and_a_trailing_slash_in_the_map(path_map):
path_map("/data/movies/=/mnt/m")
assert paths.map_path(None, "/data/movies/Foo.mkv") == str(Path("/mnt/m") / "Foo.mkv")
path_map("/data/movies=/mnt/m")
assert paths.map_path(None, "/data/movies") == "/mnt/m"
def test_no_map_leaves_the_path_as_plex_sees_it(path_map):
path_map("")
assert paths.map_path(None, "/data/movies/Foo.mkv") == "/data/movies/Foo.mkv"
def test_the_local_file_must_exist(path_map, tmp_path):
(tmp_path / "movies").mkdir()
(tmp_path / "movies" / "Foo.mkv").write_bytes(b"x")
path_map(f"/data={tmp_path}")
assert (
paths.local_media_path(None, "/data/movies/Foo.mkv")
== (tmp_path / "movies" / "Foo.mkv").resolve()
)
assert paths.local_media_path(None, "/data/movies/Missing.mkv") is None
assert paths.local_media_path(None, "/data/movies") is None # a folder is not a file
assert paths.local_media_path(None, None) is None
assert paths.local_media_path(None, "") is None
def test_a_path_that_cannot_be_resolved_is_logged_and_answered_none(path_map, monkeypatch, caplog):
path_map("")
def broken(self, strict=False):
raise OSError("dead mount")
monkeypatch.setattr(paths.Path, "resolve", broken)
assert paths.local_media_path(None, "/data/movies/Foo.mkv") is None
rec = next(
r for r in caplog.records if getattr(r, "event", None) == "plex.paths.resolve_failed"
)
# The title stays out of the log line: the path carries it.
assert "Foo" not in rec.getMessage() and rec.attrs == {"reason": "OSError"}
@@ -105,8 +105,11 @@ def test_remove_then_put_back_is_no_change(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b", "c", "d"], extra=())
client = as_user(user)
client.post(
f"/api/plex/playlists/{pl.id}/items/remove-bulk", json={"item_rating_keys": ["b", "d"]}
assert (
client.post(
f"/api/plex/playlists/{pl.id}/items/remove-bulk", json={"item_rating_keys": ["b", "d"]}
).status_code
== 200
)
assert _order(db, pl.id)[0] == ["a", "c"]
_add(client, pl.id, ["b", "d"], [1, 3])
+41
View File
@@ -8,6 +8,7 @@ Since the sweep stopped being awaited at startup it also runs CONCURRENTLY with
second half of this file pins what keeps it off a LIVE session's segments."""
import os
import sys
from pathlib import Path
import pytest
@@ -830,6 +831,46 @@ class TestProbeVideoKind:
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
class TestProbeExitStatus:
"""A probe whose ffprobe EXITS NON-ZERO is a failed probe, not an answer (E32). Each probe used to
read stdout without looking at the exit status, so a refusing ffprobe silently picked the fallback
— HW-safe, single audio, the requested offset — and skipped the warning its except branch promises.
Drives the REAL subprocess.run with the probe's own kwargs against a stand-in command that prints a
plausible answer and then fails, so the test fails whenever the exit status is ignored."""
@staticmethod
def _failing_ffprobe(monkeypatch, stdout: str):
real_run = stream.subprocess.run
script = f"import sys; sys.stdout.write({stdout!r}); sys.exit(1)"
monkeypatch.setattr(
stream.subprocess,
"run",
lambda _cmd, **kw: real_run([sys.executable, "-c", script], **kw),
)
@staticmethod
def _events(caplog) -> list[str]:
return [getattr(r, "event", "") for r in caplog.records]
def test_video_kind(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "pix_fmt=yuv420p10le\ncolor_transfer=smpte2084\n")
with caplog.at_level("INFO"):
assert stream._probe_video_kind(Path("/x")) == stream._VK_HWSAFE
assert "plex.stream.probe_kind_failed" in self._events(caplog)
def test_audio_langs(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "hun\neng\n")
with caplog.at_level("INFO"):
assert stream._probe_audio_langs(Path("/x")) == []
assert "plex.stream.probe_audio_failed" in self._events(caplog)
def test_first_pts(self, monkeypatch, caplog):
self._failing_ffprobe(monkeypatch, "12.5\n")
with caplog.at_level("INFO"):
assert stream._probe_first_pts(Path("/x")) is None
assert "plex.stream.first_pts_probe_failed" in self._events(caplog)
class TestOpenclPresent:
"""The OpenCL-runtime probe (_opencl_present) — the precondition for the HW HDR tonemap graph. Drives
the REAL glob against a temp ICD dir instead of monkeypatching the function, so a regression to the
+1 -1
View File
@@ -208,7 +208,7 @@ def test_an_unshowable_track_says_why_and_maps_to_a_status(client, monkeypatch):
raise subtitles.SubtitleUnavailable("bitmap")
monkeypatch.setattr(subtitles, "_extract_and_publish", _bitmap)
client.get("/api/plex/subtitle/98614/0/status")
assert client.get("/api/plex/subtitle/98614/0/status").status_code == 200
pool.run_all()
st = client.get("/api/plex/subtitle/98614/0/status").json()
assert st["state"] == "failed" and st["code"] == "bitmap"
+277
View File
@@ -0,0 +1,277 @@
"""The daily full watch reconcile (`run_plex_watch_reconcile`) — the one pass that DELETES watch state:
a mirrored row Plex no longer has a record for is read as "Plex un-watched it" and cleared. E32 S2
found it trusting a scan a Plex hiccup had cut short; these hold what it may and may not clear.
"""
import pytest
from sqlalchemy import select
from app import sysconfig
from app.models import PlexItem, PlexLibrary, PlexLink, PlexState, User
from app.plex import watch_sync
class _FakePlex:
"""A PlexClient stand-in: one movie section whose listing reports `total` (more than it hands
over = a walk cut short), plus any `more` sections as `(type, leaves, total)` — listed AFTER
the movies — and a record of every write the reconcile sends back to Plex."""
def __init__(self, leaves: list[dict], total: int | None = None, more=()):
self.listings = {"1": ("movie", leaves, len(leaves) if total is None else total)}
for i, (stype, extra, extra_total) in enumerate(more, start=2):
self.listings[str(i)] = (stype, extra, extra_total)
self.calls: list[tuple] = []
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def sections(self):
return [{"type": stype, "key": key} for key, (stype, _l, _t) in self.listings.items()]
def section_items(self, key, item_type, start, size):
_stype, leaves, total = self.listings[key]
return leaves[start : start + size], total
def scrobble(self, rk):
self.calls.append(("scrobble", rk))
def unscrobble(self, rk):
self.calls.append(("unscrobble", rk))
def set_timeline(self, rk, time_ms, duration_ms):
self.calls.append(("timeline", rk, time_ms))
@pytest.fixture
def world(db):
"""Five mirrored movies and the owner's link; each test sets the local states it needs."""
sysconfig.set_value(db, "plex_enabled", True)
lib = PlexLibrary(plex_key="1", title="Movies", kind="movie", enabled=True)
user = User(email="reconcile@example.com")
db.add_all([lib, user])
db.flush()
items = {}
for rk in ("m1", "m2", "m3", "m4", "m5"):
items[rk] = PlexItem(
rating_key=rk, library_id=lib.id, kind="movie", title=rk, duration_s=6000
)
db.add(items[rk])
db.add(
PlexLink(
user_id=user.id,
uses_admin=True,
sync_enabled=True,
initial_import_done=True,
plex_account_id=1,
)
)
db.flush()
def state(rk, **kw):
db.add(PlexState(user_id=user.id, item_id=items[rk].id, **kw))
def rows():
db.expire_all()
by_id = {i.id: rk for rk, i in items.items()}
return {
by_id[s.item_id]: (s.status, s.position_seconds, s.synced_to_plex)
for s in db.execute(select(PlexState).where(PlexState.user_id == user.id)).scalars()
if s.item_id in by_id # the five movies; a test's own extra items it reads itself
}
return state, rows
def _run(db, monkeypatch, plex):
monkeypatch.setattr(watch_sync, "PlexClient", lambda _db: plex)
return watch_sync.run_plex_watch_reconcile(db)
def _local_states(state):
state("m1", status="watched", synced_to_plex=True) # mirrored; Plex has no record now
state("m2", status="watched", synced_to_plex=False) # local change that never reached Plex
state("m3", status="hidden", synced_to_plex=True) # Siftlode-only, never cleared
state("m5", status="new", position_seconds=300, synced_to_plex=True) # Plex says otherwise
def test_a_complete_scan_clears_what_plex_unwatched_and_keeps_the_rest(db, monkeypatch, world):
state, rows = world
_local_states(state)
db.commit()
plex = _FakePlex(
[
{"ratingKey": "m3"}, # listed, no watch signal
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
]
)
out = _run(db, monkeypatch, plex)
assert rows() == {
"m2": ("watched", 0, True), # re-pushed, then settled
"m3": ("hidden", 0, True),
"m4": ("watched", 0, True), # pulled in
"m5": ("new", 120, True), # Plex's position wins
}
assert ("scrobble", "m2") in plex.calls
assert out["cleared"] == 1 and out["repushed"] == 1 and "incomplete" not in out
def test_a_scan_cut_short_clears_nothing(db, monkeypatch, world):
state, rows = world
_local_states(state)
db.commit()
# The server said 5, the walk got 3: m1 may simply not have been reached.
plex = _FakePlex(
[
{"ratingKey": "m3"},
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
],
total=5,
)
out = _run(db, monkeypatch, plex)
after = rows()
assert after["m1"] == ("watched", 0, True) # NOT un-watched on a partial picture
assert after["m4"] == ("watched", 0, True) and after["m5"] == ("new", 120, True) # pulls stand
assert after["m2"] == ("watched", 0, True) # the dirty row is still pushed up
assert out["cleared"] == 0 and out["incomplete"] == 1
_LISTED = [
{"ratingKey": "m3"},
{"ratingKey": "m4", "viewCount": 1, "lastViewedAt": 1_700_000_000},
{"ratingKey": "m5", "viewOffset": 120_000, "lastViewedAt": 1_700_000_000},
]
def test_a_genuinely_empty_second_section_still_lets_the_reconcile_clear(db, monkeypatch, world):
# The server SAYS the shows section holds 0: that is a complete listing, not a hiccup — before
# the E32 review it read as "incomplete" forever and un-watches never synced again.
state, rows = world
_local_states(state)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
assert "m1" not in rows()
assert out["cleared"] == 1 and "incomplete" not in out
def _watched_episode_in_a_shows_section(db) -> int:
"""A second mirrored section (key "2") holding one episode the owner has watched, synced."""
shows = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(shows)
db.flush()
ep = PlexItem(rating_key="e1", library_id=shows.id, kind="episode", title="e1")
db.add(ep)
db.flush()
user_id = db.execute(select(PlexLink.user_id)).scalar_one()
db.add(PlexState(user_id=user_id, item_id=ep.id, status="watched", synced_to_plex=True))
return ep.id
def _state_of(db, item_id: int) -> str | None:
db.expire_all()
st = db.execute(select(PlexState).where(PlexState.item_id == item_id)).scalar_one_or_none()
return st.status if st else None
def test_a_section_we_mirror_that_says_0_keeps_its_rows_and_does_not_hold_up_the_rest(
db, monkeypatch, world
):
# A total of 0 from a section our items come from is a Plex that lost the library for a moment,
# not one that un-watched all of it (E32 review round 2): its rows stay. And it is decided per
# section: before round 3 one such section (a library emptied on Plex, its mirrored items kept)
# made EVERY run incomplete, and un-watches in every other section never synced again.
state, rows = world
_local_states(state)
ep_id = _watched_episode_in_a_shows_section(db)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
assert _state_of(db, ep_id) == "watched"
assert "m1" not in rows() # the movies section was walked whole: Plex un-watched m1
assert out["cleared"] == 1 and out["incomplete"] == 1
def test_a_library_the_allowlist_leaves_out_keeps_its_watch_states(db, monkeypatch, world):
# Its items are absent from every scan because the scan never walks it — not because Plex
# un-watched them. It cleared them on every run (E32 review round 2).
state, _rows = world
other = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(other)
db.flush()
ep = PlexItem(rating_key="e1", library_id=other.id, kind="episode", title="e1")
db.add(ep)
db.flush()
user_id = db.execute(select(PlexLink.user_id)).scalar_one()
db.add(PlexState(user_id=user_id, item_id=ep.id, status="watched", synced_to_plex=True))
_local_states(state)
sysconfig.set_value(db, "plex_libraries", "1")
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], 0)]))
db.expire_all()
left = {
rk
for rk in db.execute(
select(PlexItem.rating_key).join(PlexState, PlexState.item_id == PlexItem.id)
).scalars()
}
assert "m1" not in left # the walked section still clears
kept = db.execute(select(PlexState).where(PlexState.item_id == ep.id)).scalar_one_or_none()
assert kept is not None and kept.status == "watched"
assert out["cleared"] == 1
def test_a_cut_short_section_is_not_rescued_by_a_complete_one_after_it(db, monkeypatch, world):
# Each section settles on its own walk: movies cut short (5 said, 3 handed), then a complete
# shows listing — m1 may simply not have been reached.
state, rows = world
_local_states(state)
db.commit()
shows = [{"ratingKey": "s1"}]
out = _run(db, monkeypatch, _FakePlex(_LISTED, total=5, more=[("show", shows, 1)]))
assert rows()["m1"] == ("watched", 0, True)
assert out["cleared"] == 0 and out["incomplete"] == 1
def test_a_listing_without_a_total_is_never_complete(db, monkeypatch, world):
state, rows = world
_local_states(state)
ep_id = _watched_episode_in_a_shows_section(db)
db.commit()
out = _run(db, monkeypatch, _FakePlex(_LISTED, more=[("show", [], None)]))
assert _state_of(db, ep_id) == "watched" # its section never said how much it holds
assert "m1" not in rows()
assert out["cleared"] == 1 and out["incomplete"] == 1
def test_an_empty_listing_is_not_a_library_of_unwatched_titles(db, monkeypatch, world):
state, rows = world
state("m1", status="watched", synced_to_plex=True)
db.commit()
out = _run(db, monkeypatch, _FakePlex([]))
assert rows() == {"m1": ("watched", 0, True)}
assert out["cleared"] == 0
def test_a_dirty_resume_is_re_pushed_as_a_timeline_and_a_dirty_reset_as_an_unscrobble(
db, monkeypatch, world
):
state, rows = world
state("m1", status="new", position_seconds=300, synced_to_plex=False)
state("m2", status="new", position_seconds=2, synced_to_plex=False) # under the 5 s floor
db.commit()
plex = _FakePlex([{"ratingKey": rk} for rk in ("m1", "m2", "m3", "m4", "m5")])
_run(db, monkeypatch, plex)
assert ("timeline", "m1", 300_000) in plex.calls
assert ("unscrobble", "m2") in plex.calls
assert {rk: synced for rk, (_s, _p, synced) in rows().items()} == {"m1": True, "m2": True}
def test_watch_fields_map_to_a_state_only_when_plex_holds_a_signal():
to = watch_sync._plex_watch_to_state
assert to(None, None, None) is None
assert to(0, 4_000, None) is None # a stray few seconds is not "in progress"
assert to(0, 90_000, 1_700_000_000)[:2] == ("new", 90)
assert to("2", 90_000, 1_700_000_000)[:2] == ("watched", 0) # watched wins over an offset
+2 -2
View File
@@ -9,7 +9,7 @@ independent literal, there was no `.python-version`, no `requires-python`, and r
**The contract is a chain, and it is split across two test files because no single runner can see
every link.** This suite runs inside the test image, which mounts `backend/` and nothing else, so it
cannot read the repo-root Dockerfile or the workflow. It therefore holds the link it CAN see, and the
cannot read the repo-root Dockerfile. It therefore holds the link it CAN see, and the
one that matters most here:
the interpreter actually running == backend/pyrightconfig.json == backend/ruff.toml
@@ -17,7 +17,7 @@ one that matters most here:
and `frontend/src/lib/pythonEngine.contract.test.ts` — which runs on the host, with the whole tree —
holds the rest of the chain:
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test == check.yml
/.python-version == pyrightconfig.json == ruff.toml == Dockerfile == Dockerfile.test
Together they close the loop. To move the version: edit `/.python-version`, then follow the failures
in both files. Every place they name is a place that would otherwise have been forgotten.
+46
View File
@@ -0,0 +1,46 @@
"""`quota_events` has a retention: the daily history-cleanup job (`audit_gc`) prunes rows older than
`quota_retention_days`, as it does the audit log. Without it the table only ever grew — one row per
API spend plus a zero-unit row per live search."""
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app import scheduler, sysconfig
from app.models import QuotaEvent
def _event(days_ago: int) -> QuotaEvent:
return QuotaEvent(
user_id=None,
action="videos.list",
units=1,
created_at=datetime.now(timezone.utc) - timedelta(days=days_ago),
)
def _remaining(db) -> list[int]:
db.expire_all()
return sorted(e.units for e in db.execute(select(QuotaEvent)).scalars())
def test_the_history_cleanup_prunes_quota_events_past_the_retention(db):
old, recent = _event(400), _event(10)
recent.units = 7
db.add_all([old, recent])
db.commit()
scheduler.JOB_FUNCS["audit_gc"]()
assert _remaining(db) == [7] # the default keeps a year, which the admin usage view can show
def test_a_zero_retention_keeps_quota_history_forever(db):
db.add(_event(4000))
db.commit()
sysconfig.set_value(db, "quota_retention_days", 0)
db.commit()
scheduler.JOB_FUNCS["audit_gc"]()
assert _remaining(db) == [1]
+6 -1
View File
@@ -51,7 +51,12 @@ def test_a_link_whose_account_was_deleted_is_reported_dead(client, db, make_user
def test_a_spent_link_is_reported_dead(client, db, make_user):
raw = _issue(db, make_user(email="spent@example.com", password=_PW))
client.post("/auth/password-reset/confirm", json={"token": raw, "password": _NEW})
assert (
client.post(
"/auth/password-reset/confirm", json={"token": raw, "password": _NEW}
).status_code
== 200
)
assert client.post("/auth/password-reset/check", json={"token": raw}).json() == {"ok": False}
+58
View File
@@ -0,0 +1,58 @@
"""No test sends a request and throws the response away (E32 S2).
A discarded `client.post(...)` passes whatever it got back. As a SETUP step, a failure leaves the
test asserting on a state it never reached; as the step UNDER test, the test proves nothing. E32
found `test_youtube_sources_never_consult_the_guard` passing on a 400 that stopped the request
before the code it meant to exercise, and 53 more discarded responses across the suite. Each one
now states the status it expects; this keeps it that way.
"""
import ast
from pathlib import Path
TESTS = Path(__file__).resolve().parent
VERBS = {"get", "post", "put", "patch", "delete", "head", "options", "request"}
def discarded_requests(source: str) -> list[int]:
"""Line numbers of the statements that are a bare request call on an app path."""
hits = []
for node in ast.walk(ast.parse(source)):
if not (isinstance(node, ast.Expr) and isinstance(node.value, ast.Call)):
continue
func = node.value.func
if not (isinstance(func, ast.Attribute) and func.attr in VERBS):
continue
path = node.value.args[0] if node.value.args else None
literal = path.value if isinstance(path, ast.Constant) else None
if isinstance(path, ast.JoinedStr):
first = path.values[0] if path.values else None
literal = first.value if isinstance(first, ast.Constant) else None
if isinstance(literal, str) and literal.startswith(("/api", "/auth")):
hits.append(node.lineno)
return hits
def test_the_detector_sees_a_discarded_request_and_passes_a_checked_one():
src = (
"client.post('/api/x')\n"
"client.get(f'/auth/{u}')\n"
"assert client.post('/api/x').status_code == 200\n"
"r = client.get('/api/y')\n"
"items.get('/api/not-a-request-but-a-key')\n"
)
# The last line is a false positive by design: a dict keyed by an app path is not a thing the
# suite does, and a narrower match would miss a client held under any other name.
assert discarded_requests(src) == [1, 2, 5]
def test_no_test_discards_a_response():
offenders = [
f"{p.name}:{line}"
for p in sorted(TESTS.rglob("test_*.py"))
for line in discarded_requests(p.read_text(encoding="utf-8"))
]
assert offenders == [], (
"assert the status these requests expect (`assert client.post(...).status_code == 200`), "
f"or keep the response and check it: {offenders}"
)
+38 -13
View File
@@ -100,15 +100,22 @@ def test_a_later_registration_never_touches_an_existing_password(db, client, smt
have installed a password where the whole point is that there is none. Reverted; the owner's
recourse is password-reset, which proves the mailbox. Residual logged as SEC-16.
"""
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the first password"}
assert (
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the first password"}
).status_code
== 200
)
db.expire_all()
user = db.execute(select(User).where(User.email == "victim@example.com")).scalar_one()
before = user.password_hash
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the second password"}
assert (
client.post(
"/auth/register",
json={"email": "victim@example.com", "password": "the second password"},
).status_code
== 200
)
db.expire_all()
@@ -116,8 +123,11 @@ def test_a_later_registration_never_touches_an_existing_password(db, client, smt
def test_a_later_registration_leaves_a_link_already_in_flight_alive(db, client, smtp_on, wire):
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the first password"}
assert (
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the first password"}
).status_code
== 200
)
db.expire_all()
user = db.execute(select(User).where(User.email == "victim@example.com")).scalar_one()
@@ -126,8 +136,12 @@ def test_a_later_registration_leaves_a_link_already_in_flight_alive(db, client,
# would go green while asserting nothing the day registration stops issuing a token.
assert len(first) == 1
client.post(
"/auth/register", json={"email": "victim@example.com", "password": "the second password"}
assert (
client.post(
"/auth/register",
json={"email": "victim@example.com", "password": "the second password"},
).status_code
== 200
)
db.expire_all()
@@ -148,8 +162,12 @@ def test_a_stripped_account_keeps_its_recovery_stamp(db, client, smtp_on, make_u
password_revoked_at=datetime.now(timezone.utc),
)
client.post(
"/auth/register", json={"email": "stripped@example.com", "password": "an attacker password"}
assert (
client.post(
"/auth/register",
json={"email": "stripped@example.com", "password": "an attacker password"},
).status_code
== 200
)
db.expire_all()
@@ -169,7 +187,9 @@ def test_a_denied_sibling_is_not_reopened_while_a_pending_row_exists(db, client)
)
db.commit()
client.post("/auth/request-access", json={"email": "pairone@gmail.com"})
assert (
client.post("/auth/request-access", json={"email": "pairone@gmail.com"}).status_code == 200
)
db.expire_all()
rows = {i.email: i.status for i in db.execute(select(Invite)).scalars().all()}
@@ -195,7 +215,7 @@ def test_an_env_granted_address_cannot_be_shadowed_by_a_stranger(db, client, mon
# env grant. With the env list matched canonically the request never opens a row.
monkeypatch.setattr(settings, "allowed_emails", "bob@gmail.com")
client.post("/auth/request-access", json={"email": "b.ob@gmail.com"})
assert client.post("/auth/request-access", json={"email": "b.ob@gmail.com"}).status_code == 200
assert db.execute(select(Invite)).scalars().all() == []
assert auth_core.is_allowed(db, "bob@gmail.com") is True
@@ -331,7 +351,12 @@ def test_two_admins_two_languages_two_mails(db, client, make_user, smtp_on, wire
make_user(email="hu.admin@example.com", role="admin", preferences={"language": "hu"})
make_user(email="en.admin@example.com", role="admin", preferences={"language": "en"})
client.post("/auth/request-access", json={"email": "asking@example.com", "lang": "en"})
assert (
client.post(
"/auth/request-access", json={"email": "asking@example.com", "lang": "en"}
).status_code
== 200
)
notices = [w for w in wire if "asking@example.com" in w[1]]
assert len(notices) == 2
+17 -2
View File
@@ -53,8 +53,8 @@ def test_default_is_per_source(db, monkeypatch, override_dep):
pid = c.post("/api/saved-views", json={"name": "P", "filters": {}, "source": "plex"}).json()[
"id"
]
c.patch(f"/api/saved-views/{fid}", json={"is_default": True})
c.patch(f"/api/saved-views/{pid}", json={"is_default": True})
assert c.patch(f"/api/saved-views/{fid}", json={"is_default": True}).status_code == 200
assert c.patch(f"/api/saved-views/{pid}", json={"is_default": True}).status_code == 200
# Making the Plex view default must NOT clear the feed view's default (different source).
assert c.get("/api/saved-views?source=feed").json()[0]["is_default"] is True
assert c.get("/api/saved-views?source=plex").json()[0]["is_default"] is True
@@ -67,3 +67,18 @@ def test_duplicate_name_within_a_source_conflicts(db, monkeypatch, override_dep)
b = c.post("/api/saved-views", json={"name": "Dup", "filters": {}, "source": "plex"})
assert a.status_code == 200
assert b.status_code == 409 # same (user, source, name)
def test_a_feed_views_tag_pick_must_be_a_short_list_of_ids(db, monkeypatch, override_dep):
"""The autotag job reads every view's tag pick (E32 review): the route holds it to the shape the
feed sends, on create and on update."""
user = _user(db)
c = _client(db, user, monkeypatch, override_dep)
for bad in (5, "1", [1, "2"], [True], list(range(501))):
r = c.post("/api/saved-views", json={"name": "Bad", "filters": {"tags": bad}})
assert r.status_code == 422, (bad, r.text)
ok = c.post("/api/saved-views", json={"name": "Ok", "filters": {"tags": [1, 2]}})
assert ok.status_code == 200, ok.text
vid = ok.json()["id"]
assert c.patch(f"/api/saved-views/{vid}", json={"filters": {"tags": 5}}).status_code == 422
assert c.patch(f"/api/saved-views/{vid}", json={"filters": {"tags": []}}).status_code == 200
+178
View File
@@ -0,0 +1,178 @@
"""youtube/search_scrape — the zero-quota InnerTube search (E32 S2: the module was 20% covered).
Everything here breaks silently when YouTube moves: a parser that stops finding videos, a cursor that
stops paging, a fallback key that outlives its usefulness. The network is an httpx MockTransport.
"""
import json
import httpx
import pytest
from app.youtube import search_scrape
RESULTS_HTML = (
'<script>{"INNERTUBE_API_KEY":"live-key","INNERTUBE_CLIENT_VERSION":"2.20261001"}</script>'
)
def _video(vid: str, title: str, *, simple: bool = False) -> dict:
return {
"videoRenderer": {
"videoId": vid,
"title": {"simpleText": title}
if simple
else {"runs": [{"text": title[:3]}, {"text": title[3:]}]},
"ownerText": {
"runs": [
{
"text": "Chan",
"navigationEndpoint": {"browseEndpoint": {"browseId": "UCchan"}},
}
]
},
"thumbnail": {"thumbnails": [{"url": "small.jpg"}, {"url": "big.jpg"}]},
}
}
PAGE = {
"contents": {
"sectionListRenderer": {
"contents": [
{
"itemSectionRenderer": {
"contents": [
_video("v1", "First video"),
{"channelRenderer": {"channelId": "UCx"}}, # not a video: skipped
{
"reelShelfRenderer": {
"items": [{"reelItemRenderer": {"videoId": "s1"}}]
}
},
_video("v2", "Second", simple=True),
_video("v1", "First video"), # a repeat in another shelf: once
{"videoRenderer": {"title": {"simpleText": "no id"}}},
]
}
},
{
"continuationItemRenderer": {
"continuationEndpoint": {"continuationCommand": {"token": "NEXT"}}
}
},
]
}
}
}
@pytest.fixture
def net(monkeypatch):
"""Route every httpx.Client the module opens through a handler the test sets."""
calls: list[httpx.Request] = []
state: dict = {"handler": None}
def transport(request: httpx.Request) -> httpx.Response:
calls.append(request)
return state["handler"](request)
real = httpx.Client
monkeypatch.setattr(
search_scrape.httpx,
"Client",
lambda **kw: real(transport=httpx.MockTransport(transport), **kw),
)
monkeypatch.setattr(search_scrape, "_config", None)
state["calls"] = calls
return state
def _serve(results_html=RESULTS_HTML, search=PAGE, status=200):
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/results":
if isinstance(results_html, Exception):
raise results_html
return httpx.Response(200, text=results_html)
if isinstance(search, str):
return httpx.Response(status, text=search)
return httpx.Response(status, json=search)
return handler
def test_a_page_flattens_its_videos_in_order_and_hands_back_the_cursor(net):
net["handler"] = _serve()
page = search_scrape.search("cats")
assert page["next_page_token"] == "NEXT"
assert [i["id"] for i in page["items"]] == ["v1", "v2"]
assert page["items"][0] == {
"id": "v1",
"channel_id": "UCchan",
"channel_title": "Chan",
"title": "First video",
"published_at": None,
"thumbnail_url": "big.jpg",
"live_broadcast": "none",
}
assert page["items"][1]["title"] == "Second"
post = net["calls"][-1]
assert post.url.params["key"] == "live-key"
body = json.loads(post.content)
assert body["query"] == "cats" and body["params"] == search_scrape._TYPE_VIDEO
assert body["context"]["client"]["clientVersion"] == "2.20261001"
def test_the_next_page_sends_the_cursor_not_the_query_and_the_end_has_none(net):
net["handler"] = _serve(search={"onResponseReceivedCommands": [_video("v9", "Last")]})
page = search_scrape.search("cats", continuation="NEXT")
body = json.loads(net["calls"][-1].content)
assert body["continuation"] == "NEXT" and "query" not in body
assert [i["id"] for i in page["items"]] == ["v9"]
assert page["next_page_token"] is None
@pytest.mark.parametrize(
("search", "status"),
[
("<html>consent</html>", 200),
({"error": "blocked"}, 429),
({"responseContext": {}, "alerts": ["unusual traffic"]}, 200),
([], 200),
],
ids=["not-json", "http-error", "unknown-shape", "not-an-object"],
)
def test_a_blocked_or_garbled_answer_is_an_error_not_an_empty_page(net, search, status):
net["handler"] = _serve(search=search, status=status)
with pytest.raises(search_scrape.ScrapeError):
search_scrape.search("cats")
def test_a_search_with_no_results_is_an_empty_page_not_an_error(net):
net["handler"] = _serve(search={"contents": {"sectionListRenderer": {"contents": []}}})
assert search_scrape.search("zzzzqqq") == {"items": [], "next_page_token": None}
def test_the_live_key_is_fetched_once_and_cached(net):
net["handler"] = _serve()
search_scrape.search("a")
search_scrape.search("b")
assert [c.url.path for c in net["calls"]].count("/results") == 1
@pytest.mark.parametrize(
"results_html",
[httpx.ConnectError("down"), "<html>a consent page, no keys</html>"],
ids=["fetch-failed", "no-key-on-page"],
)
def test_a_failed_key_fetch_falls_back_for_now_and_tries_again_next_time(net, results_html):
"""The fallback pair used to be CACHED for the life of the process: one failed fetch pinned
every later search to a hard-coded key, and once that went stale search returned nothing,
with no error anywhere (E32 S2)."""
net["handler"] = _serve(results_html=results_html)
search_scrape.search("a")
assert net["calls"][-1].url.params["key"] == search_scrape._FALLBACK_KEY
net["handler"] = _serve()
search_scrape.search("b")
assert net["calls"][-1].url.params["key"] == "live-key"
+6 -2
View File
@@ -68,8 +68,12 @@ def test_using_the_link_evicts_whoever_held_the_old_password(db, as_user, make_u
admin = make_user(role="admin")
db.commit()
token = _link(as_user(admin), target.id)
client.post(
"/auth/password-reset/confirm", json={"token": token, "password": "the owner's own choice"}
assert (
client.post(
"/auth/password-reset/confirm",
json={"token": token, "password": "the owner's own choice"},
).status_code
== 200
)
db.refresh(target)
+44
View File
@@ -0,0 +1,44 @@
"""youtube/shorts.probe_is_short — three answers, and the third is not the second (E32 S2: 33% covered).
A Short answers 200 at /shorts/<id>; a regular video redirects to /watch. Anything else — an error
page, a network failure — is "undetermined, ask again", never "not a Short": that would file a Short
in the normal feed for good.
"""
import httpx
import pytest
from app.youtube import shorts
def _client(respond) -> httpx.Client:
return httpx.Client(transport=httpx.MockTransport(respond))
@pytest.mark.parametrize(
("status", "expected"),
[(200, True), (301, False), (302, False), (303, False), (307, False), (308, False)]
+ [(404, None), (429, None), (500, None)],
)
def test_the_status_decides_short_regular_or_undetermined(status, expected):
seen: list[httpx.Request] = []
def respond(request: httpx.Request) -> httpx.Response:
seen.append(request)
return httpx.Response(status, headers={"location": "https://www.youtube.com/watch?v=x"})
assert shorts.probe_is_short(_client(respond), "abc123") is expected
assert str(seen[0].url) == "https://www.youtube.com/shorts/abc123"
assert len(seen) == 1 # a redirect is the answer, never followed
def test_a_network_failure_is_undetermined_not_regular():
def respond(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectTimeout("slow", request=request)
assert shorts.probe_is_short(_client(respond), "abc123") is None
def test_the_client_skips_the_consent_interstitial():
with shorts.make_client() as c:
assert "SOCS=" in c.headers["cookie"]
+145
View File
@@ -0,0 +1,145 @@
"""The size / complexity RATCHET for the backend (E32, owner 2026-10-03 — "class size" from the quality
checklist, measured rather than argued). The frontend's twin is src/lib/sourceSize.contract.test.ts.
Two limits, each with today's offenders as a CEILING:
* a module over MAX_LINES lines — a listed one may not grow past its line count; any other module
may not cross the limit;
* a function over ruff's C901 limit (cyclomatic complexity, MAX_COMPLEXITY) — a listed one, by its
NAME (`module::function`), may not rise; no other function may cross the limit. (Ranked per
module until the E32 review: one function could grow by what another in the same module lost.)
A rename reads as a new offender plus a stale entry: move its row.
Shrinking is free; a module or function that drops UNDER the limit must be taken off its list (red
until it is), so the lists only get shorter. Raising a ceiling is a decision for the owner, with the
reason in docs/gates.md.
C901 is measured here rather than switched on in ruff.toml because ruff has no per-function
baseline: on, it would need a `noqa` on each of today's offenders, and a `noqa` lets the function
grow without limit. This keeps the ceiling on the number.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
BACKEND = Path(__file__).resolve().parent.parent
MAX_LINES = 800
MAX_COMPLEXITY = 10 # ruff's C901 default (mccabe max-complexity)
BIG_MODULES: dict[str, int] = {
"app/downloads/edit.py": 833,
"app/downloads/pipeline.py": 870,
"app/logs.py": 839,
"app/models.py": 1607,
"app/plex/stream.py": 1128,
"app/plex/subtitles.py": 1092,
"app/routes/admin.py": 864,
"app/routes/auth.py": 1353,
"app/routes/downloads.py": 1237,
}
COMPLEX: dict[str, int] = {
"app/bodylimit.py::__call__": 11,
"app/discover/hero.py::pick": 14,
"app/discover/pins.py::_shelf_card": 12,
"app/downloads/edit.py::normalize_edit_spec": 16,
"app/downloads/edit.py::run_ffmpeg": 15,
"app/downloads/formats.py::build_ydl_opts": 11,
"app/downloads/gc.py::_sweep_orphan_files": 15,
"app/downloads/gc.py::run_download_gc": 11,
"app/downloads/og.py::_jpeg_size": 12,
"app/downloads/og.py::_watch_tags": 11,
"app/downloads/pipeline.py::_download": 14,
"app/downloads/pipeline.py::_process_edit": 12,
"app/feedquery.py::feed_summary": 14,
"app/feedquery.py::filtered_query": 28,
"app/plex/query.py::meta_filter_conds": 13,
"app/plex/stream.py::start_session": 14,
"app/plex/subtitles.py::_ffmpeg_extract": 12,
"app/plex/sync.py::sync": 13,
"app/plex/tmdb.py::parse_tmdb": 11,
"app/plex/watch_sync.py::run_plex_watch_sync": 12,
"app/routes/auth.py::callback": 19,
"app/routes/debug_sessions.py::send_issue": 12,
"app/routes/downloads.py::resolve_source": 11,
"app/routes/plex/browse.py::facets": 25,
"app/routes/plex/browse.py::per_table": 13,
"app/routes/plex/browse.py::unified_library": 18,
"app/routes/plex/detail.py::item_detail": 13,
"app/routes/search.py::search_youtube": 15,
"app/sync/playlists.py::merge_playlist": 19,
"app/sync/runner.py::run_rss_poll": 13,
"app/sync/tagsuggest.py::compute_tag_suggestions": 14,
"app/worker.py::_run_supervised": 18,
}
def _line_counts() -> dict[str, int]:
return {
p.relative_to(BACKEND).as_posix(): len(p.read_text(encoding="utf-8").split("\n"))
for p in sorted((BACKEND / "app").rglob("*.py"))
}
def _complexity() -> dict[str, int]:
"""`module::function` -> its complexity; a name taken twice in a module gets ` 2`, ` 3` in
source order."""
out = subprocess.run(
[
sys.executable,
"-m",
"ruff",
"check",
"--config",
str(BACKEND / "ruff.toml"),
"--select",
"C901",
"--output-format",
"json",
"--exit-zero",
str(BACKEND / "app"),
],
capture_output=True,
text=True,
check=True,
).stdout
found: dict[str, int] = {}
for m in sorted(json.loads(out), key=lambda m: (m["filename"], m["location"]["row"])):
value = re.search(r"\((\d+) > \d+\)", m["message"])
name = re.search(r"`([^`]+)`", m["message"])
assert value and name, m["message"]
rel = Path(m["filename"]).resolve().relative_to(BACKEND).as_posix()
key, n = f"{rel}::{name.group(1)}", 2
while key in found:
key, n = f"{rel}::{name.group(1)} {n}", n + 1
found[key] = int(value.group(1))
return found
def test_no_module_crosses_the_line_limit_and_a_listed_one_does_not_grow():
over = [
f"{f}: {n} lines (ceiling {BIG_MODULES.get(f, MAX_LINES)})"
for f, n in _line_counts().items()
if n > BIG_MODULES.get(f, MAX_LINES)
]
assert over == []
def test_lists_no_module_that_is_back_under_the_limit():
lines = _line_counts()
assert [f for f in BIG_MODULES if lines.get(f, 0) <= MAX_LINES] == []
def test_no_function_crosses_the_complexity_limit_and_a_listed_one_does_not_rise():
worse = [
f"{fn}: {v} (ceiling {COMPLEX.get(fn, MAX_COMPLEXITY)})"
for fn, v in _complexity().items()
if v > COMPLEX.get(fn, MAX_COMPLEXITY)
]
assert worse == []
def test_lists_no_function_that_is_back_under_the_limit_or_gone():
actual = _complexity()
assert [fn for fn in COMPLEX if fn not in actual] == []
+6 -1
View File
@@ -229,7 +229,12 @@ def test_registering_again_does_not_create_a_second_account(db, client, monkeypa
_smtp_on(monkeypatch)
_capture(monkeypatch, "send_verify_email")
for _ in range(2):
client.post("/auth/register", json={"email": "twice@example.com", "password": _PW})
assert (
client.post(
"/auth/register", json={"email": "twice@example.com", "password": _PW}
).status_code
== 200
)
db.expire_all()
rows = db.execute(select(User).where(User.email == "twice@example.com")).scalars().all()
+109
View File
@@ -0,0 +1,109 @@
"""The worker obeys the two admin knobs that concern it — `download_worker_concurrency` and
`log_level` — as stored in the settings page, not only the env it booted with. It used to read
`settings.download_worker_concurrency` and `settings.log_level` once at start, so the admin page
showed values the worker never applied. No DB or real threads of work here: the stored values come
from a patched `sysconfig`, and the loop threads are stand-ins that only record their index."""
import threading
import pytest
from app import worker
class _NoSession:
def __enter__(self):
return None
def __exit__(self, *exc):
return False
@pytest.fixture
def stored(monkeypatch):
"""The settings page's values, served to the worker instead of a database."""
values = {"download_worker_concurrency": 3, "log_level": "DEBUG"}
monkeypatch.setattr(worker, "SessionLocal", _NoSession)
monkeypatch.setattr(worker.sysconfig, "get_int", lambda _db, key: int(values[key]))
monkeypatch.setattr(worker.sysconfig, "get_str", lambda _db, key: str(values[key]))
return values
@pytest.fixture
def levels(monkeypatch):
applied: list[str] = []
monkeypatch.setattr(worker.logs, "set_level", applied.append)
return applied
@pytest.fixture
def loops(monkeypatch):
"""Stand-in loop threads: each records its index and parks until the worker stops."""
started: list[int] = []
lock = threading.Lock()
def fake_loop(idx: int) -> None:
with lock:
started.append(idx)
worker._stop.wait()
monkeypatch.setattr(worker, "_worker_loop", fake_loop)
yield started
worker._stop.set()
@pytest.fixture(autouse=True)
def _clean_stop():
worker._stop.clear()
yield
worker._stop.clear()
def test_worker_boots_with_the_stored_concurrency_and_log_level(monkeypatch, stored, levels, loops):
monkeypatch.setattr(worker.settings, "download_worker_concurrency", 2)
monkeypatch.setattr(worker.settings, "worker_enabled", True)
monkeypatch.setattr(worker.logs, "configure_logging", lambda _source: None)
monkeypatch.setattr(worker.signal, "signal", lambda *_a: None)
monkeypatch.setattr(worker, "_wait_for_schema", lambda: True)
monkeypatch.setattr(worker.logsink, "start", lambda: None)
monkeypatch.setattr(worker, "_recover_orphans", lambda: None)
threading.Timer(0.5, worker._stop.set).start()
worker.main()
assert sorted(loops) == [0, 1, 2] # the stored 3, not the env's 2
assert levels and levels[-1] == "DEBUG"
def test_a_changed_knob_applies_without_a_restart(stored, levels, loops):
threads: dict[int, threading.Thread] = {}
level = worker._sync_knobs(threads, None)
assert level == "DEBUG"
assert worker._target == 3
stored["download_worker_concurrency"] = 5
stored["log_level"] = "WARNING"
level = worker._sync_knobs(threads, level)
assert worker._target == 5
assert sorted(threads) == [0, 1, 2, 3, 4]
assert level == "WARNING" and levels[-1] == "WARNING"
# Lowering it retires the surplus threads after their current job: they see the lower target.
stored["download_worker_concurrency"] = 1
worker._sync_knobs(threads, level)
assert worker._target == 1
def test_an_unreadable_database_keeps_what_runs(monkeypatch, levels, loops):
def boom():
raise RuntimeError("db down")
monkeypatch.setattr(worker, "SessionLocal", boom)
monkeypatch.setattr(worker, "_target", 2)
threads: dict[int, threading.Thread] = {}
assert worker._sync_knobs(threads, "INFO") == "INFO"
assert worker._target == 2
assert sorted(threads) == [0, 1] # the loops still run on the boot value
assert levels == []

Some files were not shown because too many files have changed in this diff Show More