Compare commits

..
863 Commits
Author SHA1 Message Date
peter 6ea750a698 test: the commit check starts every request clean, and the fence docs claim what it holds
The test's own writes are committed before a request, so a handler's Core DML can no longer hide
in a transaction the test had already opened. The entry-point probe answers on a marked line the
API's logging cannot mix into. A listed commit whose function no longer commits fails the fence.
CLAUDE.md and the backend README now name what an AST cannot see; the route-commit tests no longer
claim their read-back holds the commit.
2026-10-20 01:56:31 +02:00
peter c5f121139b fix: every first watch state survives a write beside it, and a folded Watch later keeps its pin
A season marked watched, a pull from Plex and a YouTube resume point made their first row by a read
then an add, and failed on the unique key when a first checkpoint or the watched mark made it in
between. They go through the same insert-then-read as the checkpoint (the resume point is one
upsert); a pull that meets a local change still waiting for its push leaves it.

Folding duplicate Watch later lists moved their videos but not a Home pin, which the cascade then
deleted; the first pin of a dropped copy moves onto the kept list.

Marking one notice read trimmed before its own flag was flushed, so the trim left one past the cap.
The route test now crosses the cap and reads the result from a session of its own.
2026-10-20 01:56:30 +02:00
peter b04b2cbd62 test: the rewritten write handlers commit, and three more owner filters hold
Each handler that now calls a service and commits is driven through HTTP once.
A second account's watch state, Watch later list and private download profile
are never read or changed. The autotag job's orphan prune is seen committed.
2026-10-20 00:45:22 +02:00
peter 3a5212cd8a fix: a first watch state and a first Watch later survive two requests at once
The first checkpoint of an item and the final flush beside it both read no
state and both inserted; the loser failed on uq_plex_user_item. Two first saves
each made a Watch later list. Both are created by an ON CONFLICT DO NOTHING
insert and then read; a partial unique index holds one Watch later per user, and
the migration folds an existing second list into the first. Plex keeps one
per-item state map (rollup.item_state_map).
2026-10-20 00:45:22 +02:00
peter b376373af5 fix(files): a folder's kind is answered before its new name, as before
A blank name on a source folder answered 400 name_required; it answers 409
not_own again.
2026-10-20 00:45:22 +02:00
peter 8206384c91 test(fence): owner filters in any form, statement writes under any name, core commits by list
The owner-filter check missed an aliased() model, an import alias, .in_() and
filter_by; four live inline filters on aliased VideoState go through owned_by now.
Cross-unit writes are seen under any import name, through sa.update and via
__table__. Core modules may commit only where CORE_COMMITS names the reason;
trim_read and audit.gc flush now and their callers commit. The pin target, the
debug-session header and a link's target fetch through owned_row. The docs
claim what the fence holds, and say what is beyond it.
2026-10-20 00:45:21 +02:00
peter e27656ee7e test: the request commit check sees Core DML and row locks
A db.execute(update/delete) fires no flush and stages nothing, so a handler whose
only write was Core DML could forget its commit and stay green. The check now
asks Postgres whether the request opened a writing transaction. It found one
real case: a repeated issue send returned with the session row still locked.
2026-10-20 00:45:20 +02:00
peter d73cdcee97 fix(capture): every process registers every table before its first query
The models now live in their units' packages, and a core model names a unit's by
string (User.subscriptions -> Subscription). The download child imports neither
catalog nor app.tables before db.get, so mapper configuration failed and every
download, clip, edit and upload job died with child_exit. Each entry point now
imports app.tables, and a test imports each one in a clean interpreter.
2026-10-20 00:45:20 +02:00
peter 5ce292cd28 test(fence): the unit fence has no exemptions, and the owner filter is held in every module
The per-file pin list and the unit-by-unit switch go: every route file is held
to zero database work, every unit's package to its edges, and core code says
whose a row is through userscope too. CLAUDE.md, the backend README and the
gate register describe the units as they now are.
2026-10-19 23:36:01 +02:00
peter ad6ccbac05 refactor(core): the core's owner filters go through userscope, each held by a second account
Tokens, notifications, quota, debug sessions and the reset-link burn filtered
their owner inline. Ten of the fifteen filters had no test that noticed one
going missing; each now has a test where another account's rows must not
count, show or change.
2026-10-19 23:36:01 +02:00
peter 531eb2eef1 refactor(plex): playlists, playback, Discover and admin through the unit; the plex package is fenced
The last Plex HTTP files do no database work of their own, so ALLOWED is empty: no route builds
SQL, touches the session or a unit's table. Discover's rails move whole into plex/discover.py; a
user's playlists, watch-state writes (the race-safe finish, an early checkpoint, a status, a
season or show at once) and the watch-sync link go through plex/queries.py and plex/service.py.

The plex package joins FENCED: it reads no other unit's tables, says whose a row is through
userscope (the rollup and the watch sync's 8 inline filters moved to owned_by), and commits only in
its named jobs — the library sync, TMDB enrichment, the watch sync and its background pushes.
A playlist's has-item probe keeps the plain key lookup, so a NUL in it is still the 422 a test pins.

Pins 178 -> 0; basedpyright baseline 118 -> 106.
2026-10-19 22:42:46 +02:00
peter 2cf1b688d9 test(plex): a collection's editable flag and the mirror's reads, committed through HTTP
The editable endpoint had no route test: dropping its commit left every test green.
2026-10-19 22:11:57 +02:00
peter 3b0e365e26 refactor(plex): the browse, detail, people and collection pages read through queries and write through the service
The HTTP files browse, detail, people, collections and _shared do no database work of their own
now. New plex/queries.py (the rows a page shows, a user's watch state through userscope),
plex/service.py (the info pages' enrichment, a collection's editable flag, the row kept after a
failed follow-up read, the person bio's background fetch; none commits), plex/library.py (the
unified browse feed, its search and facets) and plex/collections.py (a collection's source,
editability, card, certificate and member order). Every watch-state join in the moved code names
its owner through owned_by.

Pins 354 -> 178 in 4 files; basedpyright baseline 132 -> 118.
2026-10-19 22:11:20 +02:00
peter 533571f572 refactor(plex): the unit's tables move into plex/models
PlexLibrary, PlexShow, PlexSeason, PlexItem, PlexPerson, PlexCollection, PlexState, PlexLink,
PlexPlaylist and PlexPlaylistItem leave app/models.py for app/plex/models.py; app.tables registers
them. Every import is re-pointed; no schema change.
2026-10-19 21:42:07 +02:00
peter 4b6db47083 test(catalog): the channel manager's local writes, committed through HTTP 2026-10-19 02:36:19 +02:00
peter 5229fd96be refactor(catalog): the HTTP layer reads through queries and writes through the service
Channels, search and sync do no database work in their routes any more: reads go through
catalog/queries, writes through catalog/service, and a live search's ingest is catalog/search.
What a catalog route needs from Reader (a playlist's channels, a user's tag links, the feed
projection of results) it asks reader/queries and reader/service. The scheduler and the runs page
read channel titles and the work queue through catalog/queries. Catalog is fenced below its HTTP
layer: its owner filters go through userscope, it reads no other unit's tables, and its ingest
jobs commit only where they are named.
2026-10-19 02:32:20 +02:00
peter 238e6f8a61 refactor(catalog): the unit's tables move into catalog/models
Channel, Subscription, Video, SearchFind, ExploredChannel and BlockedChannel (and LIVE_OR_UPCOMING)
leave app/models.py for app/catalog/models.py; app.tables registers them. The User↔Subscription
relationship names its other side as a type only. No schema change.
2026-10-19 02:03:14 +02:00
peter a35eece652 fix(e2e): the seed imports Reader's tables from reader/models 2026-10-19 01:33:32 +02:00
peter b703234c17 test(reader): the service tests hold each rule their mutant broke
A reorder's left-out item, a playlisted video beside one not asked about, another user's tag, and
the user's own channel link committed before the auto-tagger reads the table.
2026-10-19 01:28:38 +02:00
peter dbb4065e7b refactor(reader): the HTTP layer reads through queries and writes through the service
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
2026-10-19 01:20:14 +02:00
peter e82f87acb0 refactor(reader): the unit's tables move into reader/models
VideoState, Playlist, PlaylistItem, SavedView, ComfortPin, Tag and ChannelTag leave app/models.py
for app/reader/models.py; app.tables registers them. No schema change.
2026-10-19 00:53:44 +02:00
peter e9c60f72cd refactor(capture): the HTTP layer reads through queries and writes through the service
Capture's routes (downloads, files, public watch links) and the token list no longer build SQL, call
the session or set a model's fields: reads are downloads/queries, writes the unit's service (service,
folders, links, quota), and each write flushes for the handler to commit. Whose a row is goes
through userscope (owned_by, owned_row), which owned_or_404 now uses too.

The fence holds Capture below its routes as well: it reads only Catalog's and Plex's tables, writes
no other unit's, filters owners through userscope and commits only in its jobs and the upload
ingest. A test per service use case, among them that the tray lists only the caller's files.
2026-10-19 00:19:03 +02:00
peter 3a688fa0f5 refactor(capture): the unit's tables move into downloads/models
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
2026-10-18 23:45:53 +02:00
peter d66174ed16 test(arch): a unit's HTTP layer does no database work beyond its pin
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
2026-10-18 23:01:12 +02:00
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
peter 4b1f6d7ec2 release: 0.107.3 (improvement/dev-tooling) 2026-10-17 21:34:53 +02:00
peter 51bf780100 chore(release): 0.107.3 — text on glass holds 4.5:1; the e2e suite runs axe, the gate lints every shell script 2026-10-17 21:21:37 +02:00
peter 3ada34073b fix(theme): text on glass holds 4.5:1 — light --muted kept at 6:1 on the page, tonal fills opaque
Pixel-measured over every scheme x mode x backdrop: the item count fell to 3.8:1 (dark too: its 8%
fill let the toolbar glass and the thumbnails under it through), Channels' tabs to 4.1:1 and the
current Files row to 4.4:1. Light --muted moves 15% toward --fg in every scheme; the badges, the item
count and the current rail row paint opaque bg-tint* fills mixed into the surface. The token contract
now asks 6:1 of --muted on the bare page and checks those three sites use the tints.
2026-10-17 20:06:54 +02:00
peter d74f2a9ca1 fix(deps): source-map-js 1.2.2 — the indexed-source-map denial-of-service advisory (dev tooling only) 2026-10-17 17:51:10 +02:00
peter a8dddc8c24 test(e2e): axe-core's WCAG A + AA rules over every page and the player (chromium only); contrast stays with the visual check 2026-10-17 17:48:53 +02:00
peter 460bdec406 chore(tooling): drop scripts/lint-changed.sh — the shared gate runner's Stop hook lints the turn's files, shellcheck included 2026-10-17 17:22:35 +02:00
peter 66181d1394 chore(gate): shellcheck joins the gate — every tracked shell script, the host binary held to a pin 2026-10-17 15:31:34 +02:00
peter 0a1fef28ac docs(claude): the YouTube write-error helper, the overlay and z-scale rule, and px-frozen sizes join the canonical patterns 2026-10-17 14:47:57 +02:00
peter fe572d9207 docs: a copy-only change runs the Conformance check's step 4 alone, in both languages 2026-10-17 14:27:45 +02:00
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
peter 2b6d968833 release: 0.107.1 (improvement/gate-timing) 2026-10-16 00:45:45 +02:00
peter 95e6b2c3d9 chore(release): 0.107.1 — a failed download's explanation says press Retry 2026-10-16 00:33:59 +02:00
peter 80c55fbf1b merge bug/download-retry-copy into improvement/gate-timing 2026-10-16 00:33:38 +02:00
peter 011f269152 chore(gate): every lane reports how long it took, and a run can log the times for comparison 2026-10-16 00:13:47 +02:00
peter dcb244dd04 fix(downloads): a failed download's explanation says press Retry, not retry it from a list that is gone 2026-10-15 22:56:40 +02:00
peter 08a9f4e255 release: 0.107.0 (feature/e39b-machine-access) 2026-10-15 06:20:35 +02:00
peter 1673134c36 chore(release): 0.107.0 — API tokens for your machines, share a whole folder by link 2026-10-15 06:09:23 +02:00
peter 27427eef32 fix(files): ending every session and making a token lock the user without holding up rows that refer to it 2026-10-15 06:05:51 +02:00
peter a248ca6c13 fix(files): the token list is fresh after the sessions end, a copy again reads Copied again, the Copied button says so 2026-10-15 05:47:00 +02:00
peter 620f84dba4 fix(files): a token made as every session ends goes with them, and a file being shared holds its job 2026-10-15 05:47:00 +02:00
peter 2efa7d5a75 fix(files): an upload is said once in any order, copy buttons speak, a link list that failed says so
- The queue holds an upload it sees land done while this tab's upload still waits for its answer, so
  the dialog's notice is the only one whichever comes first; another device's upload follows after.
- One `useCopyFeedback` for every Copy beside a value (token, share link, set-password link): Copied
  for a moment, a notice a screen reader hears, a warning when the clipboard refuses; the token
  dialog's two Copy buttons are named by what they copy.
- The share dialog's link list shows loading, and a load error with Retry, never "no links" — and no
  New link beside a list it has not seen.
- The settings texts say that signing out elsewhere or a password change revokes the API tokens.
- DESIGN.md logs the folder-link decisions; the design lab keeps only the chosen renderings of the
  Links, Tokens and Files rounds.
2026-10-15 04:58:17 +02:00
peter 827df7494c fix(files): ending every session ends every API token, and the share links hold their folder
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
  account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
  foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
  0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
  links, key change, the cleanup job's token warnings).
2026-10-15 04:58:16 +02:00
peter 86712b6b0e feat(files): share a folder by link — its page lists what is in it, plays a video above the list
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.

The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.

Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
2026-10-15 02:23:24 +02:00
peter 90567718b2 feat(files): share-link tokens are kept only hashed and encrypted, and every link change is audited
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
2026-10-15 01:02:29 +02:00
peter c31d7f9d65 fix(files): an upload is announced once, as an upload, and a finish shows in Files at once
An upload that came back done was announced twice: by the upload dialog ("File uploaded") and
again by the tray, which saw a job turn done ("Downloaded: …"). The dialog now marks its job as
announced, so the queue keeps it for the tray but does not say it again. An upload the worker
converted first is still the tray's to announce, now as "Uploaded: …".

A finished job's notice offered "Open" while Files still listed the folder without it (its own
poll runs every 15 s): the tray now refreshes the library the moment it sees a finish.
2026-10-14 23:54:19 +02:00
peter db56f25387 feat(files): API tokens in Settings, the Hub folder in Files, a notice before a token runs out
Settings gains an API tokens tab: a row per token (name and prefix; each
folder with its rights; last use and expiry, "expires soon" marked),
revoked from its menu behind a confirm. A new token is the Hub exchange in
one click, or folders and rights picked per folder (no Write where an
upload cannot land); expiry 30/90 days, 1 year or never (warned). The
token shows once in the same dialog, with Copy and a ready curl line.

The Hub folder closes the Sources in the Files rail, takes uploads and
moved-in files. A token about to run out warns its user once, a week
before (the Files GC's sixth pass); the notice opens the tokens tab.
2026-10-14 23:33:34 +02:00
peter 4f58d15d3f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 21:19:14 +02:00
peter 2c66ce1cf4 release: 0.106.2 (improvement/thumbs-and-apt-refresh) 2026-10-14 21:17:39 +02:00
peter ce45445d17 chore(release): 0.106.2 — pictures that fail to load retry instead of breaking 2026-10-14 21:06:18 +02:00
peter 383d394e32 fix(ui): a picture that fails to load shows the card's own frame, then retries
A bare <img> sent a Referer that Google's image CDNs refuse and had no retry and no fallback, so a
Home shelf card showed the browser's broken-image mark. ui/RemoteImage gathers what Avatar, the
channel banner and the download rows had each hand-rolled: no Referer, up to three retries past the
browser's negative cache, the caller's fallback (by default the card's empty frame) instead of the
broken mark, the failure remembered by URL. Every picture now goes through it; a lint rule refuses
a bare <img>.
2026-10-14 20:51:08 +02:00
peter 06e8291896 build(image): rebuild the apt-get upgrade layer when APT_REFRESH changes
A build cache kept the upgrade layer for as long as nothing above it changed, so a publish shipped
curl and librsvg a release behind fixes already in the Debian archive (the CVE gate refused it).
The release build passes the day; empty, the default for a self-hosted build, leaves the cache alone.
2026-10-14 20:51:07 +02:00
peter db6b60029f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 20:17:05 +02:00
peter 02134799cc release: 0.106.1 (bug/plex-settings-outside-click) 2026-10-14 20:03:38 +02:00
peter c8bc843517 chore(release): 0.106.1 — the Plex player survives a click outside its Settings panel 2026-10-14 19:44:25 +02:00
peter 759a7536b9 fix(history): an overlay opened while a back() is in flight waits to push its entry
The Plex player's ⋯ → Settings closed the menu (its entry went back) and opened the panel a task
later, before that back() had landed. The panel's push was the entry the landing back() took, so
the books counted one entry more than the history held, and closing the panel by a click outside
it stepped back over the player's own entry: the player closed.

A push now waits while a back() of ours is in flight; the popstate that lands the last one
reconciles again.
2026-10-14 19:43:51 +02:00
peter b2d34e6e6c feat(files): API tokens — a machine reaches the library folder by folder
A user can make a token for a machine of theirs (`/api/tokens`, signed-in only): `sfl_` + 32
random bytes + a checksum, only its SHA-256 kept, shown once. Sent as a Bearer header it is
honoured on six `/api/files` routes only — folders, list, file, hash, upload, delete — and there
only in the folders its grants name, with the rights they give (list / read / write / delete).
One check, `FilesCaller.may`, in every route; a folder or file out of reach answers like a
missing one; the files shared with the user are never in reach. A token beats a cookie on the
same request.

The Hub preset makes the user's one Hub folder (a new folder kind) and grants it every right.
Refused tokens are slowed down per client, a runaway machine per token; last use is stamped at
most once a minute; making and revoking a token is audited, and a token's uploads and deletes
name it in their audit rows.
2026-10-14 18:40:23 +02:00
peter c335d54929 release: 0.106.0 (feature/e39-files) 2026-10-13 03:34:56 +02:00
peter 7188ad7d71 chore(release): 0.106.0 — the Files library 2026-10-13 03:24:14 +02:00
peter ebbae6794c fix(files): review round 3, the page — a search waits for the server, the viewer keeps its file, a drop ends the drag
- past the list's cap a search waits in a skeleton, never showing the newest files' 'no match' for a
  file the server is about to find; a sort keeps its stand-in rows
- the viewer keeps the file it was opened on when a server answer replaces the rows under it
- a drop another zone took still clears the page's drop overlay
- 'The first N items' only for a server sort; the past-the-end notice no longer claims the file exists
- a failed server search is tested to say so
2026-10-13 03:15:59 +02:00
peter c9baf68b16 fix(files): review round 3, the server — a fetched locked re-read, a finish that follows its file, an eviction that counts again
- set_retention and delete_own re-read the folder under its lock and use that row (an unfetched ORM
  result never refreshes the object); a folder gone meanwhile is refused (409), and the route audits
  the retention actually replaced
- a finishing download is stamped only while it is still in the folder (and keep) it was read with:
  a move or a folder delete landing meanwhile sends the read round again, so no file is done with no
  expiry
- the eviction pass counts the disk before each candidate and skips one asked for since it began
- docs: source builds get their own volume-copy snippet
- race tests assert the lock wait; one built on a committing delete never raced
2026-10-13 03:15:59 +02:00
peter 5bb101e493 fix(files): review round 2, the page — the viewer's Escape, the dialog's drop, the capped list's honesty
- Escape closes the file viewer again: its layer sits above its dialog's, so it takes the Escape.
- The page's drop guard leaves a drag the open Upload dialog's zone accepted alone (it ran last and
  refused that drop).
- A capped list's server sort or search keeps the rows on screen until its answer is in (no skeleton
  under the pressed header), says "The first N items" when that answer is cut too, and an arrival is
  decided on the rows shown: one past the end of a cut list is said to be so, not gone.
- The search an arrival clears is taken at once.
- Tests: confirm names asserted per confirm, a fitting text pages under the arrows, a deleted file
  leaves the tray's finished list.
2026-10-13 02:32:50 +02:00
peter 54a6bbf07f fix(files): review round 2, the server — locks the GC and folders read under, the mount check, the old mount name
- GC: the expiry warning marks each file conditionally on the expiry it saw, one commit each (a file
  deleted meanwhile no longer fails the run; one re-timed keeps its fresh expiry unwarned); eviction
  counts the disk again when a candidate was freed meanwhile, so it never evicts for space already free.
- Folders: a retention change locks the folder and its files, and a finishing file reads the retention
  under a share lock, so neither misses the other; a folder delete locks the files it takes home (a
  file moved out meanwhile stays put, one deleted meanwhile no longer fails the delete).
- Upload: the folder is checked before the probe and held only once the job names it.
- Dedup enqueue locks the cached asset, so a release meanwhile yields a fresh one instead of a 500.
- Mount check reads the kernel's mount table instead of comparing devices (the vfs storage driver gave
  a mounted volume the device of /), and only inside a container; the API also logs a fresh, empty
  volume in the library's place.
- Compose files fall back to DOWNLOAD_HOST_PATH; the upgrade notes cover the from-source pull.
- Folder routes call owned_or_404 directly.
2026-10-13 02:32:49 +02:00
peter 8b2260c9a8 fix(files): review round 1, the page — honest states, the keyboard, the tray's words
- A library past the list's cap is searched and sorted on the server; the count says "the newest".
- Each folder and each search opens on its first page (DataTable pageScope, useCardPager scope).
- A file dropped where the page takes none never reaches the browser (which would unload the app).
- The viewer's arrows wait while a layer opened from it is on top, and scroll a wide text line;
  a text shows its first 5000 lines and says where the rest is.
- Confirms name the file (no more "your downloads"); the asking menu rows end in "…".
- The tray names a running step beside its bar, shows a pause mark for paused work, polls slowly
  while only paused work waits, and forgets a finished file that was deleted.
- An unchanged rename sends nothing; a paste opens the download dialog only with nothing on top;
  the link box never replaces what was typed.
- Honest states: folders that failed to load, a file no longer there, the storage figure; the rail
  says "nothing downloaded" only when there is nothing.
- Focus after deleting the open folder and when the format form opens; the format name's label.
- ui/Meter (one danger line for both quota bars) and ui/RenameField in the platform layer.
- Copy fixes, stale comments and the speculative hub key; a contract test for the list's
  parameters; e2e: the Files page and the paste intake.
2026-10-13 00:14:09 +02:00
peter 815083bcc4 fix(files): review round 1, the server — the tray's window, the GC's locks, a file's origin, the upgrade path
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming,
  keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished.
- GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit
  one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a
  second release that frees a file another user still holds.
- A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit
  whose parent expired still goes back to its folder, not to Uploads.
- A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes
  that file home too, instead of failing on the foreign key.
- The hash route reads at most two files at once (429 past that) and holds no connection meanwhile.
- 0104's backfill dates old files by their request date only: no notice prod wrote names its job.
- The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the
  worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up.
- Stale comments about the /downloads mount and the Downloads page.
2026-10-13 00:14:09 +02:00
peter 96509fe356 fix(files): a file is added when it is done, not when it was asked for
A download that failed for days and then went through sat days down the library, under its
request date, and read as missing. A job now records when it became done (`ready_at`), stamped
where the expiry already is (`folders.stamp_done`, `finalize_if_running`) and on an insert that is
already done; a check constraint holds every path to done to it. The library sorts and dates its
"Added" by it; the queue keeps the request order.

The migration dates a file from before by its "ready to watch" notice where the inbox still holds
one, else by its request date, where it already sat.
2026-10-12 22:19:37 +02:00
peter 132fb56428 fix(downloads): a site the downloader no longer understands says so, and offers a retry
yt-dlp asks for a bug report on an extraction failure it did not expect: the site answered in a way its extractor does not read (measured on a TikTok link: fails on every attempt under yt-dlp 2026.07.04, downloads under 2026.08.19). That failure read as "Something went wrong" with no retry; it is now its own category, site_changed, with a Retry in the failure window. A contract test holds every failure category to its copy in both languages.
2026-10-12 20:54:47 +02:00
peter f6bce23a94 fix(files): the Storage tab drops its user picker — every user already has a row and a menu
The list holds every user, files or not, so the picker above it only repeated a row's Edit quota. The empty text names what the list holds: users.
2026-10-12 20:06:30 +02:00
peter fb8fb980e5 feat(files): a file that is not media opens in a viewer — an image, a PDF, text as text
- GET /api/files/{id}/preview: an allow-list on the stored extension (downloads/preview.py) picks
  image / PDF / text; served inline under `CSP: sandbox; frame-ancestors 'self'`, the type from the
  table; HTML and SVG arrive as plain text; a text is its first 1 MiB (X-Preview-Truncated), sent
  as UTF-8 whatever it was written in (BOM, UTF-8, else cp1250); a NUL means binary: 415
- the list says which preview each file gets (asset.preview); FE/BE contract on kinds and header
- the viewer (shared dialog shell): ← / → through the list's such files in its order, Save, open in
  a tab, the file's menu; one stage height for every kind; a no-preview card offers the file
- the name, the card's picture and the menu's Preview open it; an image up to 8 MB is its own
  thumbnail
2026-10-12 15:28:31 +02:00
peter 6707550513 fix(files): a steady list header, every user on the Storage tab, the name gets the row's room
- The Files list starts at the scroller's top: the padding above it let the sticky header ride up
  12px on the first scroll, then stop.
- Users -> Storage lists every user with their quota, not only those already holding a file — a
  quota is set before anyone downloads.
- On a wide Files list only the "…" holds a place; Copy link, Share and Save come up over the date
  while the row is hovered or holds the keyboard (a card keeps them in its line). Keeps is narrower.
2026-10-11 20:33:48 +02:00
peter 078194731b feat(files): the Downloads page goes — the tray, a Storage tab and a Downloads settings tab take its place
- The Downloads module, its rail icon and badge are gone. A bookmark, the remembered page and an old
  history entry that name it land on Files (urlState.resolvePage); `g d` goes to Files too.
- Users page: a Storage tab — the disk in numbers, every user's use against their own limit with
  their file count, every user's downloads; aligned tables, one menu per row (edit / reset quota,
  open source, copy, the owner's quota). The admin storage rows carry quota + file count, the admin
  job list its owner.
- Settings: a Downloads tab — the download formats in one list (duplicate a built-in, edit or delete
  your own) and the site sign-ins, moved from Account.
- "View" on an added download, a clip or a converting upload opens the tray on that row (unfolded,
  back if closed; a finished one opens as a file; under a player it opens without taking focus).
- The download dialog opens on the format used last.
- DataTable: `pager={false}` for a short list, `cardRow` to give a lone "…" its own card line.
- Welcome and README speak of Files; a new Files screenshot.
2026-10-11 19:37:13 +02:00
peter 1c4f0042d2 feat(files): download from anywhere — a link box, a pasted link, the palette; the running downloads in a tray
- Files toolbar link box (paste button, Enter); a lone http(s) link pasted onto any page opens the one
  app-wide download dialog, which now names the link; the palette's "Download from a link…" focuses
  the box on its history entry.
- The download tray, bottom right on every page: what waits, runs, is paused or failed (reason +
  Retry, the queue's right-click / "…" menu), and what finished in this tab as "Done · Open" until
  closed; folds to a pill (per-account pref); closed, it stays shut — across a reload — until the
  work changes or something finishes. Back to top steps up over it, the page end keeps clear.
- A finish is announced once: "Downloaded: …", whether a phone plays it, Copy link + Open (toasts
  and notices may carry a second action).
- The inbox's ready notice opens the file in its folder: download_done carries job_id + folder_id.
- GET /api/downloads is the queue only (in flight, failed, finished in the last 10 minutes), with
  each row's source key.
2026-10-11 16:56:13 +02:00
peter d22c64f4dd feat(files): folders managed in place, a grid per folder, drop to upload, open on one file
- Own folders are made, renamed and deleted from the rail (right click, long press,
  Shift+F10) or the "…" beside an open folder's name; any folder's retention is a
  submenu (7/30/90 days, until deleted). Deleting confirms where its files go.
- List or grid per folder, sharing one sort; the grid is the one media card.
- A file dropped anywhere on the page opens Upload with it, bound for the folder
  on screen.
- A link (?item=) or the upload notice opens Files on the file: its page turned
  to, the keyboard on it, a ring round it.
- An in-place name field left by Enter or Escape hands the focus back to what
  replaced it (the row, the entry, the "…") instead of dropping it on <body>.
- The table's saved page size is per account.
2026-10-11 15:59:12 +02:00
peter 85aa917b38 feat(files): the Files page — the library in one aligned list, its folders in a rail
Files is the one library: every done file, the ones shared with me, or one
folder's, picked in a side rail (sources, own folders, the storage meter) and
mirrored to ?folder=. One table with every kind of value in its own column;
Copy link, Share and Save in sight, every action on a right click: play,
cut, rename in the row, move to an own folder or back home, keep, details,
re-download, delete.

The Downloads page keeps only the queue until the tray replaces it. The old
/api/downloads library routes are gone: list shared, upload, the details
PATCH, delete, the file and the usage now live under /api/files, and a
listed file says which source it came from. The job's "expired" flag is gone
with the state it described.
2026-10-11 05:34:25 +02:00
peter 1a1665b8ec feat(files): the Files library API — folders, moving and keeping files, upload into a folder, the file's hash
/api/files is the library's home: list a user's done files (by folder, searched, sorted, or the
ones shared with them), rename / move / keep / delete a file, make, rename and delete own folders
(a deleted folder's files go back to their source folders), and set how long any folder keeps a
file — done files shift by the difference, never closer than the GC's warning window.

An upload can name its folder (an own folder or Uploads), checked before the body is read and again
before the file lands; it is hashed while it streams, and any file can be hashed on first ask. The
download says the hash (Repr-Digest) once it is known; a new file at an asset's path forgets it.

Uploads, deletes, keeps, moves and folder changes are audited (ids only). The fileless "expired"
rows of the old retention are deleted by migration 0103: an expired file now leaves the library.

The shared route helpers moved to routes/files.py; the old /api/downloads library routes delegate
to the new ones until the client moves over.
2026-10-11 04:42:55 +02:00
peter bf99444ded feat(files): every file sits in a library folder, and keeps its own expiry
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.

The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.

safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
2026-10-11 03:30:34 +02:00
peter 07e0990ecf design(files): the one-library round in the lab — aligned list columns, the tray, the done notice
The Files page replaces the Downloads page: list rows with one column per value kind and a
right-click menu, an app-wide download tray, a done notice that copies the link. Picked
variants are logged in DESIGN.md.
2026-10-11 02:59:51 +02:00
peter e2ed2bf874 merge: chore/e2e-retry-connect into E39 (rides the next release) 2026-10-11 02:30:16 +02:00
peter b9eb8458b3 test(e2e): a localhost connect the machine dropped is tried again on the test side
On a Windows host a burst of test traffic can drop a loopback connect now and then (every port,
any program — measured on the dev desktop). The browser resends it and only waits; Node (libuv)
sends a loopback SYN once and fails at ~300 ms with "connect ETIMEDOUT". `e2e/retryConnect.ts` wraps
Playwright's request context so that one error is tried again (0.5 / 1 / 2 / 4 / 8 s, the system's
own retry waits) while every other failure goes straight to the test; global.setup's health check,
login and /api/me and speed.measure's share probe go through it. Unit-tested; e2e 61/61.
2026-10-11 00:45:13 +02:00
peter 891507dc75 release: 0.105.0 (feature/e38-ad-hoc-clips) 2026-10-10 23:38:13 +02:00
peter e6894ac0fc release: 0.105.0 — ad-hoc clips 2026-10-10 23:13:22 +02:00
peter 663a7e8012 fix(player): the Plex player's fills hold on its chrome: an amber head on a white played part
A light theme's accent as a fill on the always-dark chrome was ~2.8:1 (Slate), under the 3:1 a
control needs. A native checkbox and slider tint with `accent-chosen` (the accent elsewhere, the
media amber under `media-marks`), and so does the skip button's countdown. The progress bar plays
white with an amber head: amber there marks a place, as the intro / credits markers do, and an amber
played part made a marker ahead of the head read as already played (owner's choice of three).
2026-10-10 22:56:09 +02:00
peter 93192dadd8 fix(player): a chosen tab's underline and an open or pressed icon read the scope's chosen colour
The underline named `--color-chosen`, which is resolved once on :root, so it stayed the accent under
`media-marks` (seen in the browser: fuchsia on Slate light); it reads `--chosen` where it is used
now. An icon button pressed or with its menu open tinted its glyph with the accent; it takes the
chosen colour — the accent everywhere but the player chrome, where it is the media amber.
2026-10-10 22:26:22 +02:00
peter 5b48018e92 fix(downloads): a produced file is held to the space left by its real size
yt-dlp's `max_filesize` only stops a plain HTTP file that announces its size: YouTube's fragmented
formats never meet it — measured on localdev, a 718 MB download ran to the end with 1 MB left and
put its owner four times over the cap. The download now re-reads what is left once its file is made
(after the phone-safe pass) and refuses one larger than that as the owner's refusal; an editor clip,
which nothing checked, is held the same way, as a player clip already was.
2026-10-10 22:10:56 +02:00
peter 8312c6d592 fix(player): what is chosen on the player chrome is marked amber, not written in the accent
A light theme's accent as text on the always-dark chrome was 2.84:1 (Slate). `--chosen` is the
accent unless a scope names its own; `media-marks` (was media-focus) points it, with the focus
ring, at the media amber. A chosen track stays white with an amber tick (MenuCheck reads
`text-chosen`); a chosen settings tab or toggle is white with an amber underline (MEDIA_CHOSEN).
The theme contract test holds the token, the tick and the player components free of accent text.
2026-10-10 21:56:47 +02:00
peter e1d325b942 fix(downloads): a download larger than the space left says so, not "processing failed"
yt-dlp refuses a file announced over `max_filesize` by logging one line and returning with no file
(measured, and pinned in test_formats against a local server): the worker found no output and
failed the job as `postprocess`. `formats.watching_size_cap` now sees that line through our yt-dlp
logger, and the download (and, should it ever happen, a YouTube clip's range fetch) fails as the
owner's refusal with DOWNLOAD_TOO_BIG_MESSAGE, classified `quota_exceeded`. The download's quota
copy says it does not fit in what is left — true at a full cap too — instead of "Storage full".
A pre-existing type error the change touched is fixed (baseline down by one).
2026-10-10 21:56:47 +02:00
peter 977ba244fa fix(clip): a time's error line sits in the flow, the remove button off the cut, and every player surface rings amber
- useTimecodeField holds a time field's draft for its caller, which places TimecodeError (role
  alert) in its own flow under the fields' row: it no longer lies over the crop row taking its
  clicks, runs out of the editor's card, or joins the field's <label> name.
- The editor's remove-cut buttons sit in a row under the track at their cut (trackLeft), off the
  cut's grab area, where a press drags it; RangeTrack's adornment slot is gone.
- media-focus re-points --focus to the media amber on the Plex player, the YouTube stage and every
  bg-media-chrome surface; a contract test holds each such class string to media-focus or
  media-scope, the field's focus edge to the focus token, and --color-focus to the inline theme.
- An upload's quota headline says the converted file does not fit, not "Storage full".
- DESIGN.md: the tokens paragraph says what is re-pointed where; the decision row measures all
  three colours on the chrome.
2026-10-10 21:29:55 +02:00
peter 89d18b46a6 fix(downloads): a converted upload too big says so, and a suspended owner's waiting rebuild is not "preparing"
- A converted upload over the space left (not zero) fails with CONVERTED_TOO_BIG_MESSAGE, not
  "storage is full"; it classifies as quota_exceeded by the phrase it shares with the clip's.
- The public watch page says "preparing" only for a job a worker will take: a suspended owner's
  queued job waits for as long as the suspension lasts, so the link says the video is gone, as
  for a paused rebuild.
- The worker's claim docstring names the restart case: a running job put back in the queue waits.
2026-10-10 21:29:55 +02:00
peter c27cfdcb9c fix(clip): typed times belong to their value, and every mark on the player chrome can be seen
TimecodeField drops a draft when the value moves under it (another segment selected, a nudge), so
leaving the field never commits it to another cut; it is invalid only once offered, with a line
under it saying how to type a time. On the always-dark chrome media-scope points every focus ring
at --media-mark and errors at the new --media-danger (a light theme's accent and danger were under
3:1 there). A track whose dragged handle unmounts ends the drag on the pointer's release instead of
locking up; a remove refused while clips are queued leaves no focus to take later; each remove-cut
button names its cut and is a 24px target. DESIGN.md records the media tokens; stale comments fixed.
2026-10-10 20:28:52 +02:00
peter a867145bcb fix(downloads): a suspended user's queued jobs wait, and a clip too big for the space left says so
A suspension now blocks the user's downloads as quota.resolve already advised: the worker leaves
their queued jobs unclaimed until it is lifted (the confirm says so). A cut clip larger than the
remaining storage is refused with its own message (not "storage is full"), and a clip's quota
refusal reads as a clip's in both languages. The cancel poll compares to JobStatus.RUNNING.
2026-10-10 20:28:52 +02:00
peter 4d4eea2c54 fix(clip): marks on the player chrome keep their amber in a light theme, and a removed cut hands the focus on
- A media amber (`--media-mark`), the same in both themes, for what marks a place on the always-dark
  chrome or over a video: the clip strip's playhead and focused end, the crop frame's ring, Plex's
  intro / credits markers. A light theme's warn was a dark amber there, under 3:1 (contract-tested).
- The editor's remove-cut button rings inside itself (the track clipped its ring), and once it goes
  with its cut the focus moves to the cut beside it, or to Split here, not to <body>.
2026-10-10 15:57:02 +02:00
peter c489b60c8b fix(clip): clip mode keeps its range, its focus and its keys through the player's own moves
- A mark on the very end seeks a moment short of it; while a clip is marked the end of the item
  neither closes the player nor steps the queue, and Plex's auto-skip waits.
- Plex: Escape and Back peel what is over the picture (info, up-next, the countdown) before the
  clip; Back under a dialog is the dialog's; no range opens while the length is unknown.
- YouTube: the focus goes back to the scissors the user can see, else the player; the panel keeps
  its name, Loop and focus when the view is maximised or restored under it.
- A typed time stays in its field until taken, then shows where it landed; unreadable stays marked
  invalid; a decimal comma is read (ui/TimecodeField, the editor's fields too).
- RangeTrack: a drag is its pointer's, a cancelled drag reports final, an adornment sits beside the
  slider, a caller-chosen focus ring (amber on the clip strip). CropFrame: a drag is its pointer's.
- The editor's remove-cut is a real button at the cut's foot; a segment's keep toggle sits over the
  cut's hit area.
2026-10-10 15:35:21 +02:00
peter b4bf71ea64 fix(clip): a clip is held to its owner's storage cap, and its limits and failures say the server's truth
- The cut file is measured against what its owner has left, for a YouTube range (whose fetch never
  reads yt-dlp's size ceiling) and for a Plex clip (which fetches nothing), before the fetch and
  again before it is placed; a refusal leaves the asset pending for a Retry.
- A silent Plex file is cut silent; a file the probe cannot read says so.
- A second job on a failed or renamed clip gets the same YouTube diagnosis and the fetched title.
- The "too long" refusal names the server's cap; a contract test holds the client's copies of the
  clip limits to clip.py; the shipped player clients are pinned against the installed yt-dlp.
- One HDR transfer set and one rating_key lookup; job status words from JobStatus; the display-name
  guard sees subscript writes and the clip route; stale comments and types follow the clip job.
2026-10-10 15:35:12 +02:00
peter 72df11f2c0 fix(ui): a small crop frame takes smaller handles, so it can still be grabbed to move
The handles reach 10 px into the frame from every side; a frame at its minimum on a small picture
(the Downloads editor at 800 px: 41 x 23) was then nearly all handle - 74 of its 920 px still moved
it. Below 64 px on its shorter side the frame takes 12 px corners and 8 px edges: 262 px move it.
2026-10-10 13:54:04 +02:00
peter 0a16bc40ae fix(ui): a crop frame grabbed with the pointer takes the focus, so the arrows go on from there
The press is held back from the page (no text selection, no play / pause underneath), which also
kept it from focusing: after a drag the arrows still went to whatever had the focus before, and
Escape did not reach the frame. The press now focuses the frame or the corner it grabbed (an edge
focuses its frame).
2026-10-10 13:38:49 +02:00
peter 520b0afb90 feat(clip): crop a clip to a frame drawn on the picture, and give the clip panel the room it needs
An optional crop in clip mode: a Crop toggle puts a frame on the picture (free, 16:9, 9:16 or 1:1;
the whole picture stays the default). The frame is sent in fractions of the box it was drawn in,
with that box's shape; the worker fits the real picture into the box by its display aspect (sample
aspect and rotation aware), drops the part over the bars, and crops first in the filter chain. A
frame covering the whole box is no crop at all, and one off the picture is refused by name.

The crop frame is a ui primitive (CropFrame) shared with the video editor, keyboard-operable.
YouTube: the wide side panel shuts while clipping and comes back after; maximised, the clip panel
docks in the stage. Plex: the picture sits above the clip panel so the frame lies on it.
2026-10-10 12:52:59 +02:00
peter e301632765 fix(clip): a clip refusal speaks the user's language, and a choice in the clip form no longer sends it
The backend refuses a clip with a structured reason, but the client showed every one as the generic
"couldn't make the clip": each reason now has its own message in both languages, held to the
backend's reasons in both directions by the contract test. A segmented toggle's and the priority
stepper's buttons had no type, so inside a form one of them submitted it.
2026-10-10 12:52:12 +02:00
peter 50157cc2fd feat(player): clip mode - mark a range of what is playing and cut it into a download
Both players get a scissors in their control row (Plex: the control row; YouTube: the actions and
the docked bar), `c`, and `[` / `]` to mark at the playhead. The clip panel lives in the player, so
the watched video is the preview: an overview of the whole item and a zoomed strip around the
selection whose ends are sliders (arrows step a tenth, Shift a second), Start / End fields with
nudges and "to the playhead", the length, a looping Play selection, a name, Cut. Escape closes the
panel before the player and the focus goes back to the scissors. A cut is a toast with a Downloads
link plus the usual Ready notification.

- ui/RangeTrack + RulerTicks: the shared, domain-free time track (regions, keyboard-steppable
  handles with a 1.5rem hit area, a playhead, a background slot, over a window of the item). The
  Downloads editor's cuts moved onto it, so they are keyboard-steppable now.
- lib/clipRange holds the marking rules (1 s to 10 min, the server's), lib/timecode the editing
  timecode the editor's private tc/parseTc became.
- keymap: a symbol typed through AltGr answers its bare-symbol row - on the HU layout `[` is
  AltGr+F, which arrives as Ctrl+Alt; letter and Mod rows still never match Ctrl+Alt.
- The YouTube stage sits in a wrapper with the panel, so the maximised stage's blur escape matches
  at any depth.
- media-scope: fields on the always-dark player chrome read white in a light theme too.
2026-10-10 03:58:27 +02:00
peter f221c1962a fix(downloads): a YouTube clip goes with the user's own cookie and is diagnosed like a download
A clip fetched its range signed out, so a video behind YouTube's per-video "not a bot" wall could
never be clipped while its download went through with the user's cookie - and the failure copy told
the user to add a cookie the clip would not use. The clip asset is private by construction
(`clip.asset_ref` names its user), so `usable_clip_cookie` attaches the user's YouTube cookie on that
identity, and a failed range fetch now carries the download catalog's diagnosis
(`youtube.bot_wall` / `youtube.cookie_rejected`) instead of a bare category.
2026-10-10 03:58:27 +02:00
peter d466ae915b fix(downloads): a YouTube clip outside the catalog takes its title from the fetch
It was queued as "Clip (clip)" and stayed so: nothing renamed it once yt-dlp had answered.
2026-10-10 02:43:13 +02:00
peter 14598ec320 feat(downloads): ad-hoc clips cut straight from a YouTube video or a Plex item
POST /api/downloads/clip takes a source (a YouTube id or a Plex rating key), a range of at most
ten minutes and, for Plex, the audio track being heard. The clip becomes a per-user asset in the
library like an editor clip, shareable as soon as it is ready. A YouTube clip fetches only its
range (yt-dlp download_ranges: 3-7 s whatever the video's length); a Plex clip reads the library
file in place. Both then go through one re-encode: frame-exact at the marks, H.264 at most 1080p
with HDR tonemapped to SDR, stereo AAC, no chapter track, .mp4.

Job kinds are a JobKind StrEnum now, and Re-download refuses every kind but a download.
2026-10-10 02:31:38 +02:00
peter b16cf7668c fix(downloads): YouTube downloads reach their profile's height again
The primary player client, android_vr, was still offered 1080p and 2160p, but every fetch of
those formats died with HTTP 403: it now needs a stream PO token the bgutil sidecar cannot mint.
The fallback set then served 360p only, so every download finished at 360p without failing.
tv_simply is offered the full ladder and its fetch completes (measured: 1080p, and a localdev
download through the pipeline reached the default profile's 720p). yt-dlp 2026.8.19 marks
the clients that need such a token, so it stops fetching into the 403.
2026-10-10 02:08:43 +02:00
peter 3438aeb695 fix(player): the player's own focus grabs leave a tabbed-to control alone
Turning ready, the pointer leaving the video, the window regaining focus, a wheel over it and
leaving fullscreen each put the focus back on the card unconditionally, so a viewer who had
tabbed to a control right after opening started over from the first button on the next Tab.
They now take it back only when it sits in the iframe or off the card.
2026-10-10 01:14:31 +02:00
peter c003a48244 release: 0.104.0 (improvement/adhoc-batch-oct) 2026-10-10 00:24:27 +02:00
peter 8a2334e198 release: 0.104.0 notes and version 2026-10-10 00:15:23 +02:00
peter 1f010328df fix(plex): plain text survives a decimal reference too long to parse
int() refuses a decimal run past 4300 digits, so one such reference in a summary raised out of the sync and the text migration. Any number past seven significant digits is beyond U+10FFFF, so it becomes U+FFFD like any out-of-range reference; leading zeros are dropped first. The rails test's comment now says what it guards: both rails agree with the rollup.
2026-10-10 00:03:08 +02:00
peter 77638f7a0b fix(player): a link that opens the cut text is scrolled back into view
Focus had scrolled the clamped box itself; opened, the link dropped by that much, out of a scrolling panel's sight. DESIGN states where Close sits in the stacked layout; a stale test comment goes.
2026-10-09 23:46:40 +02:00
peter 2cff4ad954 fix(feed): an override yields only to a refetch that set out after its write was answered
Send order let a refetch read before an unanswered mark committed and drop it. A reset's position 0 is written again after a refetch that set out before the server took the reset. The player-session refetch also waits for the player's closing checkpoint, which the player now tracks (progressFlushes); refetched first, the list put the older position back.
2026-10-09 23:46:39 +02:00
peter 8776586ca3 fix(plex): a matched episode and a collection member offer Details in their menu
Their titles open the item's page, a pointer-only door; the card menu is the keyboard's.
2026-10-09 23:46:39 +02:00
peter 92a17aba83 fix(plex): a reshuffle survives a refresh landing mid-deal; the ↻ spins for a deal only
The ↻ intent was spent as its fetch set out, so a watched toggle's resync cancelling that fetch re-ran it as a re-read of the old draw. It now lasts until the latest deal answers. The button spun on every re-read of the same titles; it now follows the deal. Comments that said the Surprise draw is never refetched are brought in line.
2026-10-09 23:46:39 +02:00
peter fc53bc03bc fix(plex): the rails' Restart comes from the one rollup; plain text decodes whole entity names only
Both Continue-watching rails built a show's card state inline; they now take it from rollup.show_state, so a show part watched with nothing to resume offers Play alone there too. A named reference counts only when its whole name is an HTML5 entity: html.unescape still decoded the legacy "&not" inside "&notice;". Tests cover the grid's in-progress-but-not-resumable case through its SQL and both rails.
2026-10-09 23:46:39 +02:00
peter 6df257d7df fix(plex): Restart only where an episode has a resume point; the Surprise draw is re-read, not redrawn
Show cards carry resumable (ShowState, from the same rollup) so a part-watched
show whose next episode is untouched offers Play, not a Continue and a Restart
that do the same thing; plexStarted is the one rule for cards and menus. The
Surprise rail's key sits under the discover root: whatever refreshes the rails
re-reads the draw (keep=) with its watch state now, kept past the 5-minute gc,
and only the reshuffle deals. Closing the player refreshes the Discover rails.
A show card's play acts on the last click only, and only where it was made.
2026-10-09 22:01:28 +02:00
peter 5572499b56 fix(player): the shut panel keeps the dialog's name; Tab onto a cut link opens the text
With the side panel shut the title heading was gone, and the dialog it named
went unnamed: the strip carries an sr-only copy. A link below a clamp's cut
stayed tabbable out of sight; reaching it now opens the text, measured from the
text's top since focus scrolls the clamped box. DESIGN describes the modal as
built, with log rows for its chrome and the wheel rule.
2026-10-09 22:01:27 +02:00
peter 76738f8ea9 fix(plex): plain text decodes only well-formed character references
html.unescape also decodes the legacy forms without a semicolon, so prose like
"AT&notice" came out as "AT¬ice". The show and collection sync paths are
now tested too.
2026-10-09 22:01:27 +02:00
peter 568bf93510 fix(feed): an optimistic write outlives only the server answers older than it
A reset wrote a position mask over the cached list, so a replay's saved
position stayed hidden: a reopen resumed at 0 and its first checkpoint wrote
over the progress. The reset now writes 0 into the cache, where the player's
closing patch writes the next one. Overrides carry their write order, and a
landing refetch drops only those written before it set out. The player
session ends after the modal unmounts, so a clip auto-marked watched as it
closes still refetches the list.
2026-10-09 22:01:27 +02:00
peter 942154cf49 fix(plex): a Surprise draw stays put until its reshuffle
The rail's query was always stale, so Back from a card's player or page remounted it into a new
draw, as did a return to the window: the card just looked at was gone. It now stands until the
reshuffle (or another scope, language or filter) asks for a new one.
2026-10-09 01:29:12 +02:00
peter 3eb87e7108 feat(plex): every show card plays the episode it would put on
A show card's picture opened the show page everywhere but Home, whose shelves carried the
resolved `next`. Now every Plex picture plays: a show asks `GET /api/plex/show/{id}/next`
(the in-progress episode, else the first unwatched, else S1·E1 — the rule Home and the show
page's Resume follow) on the click, fresh each time. A started show shows Continue + Restart;
its title, an ⓘ and the menu's "Open show" row open its page; one with no episodes opens there.

The question is asked per show, never carried on every card: three probes a show would cost a
grid page up to 180 queries. The hover preview asks it once it opens and names the episode
("Next: S6·E16 — Title"). Related shows on a show page play the same way.
2026-10-09 01:22:00 +02:00
peter 851b97b09d fix(ui): a card's hover chrome answers keyboard focus, not any focus
Back from a channel page hands the card its focus again after a mouse trip;
revealed on focus-within, its hover row stayed up under no pointer until a
reload (owner). The row, the Play layer, the Continue + Restart pair and the
global .reveal rule now show on hover or on :focus-visible on the card or
inside it, so Tab and the arrow keys still reveal it.
2026-10-09 00:44:46 +02:00
peter ee94d1949a fix(feed): a video marked watched in the player stays out of an Unwatched view
Marked watched in the player (by hand, or by the player at its end), a card
left an Unwatched list at once, then came back as "new" when the player
closed, until a reload. The player's closing position patch writes into the
cached feed; the feed took the bumped dataUpdatedAt for fresh server data,
dropped its optimistic overrides, and nothing refetched. The overrides now
clear only when a refetch lands, and a player session that changed a state
refetches the list and its count by the view's filters on close, once its
writes have landed.
2026-10-09 00:44:46 +02:00
peter 4f076b5d45 feat(feed): a card's menu filters the feed by its channel, and back
Under "Open the channel" a video card's menu (the right-click on its
channel name included) adds the channel to the feed's channel filter, or,
when it is already in, takes it out. Channels OR together, so it widens a
filter that names others, as "Show in feed" does. Not on a channel page,
whose view is that channel already, nor in a live YouTube search or on Home.
2026-10-09 00:44:46 +02:00
peter ece74a42ab feat(player): a roomier YouTube player, its panel shut on demand
- The full-height previous/next strips beside the card are gone: the panel's
  transport row and Shift+arrows step on their own, and the video gains the
  7rem the strips' gutters took.
- Close is one icon in the card's top-right corner (the wide panel's head,
  the stacked card's title row).
- The panel's description is cut to three lines with More / Less
  (ClampedText now takes the linked text as children).
- Wide, the panel shuts to a 2.75rem strip (close, open, previous, next) for
  a bigger video; it stays shut while the queue steps on, and a video opened
  from a card starts it open. Focus moves to the counterpart toggle.
- The wheel has one meaning per place: over a panel (or the stacked card)
  with something to scroll it only scrolls, never falling through to the
  volume at the end; over the video, or a panel that fits, it is the volume.
- A long channel name is cut on the panel's line instead of running the
  20rem panel into a sideways scroll.
2026-10-09 00:27:29 +02:00
peter 600d5c39da feat(ui): a button over a picture answers the pointer with an accent ring
The media variant's hover darkened its body from 60 to 75% black, which no
one could see on a poster; a rail's scroll arrow over a card could not be told
from the card's own Play under it. Its hover is now a 2px accent ring over an
80% body (owner pick B), on every card control, Plex and YouTube, and the rail
arrows. DESIGN.md B3 names it beside the edged pills, with its log row.
2026-10-09 00:27:28 +02:00
peter 4c7f73af9d feat(plex): a title opens its page, the picture plays; an episode links up
A card title lit like a link but played the item. Now the picture plays and
the title opens the page of what it names: a movie's or an episode's info
page, a show's page, and for an episode standing as its show (Recently
watched) the show. Every poster and episode card carries it (grid, rails,
person, collection, season, related, Home), a card with no page keeps a plain
title, and the season page gains the menu's Details row too.

The episode page had no way up: the detail now serves the show's and the
season's rating keys, and on the page the show title opens the show and
"S1·E25" the season (Back returns to where the page was).
2026-10-09 00:06:18 +02:00
peter 5f42ff61c8 feat(plex): a Surprise reshuffle deals the previous draw last
The rail draws three unseen titles per kind under the sidebar filters; on a
small filtered pool every reshuffle dealt the same few again. The redraw now
sends the draw on screen (exclude=), the server deals the rest of the pool
first and tops up from the excluded titles only when the pool is short, so a
small pool still fills the rail. (The two discover.py baseline entries went
with the rewritten draw.)
2026-10-09 00:06:11 +02:00
peter 6dd82ef5e2 fix(plex): summaries and taglines are stored as plain text
A Plex agent can hand a summary over as HTML (<p> paragraphs, &aacute;
entities); the info page and the hover card showed the tags. One cleaner,
app/plex/plaintext.py, now runs on every summary and tagline the sync, the
Hungarian overlay and the rich-detail fetch store, and migration 0101 cleans
the rows written before through the same function. Paragraph breaks survive
as newlines, which the info page and the hover card now keep.
2026-10-09 00:06:02 +02:00
peter 75f4efde98 release: 0.103.0 (bug/yt-fullscreen-button) 2026-10-08 22:50:52 +02:00
peter c0059a0c4b release: 0.103.0 notes within the item length cap 2026-10-08 22:36:15 +02:00
peter a29cadf03e release: 0.103.0 notes and version 2026-10-08 22:30:56 +02:00
peter 83886494ce fix(ui): a membership menu re-homes focus a frame later, once its new rows are on the list
Rows that just mounted join the menu's navigation list a render after the status effect ran: a row
focused that early was unknown to it, and the arrows did nothing from it (seen live after a keyboard
Retry). The retry test now arrows on from the row it lands on.
2026-10-08 22:16:47 +02:00
peter 4eaa55385f fix(plex): a collection kept after a failed refresh holds its seed title
When Plex made the collection and only the read after it failed, the kept row had no members: the
title showed unticked in the collection it had just been added to, the collection empty, and the
name stayed in the field for a retry that could only be refused as taken. The seed's row now names
the collection (child_count 1), and the panel clears the field on that refusal as on success. Also
held: the panel's lines sit beside its list, and a keyboard Retry lands on the first row.
2026-10-08 22:12:43 +02:00
peter 7541b1b016 fix(ui): membership menus keep their lines beside the list and focus in the panel
The loading / failed-load lines and their Retry sat inside role=menu, which may own its items only:
the three surfaces now open with `listInside` and the rows sit in a PopoverList. A keyboard Retry
unmounted the focused button and left focus on <body> with the menu open, the arrows dead: focus is
put back on the first row, else the field, else the new Retry. The channel tag editor said a failed
tag load as "no tags yet"; it reads the catalog query's state now. The playlist panel's name and the
tag editor's error state are held by tests.
2026-10-08 22:12:43 +02:00
peter 24ad17fc7d test: hold the show collection's type, an unknown title's 404 and Brave's fullscreen stand-in 2026-10-08 21:18:48 +02:00
peter 48bd038c5f fix(plex): a collection created on Plex keeps its editable row when the refresh read fails
The rollback dropped the new row: the next sync recreated it read-only, out of every 'add to' list, and the duplicate-name check let a retry make a second same-named collection on Plex.
2026-10-08 21:18:48 +02:00
peter 0fce732497 fix(plex): a playlist row's menu adds its title to a collection too
The row menu called the title's action list without the anchor the panel opens beside, and the optional parameter dropped the row silently. The anchor is now required, so every host menu decides.
2026-10-08 21:18:48 +02:00
peter 7e94da72b9 fix(ui): one membership menu for playlists, collections and tags
Three hand-copied panels had drifted: Add to playlist said a failed load as 'no playlists yet'. ui/MembershipMenu now holds them: a loading line and an error with Retry instead of a blank or an empty list, the up/down arrows step from the name field into rows that arrived after the menu opened, and the panels opened from a menu carry their own accessible name. A write that landed flips its tick at once, so a second tap before the refetch undoes it instead of repeating it; a collection edit also refreshes a show's own page.
2026-10-08 21:18:48 +02:00
peter fadb4f17a9 fix(plex): subtitles draw only the player's own track, lifted in the player's own pixels
A direct-played file's own subtitle stream (WebKit lists an mp4's mov_text) was switched to hidden and drawn under any choice, Off included: only the player's <track> is kept hidden and read, other subtitle tracks are switched off. The lift was measured in screen pixels and applied inside the x1.25 windowed player, and re-measured off a box carrying its own (or a mid-slide) transform: an untransformed outer box is measured, the distance divided by the scale, and a cue that just appeared is placed, not slid from where the last one was left.
2026-10-08 21:18:48 +02:00
peter 57a81c5643 fix(plex): subtitles survive hls.js and a remounted track
hls.js emptied the cues of every text track on the element when a manifest
loaded or the media attached, our own <track> included, so a VTT that loaded
first showed nothing for the session; its subtitle and caption controllers are
no longer built. A remounted default <track> was switched to showing by the
browser after the hide had run, drawing the cues natively under ours; the hide
now follows every added track and mode change.
2026-10-08 19:35:47 +02:00
peter dae0b80f6a feat(plex): add a movie or show to a collection from its card menu
"Add to collection…" opens a panel beside the card: the title's library's
manual collections tick it in or out, and a name + Enter creates a new one with
the title in it. It used to take the Collections page, a new collection, its
name and a title search, then another search per title. A lean endpoint lists
the collections with whether each holds the title. Admin only, as on the
title's page.

Also: a collection seeded with a show is made as a show collection (Plex type
2). Measured: as a movie one it came out empty and refused the show after.
2026-10-08 03:09:55 +02:00
peter 75093458c9 feat(plex): subtitles rise above the control bar while it shows
On an iPad the bar covered the text whenever it was up: the browser places a
native cue by its line, which nothing animates. The track now stays loaded as
hidden and PlexSubtitles draws the active cues, measured against the bar and
lifted above it in 200 ms, back down when it hides. The viewer's size, colour,
background and outline settings style the text directly. Also fixes a test
typing slip from the search pill change.
2026-10-08 02:59:13 +02:00
peter 5db6dc8ee6 docs(design): log the touch header search 2026-10-08 02:52:39 +02:00
peter 45a9ceae25 fix(ui): the header search has room on a tablet
On touch the 44 px key cap and Go filled a 44 px pill edge to edge and the
placeholder was cut. The key cap is left out on touch (⌘K still works), the
pill grows to 52 px so a ring shows round Go, the box's floor counts the
controls beside the text, and Go drops to its icon by the column's width,
not the screen's.
2026-10-08 02:52:32 +02:00
peter 5249c41edc fix(player): a fullscreen button the browser cannot honour is not shown
Brave on iPadOS switches element fullscreen off, patches requestFullscreen with
a video-only stand-in and fakes fullscreenEnabled, so the Plex fullscreen button
and the YouTube docked bar's did nothing. Support is now read off
Element.prototype and the buttons are left out without it (owner). A stand-in
that returns a boolean no longer throws. A test's bare matchMedia stub no
longer outlives its describe.
2026-10-08 02:50:51 +02:00
peter c3664826fe fix(ui): a caption never opens under a finger
On an iPad a tap's synthetic mouseenter opened the Tooltip; WebKit then took the
tap as a hover and swallowed its click, so every rail button needed two taps,
and with no mouseleave from a finger the caption stayed up. Tooltip now opens
for a mouse and a key's focus only (owner: no captions on touch).
2026-10-08 02:45:48 +02:00
peter c9b8fe15a3 fix(ui): a scroller with no edge to fade carries no mask
iPad WebKit kept a scrolled container's mask stuck to its content: at the
bottom of a scrolled page the last strip, a table's pager, sat under a dark
band. With nothing to fade there is no mask; isolation keeps its layer.
2026-10-08 02:43:58 +02:00
peter 1044eaf8a9 fix(plex): the player holds the page still and covers it whole; the row drops the countdown
On an iPad a drag on the video scrolled the page behind the player, and in Brave
dvh did not follow the toolbar, so a strip of the page showed under it. The
page is now locked without overscroll on a black root while the player is open,
and the player is sized from its fixed box. The time row keeps the position,
the length and the end clock; the countdown is gone (owner).
2026-10-08 02:42:31 +02:00
peter ca949d22de fix(player): the channel line wraps instead of squeezing the meta into a column
On a narrow card (a tablet, a zoomed window) the unwrapped row of name, meta and
actions left the meta a one-word column. The row now wraps: the meta keeps at
least 12rem, and the actions move down together.
2026-10-08 02:25:58 +02:00
peter 20178f0ec4 release: 0.102.0 (improvement/e37-player) 2026-10-08 01:58:01 +02:00
peter 49a9319f84 fix(player): the click layer clears only the fullscreen button's column, not a full-width band
Measured: across the full width YouTube needs only its seek bar (up to 96 px
from the bottom); the 140 px is the fullscreen button's, at the right end. A
full-width 140 px band handed clicks on the picture to the iframe, which then
takes the keyboard focus and our shortcuts with it. The layer is now two boxes,
hidden from assistive tech (a mouse surface; the keys are its shortcuts), and
the tests render the big view too, where the docked bar sits under the iframe.
2026-10-08 01:42:39 +02:00
peter 70d56ab939 fix(player): YouTube's fullscreen button stays clickable on a short player
The click layer over the embed left the bottom 22 % of the stage free. YouTube's
new embed keeps its fullscreen button 80-128 px above the iframe's bottom, so on
a stage under ~580 px (a small window, a browser zoomed to 150 %) the layer lay
over it and a click paused instead. The cutouts now have px floors (56 top, 140
bottom) and are measured against the iframe's own box.
2026-10-08 01:36:36 +02:00
peter 3865689c26 chore(release): 0.102.0 2026-10-08 01:33:39 +02:00
peter c32429ba69 fix(plex): an episode with no number is named by its title on every card
The cards, the up-next card and the mark-next label printed a question
mark where Plex has no number; the player's header already showed the
title alone. One helper (episodeNumbers/episodeName) for every place.
2026-10-08 01:16:30 +02:00
peter a504550196 docs: the superseded fallback answers its own session; who uses beacon() 2026-10-08 01:16:30 +02:00
peter 5095047c81 test(player): pin the stop URL's ids, the beacon's settle, the kept point's edges
The stop-URL contract passed with the two ids swapped and read the backend
from the working directory; nothing called a real beacon's onSettled or its
library pass-through; the kept-point rule was held by its constants only.
2026-10-08 01:16:30 +02:00
peter 4e70f168b1 fix(player): a layer that opens by itself takes no history entry
The auto-skip countdown and the up-next card pushed an entry without a
user gesture (Chrome's Back button skips those), and one pushed under the
open shortcuts sheet sat above it, so Back closed the hidden layer.
2026-10-08 01:16:30 +02:00
peter 7f17f488ab fix(nav): a sub-view step closes the overlays standing over it first
Leaving the Plex player while one of its layers was up (the info panel when
the film ended, the countdown under the back arrow) landed as a user Back:
the overlay listener spent it closing the top layer and the player stayed.
useHistorySubview's open/back now go through closeOverlaysThenNavigate,
like setPage. The RECLOSE comment no longer cites the shared Plex entry.
2026-10-08 01:16:30 +02:00
peter d1704a5519 revert: the Modal Escape guard for a fullscreen own-content video
Chrome does not hand the page the Escape it spends on leaving fullscreen (checked with a real
keypress: the first Esc leaves fullscreen, the second closes the player). The guard defended a case
that does not happen, under a comment saying it does. This reverts 121ebeeb.
2026-10-07 23:59:32 +02:00
peter 121ebeebc1 fix(ui): an Escape while a dialog's own video is fullscreen does not close the dialog
The page gets the Escape keydown with the video still the fullscreen element. When that element is
the dialog's own content (the library player), closing the dialog took the player away with the
fullscreen. A dialog opened over something else that is fullscreen still closes.
2026-10-07 23:57:04 +02:00
peter ee5290ba01 refactor(plex): the item progress body extends the shared checkpoint
ItemProgressIn repeated ProgressIn's config, fields and bounds; it now adds only `final`. The
threshold comment points where the constants live now, and plexProgressBeacon's comment is back
on it.
2026-10-07 23:16:38 +02:00
peter dd6a666ad1 test(player): hold the session stop and the kept-point rule to the server
- The client's resume thresholds equal the server's (read from both sources).
- The session stop's URL matches the route the server serves it on.
- The player stops its server session by its item and its own id on an unload and on close,
  and keeps it when the page only goes into the back/forward cache.
2026-10-07 23:16:38 +02:00
peter 392099770a fix(plex): an episode Plex has no number for is named by its title alone
The OS media title printed the missing number as "Enull", the player and info headers as a bare
"E". One helper for the three lines.
2026-10-07 23:16:38 +02:00
peter 2aa3ac6e85 fix(player): the browser Back peels each layer over the Plex video, one press at a time
All the layers shared one history entry, so the first Back spent it on one layer and the second
left the player with the stats readout (or the countdown) still up. Each layer now holds its own
entry; Back closes the one opened last.
2026-10-07 23:16:38 +02:00
peter 891dc628f3 fix(library): a listing poll can no longer put back an older resume point
A poll already out when the player closed answered with the point the server held before the
closing beacon, over the row's patch; Continue then opened there and wrote it back. The close now
cancels the listings' fetches before patching and reads them again once the beacon has landed.
2026-10-07 23:16:25 +02:00
peter 0a35f2a3b1 fix(player): a stale session answer can no longer stop the playback that won
- The engine tracks the request out per item: A, then B, then back to A sent a second request
  for A beside the one still out.
- A hardware transcode superseded during its fallback answered with the NEWER session; the losing
  request's player then stopped it as stale. It now answers with its own (dead) session.
- A replaced hls.js instance's late manifest no longer marks the player ready or plays the
  outgoing media.
- A pause given after the answer stands: the manifest and the first canplay read the live intent.
- The debounced A/V-offset restart is dropped on an item change instead of restarting the
  previous item under the next one's title.
2026-10-07 23:16:25 +02:00
peter d25a132495 fix(ui): a dialog whose own video goes fullscreen no longer crashes the app
The portal host followed the fullscreen element into it. When that element is the dialog's own
content (the library player's <video>, by its f key or its own button), the move is into the host's
own descendant, which the DOM refuses: HierarchyRequestError in a layout effect, and the crash
screen replaced the app. The host now stays where it is then — the content already shows.
2026-10-07 23:16:25 +02:00
peter 61d3a1a189 chore(db): drop plex_items.markers, a column nothing writes or reads
Markers are read live from Plex for the item the player opens; the cached copy was never filled.
2026-10-07 02:23:37 +02:00
peter d482a5e8e2 chore(dev): the dev proxy answers a failed connect with 502, as the prod proxy does
Vite's own answer was a bodiless 500, which the app reads as a real server fault and shows the blocking error dialog; a 502 is the passing connection loss it is.
2026-10-07 02:23:37 +02:00
peter 2064373ac2 fix(history): a step back never leaves the app for an older page
Chrome keeps 50 entries a tab, and when the list is full a new entry drops the
oldest one the page left without a user gesture, which can be the page's own
entry under a sub-view or an overlay. A programmatic back() then left the
document; once it landed on an older entry whose restored Plex player started
playing on its own. Every programmatic back() now asks the Navigation API first
and, with no entry of this document below, settles in place: the sub-view turns
to its root, an overlay's entry drops its mark, the channel page and the YouTube
search clear.
2026-10-07 02:23:37 +02:00
peter 0cda5b6dd4 fix(player): only the credits that end an item move on to the next one
An item can carry several credits blocks with mid- and post-credit scenes between
them, and Plex flags the one that really ends it. The player left at the first
block. Now the Plex-final block moves on (by the auto-skip or the Skip button), an
earlier block is only skipped to its end by the button, and with no final flag the
last block ends the item by the button alone.
2026-10-07 01:48:10 +02:00
peter f3c94641a7 fix(player): a library item played from the start keeps its point
A cached file that autoplays from 0:00 fires its one playing event before the resume hook
subscribes, so the hook never counted it as played and reported nothing - not on pause, not on
close. An element already playing when the hook subscribes now counts as played.
2026-10-07 00:36:28 +02:00
peter 59b27467ab feat(player): the OS media controls, the basic keys and resume for the library and shared-link players
Plex, library and /watch tell the operating system what plays (title, line, picture) and take its
play / pause / skip / seek through one hook (lib/mediaSession); the Plex player drives it with its own
seek and clock, so the lock screen shows the film's position, not the stream's. The YouTube player is
left alone: its iframe already publishes all of it, and the OS routes to the frame that plays.

The library and /watch players get the basic keys (lib/mediaKeys, the keymap's mediaPlayer rows) and
open where the viewer stopped (lib/resume): the library keeps the point on the server, per user and
library item (library_states, POST /api/downloads/{id}/progress, listed back on both listings), the
shared link in this browser. A started library row continues, its menu can start it over, and its
picture carries the cards' progress bar. The resume rule (too early / near the end = from the start)
is one function shared with the YouTube route.

The library dialog opens focused on its panel (Modal initialFocus "player"): on the ✕, Space closed
the film. While the media element itself has the focus, Space and the arrows stay its own controls'
— the browser acts on them whatever a listener does, and one Space played and paused.
2026-10-06 23:55:41 +02:00
peter 02feaa9a1c refactor(player): one panel at a time over the Plex video, and one Back cascade
The ⋯ menu, the tracks menu, the tuning panel, the info overlay and the up-next card were a flag
each, so two could stand on top of each other, and Back/Escape were written out three times. They
are one `panel` value now; the stats readout stays a separate toggle beside it. One `back()` peels
the countdown, the panel, then the stats, then leaves — for the arrow, ⌫, Escape and the mouse
Back alike. The mouse Back now also closes the ⋯ menu instead of leaving the player.
2026-10-06 23:04:35 +02:00
peter c1058f1d5a refactor(player): the Plex session machine is a tested engine, one request per item in flight
PlexPlayer's session logic (generation, pending play intent, clock ownership, session offset, reclaim
budget, listener teardown) moves into lib/hlsEngine.ts, a plain class with hls.js behind lib/hlsPort.ts
and its own unit tests.

- A seek, track change or reclaim asked for while a session request is out replaces the queued
  target; it is sent once the in-flight answer settles. A burst of 20 presses past the buffer was 20
  requests and 20 ffmpeg spawns; it is now 2-3, and plays again ~2 s sooner.
- The player stops its server session when it leaves the item, closes or the page unloads
  (POST /api/plex/stream/{rk}/session/{sid}/stop, identity-aware). A left transcode used to run to
  the end of the film first.
- A restart asked for after a new session attached but before it played no longer reads the emptied
  media as the viewer's pause.
2026-10-06 22:31:48 +02:00
peter 23f16b69df release: 0.101.0 (improvement/e36-delivery) 2026-10-06 21:14:35 +02:00
peter c813814c3c chore(release): 0.101.0 2026-10-06 21:04:19 +02:00
peter ed6facb703 fix(downloads): a conversion's commit, a stopped upload, the fast join and a clip's copy hold up
- The fast join gives each ffconcat entry its own media-only format whitelist, so a library file
  can no longer be opened as a nested concat list (the list's whitelist was handed down to it).
- A file the guarded probe cannot read is a download-only file, not a guess from yt-dlp's info.
- A conversion's or an ingest's commit that raises is re-read in a fresh session before any file
  is removed: a commit that landed keeps the file its row now names.
- An original that cannot be removed after conversion is logged; the upload stays ready.
- An upload whose conversion is cancelled or deleted midway is freed once nothing holds it; a
  paused one is kept.
- A shared link says "preparing" only while its video's job is queued or running.
- The admin storage view counts every file on disk ("Files on disk"), not only ready ones.
- A clip's failure title, unknown failure and cancel dialog, and a clip's and an upload's delete
  dialog speak of what the job made; an upload's quota refusal explains that the original is kept.
- /watch announces "preparing" as an alert and moves focus to the new Retry after an action.
2026-10-06 21:02:07 +02:00
peter d44d8d6dda fix(gate): knip sees the watch entry, the build must emit watch.html, and two hand copies are held
- The raw-source contract tests read files from disk (rawSources.testkit)
  instead of import.meta.glob, which knip counted as importing every src file —
  its unused-files lane could never name an orphan. knip now names
  src/watchMain.tsx as an entry (registered in docs/gates.md).
- vite build fails when index.html or watch.html is not emitted.
- The watch-entry import walk follows dynamic import() too.
- audit_metrics.py's plan-code patterns are held equal to the guard's.
- CrashBoundary is domain-free: it moves into components/ui.
2026-10-06 03:19:33 +02:00
peter b20e9a3947 fix(player): the stacked layout's minimum width no longer clips the wide one
The 36rem floor guards the stacked layout's width/height feedback loop; the
wide side-panel card has no such loop, and on a short wide window the floor
made the video taller than the card, cutting off YouTube's control bar. The
floor now applies to the stacked layout only; e2e covers a 1400x380 window.
2026-10-06 03:19:33 +02:00
peter 6073a006b8 fix(downloads): an upload's cancel, progress, failure and notice speak of an upload
- Cancelling an upload asks "Delete upload?" and says the uploaded file is
  removed from the server with it.
- The phone-safe conversion's real ffmpeg % shows as a bar ("Converting for
  phones"), not an indeterminate pulse.
- A failed upload is titled "The upload isn't ready", with its own reasons
  (convert_failed, retryable; upload_missing) instead of download copy.
- The inbox says "Upload ready to share" for a converted upload.
2026-10-06 03:19:32 +02:00
peter dba4758c59 fix(watch): a link whose video is being made again says so, in the page's language
The public watch route answers video_preparing for an asset that is pending or
downloading, and the page shows "being prepared — try again" with a Retry
instead of "this link is gone". watch.html ships one lang; the page now sets
documentElement.lang to the language its strings were picked in.
2026-10-06 03:19:32 +02:00
peter 68cfc7e25d fix(downloads): an upload's conversion is guarded, counted, named and cleaned up
- Every ffmpeg/ffprobe over a library file names the demuxers it may use
  (edit.INPUT_GUARD; a downloaded thumbnail edit.THUMB_GUARD): a playlist-type
  demuxer would read local paths named inside a user's upload.
- A re-encode ends on an even frame (edit.EVEN_FRAME): an odd 4:4:4/RGB source
  failed its conversion; a clip stores the frame its file has.
- An upload counts against its owner's cap from the moment it lands (queued, or
  kept after a failed conversion) — the footprint, the GC's cap and the admin
  view read one ON_DISK predicate — and a conversion that outgrows the cap is
  refused, keeping the original.
- The converted name keeps _a{id} (a dotted bare name once lost it and two
  uploads shared a file); a converted file placed but not committed is removed,
  and the orphan sweep enters .uploads; the LRU cap never evicts an upload.
- Ingest judges the extension the file is stored under; a failed conversion is
  convert_failed / upload_missing, reports the "converting" phase, and notifies
  as an upload. Job statuses in routes/downloads use JobStatus.
2026-10-06 03:19:32 +02:00
peter 00886d92fe fix(player): the YouTube player keeps its size on a short window and when maximised
Two layout bugs since the player grew a side panel, both reported from prod.

Maximised (F) on a wide window, the video was a thin line across the screen: the side-panel layout
aligns the stage `self-start`, and Chrome applies a flex item's self-alignment to an absolutely
positioned box too, so the fixed stage took its content's height — 0 px. The maximised stage now
stretches between its insets whatever the layout asks of it.

On a short window under 1280 px (a laptop at 100%, a 1080p screen at 175%) only the side strips were
left: the card narrows so the chrome under the video fits, but that chrome grows as the card narrows
(its text wraps), and the loop ran the card down to 2 px. The video now never goes narrower than
36rem to make room; past that the card scrolls.

Measured before/after (localdev, real YouTube iframe): 1097×520 open 0×0 → 610×343; 1920×911 maximised
1920×0 → 1920×911; 1440×760 and 1366×650 maximised fill the window; leaving maximise restores the card.
Held by an e2e test at both sizes (red on the old code: 0 px wide, and 911 ≠ 0), green on Chrome,
WebKit and iPad.
2026-10-05 23:40:13 +02:00
peter e7d2eb8cf6 test(downloads): what a share link serves passes the phone-safe rule
A link serves an asset once it is ready, so the contract is pinned on both sides. The code: every
function in app/ that marks something ready must run the rule (`_ensure_phone_safe`, or the
upload's own `playable.plan`) — an AST scan names any that does not, and fails blind if it stops
finding the four paths known today (download, editor clip, upload conversion, upload kept as is).
The bytes: the files an iPhone + iPad refused (HEVC tagged hev1, AV1, Matroska, MP3 inside an
.mp4) and the ones other browsers cannot be trusted with (VP9 + Opus, 10-bit H.264, Opus in Ogg)
go through the upload route, the worker's conversion and the public file route; what the link
hands out must be served as mp4 / m4a / mp3 and pass the rule.

Seen red: a planted ready-marking function in service.py is named; with the conversion step
short-circuited, all seven files fail (on the rule, or on being served as quicktime / webm / ogg).
2026-10-05 21:05:22 +02:00
peter e886890d6e perf(watch): a shared link loads the player, not the app
A /watch link is opened by whoever it was sent to — signed out, often on a phone on mobile data —
and it booted through main.tsx: every app provider, the app's i18n (every locale of every area),
the flag sprites, ~392 KB gzip before the player could start. It now has its own HTML entry,
watch.html → watchMain.tsx, which imports WatchPage and what it renders; the server hands every
/watch/ path that page (the OG block injected as before), and vite dev does the same.

Measured on the build: /watch JS 290 → 87 KB gzip, CSS 104 → 19 KB; the app's own bytes unchanged.
Two leaks had to go for that: lib/format imported the app's i18n for one "just now" string (the
relative-time formatters now take `t`, like the file's other t-formatters), and the boundary's
crash copy came from the same i18n (CrashBoundary takes its copy; ErrorBoundary is the app's
wrapper, the watch entry gives it the page's own strings). The page's popover tree went too: it
renders no popover.

Held by: a contract test walking the entry's runtime imports, naming the path to anything only the
app may load (red on a planted i18n import in lib/format), and holding watch.html's head equal to
index.html's; a DB-lane test that every /watch/ path gets the player's page with the link's OG tags
and app paths get the app (red on the old routing). Screenshots at 375 px, EN and HU, video / audio /
password / gone / wrong password / unlocked: identical to the old entry's, bar the video's buffered bar.
2026-10-05 20:51:39 +02:00
peter 5561d9a379 feat(downloads): every ready file is phone-safe — uploads and clips too
A share link is opened on whatever the friend holds. Measured on an iPhone and an iPad: AV1,
Matroska, HEVC tagged hev1 and MP3 inside an .mp4 fail there; Chrome and Firefox add HEVC and
10-bit / 4:4:4 H.264. The old check judged the codec NAME only and ran on downloads only, so a
10-bit H.264 passed, an upload went out exactly as it came (an iPhone's HEVC .mov, a webm, an mkv)
and a fast trim kept its source's codecs.

One rule now, `downloads/playable.py`: video is H.264 8-bit 4:2:0 + AAC in .mp4, audio-only is
.mp3 or AAC .m4a; `plan` says what must change, `build_cmd` re-encodes only the failing stream (or
remuxes). The worker applies it to a download, an editor clip and — new — an upload: media that
fails the rule is queued (asset pending, job queued) and the worker converts it, keeping only the
converted file (owner's choice). Until then the original sits at rel_path, so Retry starts from it
and a delete removes it; an upload that lost its last holder mid-conversion is reaped by the GC.
Every libx264 run (edit, concat, conversion) now pins `-pix_fmt yuv420p` — without it a 10-bit
source came out High 10. The download cache signature moves to v5 so a cached 10-bit asset
re-derives. The upload toast says when the file is still converting (HU/EN).

Tests: the plan for every measured class; real ffmpeg turning a 10-bit Matroska + Opus and an
mp3-in-mp4 into files the rule accepts (red without the pixel-format pin); the queued upload's
delete, cancel-then-GC, failed conversion and real worker conversion (the cleanup tests red
without their rules); the toast (red without the branch). Live on localdev: 9 fixture uploads
converted, every served file probed H.264 High yuv420p + AAC, or mp3 / m4a.
2026-10-05 03:16:27 +02:00
peter 2768ab5bdd fix(gate): the plan-code guard catches decision codes, and the purge takes them
A plan's decision ids (`D1`, `D4`, `D6-A`) cited in 11 comments and test titles named a document
no reader of this repo has, and the guard had no pattern for them. It has now (mirrored in
audit_metrics), with cases for each form the citations took and for the look-alikes it must leave
alone (a JPEG marker, "3D"); the 11 lines say their why without the id. Case test seen red without
the pattern, the scan red on a planted `D9:`.
2026-10-05 03:16:13 +02:00
peter 8c79590165 release: 0.100.0 (improvement/e35-tooling) 2026-10-05 01:42:39 +02:00
peter eef3ac9c5c chore(release): 0.100.0 2026-10-05 01:19:51 +02:00
peter 4ba7c966d7 fix(nav): the dead-page rewrite does not restore a spent focus stamp
It spread the event's snapshot, which still carried the sfNav stamp lib/history's own listener had
just taken off the entry, so a later landing there (an overlay closing) refocused a stale item. It
now spreads the live entry (test red on the snapshot). Also: format.test's comment named a test
image that never runs vitest; it is the gate host's locale.
2026-10-05 01:13:06 +02:00
peter 513dd8175a fix(auth): becoming admin through ADMIN_EMAILS drops the password, on any sign-in
The promotion revoke guessed from today's SMTP state whether the password was trustworthy, and ran
only when the account was adopted. Both guesses failed: a row verified while SMTP was off stays
"verified" after SMTP is configured, and a later sign-in matched by Google identity promoted the
account with no revoke at all. Either way a stranger's password and session could become admin.

Now one rule, where the promotion happens: when ADMIN_EMAILS turns a non-admin account into an
admin and it holds a password, the password goes and every session ends; the owner, signed in
through Google at that moment, sets a new one in Settings. An account that already is admin keeps
its password. Tests: the activated squat, the SMTP case, the later sign-in (all three red without
the rule) and an admin's own sign-in.
2026-10-05 01:13:06 +02:00
peter c4298b0a73 docs(comments): no WebSocket and no CI in the present tense
The chat cut removed the only WebSocket and CI went before it, but comments still leaned on both:
`?account=` was justified by the socket, job polling by an in-process WS registry, the e2e flows as
"proxy-safe (none use the WebSocket)", and pyrightconfig named a CI container the engine contract
test never checked. Each now says what holds today (the wallet gate, no push channel, the two
contract test halves, the image build's `npm ci`, the gate).
2026-10-05 00:50:36 +02:00
peter 5ca2a0e6ee fix(gate): the plan-code guard catches bare sprint and phase codes, and the purge takes the P ones
The sprint alternative matched only `(S6)` or `S3e2`, so the commonest purged form (`# S6:`,
`S3's`, `"""S6 —`, `S1 justified`) and every phase code (`P1`) would come back unflagged; the phase
codes had in fact survived the purge in 16 lines (Plex client sections, the upload's header shape),
now reworded around them.

One pattern now covers epic, roadmap, sprint and phase codes; the named exceptions are TV episode
notation, a fixture's season or episode title (`title="S1"`) and ruff's `E4/E7/E9`, and a ruff rule
like `S110` does not match. The case test lists the purged forms and no longer claims every form.
audit_metrics copies the definition (ASCII, as JavaScript's \b is) and reports it as
plan_code_lines / plan_code_files.
2026-10-05 00:50:36 +02:00
peter 5257d57196 fix(nav): Back onto a removed page also rewrites the entry to the feed
The fallback rendered the feed but left the history entry and the address bar on the dead page, so
a Feed click was a no-op (the same-page guard), filter mirroring kept `?page=messages` and the nav
diagnostics logged a page that does not exist. The entry is now replaced with the feed's, keeping
every other URL key (test red without it).
2026-10-05 00:50:35 +02:00
peter 58ad95263c fix(auth): a Google sign-in that makes an account admin drops a password nobody proved
The never-activated rule left one order of events open: on an instance without SMTP, a stranger
who registered an ADMIN_EMAILS address (or a sibling spelling) and was then activated by another
admin kept their password and live session, and the owner's first Google sign-in raised that row
to admin, the stranger's cookie with it. Activation vouches for access, not for who chose the
password, and without SMTP nothing else ever proved the mailbox.

When ADMIN_EMAILS promotes an adopted account that holds a password on an instance without SMTP,
the password goes and the session epoch is bumped. With SMTP the verified flag is a proven mailbox,
and an account the sign-in does not promote keeps its password, so the "your password was cleared"
detour stays off the default path. Tests: the activated squat (red without the rule; the squatter's
session is signed out) and the SMTP case that keeps the password.
2026-10-05 00:50:35 +02:00
peter 96ad1ef306 docs(comments): correct comments the cuts left stale
The registration language is read today (preferences.language, the mail language); the Modal and
the players order against popovers through layerStack, not the FloatingTree; the upload's auth
dependency is current_user; the module of access tests covers onboarding and access, not only the
canonical email. The dev proxy's WebSocket upgrade served the removed messages socket and goes; the
test compose header says a pure-logic run needs SIFTLODE_TEST_REQUIRE_DB=0; the speed measure no
longer reports an unreachable database as "no share link".
2026-10-05 00:19:50 +02:00
peter d1bbe4524c fix(gate): close the gaps in the plan-code guard, the button lists and the tooling
- planCodes: the quote and slash lookarounds let the commonest form through (a docstring or test
  title opening with a code, a slash pair); the two real exceptions (an episode titled "E1", ruff's
  E4/E7/E9) are now named and cut first, with cases that fail on the old pattern. audit_metrics
  counts by the same definition.
- button-primitive: both file lists live in the rule body, out of sight of the snapshot and the
  register; each is now pinned to its exact set of ui/ files (red when either is widened).
- check.sh --brief shows a short log whole and keeps the line before each hit, so ruff's rule and
  eslint's file path survive the excerpt.
- lint-changed runs prettier over every changed frontend file, as the gate does, not only code.
- CLAUDE.md and gates.md no longer claim the basedpyright baseline was found harmless: it was never
  triaged, and the reportOptional* entries are the likeliest real bugs.
2026-10-05 00:19:50 +02:00
peter eecc195b7e fix(migrations): dropping the demo also takes its reset job's interval-change audit rows
0098 deleted the reset job's run rows but not the scheduler_interval rows that changing its interval
wrote; with the job's label gone they rendered as a raw id. The delete now takes every job-targeted
row for it, and a test seeds both kinds below 0098 and upgrades (red on the old filter). The test
has its own file so the removed-features guard can name it.
2026-10-05 00:19:50 +02:00
peter 2f901ad643 fix(nav): Back onto a page this build no longer has lands on the feed
A history entry pushed while the Messages page existed still names it after a reload onto the new
build; popstate took sfPage unchecked and the shell's module lookup threw. It is now validated with
isPage, like every initial-load path already was.
2026-10-05 00:19:50 +02:00
peter 1f357b0c21 fix(auth): Google adoption revokes the password of a never-activated account
Without SMTP, registration marks the address verified with no proof, and an ADMIN_EMAILS /
ALLOWED_EMAILS address gets no invite row to gate it. So a stranger who registered such an address
(or a sibling spelling) before its owner kept their own password when the owner's first Google
sign-in adopted the row, activated it and made it admin, and could then sign in with it.

The adoption now treats a never-activated row like an unverified one: the password goes,
password_revoked_at is stamped and the session epoch bumped. An account already let in keeps its
password. The helper is renamed to say what it now judges (revoke_untrusted_preset_password).

test_google_callback.py drives the callback's account-matching branches through the real route,
none of which a test ran before: the squat (red on the old rule), adoption by mailbox, the
sub-collision and unverified-address refusals, the refused sign-in recorded in Google's locale, and
the link flow's conflict, mismatch and attach answers.
2026-10-05 00:19:50 +02:00
peter fb962682eb chore(blame): skip the plan-code purge in git blame
It touched 671 files' comments and nothing else; blame should keep naming the commit that wrote each line.
2026-10-05 00:19:50 +02:00
peter 4e71723a07 test(comments): no plan code lands in the code again
planCodes.contract.test.ts scans every tracked text file and every new one not yet added (comments,
docstrings, test titles, lint messages alike) for an epic or roadmap code (`E28`, `E9.5`, `R14`) and
for a sprint or review round cited on its own (`(S6)`, `S3e2`, `round 4`, `deep-review`). Not
`E2E`, not a ruff rule family (`E4/E7/E9`), not a TV episode (`S2 · E5`, a test episode titled
"E1"), not `S3h` (the media-card decision docs/DESIGN.md is headed by). Markdown and the Alembic
migrations keep their history and are not scanned.

One sweep test rather than an ESLint rule plus a ruff one: the codes sat in Python, TS, CSS, shell,
TOML, YAML and lint messages, and no single linter reads all of them. Seen red on a planted
backend/app file (all three plan-code lines named, its allowed line not).

audit_metrics.py counts epic codes by the same definition (it used to miss `E9.5` and `R6`), and
CLAUDE.md's "comment the why" rule names the test.
2026-10-05 00:19:39 +02:00
peter 4aaaaa7860 chore(comments): drop epic, sprint and review-round ids from the code
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").

Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.

Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).

One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
2026-10-05 00:19:39 +02:00
peter c73d8be094 docs(welcome): retake the landing screenshots on today's UI
The seven gallery shots dated from 2026-07 and still showed the old
labelled sidebar with the removed Messages item. Retaken at the same size
(1200x675 @2x, dark) from the localdev stack with the E2E account (no
personal account name on screen), filled for the shoot with a copy of a
real account's subscriptions, playlists and finished downloads, removed
again afterwards.
2026-10-05 00:19:39 +02:00
peter 30545d0d88 fix(gate): --brief leaves the test database's own log out of an excerpt
The DB lane provokes constraint errors on purpose, and their ERROR: lines
from the testdb container came first in a red pytest excerpt, ahead of the
failing test (seen in E35 S3). The excerpt now skips the testdb-N lines.
2026-10-05 00:19:39 +02:00
peter ac702192c9 docs(claude): drop the canonical-pattern rows a lint rule already holds
Buttons, text fields and native controls, glass surfaces, the native
<select> and network calls are each refused by a lint rule whose message
names the primitive to use (button-primitive, input-primitive,
surface-role, no-restricted-syntax, no-restricted-globals). Their table
rows repeated that, so a change had to be made twice. One paragraph now
says they are lint-held and lists the primitives. The rows nothing
mechanical holds stay, as do ORM and current time, now test-held.
2026-10-05 00:19:39 +02:00
peter 24c47d12bd refactor(backend): select() and an inline now everywhere, held by a test
CLAUDE.md's canonical table names SQLAlchemy 2.0 select() and an inline
datetime.now(timezone.utc), and both had drifted: five db.query(...) calls
remained in routes/auth.py and routes/me.py (seven before the demo went),
and four private now helpers stood in sync/videos.py, utils.py (imported
as _now by explore and maintenance), scheduler.py (_now_iso) and the e2e
seed. All are the canonical form now.

Ruff cannot ban a method call on an instance or a function by name, so
tests/test_canonical_patterns.py reads every app module's AST and fails on
a .query(...) call or a now/_now/now_utc helper — seen red on a planted file.
2026-10-05 00:19:39 +02:00
peter ab76166878 docs(gates): basedpyright keeps new-errors-only, without a burn-down mandate
E34 read the standard-mode findings and none was a real bug, so working the
baseline down as work of its own buys nothing. The lane is unchanged: it
fails on a new error, and on a run that tightened the baseline without the
commit. CLAUDE.md and the register row say so.
2026-10-05 00:19:39 +02:00
peter a42f346a32 test(gates): the snapshot holds a config per scope, not a path per file
gates.snapshot.json listed ~800 file paths under their config, so every
new or deleted frontend file regenerated it (31% of commits, E34 S2) and
the regen diff buried the gate changes it exists to show. Now a scope is
named by path (scopeOf: src, src tests, ui, lib tests, …), the snapshot
holds one effective config per scope (2345 → 1456 lines), and a new test
fails when a file's config is not its scope's.

Not narrowed: a new file in a known place changes nothing (it gets its
scope's config, checked), a glob that moves one file into another config
fails (seen: an override on Feed.tsx → "src: src/components/Feed.tsx"), a
scope whose rules change is a snapshot diff, and a file no config lints
is still a register miss.
2026-10-05 00:19:39 +02:00
peter f2fe0fc0df build(gate)!: coverage leaves the gate and runs on demand
Both lanes printed a coverage total with no threshold (owner decision E32
D4). No real hit as a lane: the one time the number found anything (5 bugs,
E32) it was read by hand, and since then it was a line nobody acted on. It
was also the most expensive part of the vitest lane, and the run that
flaked in E35 S1. The vitest lane runs without --coverage, the pytest lane
runs plain pytest; docs/development.md gives both on-demand commands, and
the register row says why.
2026-10-05 00:19:39 +02:00
peter ca707fd390 build(eslint): focus-safe-disabled folds into button-primitive
Two rules looked at the same <button>: button-primitive refused a raw one
outside the control primitives, focus-safe-disabled refused a raw
`disabled` one everywhere but FocusSafeButton and AnchoredPopover. They are
one rule now, with the same reach: outside the primitives a raw button is
refused (a disabled one with the focus message); inside a primitive and in
a test only a raw `<button disabled>` is, and the two native-disabled owners
are named in the rule rather than in a config override. One override block,
one rule entry and two register rows go; the snapshot loses a config group.
focusSafe.contract.test.ts pins every case, the primitive and test ones new.
2026-10-05 00:19:39 +02:00
peter 2eb20ca227 build(eslint): one px-frozen box selector for box utilities and grid minimums
BOX_PX and GRID_MIN_PX were two esquery patterns with two messages for one
rule (DESIGN.md: a px box does not scale with --font-scale). They are one
alternation now, with one message that names both fixes. Nothing it refused
before passes now; pxBox.contract.test.ts pins both shapes unchanged.
2026-10-05 00:19:38 +02:00
peter adaa7dc23c build(eslint): drop the Z_RAW and EDGED_LABEL selectors
Both came out of the E28 claim audit (0521149c named z-levels, e13d93a3
static labels as flat Badges) after the audit itself had fixed the sites.
No real hit since: 0 matches in every lint run from E28 to E34 (S2/S3
measurement). Each still cost a probe test (zLayer.contract, the edged-label
half of hoverAndLink.contract) and a place in the gate snapshot. The design
rules stay in DESIGN.md (Tokens: no ad-hoc z-index; Static vs pressable),
with a decision-log row saying the selectors went and why.
2026-10-05 00:19:38 +02:00
peter 91f6a77fa3 build(eslint): drop the three React Compiler rules that cannot fire on 7.1.1
react-hooks/exhaustive-effect-dependencies, memoized-effect-dependencies and
rule-suppression were switched on in E32 "for the plugin version that wires
them up". Read in the plugin source and probed (E32 review): the first's
validation is compiled off, nothing emits the second's category, and lint
mode hands the third no suppression rules. 0 hits by construction — a rule
that cannot report reads as coverage it does not give. The comment keeps
their names and why, so a plugin upgrade can re-check them.
2026-10-05 00:19:38 +02:00
peter 36653886d5 build(frontend)!: knip drops --production; plain knip stays
The production pass (7ed2ac4e, E32) asked "is this export used by what
ships?". No real hit: its only findings were 37 exports a test imports,
answered with 36 `@internal` JSDoc tags (E34 S2) — test-only exports, not
defects. Since then it has only ever asked for another tag. The tags go
with it (36, no longer read by anything), and so do the production-only
`!…!` exclusions in knip.jsonc and their register rows. Plain knip (unused
files, exports and dependencies) keeps running in the gate.
2026-10-05 00:19:38 +02:00
peter 7e681e112e test(gates)!: the size/complexity ratchet leaves the gate; audit_metrics reports it
sourceSize.contract.test.ts, its test kit and backend/tests/test_source_size.py
held 13 FE / 9 BE file ceilings and 64 FE / 32 BE function ceilings. No real
hit (E34 S2/S4): its history is its own birth (751e1ae5), two rounds of
fixes to its own logic (e432dc27, d1df76e8), the snapshot move (182ec311)
and ceiling edits inside feature fixes (5806adda, df4a79a1, and the S2/S3
cuts here). It caught no defect, went red 8 times on its own limits and drew
5 review findings about itself. The numbers stay visible: audit_metrics
reports file size and complexity, and an epic compares before and after.
docs/gates.md: the two size rows go; the C901 row says where it is read now.
2026-10-05 00:19:38 +02:00
peter 3642397f63 fix(tooling): audit_metrics counts the demo footprint by the banned word
Its pattern also matched "demote" and the Hungarian "Demokratikus", so
the footprint read 5 after the demo was gone (and 56, not 53, in the S1
baseline). It now uses the removedFeatures contract test's pattern.
2026-10-05 00:19:38 +02:00
peter 6d8b7be6d8 feat!: remove the shared demo account
The demo account — one shared user that whitelisted emails entered from
the login page, reset on a schedule — goes end to end (owner decision,
E34): demo.py and the reset job, users.is_demo and demo_whitelist
(migration 0098 deletes the demo user, its audit rows, the reset job's
override and notices, then drops the column and table; the downgrade
restores the empty schema), the /auth/demo entry, the admin whitelist and
reset, the Demo tab, the banner, the login page's "Try the demo", and
every is_demo branch in routes, userscope, scheduler and the UI.

require_human only ever refused the demo account, so its 87 uses become
current_user (owner-approved); require_write_scope keeps its own check.
The login page's "or" divider now shows only beside the Google button:
without the demo entry under it, an instance without Google would have
said "or" to nothing.

A contract test (removedFeatures) fails on the word anywhere in the
backend, frontend, e2e or scripts — seen red on a planted hit. Size
ceilings tightened for the modules that shrank; admin.py is back under
the limit.
2026-10-05 00:19:38 +02:00
peter cca3315836 feat!: remove direct messaging (chat)
E2EE direct messages, the chat dock, the Messages module, the message
WebSocket and its realtime registry, and the messages / message_keys
tables (migration 0097; the downgrade restores the empty schema). Prod
held two welcome notes, no message between people and one key — never
used (owner decision, E34). WebSocket now has no owner module: the lint
bans it everywhere. Privacy policy updated: it no longer promises E2EE
for a feature that is gone.
2026-10-05 00:19:38 +02:00
peter 7f08c510bf feat(auth)!: remove the public access-request endpoint
POST /auth/request-access had no caller left: the login page's form was
gone, and registration and a refused Google sign-in already land in the
same admin approval. A third public write path with no protection of
its own (owner decision, E34).

Its one guard moves to where every path passes: upsert_pending_invite no
longer opens a pending row for an address the env already lets in. The
registration path lacked it — registering b.ob@gmail.com beside an
env-granted bob@gmail.com opened a pending row, and is_allowed answers
from the invite rows once any exist, so the real owner was locked out.
The tests that drove the endpoint now drive registration or the shared
function.
2026-10-05 00:19:19 +02:00
peter a499f00209 refactor(compose): localdev is an override of docker-compose.yml, not a copy 2026-10-04 20:09:58 +02:00
peter c2bf8fb7f6 feat(tooling): audit_metrics.py — size, complexity, gate weight and a speed baseline 2026-10-04 20:09:58 +02:00
peter 11d35910fd feat(tooling): lint-changed.sh — fast lint lanes on the uncommitted files, usable as a Stop hook 2026-10-04 20:09:58 +02:00
peter 27f48b7a8b feat(gate): check.sh --brief prints a red lane's first error lines only 2026-10-04 20:09:58 +02:00
peter bc856ab2af test: warm cold dynamic imports before the timed cases
Under a full coverage run on every core, a test file's first cold import
(0.6-0.8 s alone) outran the 5 s case timeout and the half-done import
failed every later case in the file: the vitest lane went red in 2 of 5
runs with no defect. A beforeAll warm-up in the 4 files of that pattern;
5 of 5 full runs green after.
2026-10-04 20:09:58 +02:00
peter 0dfb4a984b release: 0.99.2 (bug/b1-e34-defects) 2026-10-04 18:28:53 +02:00
peter 5bcd1d8d8a chore(release): 0.99.2 2026-10-04 18:19:37 +02:00
peter f49789665e fix(i18n): label the history cleanup's quota_reaped result
The scheduler row read 'quota reaped 0' — the raw key, humanised — because the new result key had no
label (found in the browser check; nothing tests result-key labels).
2026-10-04 18:03:19 +02:00
peter 0e077b84c8 fix(search): the live-search tooltip says what a search costs
It always said a live search spends the shared API quota, while the default source (the website's
own endpoint, scrape) spends none — the owner avoided a free search because of it. /api/me now names
the running search_source, and the tooltip names the cost under it: none for scrape (the enrich step
costs a unit or two), 100 units a page for the official API.
2026-10-04 17:53:19 +02:00
peter ce8ccfc3c0 fix(quota): prune quota history past a retention window
quota_events had no retention: one row per API spend plus a zero-unit row per live search, kept
forever. The daily history cleanup (audit_gc) now also prunes it past quota_retention_days (a new
Quota setting, default 365 so the admin usage view's longest window stays whole; 0 = forever). The
floor of two days keeps today's rows, which the per-user search cap counts.
2026-10-04 17:53:19 +02:00
peter 33dec56e45 fix(worker): obey the stored concurrency and log level, live
The admin page's download_worker_concurrency and log_level were stored but never reached the worker:
it read the env once at boot (worker.py) and never looked again. The worker now re-reads both every
10 s (_sync_knobs): extra loop threads start at once, surplus ones retire after their current job, and
a changed level is applied to the process. An unreadable database keeps what already runs.
2026-10-04 17:53:19 +02:00
peter 2a699e26ba release: 0.99.1 (improvement/drop-ci) 2026-10-04 06:45:48 +02:00
peter 13c8c05661 chore(release): 0.99.1 2026-10-04 06:36:06 +02:00
peter cc414de952 build: the gate is scripts/check.sh, and CI goes
The Gitea Actions gate repeated, after the merge, what a release already runs
before it, and its runner shared the git host's container: one run there
starved Gitea of memory until it stopped answering. The lanes now live in
scripts/check.sh, run the same by the release dispatcher and by hand, and the
gate-contract test holds its lines (the ruff and pip-audit checks read it
instead of the workflow). The two audits only CI ran join the gate: pip-audit
over the lock (pinned in the test image, blocking at any severity) and npm
audit of the production dependencies. The engine contracts drop the CI
containers; the register records that pushes have no server-side gate.
2026-10-04 06:20:58 +02:00
peter 0528b35f2d release: 0.99.0 (feature/e32-gate-integrity)
gate / backend (push) Successful in 8m5s
gate / frontend (push) Successful in 8m32s
2026-10-04 05:19:05 +02:00
peter cb46774851 chore(release): 0.99.0 2026-10-04 05:09:33 +02:00
peter f1cc6ce5dc fix: the reconcile scan's return type names its settled sections; the composer stays at its size ceiling 2026-10-04 05:04:14 +02:00
peter 4efe799eda fix(ui): the issue composer opens on its title, not on the Edit switch
A form dialog opens on the body's first control, and the composer's first
control is the Edit / Preview switch above the form: the focus landed there,
at open and when a retried load brought the form, and typing went nowhere. A
form can now mark the field to open on (data-initial-focus); every other
dialog keeps its first control. The Modal's comment on the re-armed watch now
says where the focus goes when the focused control is removed but others stay.
2026-10-04 04:56:57 +02:00
peter 40b27f2982 fix(sync): a channel with no uploads backfills as done, not as a failure
YouTube answers the uploads list of a channel with no uploads as
playlistNotFound. The recent and deep backfills raised on it, so the channel
was never stamped and was picked again, charged a quota unit and logged an
ERROR on every run. Both now read it as an empty last page, as explore already
does; any other failure still raises.
2026-10-04 04:51:25 +02:00
peter 0314c4cce2 fix(plex): the watch reconcile settles each library on its own walk
A mirrored library that answers "0 items" is not believed (its rows stay), but
that verdict stopped the whole reconcile: one library emptied on Plex with its
mirrored items kept made every run incomplete, and un-watches in every other
library never synced again. Each section now settles on its own walk, and only
the settled sections' rows are cleared; the unsettled ones are named in the
warning.
2026-10-04 04:49:55 +02:00
peter 182ec31120 test(gates): hold the lint and lane configs to a committed snapshot
The register test detected narrowings by modelling config shapes, and each
review round found one more shape it did not model (a basePath global ignore,
a rule src never declares, a widened glob filed under a narrow row, a substring
check on a CI line, knip.json beside knip.jsonc). It now compares instead:
gates.snapshot.json holds ESLint's effective config for every code file and the
lines that run each lane, and any difference fails until it is regenerated.
The register is read off the snapshot's configs, which surfaced the ui/ and
lib/ test scopes the sampling never reached.

The complexity ratchet also names a wrapped function by its binding
(const onSave = useCallback(() => ...)), not as "useCallback callback".
2026-10-04 04:47:10 +02:00
peter 12a39f5291 fix(plex): the watch reconcile clears only what it walked, and doubts a mirrored 0
A library left out of the allowlist is never scanned, so its watch states were cleared on
every reconcile; clearing is now limited to the sections walked. A section we mirror items
from that says it holds 0 is not believed complete. The page size is no longer read as the
section total when totalSize is missing (measured: the server always sends totalSize on a
paged request, 0 for an empty one).
2026-10-04 04:16:52 +02:00
peter 79704b0001 fix(youtube): "this playlist no longer exists" only where the user picked that list
playlistNotFound was mapped to "gone" everywhere: exploring a channel without uploads (its
uploads list answers not found) and a list deleted mid bulk pull both said a playlist no
longer exists. The reason is now "gone" only on single-playlist routes; explore reads it
as an empty back-catalogue, and the bulk pull skips that list and syncs the rest.
2026-10-04 04:16:52 +02:00
peter 4191189b4c fix(ui): a late form takes a dialog's focus once, even if it arrived elsewhere
The watch stayed armed when the form arrived while the focus was off the close button, so a
later change in the body pulled the focus off a close button the user went back to. It now
disarms on arrival and re-arms only when the focus is taken away.
2026-10-04 04:16:52 +02:00
peter 7dbda60815 fix(ui): an undo in the same tick as a rebase steps back to a rebased snapshot
rebase updated the state only, so the mirror the callbacks read still held the removed
row and an undo right after it wrote that row back for a moment.
2026-10-04 04:16:52 +02:00
peter d1df76e8e5 test(gates): the complexity ratchet names a function by what it is bound to
ESLint leaves an arrow bound to a const or a property unnamed, so three ratchet keys were
(anonymous) and ranked by position again. A tiny rule reports each unnamed function's
binding (const, property, assignment, else <callee> callback) at the head complexity
reports, and the keys line up by location.
2026-10-04 04:16:52 +02:00
peter 3f6a9efd57 test(gates): the register test parses ruff's TOML and holds pytest.ini line by line
Round 2 of the E32 review found the key whitelists blind: a line regex saw neither a
[lint.per-file-ignores] table nor a quoted key, and pytest.ini's continuation lines and
key: value form slipped past. Ruff's config is now parsed (smol-toml) and held to its
key paths; pytest.ini to its whole lines; the CI ruff steps to --config backend/ruff.toml,
which is what makes that file the whole configuration.

ESLint: every glob of a block must reach a sample (reactPlugin.ts and devOnlyGuard.ts
rode along unseen), a block's own severity is held to src, a no-restricted-* list is held
to the src block's declaration rather than Feed.tsx's effective config, named global
ignores count, an ignored sample is named, and linterOptions are checked. tsconfig.node
must keep allowJs + checkJs while it lists a .js file; knip's positive glob is pinned.
2026-10-04 04:16:52 +02:00
peter 64e8f1da3d fix(lint): E402 is on — the ignore meant for main.py had let 11 late imports in elsewhere
ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
2026-10-04 03:14:24 +02:00
peter a1224e9a8c test(gates): the repo-root scripts are linted and format-checked
scripts/check-node-modules.mjs and gen-changelog.mjs run in check and release but sat outside
every lane: the eslint and prettier lanes run from frontend/, and an ESLint config cannot reach
past its own directory. A contract test now holds them to the js-configs bar (recommended rules
under Node, the frontend's Prettier style); an unused variable and the old formatting are each
red. 0 lint hits; both files reformatted. Registered in docs/gates.md.
2026-10-04 03:00:49 +02:00
peter da4b19a39a docs(lint): the config comments say which rules can report, and count what is suppressed
eslint.config.js called every zero-hit React Compiler diagnostic a tripwire. Under
eslint-plugin-react-hooks 7.1.1 three cannot report anything: exhaustive-effect-dependencies (its
validation is compiled off), memoized-effect-dependencies (nothing emits its category) and
rule-suppression (lint mode passes no suppression rules). Probed: a missing effect dependency is
reported by exhaustive-deps only, and an inline disable is named by nothing. The header's
"33 suppressions" of exhaustive-deps is 0. reactPlugin.ts said one react-hooks rule was still
off; none is (three run at warn, which --max-warnings 0 fails).
2026-10-04 03:00:49 +02:00
peter 5a74cf86b7 test(api): the wire-key checks send every filter field, and tsc says when one is new
The Plex and feed filter fixtures were typed as the filter type itself, whose scalar filters are
optional: a new optional field compiled without a test value, so its wire key went unchecked.
Both are Required<…> now (a probe field fails typecheck on each). The feed set gained the fields
it omitted (year, librarySource, the scope and show switches) and goes out with and without year,
which wins over the other date filters; a misspelt year key is now red in the wire check, which
passed it before.
2026-10-04 02:57:09 +02:00
peter e432dc2780 test(gates): the complexity ratchet holds each function by name, and its scope is registered
The ceilings were a per-file ranked list, so one function could grow by what another in the same
file lost: Home.tsx's [34, 32] passed with Home at 32 and Hero at 34. Both twins now key every
offender as module::function (an unnamed one is (anonymous); a repeated name gets 2, 3 in source
order) and re-baseline at today's measured values. Swapping two ceilings is red on both sides
(Home 34 > 32; feedquery filtered_query 28 > 14).

The ratchet measures shipped code only. docs/gates.md said C901 covered backend + scripts; it
covers backend/app. The exclusion of tests, test kits, e2e and scripts/ is now a row with its
measured hits, and the stale counts are replaced (9 BE modules over 800, not 11).
2026-10-04 02:55:07 +02:00
peter 3741d1a46d fix(playlists): a list reopened from the cache keeps its rows
Leaving the Playlists page and coming back painted the selected list empty ("0 videos") while
the rail counted its rows. The page re-seeds its undoable order from the cached list in an
effect; StrictMode runs that effect twice on the remount, so a reset was followed in the same
tick by a rebase. Since E32 S3 a reset is state only and the rebase read the mirror ref, which
still held the empty initial history, and wrote it back over the reset.

rebase is now a state updater, so it builds on whatever an earlier call in the tick installed.
Both tests seen RED (hook: [] for [10, 20]; page: no rows), and the bug reproduced in the
browser on the old hook.
2026-10-04 02:48:10 +02:00
peter 98ec1abf25 test(gates): the register test reads every config block, option and key, not eight samples and severities
E32 review round 1 (T1–T5, s1). The test that holds docs/gates.md to the effective configuration
could itself be passed by a narrowing it did not look at:

- ESLint: besides each scope's sample, every config block is read on its own. A block that targets
  files no sample reaches fails as an "unsampled block"; a no-restricted-* list re-declared without
  an entry src has, an option that lets something through (allow…/ignore…/except…), a block-level
  `ignores` and EVERY global ignores object (not only the first) count. A scope switching a src rule
  at warn off now counts too (it skipped them). Two live narrowings surfaced and are registered:
  no-floating-promises' allowForKnownSafeCalls (src and e2e) and the three network owners'
  no-restricted-globals.
- TypeScript: a strict sub-flag set false beside `strict: true`, an extends/exclude/files, a changed
  include, and a frontend-root config file no project checks. eslint.config.js is that file: listed,
  with the 43 errors strict checkJs reports in it (measured, none a defect).
- Ruff, basedpyright, pytest and knip are read as key whitelists (an unknown key fails until someone
  decides whether it narrows); pyright's information/hint count as weak; ruff's extend-select is
  pinned; pytest's three keys are pinned exactly; knip's script must run both modes.

Each new check was probed with a planted narrowing (an unsampled block, a block ignores, a second
global ignores, a warn rule off in a scope, strictNullChecks false, a ruff extend-exclude, a pytest
-m) and failed on it.
2026-10-04 02:37:33 +02:00
peter e5b155ada9 refactor(ui): reuse the store, media-query and deleted-tag rules the review found rebuilt, and name a loading pick
E32 review round 1 (F3–F6).

- Channels' open-row store and the chat dock use lib/store's createStore instead of a hand-rolled
  listener set (the chat dock gets a test of its own).
- The Plex backdrop reads prefers-reduced-motion through useMediaQuery; the query string is one
  constant (REDUCED_MOTION) for the hook and the carousel's one-off read.
- A link-restored collection filter showed its raw rating_key in the picker while the titles
  loaded: usePlexCollectionTitles now says it is pending, the picker draws a placeholder
  (labelsPending, as the channel picker does) and so does the chip row.
- The "deleted tag" rule was written twice with two i18n keys and had drifted: the channel manager
  showed NO chip for a pick while the tag list loaded, or ever after a failed load, while the pick
  still narrowed the table. One rule now (lib/tagPicks), one key (common.deletedTag); the manager's
  row shows every pick, as a placeholder while loading and as its id after a failure.
2026-10-04 02:29:03 +02:00
peter e09d613b04 fix(ui): a reviewed log takes the focus back once, and a dialog's focus survives a Retry
E32 review round 1 (F1, F2).

- Issue composer: the review's focus-return token became state in S4 and was never spent, so every
  Preview → Edit round trip remounted the attachments and moved the focus from the toggle to the
  log's Review button. The token is cleared once it has been used, as the ref was.
- Modal (initialFocus "field"): the late-form watch moved the focus to the first control once and
  stopped. A failed load's Retry then gave way to the spinner, its focus fell to <body> inside the
  aria-modal dialog, and the form's first field never got it. The watch now re-arms when the focus
  falls to <body>: it waits on the ✕ and moves to the first field when the form arrives. Focus in a
  dialog stacked on top is left alone.
2026-10-04 02:29:03 +02:00
peter 38943fa93e fix(maintenance): one clock per run, so a video phase 1 recovers is not fetched again in phase 2
E32 review round 1 (B7). Each phase took its own now; phase 2's "not checked this run" test then
let a just-recovered video through and spent another videos.list call on it.
2026-10-04 02:14:04 +02:00
peter 887b4e546c fix(feed): a malformed saved view cannot break the autotag cleanup, and a foreign private tag reads as missing
E32 review round 1 (B5, B6).

- The autotag orphan cleanup reads every user's feed views. A view whose tag pick was not a list
  ("tags": 5) raised for everyone's job, True passed as tag id 1, and ~66k ids pushed the NOT IN
  past psycopg's 65535 bind parameters. The views are now read defensively and the kept ids
  filtered in Python; the saved-views route holds filters.tags to a list of at most 500 ids, on
  create and on update.
- tags_by_category resolved any tag id, so another user's private tag landed in its real category
  and the feed summary answered differently than for an id that does not exist, which told the
  caller which foreign ids exist. It is owner-scoped now (pre-existing; S2's missing-tag branch
  made the difference visible).
2026-10-04 02:14:04 +02:00
peter df4a79a1f1 fix(sync): an empty answer the remote's own count contradicts is not believed, one level down
E32 review round 1 (B1, B2, B3, B8).

- Plex watch reconcile: a section the server SAYS is empty (totalSize 0, nothing listed) is a
  complete listing. It read as "cut short" forever, so one empty library stopped un-watches from
  ever syncing. A listing with no total at all stays incomplete (the client now returns None for
  it instead of 0), and a whole scan that listed nothing still clears nothing. Tests now cover
  completeness across two sections, both orders.
- Plex collections: no members from a collection whose listing counts some (childCount) keeps
  the membership we hold, in the full sync and in the targeted resync after a write.
- YouTube playlists: no items for a playlist whose listing counts some (itemCount) keeps the
  mirror in the bulk pull, and is refused as retryable in "reset to YouTube" and in the push
  plan/merge, where it read as "YouTube deleted them all". The rule sits beside vanished().
- A playlist deleted on YouTube answered "reconnect your account"; it now says it is gone
  (errors.youtube.gone, HU+EN).
- The pure 3-way merge moved to sync/playlist_merge.py; playlists.py is back under 800 lines and
  off the size list.
2026-10-04 02:13:53 +02:00
peter 5d10a521a8 docs(gates): react-hooks/refs is on, the compiler skips one function, and the async-cleanup idiom that keeps it so 2026-10-04 00:05:37 +02:00
peter d7fb7075b8 refactor(ui): every function compiles under the React Compiler but VirtualGrid, and react-hooks/refs is on
E32 S4 — the burn-down from 29 skipped functions to 1:

- try/finally (26 sites): the cleanup runs after a try/catch; steps that return early moved into
  their own function (push, the setup wizard's next, the share page's unlock).
- A conditional, loop or optional call inside a try skips the function too, and react-hooks/todo
  does not see it: the value is computed before the try, or the try lives in a module-level helper
  (config writes, clip queueing, clipboard, frame counts, seekable end, YouTube player times).
- The two exhaustive-deps suppressions are gone. useYtProgress takes its per-item snapshot through
  one effect event, read once when the player is built; PlexPlayer's unmount relies on the
  leaving-the-item cleanup for the generation bump it duplicated.
- What the skips hid: PlexPlayer's 17 render-phase ref writes (useLatest), its tracks popover alias,
  the loadSession self-reference (declared early, set after commit); PlayerModal's session-position
  read during render (now where the player starts), navRef/modeRef writes, and a description helper
  that ran a ref-carrying callback during render (now PlayerDescription); SavedViewsProvider's
  latest-cfg ref, and its setState in the seed effect (preferSticky now prefers the starred default
  among same-filter views, so the seed needs no sticky write — test seen red on the old helper).
- A store's hook was called `use()`, which the compiler does not know as a hook: inside the newly
  compiled usePlaylistActions it was memoized, ran on the first render alone, and the playlist rail
  threw "rendered fewer hooks" on its second render. Renamed to `useValue()` everywhere; the old
  contract test claimed the call was safe inside any `use…` function, which was false — replaced by
  a render test that is red under the old name.
- PlexPlayer's palette ref was written and never read since the Backspace guard moved to the layer
  stack: removed.

react-hooks/refs is back at error with no exception. The ratchet lists only VirtualGrid
(IncompatibleLibrary) and pins both players as compiled. No size or complexity ceiling was raised.
2026-10-04 00:05:37 +02:00
peter a486c7521a docs(gates): the compiler burn-down after E32 S3, and the two idioms that keep it
react-hooks/refs 245 -> 31 hits and 93 -> 29 skipped functions in the register and the eslint
comment. The canonical-patterns table gets the latest-value row (useEffectEvent; a handler or timer:
useLatest / useStableCallback, never a render-phase ref write) and the popover / scroll-fade row
(destructure the ref setter).
2026-10-03 23:18:33 +02:00
peter 76358a0d77 refactor(ui): the popover engine and every render-phase ref write compile under the React Compiler
E32 S3. The compiler skipped 93 functions; it now skips 29 (try/finally shapes, two suppressions,
VirtualGrid). react-hooks/refs hits 245 -> 31.

- Popover engine: `ref={pop.refs.setReference}` made the compiler treat the whole popover handle as
  a ref, so every component holding one was skipped (21 call sites, the engine and its surface
  among them). The setter is destructured now; the same for useScrollFade and useReturnFocus.
- Render-phase `ref.current = x` writes moved to `useEffectEvent` where only effects read them,
  else to the new `useLatest` (written in a layout effect) / `useStableCallback`.
- FocusSafeButton (under every Button) read the focused element during render; it reads it through
  useSyncExternalStore, which React asks on every render instead of a cached memo.

Bugs this surfaced, fixed:
- Combobox: a chip could keep showing its raw id. The label cache was a ref an effect filled after
  the render that needed it; it is state merged during render now (test seen RED).
- useUndoable rendered from a ref plus a forced re-render; compiled, that memoizes a stale value.
  The history is state now, mirrored in a ref for the stable callbacks.
- ConfigPanel and the saved-views editor re-seeded their drafts in an effect, painting the stale
  draft for a frame; both adjust during render against the remembered key.
- Toaster's handover is written after commit instead of during render; a new test pins that the
  inherited set is taken once, at the takeover (seen RED on a per-render re-read).
- IssueComposer mutated a ref prop as a one-shot focus token; it is plain state.
- PlexBrowse wrote its info-scroll memory from an effect declared before the ref, past its own
  "one writer" rule; the writer comes first now and the effect goes through it.

The compiler ratchet drops to the measured 29 and now also requires FocusSafeButton and the popover
engine to compile.
2026-10-03 23:18:26 +02:00
peter dc00b55feb test(auth): warm the Welcome import once so the first case cannot time out under load 2026-10-03 21:53:06 +02:00
peter fb6b56f121 test(api): every filter key the client sends is one the backend reads
FastAPI ignores an unknown query parameter, so a misspelt or renamed key silently stops a
filter while its chip still shows. Contract tests read routes/feed.py _feed_params and
plex/query.py LibraryFilters. The superseded feedCount/facets calls and keys are removed.
2026-10-03 21:53:06 +02:00
peter 2e2d6f5746 fix(downloads): the quota editor opens at once and says it is loading or failed
It rendered nothing until its quota arrived and nothing at all when the load failed, so the
click looked dead. The first field still takes focus when the form arrives.
2026-10-03 21:53:06 +02:00
peter 2ab9ff6a42 fix(plex): a server without a machine id refuses the collection write by name
The missing identifier became server:/// in the write URI. Client tests over a MockTransport
(errors, paging, cutoff, write URIs, image transcode) and admin-route tests (connection test,
enabled-section sampling, no server URL in a 422, the one-time watch import).
2026-10-03 21:53:06 +02:00
peter 378c4daf0f fix(youtube): the scrape search never pins a stale key, and an unknown shape is an error
A failed config fetch or a keyless consent page cached the fallback InnerTube key for the life
of the process; once it went stale, search returned nothing with no error. The fallback is now
used for that call only. A 200 JSON without a result frame returned an empty page that read as
no results; it is a ScrapeError now. Parser, cursor and Shorts-probe tests added.
2026-10-03 21:53:05 +02:00
peter 8f1208fdfe fix(feed): a filter on a deleted tag matches nothing and its chip says so
A saved view or the stored filter outlives its tag. feedquery dropped the unknown id, so the
view showed the whole feed under its old name. The id now lands in a MISSING_TAG_CATEGORY that
filters to nothing in the SQL and in feed_summary's mirror alike; the feed chip reads "Deleted
tag" (a placeholder while the tag list loads, the id if it failed). Channels' persisted tag
filter gets the same chip, first in the row so the +N overflow cannot fold it away.
2026-10-03 21:53:05 +02:00
peter 52f12e5331 test(e2e): click only a channel link that is the element hit at its centre
A card inside the viewport can still sit under the chrome above the scroller (the update
banner); the covered click made Playwright scroll and retry, moving the feed the scroll-restore
test then measured. webkit failed 3 in 24 before, 0 in 48 after.
2026-10-03 21:10:27 +02:00
peter c4abbe48c1 refactor(sync): the empty-answer rule for a mirror is one function, vanished()
The YouTube playlist guard from the previous commit put sync/playlists.py over its size ceiling;
the rule now lives beside should_prune as vanished(existing, listed), which the mirror calls in
place of its own list. The pyright baseline also drops by one (187), from the plex/overlay split.
2026-10-03 19:45:14 +02:00
peter 4fc307c426 fix(autotag): the orphan cleanup keeps a system tag a saved view filters on
Feed views store tag ids. A system tag no channel carried for a while was deleted and, once a
channel earned it again, recreated under a new id — the view's filter then pointed at nothing, the
feed dropped the unknown id, and the view showed the whole feed under its old name. The cleanup now
skips tags a saved feed view references (test seen red without it).
2026-10-03 19:37:24 +02:00
peter 328d415cfe test(e2ee): the DM crypto round-trips, refuses the wrong key, and keeps its nonces apart
lib/e2ee was 14% covered and fails silently in every way it can break. Real WebCrypto and an
in-memory IndexedDB: a message decrypts for its partner and nobody else, every message gets its own
IV, a new device unlocks with the passphrase and refuses a wrong one, the key survives a reload
until reset, a partner's rotated key is used only after forgetConv, and the key check never shares
the wrap IV (GCM nonce reuse). Mutations seen red: the shared IV, a no-op forgetConv, no persist.
2026-10-03 19:35:36 +02:00
peter 43e4929b45 fix(sync): an empty answer no longer empties a mirror; maintenance checks each video once a run
The catalog prune and the subscription import have long refused to reconcile against an empty
listing that contradicts what we hold (should_prune: a 200 with no items is a blip far more often
than a user who removed everything). Two mirrors did not: the YouTube playlist mirrors (dropped
every mirror) and the Plex collections (dropped every collection with its user-set `editable`
flag, and every member's keys — the guard landed in plex/sync.py with the previous commit). Both
now hold the same rule; test_empty_listing_guards covers each way (seen red).

sync/maintenance (19% covered) gets a lifecycle test: recover, grace, delete + one notification
per user, live stream kept, abandoned premiere. It found the rolling revalidation re-fetching the
same videos until `checked` reached the batch (15000 by default) whenever fewer videos were
available — quota spent on answers it already had; a run now skips what it has checked.
2026-10-03 19:33:38 +02:00
peter 5806adda7e fix(plex): a listing cut short no longer clears watch state or Hungarian text
The daily watch reconcile reads "absent from Plex's scan" as "Plex un-watched it" and deletes the
row — but it walked the sections without asking _paginate whether the walk reached the server's
total, so a Plex hiccup mid-walk (an empty page early) un-watched every title it failed to reach
until the next complete run. The scan now reports completeness; an incomplete one still pulls in
what it saw, re-pushes dirty rows, and clears nothing (logged as plex.watch.reconcile_incomplete).

The HU overlay had the same shape: clear every row's HU text, then refill from the listing. It now
reads the listing first and clears only after a complete one. The overlay moved to plex/overlay.py
(plex/sync.py drops under the 800-line limit and off the size ratchet's list).

Tests: plex/paths (23% -> 100%: prefix boundary, longest prefix, dead mount), the reconcile's
clear/keep/re-push rules, the cut-short scan and listing (each seen red with the guard removed).
2026-10-03 19:33:30 +02:00
peter c73761ae39 refactor(plex): one cast row for a show's page and a title's info
CastStrip (a show's page) and PlexInfo (a title's page and the player's overlay) each drew the
cast carousel by hand, and the copies had drifted: the gap, lazy pictures, the name's tooltip, the
initial's size. Both now render CastRow, which takes the overlay's white-on-art palette as a tone;
each caller keeps its own frame and heading. A DOM test holds the palette per variant (seen red
with the tone wired to "panel"). jscpd over the shipped code: 1 clone >= 10 lines -> 0.
2026-10-03 19:12:17 +02:00
peter ca9ca85b18 test: no test throws a response away; the stream session is narrowed, not baselined
The 7 baselined Optional accesses in plex/stream.py were one fact the type did not carry: the
initial spawn (replace_only=None) always supersedes, so only the fallback respawn can get None.
An assert at the call site says so; the pyright baseline drops 195 -> 188.

test_abuse_limits: the public-URL and YouTube enqueue tests stopped at the profile lookup
(400 no_profile) and passed on that unrelated refusal. They now send an inline spec and assert
the job is queued. The same shape — a request whose response is discarded — sat at 53 more
places across the suite (all returned what they should, measured with a TestClient probe);
each now asserts its status, and test_response_checked keeps it that way (seen red).
2026-10-03 19:06:47 +02:00
peter 614d728561 test(e2e): the nav spec opens the admin pages from their rail panel
E31 put the admin pages behind one rail row ("Administration", nav-admin)
that opens a panel. The spec still clicked nav-scheduler, nav-config,
nav-users and nav-audit as rows of their own, and timed out on every engine.
That went unseen from E31 to now because e2e ran in no required gate, the
exact hole E32 closes: the first run of the new release e2e step found it.

The rail test keeps the rail rows. A new test opens each admin page from the
panel and checks that the group row is current and that the reopened panel
marks the page. Seen red before the fix; the whole suite then passes, 52/52.
2026-10-03 17:41:26 +02:00
peter 783f6408f9 docs: the gate register, held to the effective config by a test
docs/gates.md lists every place a gate deliberately checks less than it
could, each with its reason and the count measured when the row was
written:
- an ESLint rule off, at warn, or not applied, by scope;
- a weak compiler flag, a ruff ignore, the basedpyright mode and baseline;
- a knip exclusion, a lane that does not run everywhere;
- an accepted CVE.
gates.contract.test.ts reads the EFFECTIVE configuration and fails when a
narrowing has no row:
- ESLint's computed config per scope;
- the three tsconfigs, ruff.toml, pyrightconfig, pytest.ini and knip.jsonc;
- .trivyignore.yaml, where an entry also needs a statement and an expiry;
- CI's pip-audit ignores.
It also pins `--max-warnings 0`, which is what makes a warn-level rule fail.
Seen red on a missing row, a new `off` rule and an allowlist entry without
an expiry.

CLAUDE.md gains the rule E32 exists for: a gate with any real hit is never
switched off or narrowed, and an "enforced by" comment is checked by running
the check. The gate descriptions there, in docs/development.md and in
backend/app/README.md name the lanes as they now are, and the "directly"
recipe points at the full list.
2026-10-03 17:26:42 +02:00
peter 751e1ae5ed test: a size and complexity ratchet; coverage reported on both sides
The "class size" item from the quality checklist, measured:
- A shipped file over 800 lines may not grow past its line count today
  (13 frontend, 11 backend). No other file may cross 800.
- A function over the complexity limit may not rise, and no new one may
  appear. The limit is eslint `complexity` 20 on the frontend (64 functions
  over it today) and ruff C901 10 on the backend (32).
- Each is a per-file ceiling. Shrinking is free; an entry back under the
  limit must be deleted, so the lists only shrink.

C901 is measured in a test rather than switched on in ruff.toml: ruff has no
per-function baseline, and a noqa would let a function grow without limit.

Coverage (owner decision D4) is printed with no threshold:
`vitest --coverage` (v8) gives 80% of lines, and the pytest lane ends with
the total for app/, 78%. The bar stays "every regression test seen red".
.coverage is ignored by git and kept out of the image's build context.
2026-10-03 17:18:53 +02:00
peter da554df7c7 build(image): the image passes a Trivy scan: OS security updates, no runtime pip
The first Trivy scan of the shipped image (0.98.0) found four fixable HIGH
advisories. One was pcre2: a Debian fix newer than the base snapshot, now
taken by `apt-get upgrade` at build time. The others were urllib3 (bumped in
the lock), and msgpack, setuptools and urllib3 vendored inside pip itself.
pip is removed once it has installed the lock, since nothing runs it at
runtime.

A local build of this Dockerfile with --pull scans clean: 0 HIGH/CRITICAL
with a fix, no secrets. .trivyignore.yaml is the publish gate's allowlist,
empty today. Each entry must carry a statement and an expiry date.
2026-10-03 17:06:28 +02:00
peter b1ff4058ac build: CI builds the bundle and audits what ships; the toolchain advisories are cleared
The server gate adds `vite build` (and with it devOnlyGuard, which until now
first ran inside the image build, after main was pushed) and
`npm audit --omit=dev --audit-level=high`. pip-audit is blocking: when it was
finally read it held three urllib3 advisories with a fix available.
urllib3 goes 2.7.0 -> 2.8.0 as a single in-place pin, passing pip check,
rather than a regeneration that would move every package.

The dev toolchain's 18 advisories are gone:
- vite 5 -> 6.4.3 (path traversal, fs.deny bypass, and esbuild <= 0.24.2
  through it);
- vitest 3 -> 4.1.11 (mocker path traversal);
- typescript-eslint 8.46 -> 8.71;
- `npm audit fix` for the rest.
tsc, eslint, knip, all 2269 tests and the build pass unchanged.

scripts/check-node-modules.mjs checks that frontend/node_modules holds
exactly what package-lock.json pins. The local gate's frontend lanes judge
the host tree, and nothing tied it to the lock the image installs.
2026-10-03 16:50:56 +02:00
peter 7ed2ac4e95 build(frontend): knip also checks what ships; strict indexing in every project
`npm run knip` now runs a second, --production pass, which asks whether the
code that ships uses an export, not just whether anything does. Of the 47
exports only tests reached:
- 34 are exported for their test and say so with @internal, naming the test.
- 11 were module-private names with a stray `export`. Two of them were not
  used at all and are deleted.
- The unused alreadyReported re-export is gone.
Test kits, the e2e suite and the two build-tooling modules are excluded from
the production project, each named in knip.jsonc.

noUncheckedIndexedAccess is now on for the node and e2e tsconfigs too (0
errors), matching the app project. tsconfig.node.json listed a
tailwind.config.js that has not existed since Tailwind 4. That entry is now
playwright.config.ts, which the typed lint finds through this project.
2026-10-03 16:44:28 +02:00
peter cff8fe9a6a test(frontend): a React Compiler ratchet over everything the build ships
compilerCoverage.contract.test.ts used to check two files. It now runs the
real compiler over all of src (minus tests) and holds the exact set it skips
today, file -> function -> category: 93 functions. A new skip or a category
change is red, and so is a listed skip that has gone, so the list only
shrinks. Channels and DataTable must also show up as compiled, so a harness
that compiled nothing cannot pass.

Measured against the audit's scan: no function this sprint touched started
to skip. PlexPersonView and useOverlayOrder now compile.

reactPlugin.ts claimed the compiler's rules were linted, which was false
while react-hooks/refs was off. That comment and the refs block in
eslint.config.js now say what holds the line and why refs is still off:
245 real hits, pinned by the ratchet, burned down from S3.
2026-10-03 16:38:46 +02:00
peter 471846d784 build(frontend): close the lint gaps the E32 audit measured
eslint now also enforces:
- the React Compiler diagnostics that `recommended` leaves out (todo,
  memo-dependencies and ten zero-hit tripwires);
- a description on every suppression, and no blanket disables;
- no focused or skipped tests (vitest and playwright);
- the typed pair no-floating-promises and no-deprecated.

Lint also covers e2e/ and the TypeScript build configs, which `eslint .`
had been skipping in silence.

Every hit is fixed or suppressed inline with a named reason:
- The 31 try statements the compiler cannot lower yet carry a suppression
  until S3 rewrites them. Five other compiler skips were plain fixes:
  a module counter bumped inside a hook, a ref read in a default
  parameter, and a local named like the <section> it renders.
- The real defects behind floating promises are fixed. ChatDock threw at
  nobody when the device key store could not be opened. VideoEditor's
  play() rejection went unhandled. loadYouTubeApi never rejected, so a
  blocked IFrame API left the player on an empty stage forever; it now
  fails on a script error or after 15 s, shows its own error with the
  way out, and lets the next open try again.
- MutableRefObject, FormEvent and lucide's deprecated brand icon are
  replaced. The YouTube glyph now lives in lib/brandIcons.
- The lint-probe harness lets its virtual file names into TypeScript's
  default project, so the contract tests still run the real config.
2026-10-03 16:34:20 +02:00
peter 3c16012524 build(backend): close the lint gaps the E32 audit measured
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.

PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
2026-10-03 16:04:06 +02:00
peter cdfe7a8986 perf(channels): the row menu opens and closes without re-rendering the table
The React Compiler silently skipped Channels and DataTable (a ref written
during render; the refs lint is off), and even once compiled it dropped the
memo of the rows and columns (they reach filterRows/optionsFromColumns past
the page's hooks) and the columns read the open row. Every open, close and
reposition of the right-click menu rebuilt the whole table: on "All" rows
~0.9 s to open and ~0.6 s to close (dev). Now: effect events instead of the
render-written refs, explicit memos for rows, columns and their inputs, a
module-scope keyOf, and the open row reaches its "…" through a small store.
Measured on 397 rows (dev): ~75 ms to open, ~35 ms to close.

A contract test runs the real compiler over both files and fails on any
skipped function. The dropped memos have no automated guard: they need
the full page rendered with its providers; the measurement above is the
evidence.
2026-10-03 05:10:52 +02:00
peter 3e96211db0 fix(plex): the backdrop crossfade toggle wins over reduced motion once set
prefers-reduced-motion hard-disabled the rotation whatever the toggle said,
and Windows reports it whenever its animation effects are off — so the
owner never saw the crossfade. Untouched, the setting now follows the
system (and the toggle shows that state); a pick made on the toggle is
stored and wins on every device.
2026-10-03 04:38:50 +02:00
peter e9c15baae0 fix(feed): a channel filter's chips name their channels from the first paint
After a reload the chips (filter panel and the row above the feed) read as
raw UC… ids until the subscription list landed. The names seen last time
are now kept per account (useChannelNames) and name the chips at once; the
list then replaces the copy, so a renamed or dropped channel heals. A
channel never named on this browser draws a placeholder bone while the
list loads; a failed list still falls back to the id.
2026-10-03 03:29:54 +02:00
peter 6edb22988c release: 0.98.0 (feature/e31-design-review-2)
gate / frontend (push) Successful in 8m29s
gate / backend (push) Successful in 9m54s
2026-10-03 02:58:46 +02:00
peter f9c3714025 release: 0.98.0 notes — design review 2 2026-10-03 02:58:07 +02:00
peter a4775e1cd0 fix(ui): the filter rail shows no tag counts until the facets land
It printed the tag's whole-library count as the current filter's while the
facets loaded, and for good when they failed: live, "Music 238" beside a
search that matched 3 of its channels. Ordering still falls back to it.
2026-10-03 02:43:52 +02:00
peter b40440b825 fix(ui): review round 2 — reveal floor, failed next page, count states, row Play
- DataTable: the reveal floor holds only in the view it was made for (Channels
  never clears its follow, so every later sort mounted rows up to it)
- Feed + Plex browse: a failed next page stops the read-ahead and shows a row
  that retries, instead of re-requesting it once a round-trip
- the YouTube Load more click uses nextPageOnClick; a contract pins NEXT_PAGE
  to the read-aheads and the click sites to nextPageOnClick
- ItemCount: a failed count says so; a stale one says updating to a screen reader
- the narrow-picture Play lift applies only where a hover row is there to clear
- PlayerModal: one description ladder for the popover and the panel, both tested
- plexImage contract reads every img src, any attribute order
- Tabs comment names who persists the sub-tab
2026-10-03 02:25:15 +02:00
peter c4c442bc5b fix(feed): a hand-made relevance cursor is refused, not a crash
The page halves compare the cursor's timestamp with the window's in Python;
a null or naive one raised there, past the route's 400. decode_cursor now
refuses a null non-nullable key and a naive timestamp.
2026-10-03 02:25:14 +02:00
peter c74ceb73e3 fix(feed): the count pill dims while a new search's count loads
keepPreviousData held the last filter's count, and a broad word's count
takes seconds: the pill showed the old number as the new search's. It now
dims and is aria-busy until the new count lands, as AdminLogs does.
2026-10-03 01:01:46 +02:00
peter e0e28c006c perf(feed): relevance ranks the newest 2000 matches, the rest follow newest first
A broad word matched most of the library and ts_rank read every match:
"a" took 4.4 s a page on prod. The window's start is read once (0.1 s) and
rides the cursor, so a video ingested mid-scroll cannot repeat a row; the
page is read as its two halves, each on an index (80 ms + 5 ms).
2026-10-03 00:28:29 +02:00
peter f8f22fc985 test(i18n): a language test unmounts its reader before switching the language back
useTags reset the language in a finally while its hook was still mounted;
the switch scheduled a render, and in a loaded gate run it landed after
the jsdom environment was torn down ('window is not defined', two
unhandled errors). auditColumns and comfortPin had the same shape.
2026-10-02 23:48:37 +02:00
peter 90e0baa870 style(db): ruff format the re-runnable index migrations 2026-10-02 23:43:21 +02:00
peter dfd43ae89c docs: the folding channel header in its e2e, and three comments brought up to date
The e2e asserted the channel chrome never moves; it folds once now. The
ui catalog lists scrollFold and progressiveRows, Tabs names the
per-tab memory its callers use, and the welcome note says a removed
language orphans its stored copies too.
2026-10-02 23:42:32 +02:00
peter 608fde2d55 fix(db): re-running 0095/0096 rebuilds an invalid leftover index
A concurrent build that failed part-way leaves an INVALID index under its
name; IF NOT EXISTS kept it on the re-run, and 0095 then dropped the old
channel_id index, leaving none usable. The leftover is now dropped and
built again, and the old index goes only once the new one is valid.
2026-10-02 23:42:32 +02:00
peter 0302c466e8 fix(i18n): a failed RSS poll over one channel reads as one, and keeps its stops
rss_all_failed is a plural pair counted on the raw total (the values are
formatted strings, which pick no plural form); both sentences end with a
stop. The Hungarian 'show N more' uses the house's megjelenítése.
2026-10-02 23:42:32 +02:00
peter a50055577a fix(plex): every small Plex picture asks for its card-size cut
The playlist rows, the search box and the collection picker still
fetched the 400px poster for a 32-80px picture. A contract test pins the
set of bare Plex pictures, and the srcset widths to imgcache.SIZES.
2026-10-02 23:42:32 +02:00
peter e16cdbd16f fix(ui): a card's Play clears its hover row at a large text size; Downloads links and the player description
At text size 1.3 a 242px feed thumbnail put the 63px Play 13px into the
hover row (the Play grows with the root font, the picture's height does
not). A picture narrower than 14rem centres the Play and the resume pair
above the row; the picture box is a size container for it.

The Downloads title and channel links were 21px tall inline boxes; they
stand on a 24px line now (B1). The player's description says a failed
load, with a retry, instead of 'No description' (both the side panel
and the popover).
2026-10-02 23:42:31 +02:00
peter 15ec67b029 fix(ui): a read-ahead waits out a background refetch, and a revealed row is mounted before it is focused
NEXT_PAGE joins ANY fetch in flight: a read-ahead during an invalidation
or a poll joined that refetch and appended no page, and nothing asked
again until the reader moved the range (the channel page during a
backfill). The read-ahead now waits while the list fetches anything
(mayReadAhead, re-run on isFetching); a Load older click cancels a poll
instead (nextPageOnClick).

A page turn to a revealed row restarted the progressive mount at 30
rows; Channels focuses that row in the next commit, and at index 40 of
a 50-row page it was not there, so focus fell to <body>. The mount now
reaches through the revealed row.
2026-10-02 23:42:31 +02:00
peter ac53a3b757 perf(search): a one- or two-letter last word is matched whole, not as a prefix
A 1-2 letter prefix matched most of the library and ts_rank scored every
match: "m" took ~4-8 s on prod against ~150 ms now. The prefix match
starts at three characters (MIN_PREFIX_LEN); shared by the YouTube feed
and the Plex library search.
2026-10-02 21:52:47 +02:00
peter e6de18b8c0 perf(plex): the facets ask the episode rollup once per call, not once per show group
A watch-state scope (watched / in progress / unwatched) joined the per-user episode rollup into
each show facet query: five passes over the user's episodes per call, half of a filtered facets
request. The shows in scope are now resolved once and every group filters on their ids; the
grid-parity test holds the counts. Localdev: 99 -> 61 ms.
2026-10-02 21:32:49 +02:00
peter fb2e3d8490 perf(plex): a card fetches its picture at a card's size, not the detail page's
Cards fetched the full cuts: a 400x600 poster (~38 KB) for a ~144px picture and the 1280x720
backdrop (~162 KB) for a ~212px shelf picture. The image proxy now also cuts thumb_sm (240x360)
and art_sm (480x270), cached apart; card pictures (grid, shelves, collection tiles, Up next) carry
both in srcset with sizes=auto, so the browser picks by laid-out width and DPR. Localdev:
17 KB a poster, 56 KB a backdrop. The detail page keeps the full cuts.
2026-10-02 21:27:13 +02:00
peter d928d50686 perf(db): each feed sort rides an index, and an ICU upgrade rebuilds the title ones
The views, duration and title sorts had no index: every page read the whole videos heap and kept a
top-N (prod 350 ms a page, 550-780 for title). Migration 0096 adds one per direction and, for title,
per reader collation, in the exact ORDER BY feedquery builds. Localdev prod copy: plain sorts
180-310 ms -> ~1 ms, page 20 3-5 ms, narrow filters unchanged. view_count/duration are only written
by the enrich, which already rewrites indexed columns, so no HOT update is lost.

The title indexes are the first on an ICU collation, and postgres:16-alpine floats: collation.heal_drift
rebuilds them (CONCURRENTLY) at startup when Postgres reports the ICU version moved.
test_feed_sort_indexes holds every videos sort to an index and checks the rebuild.
2026-10-02 21:17:41 +02:00
peter 540822d2b0 perf(ui): a next-page fetch joins the one in flight instead of sending the cursor again
react-query's fetchNextPage cancels a next page still loading and sends it again by default.
Our requests ignore the abort signal, so the server ran both: the feed's read-ahead effect
re-runs on scroll frames before isFetchingNextPage reaches it, and on prod every feed page
went out 2-3 times (a 0.8 s title sort queued to 2.3 s). Every call now passes NEXT_PAGE
({cancelRefetch: false}), lint-enforced, with a contract test.
2026-10-02 20:46:30 +02:00
peter b4ef07d28e perf(ui): a long table page paints its first rows at once and the rest in steps
The channel manager's All page (~400 rows) blocked one frame for 0.7-0.9 s on prod to mount,
and every sort and search key re-rendered every row (300-400 ms). No single cell is to blame,
measured column by column. DataTable now mounts the first 30 rows at once and 40 more per
background transition; a new page, size, sort, filter or caller-side search (narrowedBy)
starts over from the first rows. A row is its own component, so the compiler keeps the rows a
step already holds.
2026-10-02 03:16:37 +02:00
peter e995c664c6 perf(db): the channel manager and the feed sidebar read an index, not the video heap
GET /api/channels summed every subscribed channel's videos by reading each video's heap page
(681 MB on prod): 51-61 ms warm, but 2.25 s on a first open and 6.4 s under EXPLAIN once the
table had left memory. /feed/summary and /feed/years did the same (~140 ms warm, ~800 ms cold).
Migration 0095 replaces ix_videos_channel_pub_desc with the same key INCLUDING the columns
those queries read, so they are Index Only Scans over a 25 MB index. A test holds the queries
and the index's column list together.

The admin Configuration page did a db.get per setting (two for an unset one): it now reads
every override in one query (~40 ms of ~55 on prod).
2026-10-02 03:16:37 +02:00
peter 20010178b0 fix(ui): the Plex detail poster keeps its 2:3, and a long crew shows its first names
The hero's grid row is as tall as the crew column beside the text, and the poster, a flex child,
stretched to it: a 2:3 picture drawn ~175x590 and cropped. It now aligns to the top, and a wide hero
gives the movie and show poster a bigger rung (w-56). The crew, the hero's tallest part, shows its
first 6 names and a 'Show N more' toggle; a list of 8 or fewer shows whole.
2026-10-02 02:37:20 +02:00
peter 78fdcda6bb feat(ui): E31 S4c — the channel header folds, the Plex pages fill their hero, the player gets a side panel
- Channel page: the banner and identity block fold away once the grid scrolls
  (ui/scrollFold: folds past 32px only if the page can stay scrolled, unfolds
  at the very top) and the name rides in the tab row. One Back button in both
  states keeps its focus; folded at a tablet width the actions wrap instead of
  pushing the page sideways.
- Plex movie and show pages: the TMDB facts over the key crew are the hero's
  second column (plexDetailAside), the keywords sit under the text; the show
  page shows them for the first time, with the same three toggles. A
  collection's hero is sized to its text.
- YouTube player, 1280px and wider: the title, channel, actions, description
  and queue sit in a side panel beside the video, which takes the window's
  height (1438x809 at 1920x911, was ~1008 wide). The wheel scrolls that panel
  while it can and turns the volume elsewhere.
- DESIGN.md decision log: the keywords' place under the text.
2026-10-02 02:13:17 +02:00
peter 98d0ca6eeb fix(i18n): every counted string is a plural pair, so one reads singular
"1 seasons", "Synced 1 subscriptions", "1 video(s) were skipped": 21 English
strings carried {{count}} in one plain key and read plural at one. Each is a
_one/_other pair now (Hungarian keeps one form for both), and so is every other
counted string, so the next wording change is a plural too. A contract test
holds the rule for every locale file.
2026-10-02 02:13:17 +02:00
peter e4c80fb7bb fix(ui): a page's sub-tab is remembered per browser tab, not forever
Settings, Configuration, Statistics and Users kept the last sub-tab in localStorage, so every
fresh tab opened Configuration on "Plex" as if that were its home. useTabPersistedState keeps it
in sessionStorage (account-scoped): a reload and a trip to another page keep it, a new tab starts
on the first sub-tab. The pre-selects (a Review link, the Google-link return) write the same store;
a value an older build left in localStorage is cleared on first read.
2026-10-02 00:49:46 +02:00
peter be156fc52b feat(ui): E31 S4b — the admin pages behind one rail row, Configuration as a sectioned list, Statistics as tiles and bars
- NavRailGroup: one "Admin" rail row opening a panel of the admin pages (links, so a
  middle-click still opens a tab); the rail fits 133% zoom without a hidden scroll
- sysconfig.SECTIONS orders the config groups under four headings (every group in exactly
  one, test-held); the page shows them as Settings' side list (Tabs `group` headings);
  the lone registration group reads "Registration" under "Access"
- StatTiles / StatTile / StatBars: the Statistics overview's figures as tiles, the
  per-operation quota use as horizontal bars (a zero draws none)
2026-10-02 00:07:00 +02:00
peter 25905aa786 fix(audit): fold a job's successful runs whatever ran between them
The owner read a fold that broke at every other job as the same row five
times. A job's successful runs now fold into one row at the newest run's
place across other jobs and admin actions; only that job's own failed or
skipped run breaks the fold (it stays its own row, and older successes
start a new fold below it).
2026-10-01 23:05:38 +02:00
peter 87d71e3762 feat(ui): E31 S4a — rail rows are links, the channel page opens on unwatched, users and audit tidied
- Nav rail page rows are real links (/?page=<id>): a middle-click or
  Ctrl-click opens the page in a new tab; a plain click stays in-app.
- A fresh channel page shows the unwatched videos; a picked watch state
  is a remembered preference (channelShow).
- Users: the role, suspend and delete move into the row's one menu
  (the "…", right-click, Shift+F10); locked ones keep their reason.
- Audit log: back-to-back successful runs of one job fold into one row
  that opens to the earlier ones; a failed run reads in the danger colour.
- DESIGN.md decision log: the E31 picks.
2026-10-01 22:51:38 +02:00
peter ab146211fa fix(scheduler,i18n): a failure a job names reads in the reader's language
The audit log showed a failed RSS poll as the server's English sentence
under a Hungarian UI. A job can now raise progress.JobFailed(reason,
message, **params): the scheduler stores the reason and params as the
run's data (dashboard, manual-run notice, audit row) beside the English
line, and the client renders scheduler.failure.<reason>. The RSS outage
names its first channel and the reason code (HTTP 404 vs unreachable).
An upstream's own error text stays as raised (diagnostic).
2026-10-01 22:51:37 +02:00
peter ec22eefd31 fix(ui): E31 S2 — the tab names the object, the channel header wraps, the player fits a short window
- The browser tab names the place and the object: PageHeading reports the
  view's own title (lib/documentTitle), so a channel, playlist or any Plex
  view reads "<object> · <module> · Siftlode" however it was opened; the
  Feed is named too; a season adds its show (tabTitle).
- The channel page's tab + action row wraps and lets the tag lane fold into
  "+N": at 1024 it ran past the edge and the page scrolled sideways.
- The YouTube player card's width is capped from the window height less the
  chrome measured under the video: at 133% zoom (685px) it cut its channel row.
- Plex certificate words (Not Rated, Approved, Passed) are named through the
  vocabulary on the info badge, show page, collection, filter and chips.
- DESIGN.md: the zoom target is 125% and 133%, not 200% (decision log).
2026-10-01 21:17:20 +02:00
peter 2543d44860 fix(messages): the welcome reads in the reader's language, not the one it was created in
E31 S1: the welcome is rendered once, at account creation, so an account that
switched language kept it in the old one (the owner's, English under a
Hungarian UI). A stored copy is recognised by its opening line (derived from
WELCOME) and served in the reader's language; other system messages stay as
stored.
2026-10-01 21:17:09 +02:00
peter 29c0715237 release: 0.97.0 (feature/e30-app-in-hungarian)
gate / frontend (push) Successful in 6m16s
gate / backend (push) Successful in 7m19s
2026-10-01 10:44:40 +02:00
peter 6b86f99249 release: 0.97.0 notes — the app in Hungarian 2026-10-01 10:44:39 +02:00
peter 3563858fd1 fix(ui,i18n): E30 review round 3 — the stored language follows the screen, a fresh text starts collapsed, Retry waits again 2026-10-01 10:43:59 +02:00
peter 08e62bbb16 fix(plex): E30 review round 3 — a dead headshot breaks the Plex-failing streak, 410 held by a test 2026-10-01 10:43:59 +02:00
peter 9454717f47 fix(ui,i18n): E30 review round 2 — honest bio failure, a full show summary, one HU name per thing
- Person page: a bio ask that fails says so with a retry instead of reading as 'no biography'.
- Show page: the summary is a ClampedText (5 lines + More), like the person and collection pages;
  one More/Less pair (plex.textMore/textLess) for all three.
- Channels: a re-slotted row is placed in the language the server orders by (the stored
  preference, else English), not the UI's guess.
- A skipped run's body is only the reason (its title already says skipped).
- HU: Behúzási kvótatartalék; Eseménynapló ürítve / az eseménynaplót; indok; közvetlenül
  játszható / közvetlen közvetítés in the Plex hints.
- Tests: ClampedText re-measures on a resize, the show summary clamp, serverOrderLang.
2026-10-01 08:31:04 +02:00
peter 6ac9ffd45e fix(plex,audit): E30 review round 2 — only a missing file is a dead headshot, one spelling per portrait
- warm_portraits: only 404/410 mark a headshot dead (measured: Plex answers a dead CDN file 404);
  a refused token or a 5xx is Plex failing, ends the run after three in a row and marks nothing.
  'failed' counts every failure, not only the current streak.
- is_portrait_url: plain path segments only — no dot or empty segment an HTTP client folds onto
  one file; person_thumb_url uses the same predicate (27 852 stored headshots all pass).
- Tests: Plex refusing (401/403/429/5xx), failures between successes (portraits and bios), the kept
  original served from cache, the audit_clear / demo_reset / discovery_purge rows' counts as data,
  and the 0094 backfill on real rows.
2026-10-01 08:31:03 +02:00
peter fa55e01ef0 fix(ui,i18n): E30 review round 1 — audit rows in the reader's language, one HU name per thing
- audit log: a job run named from its result data (nested phases too), a skipped run says so, a
  secret's reset reads as a reset, choices named as Configuration names them, interval / batch /
  counts localized; the inbox shares the same jobRunText
- HU: remux is "Közvetlen közvetítés" (it shared "Közvetlen lejátszás" with direct play); backfill is
  behúzás, enrich is dúsítás, collection is kollekció, the allow-list is fehérlista; the last
  "jobnak" / "transzkódolás"; a test holds the options of one choice to distinct names
- contract: every skipped / reason word a job returns is named in both languages
- person page: the bio wait is timed from the open and a failing ask stops the polling; ClampedText
  (ui) shared with the collection page, whose short summary lost its dead More
- a playlist names season 0 the Specials; the channel re-slot compares in the reader's alphabet; an
  invite's date is in the admin's language; the language-ordered lists are asked again once a
  language switch is saved (saveLanguage)
- poster grids' column minimum in rem; lint refuses minmax(Npx,…)
2026-09-30 03:26:58 +02:00
peter ab963c81d2 fix(plex,audit): E30 review round 1 — the people job keeps going, HU genres everywhere, results as data
- people_enrich: a failed TMDB lookup or a dead headshot is skipped, not the end of the run (a dead
  file is marked and left for a week; three failures in a row end it); portraits wait for the Plex
  module switch; a no-Plex original keeps its own cache key, so the resize still replaces it; the job
  entry lives in the plex domain (run_people_enrich)
- person-image takes only a headshot-shaped CDN url (no query variants of one file)
- every synced row gets Hungarian genres, and a HU genre filter matches the word as picked too;
  migration 0094 fills the rows no overlay reached
- a job run's audit row carries its result as data; audit_clear / demo_reset store their counts
- one credit-column list for the prune and the enrichment order; _overlay_pass typed
- tests: batch skip + stop, HU bio stored, dead headshots, module switch, the person-image route,
  the job body, genre filter + facet write + twin fallback, tagline/summary fallbacks, audit data,
  the reader's alphabet at every remaining collated() call site
2026-09-30 03:26:57 +02:00
peter 59e6cc027d fix(plex): a short "Known for" rail sits right beside the text (E30 S6)
With one to three titles (about one person in five) the rail sat at the far edge of the hero with a gap
between it and the bio. The person's column is now capped at the bio's reading measure and a short rail
follows it directly, so the leftover is the panel's end (owner pick "b"). Four or more keep half the hero.
2026-09-30 02:27:28 +02:00
peter ca94a716d8 docs(design): composition and the reader — a chapter and a Conformance step (E30 S6)
The rules covered each piece; nothing asked whether a screen works as a whole, and the person page passed
every rule with half its panel empty. New chapter: the space is used, one focal point, worst real data,
where am I, a click answers at once (no outside service on open), zoom and text size first-class (the
owner reads zoomed in). Conformance step 5 writes one composition line per screenshot and checks 200%
zoom and the largest text size. Anti-patterns and a decision-log row to match.

HU: the download worker setting reads "Párhuzamos letöltések" (a leftover from the S5 sweep).
2026-09-30 02:19:54 +02:00
peter 63d01babe5 feat(plex): the person page in two columns; a bio on its way holds its place (E30 S6)
The biography sat at reading width in the left half of a full-width panel with nothing on the right.
Now the person (headshot, facts, the bio clamped to eight lines with More / Less when it is cut) is on the
left and "Known for" on the right; with fewer than four titles the rail takes only its posters' width.
One column where the panel is narrow (200% zoom, the largest text size).

While the server fetches a bio (`enriching`), the page shows a paragraph skeleton (a new "text" shape of
LoadingState) and asks again every 1.5 s for up to 20 s. The view is keyed by the person's name, so
another person starts fresh.
2026-09-30 02:19:54 +02:00
peter 705377c797 feat(plex): the people are local — bios and portraits fetched ahead, not on the click (E30 S6)
A person's TMDB bio was fetched only when their page was first opened, off the request path, so the first
open showed no bio and the page never asked again (2 of 18,202 linked people were ever enriched).

- people_enrich scheduler job (every 10 min, 300 per run): linked people's bios from TMDB and their
  portraits into the image cache, the most-credited people first. A failed TMDB lookup ends the batch.
- Portraits are resized through Plex's photo transcoder to a 320 box (~15 KB instead of ~170 KB; the
  person-image proxy uses the same cache key, and a full-size original is removed as its small twin lands).
- The person response says `enriching` while a bio is on its way (a person the job hasn't reached yet).
- "Known for" is ranked by vote count, without guest visits as themselves (talk shows had crowded out the
  films: Tom Cruise had 14 of 15), with directed works, from a 40-deep pool; the page shows up to 15.
  Migration 0093 un-stamps enriched people so the job refetches them.
- A phased job result (backfill, people_enrich) now counts as a change for the audit trail; before, an
  automatic backfill run that added videos was never audited.
- HU: the watch reconcile job reads "egyeztetés" (a leftover from the S5 sweep).
2026-09-30 02:19:54 +02:00
peter 2819049133 feat(audit): job runs and config changes read in the reader's language (E30 S5)
The audit log showed the server's fixed English lines ("rss_poll: ok — new=5, failed=0",
"log_level = DEBUG", "Background sync paused") to a Hungarian reader. The details are rebuilt from the
row's data on the client: a job run names the job and its result as the inbox does, a config change
names the field and its values (on/off, grouped counts, "default"), a pause/resume adds nothing the
action does not say. A failed run keeps its error text (diagnostic). parseSummary reads enrich's bare
count ("50") back as `enriched`, as the server's `_result_data` names it — the audit and old notices
keep only the one-liner. No backend change; old rows render too.
2026-09-30 01:08:29 +02:00
peter 95b94a1d72 fix(ui): a longer label fits its control — pill value truncates, headers get their room (E30 S5)
Found in the Hungarian UAT: a toolbar SelectTrigger's text value wrapped out of the pill in a narrow
place (the playlists rail) — it now truncates in its own span; the Channels "Prioritás" and "Utolsó
feltöltés" headers were clipped by their fixed columns — sized for the label, funnel and sort arrow;
a sortable header is at least 24px wide (Hungarian "Ki" was a 12px target). PRE-EXISTING, any
language: the messages unlock button and "Forgot passphrase" sat flush — one gapped row.
2026-09-30 01:08:29 +02:00
peter 12077fbebc feat(i18n): Hungarian words for the English loanwords, one name per thing (E30 S5)
The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info,
Frontend/Backend, Backfill and transcode get the Hungarian word (közvetítés, Prioritás, Feladat,
Eseménynapló, Demó, Automatikus, Infó, Felület/Szerver, videók behúzása, átkódolás). The final sweep
also took out reset/sync/reconcile/setup/logs/whitelist/email, and three features that went by two
names: access requests (kérés/kérelem), backfill (betöltés/behúzás), and the connection test
("Check the connection" beside "Test connection", in both languages). Two quota actions shared one
Hungarian name. The playlists rail title and two of its sort labels are shorter so they fit.
DESIGN.md Copy: the rule and its decision-log row.
2026-09-30 01:08:29 +02:00
peter 519fbe2a18 feat(i18n): the person page's biography in Hungarian, from TMDB's translation (E30 S4)
The /person fetch appends translations and keeps TMDB's Hungarian biography beside the instance-language
one (new plex_people.biography_hu, migration 0092). A Hungarian reader is served it, the English one when
TMDB has none. Already-enriched people are un-stamped, so their next open refetches them once.
2026-09-29 23:44:48 +02:00
peter a39504e617 fix(stats): the full-history ratio counts the requested channels on both sides
"Full history: done / requested" counted every subscribed channel whose
back-catalog was complete in the numerator - also the ones another user asked
for, or short enough to finish on their own - against only the channels this
user requested, so the Stats page read "342/329". The numerator now counts the
requested channels that are done; "channels without full history yet" still
counts every subscribed one.
2026-09-29 22:46:04 +02:00
peter aac79ffa53 fix(i18n): numbers, units, dates and order in the UI language, and after a switch (E30 S3)
Every locale formatter takes the language as an argument. Under the React
Compiler a formatter that read the i18n singleton was memoized on its other
arguments, so a card kept "2 hr 4 min" after a switch to Hungarian; plexLang()
had the same flaw and left Plex queries in the old language. Components now
take the language from useTranslation, and the signature makes forgetting it
a type error.

- ratings, file sizes, stream stats: the language's decimal mark ("8,3")
- view and subscriber counts compact in the language ("1,2 E")
- seconds as the language's unit ("6 mp") in the sliders and the stats
- a person's birth and death dates as dates, not ISO strings
- A-Z lists by the shown name in the language's alphabet (compareText),
  the tag catalog re-sorted by its translated names
- lint local/locale-format (was date-format) also refuses Intl.NumberFormat
  outside lib/format.ts, localeCompare, and a component reading the language
  off the i18n singleton
2026-09-29 22:39:57 +02:00
peter 67c9f10153 feat(i18n): every server A→Z list in the reader's alphabet (E30 S3)
The Postgres image is Alpine, whose en_US.utf8 is byte order: "Életrajzi" and
"Ötödik" sorted after "Zorró" in the feed, the channel lists, the Plex grid,
its genres and collections. `app/collation.py` names the ICU collation of the
reader's language (hu-x-icu puts "Ö" after "O" and "Cs" after "Cu"), wraps
every text ORDER BY with it - the feed's keyset cursor compares with the same
expression, so paging walks the order it sorted in - and orders the lists built
in Python (genre and studio facets, collection members, the discovery re-sort)
through the same collation. Plex title tiebreaks use the shown (HU) title.
2026-09-29 22:39:57 +02:00
peter 584d082b6b feat(inbox): a run notice from before the result rode along reads its one-liner back into one (E30 S2) 2026-09-29 21:32:29 +02:00
peter f3a85916ec feat(i18n): the server's closed English vocabularies named in the reader's language (E30 S2)
- lib/vocab + the `vocab` locale area: TMDB crew jobs and departments, auto-tags (D1 A: the data
  stays English, the catalog is named at its one source, useTags), built-in download profiles,
  YouTube topics; held key-for-key to the backend's own lists by backendIds.contract.
- Countries by ISO code: TMDB parse stores {code, name}; rows stored before are coded from TMDB's
  own country list in one request (run_enrich); Intl names them, historic codes (YU, SU, XI…)
  from the catalog.
- Player tracks named from the language tag, forced/SDH flags and codec tail, not Plex's
  server-language displayTitle; no invented "Audio N" on the server.
- Download lifecycle notices (shared / expiring / expired / evicted) named from data; the share
  notice carries the title.
- Scheduler: the job result rides along as data and is named per key (dashboard + run notice).
- Invite status named; a render crash's notice no longer shows the engine's English message.
- Fix: YouTube links …/wiki/Humour, the tagger looked for "Humor" — Comedy never came from a topic.
2026-09-29 21:21:19 +02:00
peter f6298d29af fix(plex): the recs rail's anchor and a held recommendation are named in the reader's language (E30 S1b)
Found in the S1b UAT: 'Mert megnézted: Batman Forever' kept the English title, and
the info page's TMDB recommendations kept TMDB's English titles even for titles we
hold. A held recommendation is now named as the library names it, in the reader's
language; one we lack keeps TMDB's title.
2026-09-29 02:25:36 +02:00
peter 632e39d458 feat(plex): every card rail in the reader's language; an episode in a poster rail stands as its show (E30 S1b)
The grid served Hungarian titles, but every other card surface (Discover rails, the
show page's episodes, search, collections, the person page, playlists, Home's resume
rail, hero and comfort shelf) served English. The card serializers now take a
required keyword-only lang (a forgotten call site fails instead of going English), and
each of those reads takes ?lang=hu, with the language in its query key. A Hungarian
search finds an episode by its Hungarian title too.

An episode on a poster rail showed its 16:9 still cropped into a poster, titled only
with the episode name; it now shows the show's poster and name, with S·E · title under
it (option A). Plex's generic season names (Season 1, Specials) are rebuilt from the
number in the reader's language; a season with a name of its own keeps it.

Home's comfort-shelf move wrote its optimistic order to the key without the language,
which nothing read any more; it writes the query's own key.
2026-09-29 02:12:32 +02:00
peter 13d95ef70e feat(plex): episodes, taglines and genres in Hungarian from the HU overlay (E30 S1a)
The HU TV section lists every episode with its Hungarian title and summary, but the
overlay read only movies and shows, so every episode stayed English. It now overlays
the episodes too, and the HU tagline (new tagline_hu column, served on ?lang=hu).

Plex's HU agent returns a mixed genre vocabulary (Action beside Dráma), and a title
with no HU twin kept its English list, so the HU genre list showed Drama and Dráma.
plex/genres.py maps every genre word to one Hungarian word; the overlay writes it
into genres_hu for every row, and a HU genre filter is read through the same map,
so a saved filter still holding Action keeps matching.
2026-09-29 01:52:02 +02:00
peter 0f9ed7cdbf test: a list key waits for the list's registration, not only its rows
gate / frontend (push) Successful in 6m5s
gate / backend (push) Successful in 7m10s
The server gate's frontend job went red on 0.96.0: PlexPlaylistView's j/Delete
test pressed j after the rows painted but before useListNav's passive effect
registered the list, so the key did nothing on the slower runner. Every test
that presses a list key after an async load now waits for [data-nav-list] on
its list first (PlexPlaylistView, Playlists, Home, PlexDiscover). Checked
against a 30 ms delayed registration: the old tests fail, the new ones pass.
2026-09-27 23:01:57 +02:00
peter ffcea0cb07 release: 0.96.0 (feature/e28-design-constitution)
gate / frontend (push) Failing after 6m12s
gate / backend (push) Successful in 7m22s
2026-09-27 22:46:39 +02:00
peter a3f8314c9a chore(release): 0.96.0 — the design constitution
E28: one action list per object, keyboard reorder, touch drawers and player
controls, the table pager, plus four deep-review fix rounds (playlist sync
locking, player restart and focus behaviour).
2026-09-27 22:46:39 +02:00
peter d5ccb1091a fix(player): a held key on faded bands stays one blind press; a play asked for mid-restart is the new session's
The repeats of a Space/Enter that landed on faded bands pressed the control once the bands were up;
they are swallowed with it. A play or pause while a session restart is out now sets that session's
intent instead of playing the outgoing stream. Tests: Skip intro outside the bands keeps its keys,
every restore takeover (wheel, finger, scrollbar, key) is covered, and the chip handoff test tells
the chip before from the row's first control.
2026-09-27 22:34:14 +02:00
peter 528774d640 fix(playlists): a push stops at a list deleted mid-way, and a rename made during it stays unsynced
A delete landing while a push read YouTube left a plan over no items, so the push emptied the
YouTube playlist a local-only delete keeps: the push now stops at a vanished list (PlaylistGone,
404), and a delete is refused while a sync of the list runs. The closing baseline fingerprints the
title YouTube got, not the one its lock re-reads, so a rename made during the writes stays dirty.
The other-tab test helper bounds its lock wait, so a lock-order regression fails instead of hanging.
2026-09-27 22:34:14 +02:00
peter 3a9ea9e330 test: the side panel's focus step is held in its commit; the MenuCheck contract counts chosen roles 2026-09-27 21:56:24 +02:00
peter 25f2d9d0e4 fix(ui): a removed chip's focus outlives its neighbour; a scroll restore yields to the reader
A chip handed its focus to a Clear all or +N that went in the same removal, and the focus fell to
the body. The restore now stops when the reader scrolls, and finishes the way once the Back-to-top
spacer brings the saved spot within reach.
2026-09-27 21:56:23 +02:00
peter 5ab6778b97 fix(player): a Restart of the paused film plays the new session; a key on faded bands only shows them
The same-film restart read the paused state, restarted the session without play, and resumed the
outgoing stream beside it. Space or Enter on a control the bands had faded from pressed it unseen;
now it only brings them back, and Space plays or pauses.
2026-09-27 21:56:23 +02:00
peter 26697472cf fix(playlists): every sync path locks the list before its items, and holds it to the write
The mirror refresh re-ingested inside its row lock, and that ingest's commit let go of the lock
before the replace: a remove made then was undone and the list read as synced. Push, re-pull and a
delete touched items before the list row, a deadlock beside the playlist-first route edits. A push
now re-plans from the items under the lock and derives dirty after its YouTube writes.
lock_playlists moves to the sync package. Tests pin both ids of a move's lock and the claim's
own connection.
2026-09-27 21:56:23 +02:00
peter 2abd4ffcb6 fix(player): a session superseded before it played is reclaimed, not "couldn't start"
Two session requests for one item in flight at once (a quick double seek; the
StrictMode replay of the start effect) let the server's last spawn be the older
request's. The newer request's answer then named a session already gone; its
playlist 404'd before a manifest parsed, and the player stopped with "Playback
couldn't start". The 404 reclaim now covers a session that never played too,
bounded by the same budget, and keeps the start's own autoplay.
2026-09-27 20:22:43 +02:00
peter a9ac303d3e fix(side-panel): the key's focus steps into the peeked card in the commit that opens it
A frame later never came in a page the browser was not painting (measured: a hidden page runs no
requestAnimationFrame), and the focus stayed on the inert tab under the card.
2026-09-27 19:58:18 +02:00
peter 740602b9bf fix(ui): focus, paging and state that a key or a finger can reach
- A key landing on a side panel's tab steps into the card it peeks open, not onto the hidden tab.
- The Channels cards keep their status / pager row at the list's end while the band is folded.
- A menu row's note shows on an enabled row too: the download state under 'Download…', not a title.
- A removal chip hands the focus to its neighbour before it goes, for every caller (the feed's and
  Plex's active filters); the hidden-filter cue's own copy is gone.
- Back to a spot within a page's end spacer lands at the page's end, not its top.
- A running push / revert is tracked per list: another list's push is no longer refused unseen.
2026-09-27 19:43:04 +02:00
peter e31a588f19 fix(menus): one tick for every choice; a listbox holds its options only
- The saved views, the view switcher, the year filter and the table's single-choice filter mark the
  chosen row with MenuCheck, as DESIGN asks, not a trailing tick, a dot or a filled row; a contract
  test holds every checkable menu row to it. keepSelectedPaint had no caller left and is gone.
- A combobox's panel carries its listbox role on an inner PopoverList (listInside), so the failed
  load, Loading, No matches and the 'n more' line sit beside the options, not inside the listbox. The
  Plex search box's Loading line likewise.
- The Plex sidebar's playlist picker offers Create only once the lists have loaded: over a failed load
  it pushed the error and its Retry out at the first keystroke.
2026-09-27 19:43:04 +02:00
peter cda626be10 fix(player): the playing film restarts in place; arrows stay the volume; the bands' holds end
- The info overlay's collection strip listing the film itself: Play/Restart restart or resume it in
  place instead of pausing it on a frozen 0:00.
- Up/Down on the closed ⋯ and tracks buttons stay the volume (openOnArrowKey: false).
- A keyboard focus in the bands holds them for 10 s after the last key, not the whole film; a tap
  leaving a tapped button no longer re-shows the bands its own toggle then hid.
- Next leaving under the focus hands it to its neighbour (the player keeps its focus).
- The chosen audio / subtitle track wears the MenuCheck tick.
2026-09-27 19:42:53 +02:00
peter b7ce1d467c fix(playlists): item writes lock the list first; one bulk pull per user; the claim on its own connection
- Every playlist item write (add, remove, move, reorder, rename; Plex remove and remove-bulk) row-locks
  the playlist before it touches an item, a move both lists in id order: remove beside reorder was a
  deadlock, and two adds of one video a unique-key 500.
- The scheduled mirror refresh re-reads dirty under that lock, so an edit made during its YouTube calls
  is not replaced by YouTube's order.
- A user's bulk pull holds a per-user advisory claim: two at once created duplicate mirrors. The manual
  sync answers 409 sync_all_busy instead of '0 synced', and like the re-pull spends its cooldown only
  on a pull that runs.
- Deleted mirrors are committed at once, not only when a later list commits.
- The claims open an unpooled connection of their own instead of a third pooled one per push.
2026-09-27 19:42:53 +02:00
peter 52fb999af6 feat(pickers): a list that failed to load offers Retry
A picker whose options failed to load said so, but the only way back was closing its dialog.
Combobox takes `onRetry`: a Retry beside the message for the pointer, and Enter in the field for
the keyboard (the panel is a listbox, which may hold options only, so the alert names Enter for
assistive tech). Every caller with an error state passes its query's refetch: both user pickers,
the channel picker and the two Plex pickers.
2026-09-27 17:47:01 +02:00
peter e0453fa07d fix(playlists): one YouTube sync of a list at a time, across tabs and the scheduler
The client's in-flight flag covers the menus of one page only. A push reads YouTube, then writes
both sides for seconds and commits step by step, so a row lock ends at its first commit: two tabs,
or a push and the scheduled mirror refresh, could interleave into a doubled YouTube item or a
baseline recorded over the other's writes. `sync_claim` takes a session-level advisory lock on a
connection of its own (it survives those commits; a crashed worker's lock dies with its
connection, so no claim column is left stuck). Push and revert answer 409 `sync_busy`; the
scheduled refresh skips a list being synced and picks it up next run.
2026-09-27 17:47:01 +02:00
peter 2525a12264 fix(keys): a key a focused widget already handled fires no shortcut
The player's settings tabs switched tab on an arrow and prevented its default, yet the player's
window listener still sought the video 10 s. dispatch() now skips a defaultPrevented key for
every scope; listNav's private copy of the same guard goes.
2026-09-27 17:28:10 +02:00
peter 8e09dd85f8 test: the store contract reads the last match without Array.at (the node tsconfig's lib) 2026-09-27 16:38:33 +02:00
peter addb3d1b32 fix(a11y): every side-panel pin names its panel; the tab under an open card leaves the tree
The card's own pin and the drawer's still said 'Unpin panel' / 'Pin panel open' after the edge tab
learned the title (F6). Naming them too made two buttons share a name, because the invisible tab
under an open card was still in the accessibility tree: it is now aria-hidden as well as out of
the Tab order. The generic sidebar.pin / unpin keys are gone.
2026-09-27 16:30:40 +02:00
peter 86884e368e fix(ui): the page's end scrolls out from under Back to top
The button floats over the bottom-right corner, and at the end of a long table it covered the closing
pager's '100' and 'All'. While it is up, PageScroller ends the page a button's height later (only
then, so a full-height layout gets no scrollbar). Its shown state is a shared store read through a
use… wrapper: a store's .use() called straight from a component is memoized by the React Compiler
and never re-renders (contract: store.contract.test.ts).
2026-09-27 16:28:18 +02:00
peter 51f881681b feat(tables): name a filter whose column is off screen, and let its chip clear it
A tabletHidden/narrowHidden column below its width, or any column in a card
layout, took its header funnel with it: the rows were narrowed by a filter
nobody could see. The cue is shown by the inverse of the column's own
container query, so it agrees with the table at every width and text size.
Wired into DataTable, the playlist table and the channel cards (E28 F2).
2026-09-27 16:22:26 +02:00
peter 27483ae1d6 fix(focus): a removed control's focus skips a neighbour the CSS has hidden
A display:none button refuses focus(), so handing the focus to a hidden
column's header button dropped it to <body>.
2026-09-27 16:22:26 +02:00
peter 595c29e17c style: prettier 2026-09-27 15:17:02 +02:00
peter acf93a1f36 test: the video menu's download row is 'Download…'; drop home.pin.menu (its menu is gone) 2026-09-27 15:13:19 +02:00
peter 84278293df docs(ui): the raw-button rule is 'outside the control primitives', and the README table is whole
CLAUDE.md, DESIGN.md and ui/README said a raw <button> is allowed anywhere in components/ui/; the
lint rule allows it only in BUTTON_PRIMITIVES (and refuses a bare FocusSafeButton elsewhere too).
The focus-safe rule's message no longer points feature code at FocusSafeButton. The README's
'actions & inputs' row was split in two by a stray line break.
2026-09-27 15:09:22 +02:00
peter dc24d80ed4 fix(player): a description link keeps its sentence's punctuation and its own parentheses
The trailing .,;:!?)] was trimmed off the href and dropped from the text too, and a URL ending in a
balanced ")" (a Wikipedia article) was cut short. The trim keeps a closing paren that has its
opener, and the trimmed tail is put back after the link.
2026-09-27 15:08:27 +02:00
peter 3810af66da fix(channels): every subscribe and unsubscribe refreshes the same reads
Each of the four sites kept its own invalidation list and each missed something (the channel menu's
unsubscribe left the feed showing the channel's videos; the channel page's left the counts stale).
lib/subscriptionCache invalidateSubscription is the one set; a contract test holds every call site
to it.
2026-09-27 15:07:39 +02:00
peter eb339195c7 feat(channels): the band's tag chips and status row fold on a short window (B4)
Every other page band folds its filter row below ~600px of height; the Channels band never did, so
its tag chip lane and status / pager row kept a short laptop window for themselves. They fold behind
one "Filters · n" (n = picked tags + other active column filters); the tabs and the TAGS editor stay,
and a folded table keeps its controls at the list's end.
2026-09-27 15:06:32 +02:00
peter 5d1065c1bc feat(playlists): a playlist's menu copies its name and its YouTube address
Every object's action list ends its middle group with Copy name / Copy link (E28 C1, the rows a
finger cannot select); the playlist's had neither.
2026-09-27 15:04:47 +02:00
peter 6ffb5bd8c1 refactor(pins): drop the unused ComfortPinMenu and the comment that described it
Every surface composes the pin row into its object's one action list (E28 C1); the pin-only menu
had no caller left. Also un-exports isStandInFocus, which only its own file reads since H1.
2026-09-27 15:03:28 +02:00
peter dddf79ec2f refactor(ui): FilterSelect and Combobox draw their checks with MenuCheck
They still painted their own 14px box beside the menus' 16px MenuCheck. A contract test holds the
filled-box paint to MenuCheck alone.
2026-09-27 15:02:42 +02:00
peter 5ebb4cb506 fix(a11y): chips, grips, the panel tab and Back to top keep the focus somewhere real
- F5: a chip's × pressed from the keyboard hands the focus to the next chip (else the previous one,
  else what follows) before it unmounts — in DismissibleChip, so all five chip rows get it
  (`focusNeighbour` in lib/useFocusKeeper).
- F4: a drag grip names what it lifts (`common.order.dragNamed`: panel groups, personal views);
  the two generic "Drag to reorder" keys are gone.
- F6: the side panel's edge tab is out of the Tab order while the card covers it, names its panel,
  and a keyboard pin moves the focus into the card.
- F7: Back to top is no Tab stop while hidden, and a keyboard press leaves the focus on the page.
2026-09-27 15:01:22 +02:00
peter 0b1d905866 fix(ui): Enter on a selectable title opens it even with its words selected
endsTextSelection takes the click event and exempts a keyboard click (detail 0): Enter / Space
ends no selection, so a title selected and then tabbed to opened nothing. All three callers
(FocusSafeButton, the Plex caption, the video card title) go through it.
2026-09-27 14:55:50 +02:00
peter 52c15adaf2 fix(downloads): a row whose remote picture fails too falls back to its glyph
The remote thumbnail was the last resort with no fallback of its own, so a dead CDN link drew the
browser's broken-image mark. A failed remote URL is remembered like a failed poster, and the row
shows its film / music glyph.
2026-09-27 14:54:28 +02:00
peter 3372626e34 fix(notifications): "Find in feed" drops every narrowing that could hide the video
locate() spread the reader's filters and cleared only the tag, search and age cap, so a year, a
custom range, a length, a content type (a hidden Short under "no Shorts") or the scope could still
leave the video out. It builds the filters from nothing now, keeping only the sort and tag modes.
(Also types the C1 test's pending plan.)
2026-09-27 14:53:24 +02:00
peter 98bb841a6a fix(playlists): one push or revert at a time for a list, whichever menu started it
The in-flight flag was per hook instance, and the rail row's menu and the list page header's "…"
are two at once: a push from one left the other offering a second push. It is one app-wide store
(`lib/store`), read with `get()` at the start so a second start refuses before a render; a revert
re-reads it after its confirm dialog.
2026-09-27 14:51:37 +02:00
peter 98bf0a9d66 fix(i18n): the video menu's download row says "Download…", and counted copy is pluralized
- The row opens the download dialog, so it is labelled so; the download's state ("Downloaded",
  "In queue") is its hint, not its label.
- The share dialog's expiry ("1 days") and a link's view count ("1 views") go through i18next's
  plural forms (`count`).
2026-09-27 14:50:04 +02:00
peter 17bb14fb18 fix(ui): a picker whose list failed says so, and takes no pick while one is being written
- Combobox gains `error` (said in the list, not "No matches") and a soft `disabled` (keeps the
  focus, announced, opens and changes nothing).
- UserPicker passes both: the share dialog's recipient picker is off while a share is pending, and
  the share and quota pickers say when the user list failed to load.
- ChannelPicker and the Plex playlist/collection pickers say a failed load too.
- The channel menu's Tags row says loading / failed instead of "no tags yet".
2026-09-27 14:47:46 +02:00
peter 70f98b352f fix(plex): a collection sibling is a full Plex card, and the player's info overlay plays it
- PlexInfo's collection strip gives each card the related strip's action set: the "…" menu (with
  Restart for a touch reader), the watched toggle and the playlist add.
- `onPlayItem` is required, so no variant draws a Play that does nothing; the player's info overlay
  plays the sibling in the running player (`go(id, _, fromZero)` for its Restart).
2026-09-27 14:44:38 +02:00
peter efa4aeeeda fix(plex): a card offers Restart by the menu's own "started" rule
The picture read "started" off the rounded progress %, so a resume point under 1 % showed Play
while the card's menu offered "from the start". Both use `plexStarted` now.
2026-09-27 14:38:27 +02:00
peter a7d41eb5f3 fix(plex): the player's settings panel is a tab list with named, touch-sized rows
- Its section switch is `ui/Tabs` (new `tone="media"` paint for the dark chrome) with a wired
  tabpanel; the panel opens with the focus on the selected tab.
- On/off rows are `CheckRow` (the whole row, 44 on touch), the "no background" checkbox too.
- Every slider and the four colour wells carry the name their row shows.
- The clock's date-style buttons reach the touch target.
2026-09-27 14:38:27 +02:00
peter 76eead976f fix(plex): the player's stage follows the tablet toolbar, and its bands stay out of a double-tap
- The stage is sized in dvh (80dvh × 1.25 windowed, 100dvh fullscreen); the viewport-units
  contract no longer exempts the player.
- A tap that reveals the bands keeps them pointer-events-none for the double-tap window, so the
  second tap of a skip at the bottom is not a seek.
- An arrow on the volume slider wakes the bands, and a keyboard focus in a band holds them
  (`holdOnFocus`, `:focus-visible` only, so a clicked button does not pin the bar).
- Next stays mounted (disabled) while the next episode's detail loads, keeping the focus.
2026-09-27 14:34:31 +02:00
peter 922d589b09 fix(ui): Tabs is an ARIA tab list with roving focus and a wired panel
The rail was a row of aria-pressed toggles: every tab a Tab stop, no arrows,
nothing tying a tab to its content. tablist/tab/aria-selected, one Tab stop,
arrows on either axis, Home/End, and useTabPanel for aria-controls/labelledby.
2026-09-27 14:22:37 +02:00
peter a975ab627a fix(plex): the player's ⋯ and tracks lists are real menus, and every close returns the focus
They were hand-built divs: no menu roles, no arrow keys, and Escape left the
focus on <body>. Both run on the popover engine now (media-tone rows), and the
tuning panel is a labelled dialog that takes the focus and hands it back to ⋯.
2026-09-27 14:22:37 +02:00
peter 9bedd96774 fix(nav): Back restores the focus after a click on the page's blank area
The page region is <main tabIndex=-1>, so a background click focused it and the
Back restore took that for the reader's choice. A resting place is marked and
one predicate (focusIsUnclaimed) decides "the focus is nowhere" everywhere.
2026-09-27 14:06:16 +02:00
peter 383c85231a fix(plex): a Restart does not replay on a reload or a Forward onto the player
fromStart rode in history.state._sub, so reloading the player entry restarted
the item from 0 again and the progress checkpoint overwrote its resume point.
A sub-view config can now say what the history entry keeps of a view.
2026-09-27 14:03:48 +02:00
peter de62035889 fix(playlists): an Undo puts a row back after its old neighbour, not at a stale index
A raw index recorded at removal drifted once anything changed before the Undo.
Both lists now read the list fresh and re-insert each row after the row that
preceded it (lib/reinsert). A failed watched toggle in a Plex playlist also
puts the row's state back: its local copy was never re-seeded.
2026-09-27 14:02:15 +02:00
peter b456232d00 fix(playlists): lock the playlist row before reading and writing item positions
Two adds read the same max(position) and a reorder racing an add collided with
it. Every add/insert/reorder takes the parent row FOR UPDATE; the YouTube list
breaks position ties by id and a Plex reorder renumbers rows it was not sent.
2026-09-27 13:58:23 +02:00
peter c9e95ecea6 fix(channels): a channel YouTube no longer returns is not asked for on every view
channels.list simply omits a terminated channel, so nothing was stamped and the
channel page spent a quota unit on it every time. The miss is stamped like a
fetch; the published_at sentinel that kept it stale is gone.
2026-09-27 13:58:23 +02:00
peter 7987bd8a37 fix(plex): a stray byte keeps a UTF-8 subtitle UTF-8; untagged and Serbian read as cp1250
One invalid byte sent a whole UTF-8 sidecar to a legacy code page, an untagged
file fell to Windows-1252 (ő/ű became õ/û) and Serbian was read as Cyrillic.
The cache key moves to sub3_ so a track the old rules mis-decoded is read again.
2026-09-27 13:58:23 +02:00
peter 3411265dc1 docs(design): tablet columns, glyph pager, match switch, missing picture, Copy 2026-09-27 07:12:51 +02:00
peter 098b8dbe43 feat(downloads): a bare link is titled once, a missing picture shows a glyph
A job with no name yet is titled by its short link, as the link itself;
the channel line no longer falls back to the ref and the link line does
not repeat it. No picture at all draws a film (or note) glyph instead of
an empty box.
2026-09-27 07:12:51 +02:00
peter f48f16830e feat(filters): the any/all switch sits on the panel above its picker
ui/MatchToggle replaces the Combobox popover footer (Feed tags, topics,
languages; Plex collections) and PlexSidebar's own toggle (genres,
actors): shown from two picks, one wording. A closed picker had hidden
that "all" was on.
2026-09-27 07:12:51 +02:00
peter 955c23d321 feat(tables): a glyph pager with first/last, and tablet-width columns step aside
The pager is four named arrows round the page box (first, previous,
next, last). A table hides a tabletHidden column below 64rem of its own
width, as narrowHidden does below 42rem, through one shared helper that
also hides the column's <col>. The Channels table drops Prio, About and
Length there: at 1024 the channel name had 60 px.
2026-09-27 07:12:51 +02:00
peter d3e1d8219c docs(design): Fjord stays a fifth scheme beside Midnight
The trial is over: the owner kept it. The trial wording goes from the
tokens, the scheme list, the lab and DESIGN; B2's own-hue rule still
binds any new scheme.
2026-09-27 06:46:52 +02:00
peter 4480839db0 merge: bug/channel-details-refresh into E28
# Conflicts:
#	frontend/src/components/ChannelPage.tsx
2026-09-27 06:42:37 +02:00
peter 8e25567ff2 merge: bug/hu-subtitle-encoding into E28 2026-09-27 06:42:08 +02:00
peter 8f65bd184e fix(i18n): a locale edit hot-swaps the texts instead of re-running main.tsx
The locale JSON's importer did not accept HMR, so the update climbed to
main.tsx, whose re-run called createRoot on the live container again; the two
roots tore each other's DOM down ('removeChild ... not a child' by the dozen),
each a fatal toast evicting an old notification. Dev only.
2026-09-27 06:35:40 +02:00
peter ae9c248987 fix(downloads): Retry draws the refresh glyph, not Play (E28 S4)
The first UI-review run (Downloads) found a failed row whose Retry wore the
Play glyph in the row and its menu, promising a video. DESIGN.md gains
'a glyph means one thing' and a log row.
2026-09-27 06:35:40 +02:00
peter bd6aaa1e43 feat(design): the Copy rule — names, verbs, dates and units (E28 S3g)
The owner's picks (all the recommended options) become DESIGN.md's Copy
chapter and a sweep of both languages (742 values, no key drift):

- One name per feature: the rail's label, which may shorten it (Alerts
  -> Notifications, Fiókok -> Felhasználók, Download Center -> Downloads,
  Channel manager -> Channels); one "sync from YouTube" verb.
- Four verbs, four meanings (Delete / Remove / Dismiss / Discard, and
  their Hungarian pairs), checked against what each action does.
- Sentence case, the app's places capitalised; "…" on a command that
  asks first (a popover or form title got its own key without it);
  one-sentence messages without a stop or a tick, "Couldn't …"; curly
  quotes.
- Dates only from lib/format.ts, in the UI language: formatDateTime
  names the month and stops at the minute (it printed "9/27/2026,
  8:30:14 AM"), formatDateTimeSec for the log, the audit and the
  recorder; ui/RelativeTime pairs "3d ago" with the exact minute on
  hover. local/date-format refuses toLocale*String and Intl date
  formatters elsewhere.
- Units from Intl, short (formatUnit): runtimes, totals, the Scheduler
  countdown, the stats range, the player's tap skip. "2h 13m" was
  English in both languages.
- src/i18n/copyRule.test.ts holds the machine-readable part (quotes,
  "...", ticks, bare unit letters).
2026-09-27 05:46:17 +02:00
peter 2c65edfffd fix(downloads): the failure link keeps the 24px floor on a mouse (E28 S3g)
The S3g measure found the row's failure link 20px tall (the meta line's
text height): DESIGN B1 puts nothing under 24. Coarse pointers already
grew it to 44.
2026-09-27 05:13:32 +02:00
peter 6c2c01e67b fix(design): the S3g conformance sweep — primitives at their rung (E28 S3g)
The probe re-run over the 15 pages at 1920x1080 found every button, icon
button, chip, field, tab, segment and menu row drawn by its primitive
except the S3h Plex picture, a div with the button role. It is a
Pressable now, and the button-primitive lint rule refuses an element
with role="button" outside the control primitives.

- SegmentedToggle defaults to the toolbar rung (28, text-xs): the old
  small default had put the pager's page sizes and the YouTube result
  count on 10px type beside 28px buttons. The Plex match toggle asks
  for the small rung explicitly.
- The language badge on the rail uses text-3xs (was an 8px one-off);
  the Plex playlist's watched mark and the Home hero's length use
  MEDIA_BADGE instead of hand-copied classes.
- A media picture shows the pointer whether it is a link or a button.
- DESIGN.md: a Conformance chapter (the four-step check before a UI
  change is done, with the S3g baseline), the segmented rung, two log
  rows. CLAUDE.md points at the check.
2026-09-27 05:11:34 +02:00
peter ef343dbec0 feat(design): side panels open as a drawer on touch and below 768 (E28 S3f3)
C2 as decided: on a coarse pointer, or a window below 768, the panel's
tab (shown at every width, 44 wide on touch) opens the panel as a
start-edge drawer over the content, with a scrim. It is ui/Modal's new
drawer-start placement, so Esc, Back, the scrim, the focus trap and the
scroll lock are the dialog's own. A tap never peeks; pinning is a button
in the drawer's header, offered only at 1024 and up; a pin saved on a
desktop is left alone below that. A mouse on a wide window keeps
hover-to-peek and click-to-pin.

DESIGN: C2 as built, rows answer to their own width, phone-only layout
deferred to its own epic (principle 2).
2026-09-27 04:38:42 +02:00
peter a58a398427 feat(design): the three 375 rows (E28 S3f3)
- Download row: answers to its own width (container query); below 40rem
  it stacks: picture beside a two-line title, the meta line across the
  row (one line, never wraps: zoomed on the iPad its pieces broke apart),
  then the source link and the actions on one wrapping line. The paste
  field takes a line of its own when the buttons would squeeze it.
- YT playlist row: below 42rem the length rides on the picture and the
  title runs two lines over the channel.
- Channels tag row: the suggestions take a line of their own below
  40rem (the chip lane was 0 px and the row's menu off the screen).
- Pager: on a phone the arrows drop their words (names kept), and the
  size label stays one line.
- Lint: a class split where the interpolation is the variant itself
  (${NARROW}:w-24) is refused too; it had shipped with no CSS.
2026-09-27 04:15:08 +02:00
peter d510c8a534 fix(plex): the show and season restart names its object and starts at 0 (E28 S3f2)
The header button said only "Play from start" and resumed the first
episode at its saved point. It now reads "Season from the start · S6 · E1"
or "Show from the start · S1 · E1" and asks the player for 0.
2026-09-27 03:30:04 +02:00
peter dc5206f9d3 docs(design): C3 + C5 as built, and its log row (E28 S3f2) 2026-09-27 02:12:54 +02:00
peter 89e65ca331 feat(design): players on touch and the C5 control set, restart on the card (E28 S3f2)
Plex player: a finger's tap on the picture only shows or hides the controls,
the hide timer starts with playback, a double-tap on a side third skips 10 s,
the seek bar is a 2.75rem touch-none hit area with its head shown on touch.
The row is play/pause, Back by position, Next only with a next item, mute,
time, tracks, a fixed ⋯ (settings, media info, stats, shortcuts, download)
and fullscreen; Stop, Restart and the ±10 s buttons are gone.

A started card shows the video card's Continue + Restart pair (one shared
MediaResumeButtons, beside the picture's button), and its menu lists the
restart; the player opens that item at 0 (fromStart). YouTube modal: the
description opens from an ⓘ, the queue arrows sit inside the screen.
2026-09-27 02:12:54 +02:00
peter 17916b411a docs(design): C4 as built, and its log row (E28 S3f1) 2026-09-27 01:33:37 +02:00
peter 3ef2c38eb4 feat(design): dialogs open by purpose and keep their commit row pinned (E28 S3f1)
Modal takes a required initialFocus (field / close / action), a field that
loads late takes the focus while it still rests on the close button, and
ModalFooter portals a dialog's commit row under the scrolling body from any
depth. The header and footer fades follow the body's scroll timeline, and the
viewport's interactive-widget=resizes-content keeps the footer above an
on-screen keyboard.
2026-09-27 01:33:37 +02:00
peter 4c63d6660d docs(design): the media card chapter and its log rows (E28 S3h) 2026-09-27 00:54:28 +02:00
peter a453b77f04 feat(design): one media card, and Home's hero and shelves as cards (E28 S3h)
- ui/MediaCard + mediaCardStyles: one frame for every video, Plex title and
  Home tile; the meta line pinned to the bottom edge, the length a badge on
  the picture, the media Play only where the click plays, the hover row at
  the picture's bottom-right (the owner's pick C).
- Shelf density: on any strip a one-line title over one meta line; mixed
  Home shelves are landscape; rails turn content-visibility off (a card's
  reserved height had become the strip's). Home slot 232px, so the Play
  and the hover row no longer touch; a landscape shelf takes the small Play.
- Home hero is a card (hover row, the whole menu, a touch "…"), three
  quarters wide and centred; channel / playlist tiles get Play, open and
  "…"; a show tile acts on its next episode, ⓘ opens the show.
- Plex preview: a ~0.9 s mouse rest opens a passive panel with the whole
  title, facts and synopsis (popover `pointerOnly`); the synopsis and title
  tooltips are gone.
- Compact list row B: the title leads, "…" at the end, toggles over the
  date on hover.
- Media-info layout B (collections, crew, cast beside the facts sidebar;
  an empty column is not drawn); filtering values are chips there and on
  the show page; "Cast" over the cast-only strips.
- Browse tiles hover like a card; prose capped by the `measure` utility.
2026-09-27 00:54:19 +02:00
peter 8e1004fe6f fix(channel): load the banner without a referrer
Google's image CDN refuses a banner request that carries a Referer from our origin - every
time, not only under load (measured: 6/6 refused with it, 8/8 served without). The banner
<img> now sends none, as ui/Avatar does for the same CDN; the comments that blamed a
transient burst throttle are corrected.
2026-09-26 21:45:46 +02:00
peter 663f30feeb fix(sync): refresh a channel's details once they are a week old
Channel metadata (banner, avatar, description, counts) was fetched only the first time a
channel was seen: the subscription import asked only for channels with no details, and the
channel page only for ones missing a join date. 394 of 397 subscribed channels had July
metadata. Both now also refresh details older than DETAILS_MAX_AGE (7 days), through one
shared staleness rule; channels.list costs 1 unit per 50 channels.
2026-09-26 21:45:46 +02:00
peter 6653d81a80 fix(channel): no broken-image glyph while the banner retries (E28)
The banner is retried on error; each failed attempt showed the browser's
broken-image glyph for as long as the retry waited. The <img> now stays
invisible until its source has actually loaded.
2026-09-26 20:55:34 +02:00
peter 21df7c8348 docs(design): the background menu as built, and the log row (E28 S3e4c) 2026-09-26 20:48:53 +02:00
peter 97381e65d0 feat(design): a page's background menu on the five list pages (E28 S3e4c)
A right-click on the EMPTY area of a page opens a menu of that page's own
toolbar: the feed (sort, reshuffle, view, watch state, content, source, clear
filters), the Plex library grid (sort, watch state, clear), the channel manager
(layout, the cards' sort, sync status, edit tags, clear the tag filter, read
subscriptions), a playlist (play all, group by, reset view, save order,
undo/redo, the list's own menu) and the notification center (mark all read).
No destructive row and no backfill.

PageShell listens on the page area; lib/backgroundTarget decides empty (not
text, a control, a link, media, or a row/card with its own menu). Shift+right-
click, a touch long-press and a right-click on selected words keep the
browser's menu. ui/PageMenu registers while mounted, so a loading or error
state keeps the browser's menu too. The rows mirror each control through
ui/pageMenuItems (sortSubmenu via pickSort, choiceSubmenu, multiSubmenu).
pageMenuScope.contract.test.ts pins the five pages.
2026-09-26 20:48:53 +02:00
peter f8ced6d819 docs(design): the flush-row ring, the fade at rest, the narrow table, and their log rows (E28) 2026-09-26 20:01:04 +02:00
peter f254588409 fix(playlists): hide '#' and Channel by the table's width, not the window's (E28)
Beside the playlist rail an 800 px window left the table 432 px: the fixed
columns took all of it and the title column was 0 px wide, its thumbnail
drawn over the channel. The table is an @container now and the columns hide
under 42rem of its own width (narrowHidden, was phoneHidden).
2026-09-26 20:01:04 +02:00
peter 3e7c79fa69 fix(design): the keyboard ring of a flush row stays visible (E28)
A list item's ring sat 2 px outside it. On a table row it ran under the
sticky head and past the table's sides; on a Plex playlist show header, flush
in its clipping card, it was cut away and the focus was invisible. A table row
(by its tag), a log line and the Plex show header and episode rows now ring
inside themselves, the show header with its card's corners.

The sticky header's fade-below was drawn at rest, over the first row's top
16 px, hiding the ring's top edge: it now fades in with the first 1rem of
scroll (a scroll-driven animation; no fade where unsupported).
2026-09-26 20:01:04 +02:00
peter 47c12f9405 docs(design): right-click on a selection, and the log row (E28 S3e4b) 2026-09-26 19:03:34 +02:00
peter bd0914b7ed fix(design): a right-click on selected words opens the browser's own menu (E28 S3e4b)
The row menu took every right-click, so selected text could only be copied
with Ctrl+C. On the selection (its laid-out line boxes) the event is left to
the browser; beside it the object's menu opens as before.
2026-09-26 19:03:34 +02:00
peter 079cee4e9c docs(design): titles in click surfaces stay selectable, and the log row (E28 S3e4b) 2026-09-26 18:46:24 +02:00
peter edf2f22ef9 feat(design): titles in click surfaces stay selectable (E28 S3e4b)
A button whose text is a title or a name is now `selectable` (FocusSafeButton,
under Pressable and Button): Chrome selects inside a button once it is
user-select: text (measured), and the click that ends a selection, or a press
dragged past the slop, opens nothing. 23 surfaces carry it (cards, rows, people,
channels, playlists, users, views, messages, log lines, description links); the
Plex caption div gets the same guard. A contract test pins every plain Pressable
with its reason, so a new one has to choose.
2026-09-26 18:46:24 +02:00
peter e1ef56cdbc docs(design): the non-drag reorder forms, and the log row (E28 S3e4a) 2026-09-26 18:14:28 +02:00
peter 4d0abc61d5 feat(design): reorder without a drag everywhere a grip drags (WCAG 2.5.7, E28 S3e4a)
One rule, two forms: useOrderItems (Top / Up / Down / Bottom rows of a row
menu) and OrderButtons (four buttons with the place, or a compact Up / Down
pair). A move that would go nowhere is shown off.

- YouTube playlist rows: the four moves now come from useOrderItems.
- Plex playlist rows: a block among the blocks, an episode within its show,
  off while a write is in flight; drag and menu share moveBlock/moveEpisode.
- Saved views editor: the selected view's four buttons (owner: A), and the
  grip's keyboard pick-up (its instructions promised one it did not have).
- Side panel groups in edit mode: an Up / Down pair (owner: A); a group the
  stored order does not hold yet moves too; the name wraps there.
- The grip's touch-action: none is now test-held.
2026-09-26 18:14:28 +02:00
peter 7c2b22bafa docs(design): the channel menu's Tags and Priority, MenuCheck, and the log row (E28 S3e3b) 2026-09-26 17:34:47 +02:00
peter 5e5c8a42a9 feat(design): the channel menu's Tags and Priority submenus (C1, E28 S3e3b)
Tags A-Z as in-place checks (the menu stays open for the next); Priority
as the fixed steps 10 5 3 2 1 0, an off-scale current value ticked in its
place. A pick re-sorts the list at once, moving only that row
(placeByPriority, the server's order), and the table / card pager and
the focus follow it (DataTable and useCardPager take a reveal seq,
listNav.focusNavItem lands on the row's own "…", a virtualized card
revealed first). ActionMenu rows take check: many | one. The menu reads
the row fresh from the cache and keys its target by id.
2026-09-26 17:34:47 +02:00
peter 2b3094ffc5 refactor(design): one drawn check for every checkable row that cannot be an input (E28 S3e3b)
ui/MenuCheck: a box filled with the accent for a set (a minus for a group
only partly in), a bare tick for one choice, in the row's leading slot.
Replaces seven hand-drawn marks: three box sizes, a tinted box, a cross
for "on" in the table filter, a trailing tick in the language menu.
2026-09-26 17:34:46 +02:00
peter 9c91eb5d22 feat(design): a library row keeps Share (the link copy) in sight, ahead of save (E28 S3e3)
The owner's call: sharing a download's link is the library's main errand, more
important than saving the file, so it does not hide in the row's "…". Manage
sharing stays in the menu. The "…" tests press as a mouse, so a stray long-press
timer no longer fires after teardown.
2026-09-26 07:21:21 +02:00
peter 2df60fbf07 docs(design): how the download, notification and Scheduler action lists are built, and the log row (E28 S3e3) 2026-09-26 07:10:02 +02:00
peter a2d17d693c feat(design): the download, notification and Scheduler action lists (C1) — the rows' "…", the band's and the strip's "…" (E28 S3e3)
A download row keeps play (pause / resume / retry in the queue) and the save link; the editors,
sharing, re-download, the copies and cancel / delete / remove moved into its "…"
(useDownloadActions). A link inside the row keeps the browser's own right-click menu. A
notification row keeps its link and its unread mark and shows its dismiss on hover; an activity
notice's Find in feed and Undo are in its "…". The inbox band keeps Mark all read and puts Clear
everything, now behind a confirm, in its "…". The Scheduler strip keeps Run all and Pause and puts
Purge discovery in its "…". Polled lists key their menus by id (keyOf). ui/menuGroups draws a
rule only between two non-empty groups. Find in feed also drops the tag, search and age filters
that could hide the video it looks for.
2026-09-26 07:10:02 +02:00
peter c4917619ee fix(design): a row's "…" closes its own menu when its row object is rebuilt (E28 S3e2)
The "…" told its own menu from another row's by payload identity. The Plex playlist page builds
each row's menu target afresh on every render, so a second press on the same "…" reopened the
menu instead of closing it. useContextMenu takes an optional `keyOf`, and the page keys its
targets by the row key.
2026-09-26 07:10:02 +02:00
peter 760bdae278 docs(design): how a channel's and a Plex playlist's action lists are built, and the log row (E28 S3e2) 2026-09-26 05:56:08 +02:00
peter 1b946113c6 feat(design): a channel's and a Plex playlist's one action list (C1) — the row's "…", Delete + undo, an action toast's 8 s floor (E28 S3e2)
- Channels: the row's full-history, hide and unsubscribe buttons become its "…" at the row end
  (useChannelActions: open, pin, YouTube, copy name / link, full history or the admin reset,
  hide with an undo toast, unsubscribe behind its danger confirm); the card layout docks the "…"
  beside the name (ChannelLayoutGrid moreKey). The admin reset's confirm now names the reset.
- Plex playlist page: a title, a show group and a season each open the page's one row menu (a
  title's is its usePlexCardActions list plus the removal); no X at rest; removals take an undo
  toast whose Undo puts every row back at its own index in one request (bulk add `positions`);
  j/k walk the visible rows, Delete removes the walked one; rename and delete in the header's "…".
- notify: a toast with an action lives at least 8 s (the owner's call), whatever the toast setting.
2026-09-26 05:56:07 +02:00
peter b0e51c6f89 docs(design): how a playlist's action list is built, and the log row (E28 S3e2) 2026-09-26 04:05:39 +02:00
peter 7455fff401 feat(design): a playlist's one action list (C1) — the row's "…", Delete + undo, the header's "…" (E28 S3e2)
- Playlist rows: the X column became the row's "…" (play, pin, copy title/link, move, order,
  remove). Remove runs at once, with no confirm; its toast's Undo puts the row back at its index.
  The Delete key removes a keyboard-walked row, and focus moves on to the next.
- POST /api/playlists/{id}/items takes an optional `position`, so an undo is one request that
  renumbers the list around the row.
- A new `listDelete` key scope. Only a list with `onDelete` registers it, so the help sheet does not
  offer Delete where nothing handles it.
- A playlist has one list, `usePlaylistActions.items`: rename, pin, YouTube open/push/revert,
  delete. The rail row's "…" and the page header's "…" both open it. Push and revert moved into
  the hook; the header keeps Play all.
- Phone-width playlist table: "#" and Channel are hidden below sm (`Column.phoneHidden`). Their
  fixed widths had left the title no width.
2026-09-26 04:05:39 +02:00
peter aaf1843b09 docs(design): how a Plex title's action list is built, and the log row (E28 S3e2) 2026-09-26 01:55:29 +02:00
peter a4737b92e4 feat(design): a Plex title's one action list (C1) — the poster's "…", details and the pin in its menu (E28 S3e2)
Every Plex poster list (library, Discover rails, person, show, season, collection, Home's rails)
now opens one menu per list from the right-click, the long-press, Shift+F10 and the card's "…":
play/open, watched, add to playlist, media info, copy title, the pin (usePlexCardActions +
PlexMenu, replacing the pin-only PlexPinMenu). The card keeps its state at rest and shows
watched / add to playlist / "…" on its picture on hover (ui/reveal.ts, shared with the video
card); on touch the toggles are gone and a 44px "…" sits beside the title, while the watched pill
stays. The hover ⓘ left the card; an episode's add button moved from its caption to the hover
row. Seasons get their own short menu (open, watched, add, copy), labelled "Open season" (the
card's overlay said "Open show"). Related shows are ordinary poster cards now. A collection's
admin "Remove" X moved into its menu, still behind the confirmation. The collection grid is two
columns on a phone. The unused .reveal-idle rule is gone.
2026-09-26 01:51:53 +02:00
peter 75ae810866 feat(design): a video's one action list (C1) — the card's "…", the menu holds the rest (E28 S3e1)
- useContextMenu gains buttonProps (a row's visible "…" opens the same menu under the button; a
  second press closes it) and anchor (the element the menu came from, for a panel a row opens).
- ui/MoreButton; ActionMenu rows can run inGesture (a clipboard write inside the click, Safari).
- useVideoActions: one builder for a video's menu (play / restart, channel, watched, save, add to
  playlist, download, details, copy title / link, hide, reset, pin), composed by the feed and by
  Home's rails; the playlists and details panels open anchored beside the card (VideoFollowUp).
- The card keeps watched / save / add to playlist and the "…": on the picture on hover (the
  duration steps aside), in the rows' action column; on touch only a 44px "…" beside the title.
  Download, hide, reset and the hover info card leave the card.
- The title is selectable: a click ending a selection neither plays nor follows the link.
- DESIGN.md: how C1 is built + the log row.
2026-09-26 01:23:44 +02:00
peter 9d7526ee57 fix(feed): a reset clears the card's resume position at once, not only its status
The optimistic layer overrode only the status, so after "Reset — clear watch progress/status" the
card kept its progress bar and its Continue button until the next refetch. Found in the E28 S3e1
UAT.
2026-09-26 01:23:44 +02:00
peter fa0b0edd63 docs(design): how B4, B9 and the viewport rule are built, and the log rows (E28 S3d4) 2026-09-26 00:39:10 +02:00
peter bfc6d199c0 fix(a11y): a video card's thumbnail link is named by the title, not the duration badge (E28 S3d4)
Its only text was the duration, so every card's first Tab stop was announced as "44:10".
2026-09-26 00:39:10 +02:00
peter 4d6e59b80a feat(design): the public watch page on the app's surfaces (B6) — the scheme backdrop, the title and actions on a glass panel, a glass Download (E28 S3d4) 2026-09-26 00:39:10 +02:00
peter 4646aae320 fix(design): the table header takes the B9 fade utility, and every size is dvh, not vh (E28 S3d4)
The table header carried an arbitrary-value copy of fade-below; a contract refuses a hand gradient of a surface colour. 100vh is the viewport with a tablet browser's toolbar retracted, so h-screen / min-h-screen / static vh hid the last rows under it; a contract holds dvh (the Plex player stage changes with S3f, where it is played on the tablet).
2026-09-26 00:39:10 +02:00
peter eddeca82d3 feat(design): the page band (B4) — one h1 per page, section headings, skip to content, filters fold on a short window (E28 S3d4)
- PageShell names the page with a visually hidden h1; a view with its own title renders it as
  PageHeading and the hidden one steps aside (contract: no hand <h1> inside the shell).
- Section captions are SectionLabel headings (h2, h3 in a dialog), ui/form Section included.
- "Skip to content" is the first Tab stop and focuses the scroller's <main>.
- Below 600px of height the band's filter row folds into "Filters · n" (Feed, Plex, server log),
  the same row shown in place.
- The band holds controls only: Downloads and Notifications lose their title cards, the Config and
  channel-discovery intros scroll with the content, the Stats tabs and the audit pager join the band,
  and the server log's toolbar sticks with the fade under it.
2026-09-26 00:39:01 +02:00
peter 69edeecc45 fix(a11y): a labelled rail row says its label and the exact unread count, not the capped badge first (E28)
With the labels on, the row's name was its text, badge first: 15 unread was heard as "9+ Alerts". The
name is now the visible label plus the count, as the icon-only row's is, and the drawn badge is hidden.
2026-09-25 23:59:20 +02:00
peter 6088e476a7 fix(design): the command palette's active row takes the menus' roving highlight, not an invisible bg-card; the Plex typeahead's hand copy of it goes (E28)
The palette row was ~1.07:1 on its glass (B3 names bg-card hovers a violation). ROVING_HIGHLIGHT is
exported from AnchoredPopover for the listboxes that are not PopoverItems; a contract holds that no file
paints the fill itself and every hand-built option row takes the recipe.
2026-09-25 23:51:39 +02:00
peter ce171fa954 fix(feed): a custom date bound applies once its year is a plausible upload year, not per typed digit (E28)
A native date field reports a complete date on every keystroke of its year (0002, 0020, 0202, 2026),
so one typed year refetched the feed four times. The field holds a draft while focused; leaving it
empty clears the bound, leaving it half-typed restores it.
2026-09-25 23:43:26 +02:00
peter 113da76c36 fix(design): a user row answers as a whole card — the hover layer under its badges and buttons, a click anywhere opens the account (E28)
The layer was the name button's own and ended at the row's first badge or button. The card now paints
the B3 layer under its content (isolate + a new z-under level), the name button's hit layer spans the
card, and the actions sit above it.
2026-09-25 23:43:26 +02:00
peter 4ef11901db fix(design): the nav rail's footer rows (language, About, Sign out) share the rail row and glyph (E28) 2026-09-25 17:21:01 +02:00
peter 29abf609f3 fix(design): the lab draws with the primitives' own classes, not a height-ladder copy; DESIGN no longer claims accent-color tints the colour well (E28) 2026-09-25 17:15:03 +02:00
peter e13d93a3ed fix(design): a toggle is pressed, never a bare aria-pressed; static labels are flat Badges with a tone, edged label spans lint-refused (E28) 2026-09-25 17:04:08 +02:00
peter b9850b1710 feat(release-notes): terse bilingual notes as data files — new/fixed only, the dialog lists the latest ten (E28) 2026-09-25 15:49:16 +02:00
peter 905a9f2856 feat(design): Fjord, a trial fifth scheme — Midnight's navy with a steel-blue accent and steel rules (E28) 2026-09-25 15:39:49 +02:00
peter 1d8b83c37c fix(plex): a sidecar subtitle in a legacy code page rendered its accents as U+FFFD
A Hungarian .srt beside the video (Windows-1250) was decoded as UTF-8 with
errors=replace: one episode served 1391 U+FFFD and no accented letter.
decode_subtitle now lets a BOM decide, keeps valid UTF-8, and otherwise
reads the legacy code page of the stream's language (cp1250 for Central
European, cp1251 for Cyrillic, else cp1252) — on the ffmpeg path too.

The cache has no eviction, so its key is versioned (sub2_): a track cached
broken is read again once instead of being served for good.
2026-09-25 15:20:31 +02:00
peter 62dbf35ea3 fix(player): the Plex bars hold while hovered, volume is the native slider, tooltips no longer re-layer the row (E28)
- F22: the control bar faded out from under a mouse parked on it; the hide timer
  now waits while the pointer is on either band and re-arms on leave (test seen
  RED by disabling the hold).
- The hand-drawn volume bar broke DESIGN.md A6 (native sliders): it had no hover,
  focus, keyboard or screen-reader state. It is the Slider primitive now; a
  pointer drag blurs it so Space and the arrows keep driving the player.
- The control tooltips are composited from mount (will-change: opacity): a fade
  that promoted one on hover re-layered its neighbours under the player's
  scale(1.25), and the whole row shimmered.
2026-09-25 14:48:45 +02:00
peter e9ccd9bc98 fix(player): the YouTube player's full window filled only its card — a blurred ancestor trapped the fixed stage (E28)
E28 S3d1 moved the modal's card from `.glass-card` (no blur) to the `.glass` panel role, whose
`backdrop-filter` makes the card the containing block of the fixed "full window" stage: F showed a
894×161 strip inside the card with a scrolling, wrapped control bar. While the stage is maximised the
card now drops its blur (`html .glass:has(> .player-stage--max)`, outranking the dark theme's rule);
measured live: the stage 1268×861 = the viewport, the blur back on exit.
2026-09-25 14:20:23 +02:00
peter cc8e0648db fix(design): the button and chip hover colours were never generated — an act: variant instead of a runtime prefix (E28)
buttonStyles/chipStyles glued the hover guard on at runtime (`${ACT}hover:text-fg`). Tailwind emits CSS
only for a class it finds whole in the source, so all 22 guarded hovers shipped with no rule: no text or
edge colour on hover for ghost/outline/danger buttons, no white layer on the player's control row, no
underline on a link button, no accent hover on token/suggest/ghost chips. The stylesheet had 0 rules
for `not-aria-disabled`.

- `@custom-variant act` (index.css): the same guard as the state layers; `:disabled` never matches an
  <a>, so one variant serves buttons and links and the per-element variant maps go.
- ui/tailwindScan.contract.test.ts: every class the primitives can emit stands whole in their source.
- a lint rule refuses a variant right after an interpolation; the `enabled:hover:` rule now points at
  `act:hover:`.
2026-09-25 14:13:51 +02:00
peter ff80cc96fd fix(design): rating and live colour tokens, the player's menus over its notices, the info overlay on a light theme (E28)
- F16: --rating (theme-scoped gold, 4.5:1 on the page and its chip), --rating-media, --live/--live-fg,
  --media-chrome replace 14 palette literals; a lint rule refuses palette classes; the theme contract
  measures them.
- The Plex player's notices drew over its open settings/tracks popovers: notices z-base, popovers z-menu.
- The player's info overlay rendered theme tokens on its always-dark panel: on a light theme the score
  (2.6:1), collection years (3.2:1) and cast initials (1.05:1) were unreadable; onDark variants now.
- The lint-rule contract tests share one ESLint with a one-off warm-up (a cold first lint took 8 s,
  past vitest's 5 s).
2026-09-25 13:55:26 +02:00
peter 295c1db5f1 fix(design): claim audit part 1b — whole-row switches, the field-family edge, state-layer hovers, icon links, reason-disabled controls, 24/44 targets (E28)
- F12 (owner: A): SwitchRow, the whole settings row toggles; a Config boolean toggles from its title
  and hint (a label pointed at the switch).
- F14 (owner: B): the form dropdown trigger wears the field frame (see-through, the >= 3:1 edge);
  labelled pills keep the faint edge and the accent-edge hover. DESIGN says so, with log rows.
- F6: ~15 bg-card / bg-border / opacity hovers onto the B3 state layer; a lint rule (HOVER_SWAP)
  refuses the swap, contract-tested.
- F10: bare 14px icon links onto IconLink / LinkButton; a local lint rule refuses a glyph-only <a>.
- F13: reason-disabled controls are aria-disabled (Users, Channels backfill, Discovery, Scheduler
  runs, Config tests); FocusSafeButton now swallows the click of a caller's aria-disabled; the
  Config tests say why while dirty (testDirty, HU/EN).
- F5/F11: every control measured live at 24 (mouse) and 44 (coarse) across every page; touch-target
  is a utility that wins over a caller's min-h-6; link variants carry it too; SegmentedToggle,
  Tabs, NavRail, menu rows, PriorityStepper, sliders, CheckRow, InnerInput, card links and the rest.
- F9: DismissibleChip's label reaches the chip's 24; the Combobox chips are DismissibleChips.
- F15: the "+N" / "More..." open state worded as it renders.
2026-09-25 03:58:17 +02:00
peter 0521149cf1 fix(design): claim audit part 1 — a closing key no longer re-presses its trigger, the Switch box and name, named z-levels (E28)
- In-place editors (Playlists heading + rail rename, Plex collection + playlist rename, PriorityStepper,
  Scheduler interval): a real Enter closed the editor, returned the focus to the trigger, and the same
  keystroke's activation pressed that trigger again - re-opening the editor, or in the rail selecting
  the renamed playlist. useReturnFocus.arm(e) now cancels the key's default (type-enforced); the rail and
  Scheduler editors do the same. The earlier tests dispatched a synthetic keydown, which has no default.
- Plex playlist rename: an unchanged or empty title closes with no write, as the collection rename does.
- Switch: a 36x24 box around the track (the 24 floor), 44 on a coarse pointer; `label` is required, and
  the Plex two-way-sync switch had no name.
- z-index: numeric z-10/z-20 and a literal 40 moved onto the named layer scale; a lint rule + contract.
- The /watch file card's Download is the filled primary: the page's one action.
- Stale comments: the 1024x768 floor, the retired glass tuner.
2026-09-25 02:25:54 +02:00
peter b4a6569d0d fix(design): the saved-view chip on one line, one Escape per layer, the /watch action row (E28 S3d)
- SelectTrigger lays its toolbar value out as a row; the saved-view chip passes its glyph and
  modified dot through the icon/trailing slots (an <svg> is display:block and broke the chip in two).
- One Escape in a dialog's open list closed the list AND the dialog: the list's close flushes
  synchronously and pops its layer before the dialog's window listener runs. takeEscape() is the one
  close-on-Escape test (topmost AND not yet taken); Modal, the Welcome lightbox and useLayer use it,
  and keymap.dispatch ignores a taken Escape.
- /watch: the Download sits on the toolbar rung, level with the Repeat chip.
2026-09-25 01:40:45 +02:00
peter 1833f594cb fix(design): an audit of the S3d claims — the float shadow, filter hints, one action size per row (E28 S3d)
- A glass role's shadow is one token (shadow-float); the header search's drop-shadow utility had
  replaced its top highlight, and the debug pill's shadow-lg did the same. The surface-role lint
  rule now refuses a shadow size beside a role.
- Plex: every value that filters carries a hint (genre, certificate, studio had none), and the facts
  panel's year and certificate filter like the header's. filterKeyword -> filterValue.
- Downloads: the row resized its buttons with a [&_button] override that the save link escaped
  (28px beside 32px); the size is the primitive's md rung now.
2026-09-25 01:16:20 +02:00
peter 1615838f33 fix(design): the rail's active edge bar was clipped by the scrolling list (E28 S3d3)
An overflow-y container clips on both axes; the bar sits in the rail's padding, outside the row.
NavRailList spans that padding (-mx-1 px-1) so the bar stays inside the list's box.
2026-09-25 00:59:23 +02:00
peter 7fa8f4ebde feat(design): the NavRail primitive with optional short labels, the flat surfaces onto roles (E28 S3d3, B7/B5) 2026-09-25 00:47:46 +02:00
peter 6354454cc9 feat(design): skeleton first loads by shape on the shared reveal clock (E28 S3d2, B8) 2026-09-25 00:37:45 +02:00
peter c47c193ad1 feat(design): glass surfaces by role, the static Badge, LinkButton, user pickers on the Combobox (E28 S3d1) 2026-09-25 00:34:35 +02:00
peter db10bfbbca feat(design): the header search is a 40 glass pill; a wrapper field dims for its input only (E28 S3c) 2026-09-25 00:05:31 +02:00
peter dac9d89f91 feat(design): the field family, the dropdown trigger, native controls, the no-raw-input lint rule (E28 S3c)
- ui/Input, Textarea, InnerInput + fieldClass (A4): the >= 3:1 edge, a 2px accent
  focus edge, the danger edge for invalid, md/sm/xs/heading rungs, 1rem and 44 on touch
- ui/SelectTrigger (A5): SortControl, FilterSelect, the view switcher and the date
  filter on one trigger, a pill in a toolbar, r8 in a form
- ui/NativeControls (A6): Checkbox, CheckRow, Slider, ColorInput
- every raw input swept (~100 sites); inputCls retired; local/input-primitive
- useReturnFocus: an in-place editor hands the focus back to its trigger on
  Enter/Escape (the playlist and Plex title renames, the priority number);
  the Plex playlist rename gains Escape and an accessible name
- back to top: a round 48 FAB (owner decision); DESIGN.md A2 and the log
2026-09-24 23:48:22 +02:00
peter 81936e9f56 feat(design): every button on its family, the chip primitive, the lint rule (E28 S3b2)
The second half of the button sweep: the remaining hand-rolled buttons in
feature components, the design lab and the components/ui composites now
stand on Button, IconButton, Pressable, Chip or DragHandle.

- Chip + DismissibleChip (filter / token / tag / suggest / ghost, 24px
  floor, 44 on touch) replace five chip geometries; FilterChipGroup
  renders Chips; overflow lanes measure a mirror painted by chipClass.
- The Plex control row is IconButton's player rung (36, 20px glyph,
  round, overlay; 44 on touch); the YT modal bar is on the sm rung with
  pressed cycle toggles and names that survive a folded label.
- New variants: IconButton ghost-accent, danger-soft; Button warn on a
  new state-warn utility. The .btn-{tone} recipes and btnCls are gone.
- Fixed on the way: membership rows (collection editor, Plex playlist
  add) announce checked/mixed; a one-letter tag chip was 21px wide;
  descriptionLinks moved out of lib/ (it renders components).
- local/button-primitive: a raw <button>/<FocusSafeButton> outside the
  control-primitive files is a lint error (contract-tested, seen red).
2026-09-24 23:03:23 +02:00
peter 86f467bd65 feat(design): the button families on the decided ladder, first half of the sweep (E28 S3b1)
Button is rebuilt to A1/B1/B3: explicit heights (xs 24 / sm 28 / md 32, lg 40 for a hero CTA), r8 (a
pill on media), state layers instead of opacity and bg-card hovers, a `pressed` toggle, `overlay`,
`media` and `link-muted` variants. The icon-only sizes move to the new IconButton (28 / 32, 48 only for
the Play on a picture; a required label). Pressable (a card, row or tile that is the control) and
DragHandle join them. The state-* utilities now stand down on a disabled, busy or pressed control, held
by theme.contract.test.ts.

Swept onto them: the Button icon-size callers and 16 files (Plex playlist, Plex detail, the YT player
modal chrome, Welcome, Plex and video cards with AddToPlaylist and DownloadButton, the video editor,
Feed, personal views, Scheduler, Users, the nav rail, Plex detail parts, Settings): 268 raw buttons are
153. Fixed on the way: FocusSafeButton overwrote a caller's aria-disabled (the "unavailable, here is
why" state; adminInvites caught it), a season toggle and a Plex settings switch had no name/state, and
the video card's overlay follows the C5 card (Resume pill, From the start pill, a round Play).
2026-09-24 05:25:55 +02:00
peter 7d480e2ad0 feat(design): Matrix toned down to moss, text on an accent fill held to --accent-fg
The lime accent was the loudest of the four (OKLCH L 0.85, C 0.21) and read as neon; the owner chose the
moss variant at Midnight's lightness with near-neutral greys, and the Matrix backdrops follow it. Fixed
white/black parts on a solid accent fill (two switch knobs, three play buttons, the Up next pill) now use
--accent-fg, and theme.contract.test.ts refuses a fixed white or black next to a solid bg-accent.
2026-09-24 04:49:07 +02:00
peter ad136c3fe8 feat(design): the scheme set, the colour contract and the token utilities (E28 S3a)
Four schemes (Midnight; Forest cyan, Slate fuchsia, Matrix lime), YouTube
and Starship retired with no data migration: normalizeTheme maps a stored
retired scheme to Midnight. A vitest contract measures every scheme x mode
(text 4.5:1, control edge 3:1, status text, the accent's 30-degree hue gap);
it caught light-mode danger at 4.43:1, now red-700. The state layers, the
strong edge, the touch target and the fades are index.css tokens and
utilities the lab now reads. Backdrops recoloured to the new accents, with
a grey overlay grain that actually breaks the banding (longest flat run
15-26px -> 4-8px). The glass tuner is retired.
2026-09-24 04:33:13 +02:00
peter 9c4a999e54 feat(design): C5 recorded, every E28 decision taken
The owner's lean Plex control set with both refinements (Back by position,
a touch double-tap for the 10 s skip) goes into DESIGN.md's Players
chapter and the log. The lab is now the reference only.
2026-09-24 04:12:19 +02:00
peter 3f5634a44f feat(design): C5 redrawn around the owner's reworked Plex controls
Stop, Restart and the 10 s skips leave the row, Back restarts or steps to
the previous episode, Next shows only when there is a next item, and the
card carries Resume + From the start. The lab asks back two points: how
Back tells restart from previous, and a touch path for the 10 s skip.
2026-09-24 04:05:09 +02:00
peter f4bc60d659 feat(design): batch C recorded, the player row open as C5 (E28 S2d)
C1-C4 as proposed: the action classification, overlay-drawer side panels,
touch-first players, dialog focus by purpose with a pinned footer. The
players were then played live at tablet size, which found the Plex
control row running off-screen below 1024; how it fits is open as C5.
CLAUDE.md now points at DESIGN.md.
2026-09-24 03:27:26 +02:00
peter 7055548260 feat(design): batch B recorded, batch C in the design lab (E28 S2c)
DESIGN.md: B1-B9 decided (compact density, the four-scheme set with the
accent-vs-status rule, state layers, the sticky band, surface per role, the
watch page, the slim NavRail with optional labels, loading by shape, the
fade edges) with their decision-log rows. The lab keeps the chosen A and B
renderings as reference on shared tokens (labTokens.ts) and adds batch C:
the action-placement classification for review, side panels on touch and
below 768, the Plex player on touch, and the dialog conventions.
2026-09-24 03:10:00 +02:00
peter 477271907e feat(design): batch A recorded, batch B in the design lab (E28 S2b)
DESIGN.md: the Controls chapter decided (A1-A6) with the decision-log rows;
density stays open as B1. The lab keeps only the chosen A renderings as
reference and adds batch B: density, the scheme set (measured accent hue
gaps and light contrast), state layers, the page-chrome model, surface per
role, the public watch page, the nav rail, the loading form and where the
header + fade edge goes.
2026-09-24 02:21:20 +02:00
peter 50923bb684 fix(build): keep JSX line-end spaces in dev and vitest
plugin-react recognises the React Compiler only by its package name, and
reactPlugin.ts passes the imported plugin, so plugin-react left Babel's
retainLines on in dev. With the compiler's re-printed JSX that trimmed the
space at the end of a text line (the privacy page read
"for example/api/feed" on :5173). Production builds never had it on.
Turn retainLines off through generatorOpts, which wins over the plugin's
top-level option, and pin it with a DOM test seen red without the fix.
2026-09-24 02:21:20 +02:00
peter d2e40dcffb feat(design): design lab + DESIGN.md skeleton (E28 S2a)
- docs/DESIGN.md: the written source of truth for the UI. Decided chapters
  (principles, device adaptation, action placement, text stays selectable,
  interaction rules, copy, anti-patterns, decision log); controls and
  surfaces are marked Pending until the owner picks from the lab.
- /design-lab (dev only): renders each open decision's variants with the
  app's real CSS, themes and font scale. Batch A = six control decisions.
- devOnlyGuard: a production build fails if anything under src/design-lab
  reaches the bundle (seen red with an unguarded import).
- Size contract: tablet portrait + landscape and up, replacing the
  1024x768 floor with its never-built "too small" message.
2026-09-24 01:34:12 +02:00
peter 58878958d8 merge: chore/e2e-password-from-store (E2E password from the secret store, no committed default)
gate / frontend (push) Successful in 4m37s
gate / backend (push) Successful in 6m29s
2026-09-23 23:05:17 +02:00
peter b47d22cf3b chore(e2e): no committed default for the E2E account password
The localdev e2e account's password was a hard-coded default in this public repo, so it
was never a credential. E2E_PASSWORD is now required; the ops wrapper exports it from
the secret store, and the seed/setup fail with a clear message when it is missing.
2026-09-23 02:35:19 +02:00
peter e804ff3b8e release: 0.95.0 (improvement/p1-upload-auth-first)
gate / frontend (push) Successful in 4m35s
gate / backend (push) Successful in 6m18s
2026-09-23 01:32:45 +02:00
peter cd6f119883 chore(release): 0.95.0
The P1 epic: the upload route reads its own body after `require_human`
resolves, so an anonymous caller can no longer make the server spool 500 MiB
before the auth check runs. Three deep-review fix rounds on top of it.
2026-09-23 01:32:32 +02:00
peter 321776480e fix(uploads): make the drift guards fail for the reasons they claim
Deep-review round 3, targeted at the surfaces round 2's fix commit changed.
Five of the eight findings are the same shape: the guards written to stop
half-done sweeps were themselves half-done.

test_upload.py's display-field guard walked only `ast.Assign`, so the one
writer with an unbounded REQUEST HEADER behind it — `spool_upload`'s
`display_name=display_text(...)` keyword — was invisible to the guard meant
to protect it. It also stopped at `app/routes/`, missing the four writers in
`downloads/` that actually construct the row, and asserted `not offenders`,
which enforces nothing once a refactor moves the writers out of its view.
It now walks assignments AND keywords across `app/`, and holds exact set
equality against a registry of all twelve write sites with each one's value.

test_body_limit.py's release guard used `any(... ast.walk(fn))`, which
accepts a `release_read_connection` that runs AFTER the await it exists to
precede — satisfying the letter of the rule while pinning a connection for
exactly the transfer it was supposed to free. Position is checked now, and a
late release is its own, louder failure. The registry keyed sites by file
basename while scanning recursively, and `admin.py`, `discover.py` and
`playlists.py` each exist both at the top level and under `plex/` — the same
entry-collapse the registry was widened to close. Keyed by relative path now.
And all three guards recognised a body read only by the `capped_` naming
convention, so a route spelling `await request.body()` inline declared no
ceiling, released no connection, and was invisible to every one of them.

`release_read_connection`'s precondition claimed the caller "must have
NOTHING PENDING, and that is checked rather than described" while checking
only UNFLUSHED ORM work. Measured: after `add`+`flush` all three collections
are empty and the rollback loses the row, and Core DML never enters them at
all — both of them this codebase's canonical write shapes. It now also asks
the transaction directly via `pg_current_xact_id_if_assigned()`.

`errors.tooLarge` was reworded to "try a smaller file", but that key answers
every route's body cap: the issue composer's over-size JSON draft got advice
about a file it does not have. Back to payload-neutral; the file-specific
sentence keeps its own key.

The upload dialog's new danger border outranked `dragOver`, so a user
replacing an over-cap file got no sign the drop zone was live — and with a
file picked the label shows its name rather than "drop it here", so the
border was the only feedback left.

Four further findings were refuted 3/3 and are not fixed: `aria-invalid` on
the drop zone (measured in Chromium — the AX node does carry it on a native
button), a bidi-override spoof through `display_text` (the served filename
already strips those categories, and the proposed filter would have broken
every ZWJ emoji), and `capped_body` not draining (the screenshot route's edge
ceiling equals its own cap, so that refusal is unreachable, and the log sink
is sent by beacon with nobody reading the answer).
2026-09-23 01:20:22 +02:00
peter 7c0647f87a fix(uploads): finish the sweeps round 1 left half-done
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.

* `display_text` reached three of the FIVE writers of a user-supplied name.
  `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
  unbounded string, so a 300-character name pasted into the Add-download dialog
  or the editor was a bare 500 (main.py re-raises every DataError but the NUL
  case). Four of the seven reviewers found it independently. The bound now lives
  in `models.py`, the three columns are DECLARED with it, and the guard for the
  next writer is parsed rather than remembered: every `display_name` /
  `display_uploader` assignment under `app/routes/` must go through the helper.
  That guard immediately found two more -- the Google sign-in callback writes
  the profile name into the same 255-wide column -- so it is six, not five.

* The connection release reached one of the THREE routes that read their own
  body. The screenshot route (10 MiB, from a phone) and the client log sink both
  still held a pooled connection idle-in-transaction for the length of the
  upload, which is the invariant db.py sizes the pool on. All three now call
  `db.release_read_connection`, whose precondition -- nothing pending on the
  shared session -- is CHECKED rather than described, because a whole-session
  rollback discarding a caller's staged write is a mistake this repo has already
  made once. `test_body_limit.py` holds the rule for the next such route.

* The AST drift registry keyed its sites by (module, limit expression), so a
  second route copied from the first collapsed onto the same entry and went
  uncovered -- the exact hole the registry exists to close. Keyed per call site
  now, and it carries the release requirement too.

* The dialog's own refusal was announced once and never again: two over-cap
  picks leave an identical node with identical text, and an untouched live
  region says nothing. It is keyed per pick now, the drop zone reads as invalid
  and points at the sentence, and the reason is reachable from the control
  rather than only from a one-shot announcement.

* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
  test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
  is the mutation round 1 fixed on the server side), and the header-precedence
  test only covered an exact-case duplicate while a differently-cased one would
  have been COMBINED onto the wire rather than replaced. Both fixed, and the
  merge now dedupes case-insensitively because header names are.

Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.

Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
2026-09-22 23:31:55 +02:00
peter efd7b217ca fix(uploads): make both caps answerable, and stop holding a connection
The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.

* The edge ceiling and the route's own cap are the SAME number at the default
  configuration, so the edge always answers first: a declared Content-Length
  over the cap is refused before a byte is read, and an undeclared body trips
  the middleware on the same chunk the route would have tripped on. The route's
  numbered 422 -- and the drain written for it last week -- are reachable only
  once an admin LOWERS the knob, which is the one shape the browser UAT tested.
  Measured: headroom does not fix this, because the declared-length check fires
  first for any body big enough to matter and the drain pushes the count back
  over the edge for the rest. So the edge's own refusal is made readable
  instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
  was already localized, and the dialog refuses an over-cap file BEFORE sending
  it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
  Measured in Chrome against the API directly: that 413 arrives as `onload`
  with a readable body in 43 ms.

* Reading the body behind the auth dependency put the whole transfer inside the
  request's database session -- the auth SELECT autobegins a transaction, so one
  of the twenty pooled connections sat `idle in transaction`, with nothing to
  time it out, for as long as the client cared to keep sending. That is the
  invariant db.py sizes the pool on and plex/playback.py records an incident
  for. The read transaction now ends before the await. Measured on the dev
  server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
  move, only afterwards, when the handler actually writes.

* The display name moved into a request header, where nothing bounds its length,
  while the column stayed varchar(255) -- so a long enough name was a DataError
  500 after the whole file had already transferred. One `display_text` helper
  now serves all three writers (the metadata PATCH already truncated, the upload
  ingest did not), and the dialog stops at the same number.

* The gate invariant asked `route.dependant.body_params`, which holds only the
  route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
  upload route's own idiom -- leaves it empty while FastAPI still reads the body
  before auth. It asks `route.body_field` now, which is what FastAPI itself
  consults. The AST registry of body-reading call sites had also never followed
  `capped_stream_to`, so the one route the whole invariant exists for was the
  single site the drift guard could not see.

Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
2026-09-22 22:41:22 +02:00
peter 3d7ef9e945 fix(uploads): drain an over-cap body before refusing it
Found in the browser UAT, not by the gate. Moving the cap onto the arriving
bytes meant it fires while the client is still sending, and answering + closing
mid-upload is what XHR reports as a network failure: the upload dialog said
"the server couldn't be reached" while the access log showed a correct 422
carrying the limit. A refusal the client cannot read is an outage as far as the
user is concerned.

`capped_stream_to` now reads the remainder to nowhere before raising, so the
client finishes sending and reads the real answer -- re-measured in the
browser, which now names the limit. Nothing past the cap is written, and the
request is still bounded by bodylimit's ceiling, so the cost is bandwidth an
AUTHENTICATED caller was sending anyway. The anonymous path is untouched:
auth still resolves before a byte is read.

The regression test sits on the helper, not the route, and the reason is worth
keeping: the route-level version was written first and PASSED with the drain
deleted. httpx's ASGI transport pumps the request body whether or not the app
reads it, so "drained" and "abandoned" are indistinguishable through
TestClient. Its own mutation check is what exposed it.
2026-09-22 02:11:21 +02:00
peter 7fce7a7ce5 chore(deps): drop python-multipart -- nothing parses a multipart body now
Its own entry in requirements.txt said the download center's local-file upload
was "the only multipart endpoint in the app". Since that route reads a raw
body, the claim is false and the package is dead weight: no route declares
File/Form, and nothing calls `request.form()`.

Removed from requirements.txt and from its single pinned line in
requirements.lock. The lock is generated and normally regenerated wholesale,
but a full regeneration would bump every transitive dependency, which has no
business riding inside a security fix -- the next legitimate regeneration
reproduces exactly this state, since the package is gone from the source.

The `multipart` entry in logs.py's third-party level map goes with it: a level
for a logger that can no longer exist.
2026-09-22 01:59:09 +02:00
peter f5018bdcbc fix(uploads): read the body after authenticating, not before it
`POST /api/downloads/upload` declared `file: UploadFile = File(...)`, and
FastAPI parses and spools a declared body before the first dependency runs --
`require_human` included. An unauthenticated caller could therefore make the
process write the whole per-file allowance (500 MiB by default) to temp, once
per request. SEC-20's ceiling bounded that number but could not stop the
writing, which is why E27 parked it rather than closing it.

The route now takes no body parameter. `spool_upload` names `require_human`
first and only then streams `request.stream()` to the staging temp file, so a
stranger is refused before a byte moves -- the shape the screenshot route has
used since E16. Filename and display name travel as percent-encoded headers
(latin-1 on the wire, so a Hungarian filename needs the encoding) instead of
multipart parts.

`capped_stream_to` joins `capped_body` in routes/_common.py: the same counting
rule, written to a file descriptor rather than accumulated, because holding
500 MiB would trade a disk problem for a memory one. Its refusal is the
caller's, so the upload route keeps answering 422 `upload_too_large` with the
admin knob's own number.

With no envelope to allow for, the edge is now exactly that knob, and
MULTIPART_ENVELOPE_ALLOWANCE is gone.

The invariant stops being prose: test_body_limit.py asks FastAPI itself
(`route.dependant.body_params`) and fails the gate if any route with a raised
ceiling declares a body. A comment could not have prevented the defect it
described; that test would have.

Both new tests were seen red for the right reason -- before the fix the
anonymous probe got its 401 only after the server had read 4236 bytes of a
multipart body, and re-declaring the File parameter turns the invariant test
red naming the route and the parameter.
2026-09-22 01:59:03 +02:00
peter b82c22b888 release: 0.94.0 (improvement/e27-contract-hardening)
gate / frontend (push) Successful in 4m32s
gate / backend (push) Successful in 6m8s
2026-09-22 00:34:12 +02:00
peter f839a461e6 chore(release): 0.94.0 — framework & contract hardening 2026-09-22 00:34:06 +02:00
peter 8525666913 test: round 4 — an assertion that could not fail, and a comment that did not add up
Two findings, both in the review's own work; nothing in the product code. That is
the shift that ends the loop: 22 → 13 → 6 → 2, and the last two are a test and a
comment rather than a defect.

The "an override that is fine is still honoured" leg of the bounds-fallback test
stored `spec.max` — which, for this key, IS the env default, because the spec's
max is derived from it. So the assertion was satisfied by the fallback it existed
to rule out: mutating `_coerce` to reject EVERY int override left all three legs
green. It stores `spec.max - 1` now and asserts the value differs from the
default, so the leg can actually fail.

And the shrink test's comment claimed `3 - 5 = -2` for an expression that computes
`chips.length - visible` = `2 - 3 = -1` — naming neither operand nor the result,
in a comment that was rewritten one round earlier precisely for miscounting.
2026-09-22 00:22:30 +02:00
peter 3f24248e01 fix: round 3 — a vacuous ordering test, a clamp that was a silent rewrite, a regex that missed
Round 3 over round 2's fixes. The trend is 22 → 13 → 6, and the character has
shifted: these are mostly defects in the REVIEW's own work, not in the epic.

The ordering test I added to hold the middleware's position could not fail for
either arrangement it names. All four `@app.middleware("http")` layers — the
access log, the two header decorators AND the setup gate that must be inside —
register as the same `BaseHTTPMiddleware`, so asserting on the class passes for
the pre-move stack and for a cap moved back outside CORS. It names the layers by
their dispatch function now, which is the only thing that distinguishes them.

The `sysconfig` bounds check CLAMPED, and clamping is a silent rewrite — the thing
this module refuses to do everywhere else, where an unusable override falls back
to the env default and logs once. On the max side the difference is cosmetic; on
the min side it is a live bug. `download_upload_max_bytes` had `min=0` before E7,
and a stored 0 means "no per-file cap" to the route that reads it. Clamped to 1 it
becomes a ONE-BYTE cap: uploads dead, nothing in the log, the admin page showing
"1". It raises `OutOfRange` now and takes the existing fallback, with its own
event code.

The `capped_body` sweep grepped for an identifier as the limit argument, so a site
written `capped_body(request, 4 * 1024 * 1024)` — or with the parameter not named
`request` — was MISSED, and a miss reads as "no new sites". Parsed with `ast`
instead: every call is seen whatever it looks like, and an unfamiliar argument
shape fails the equality instead of vanishing.

Frontend: the shrink test drove the mock in a way the real hook cannot produce
(it moved `visible` in the same step as the shrink, and the comment miscounted the
tags), so it reproduced the right END STATE by accident rather than the sequence.
`visible` is now left alone across the shrink, which IS the precondition.

Ops: the baseline guard answered "did this run rewrite it?" and therefore went
green on the second run over an already-tightened, uncommitted baseline — the
exact state its own comment says marks the localdev stack stale ever after. It
asks both questions now. And `.dockerignore` joins BAKED_PATHS: it decides what
the COPY lines actually take, so it is an input to the build context exactly as
the copied trees are.
2026-09-22 00:12:30 +02:00
peter 6b86673af4 chore: untrack tsconfig.tsbuildinfo, and hold the CI baseline the way the local gate does
`frontend/tsconfig.tsbuildinfo` was swept into the previous commit by a `git add
-A`. It is a machine-local incremental artifact carrying absolute Windows paths,
no lane reads or regenerates it, and it was already stale in the commit that added
it — it does not list the contract test that same commit created. Untracked, and
`*.tsbuildinfo` ignored, because the day anything does start rewriting it a green
gate would leave a tracked file dirty: the exact hazard the basedpyright baseline
just got a guard for.

And CI ran bare basedpyright while the local lane gained that guard, so a
burn-down would have been green on the server and red on the developer's machine
over the same tree. Two gates, one tree, one answer.
2026-09-21 23:59:59 +02:00
peter d924c665f8 fix(frontend): round 2 — the popover latch reset read a stale, negative count
The reset I added guards on `hidden === 0`, and `hidden` can be NEGATIVE.
`useChipOverflowFit` keeps `visible` in state and re-decides in a layout effect,
so the render in which the chip set SHRINKS still carries the previous
measurement — delete three tags under an open "+N" and `chips.length - visible`
goes to -2, the equality never matches, and the flag survives the one window the
reset exists to close. The next re-fit brings the menu back on its own, which is
the original defect with an extra step. Clamped at all three lanes, with a test
that drives the shrink through the stale value.

Also: the Scheduler characterization banner said "the four tests below" and now
sits above six, the last two of which are genuine regression guards for the
range-refusal fix. A future mutation check would have read it, seen them go red,
and concluded the harness was broken rather than that the guard worked.
2026-09-21 23:59:59 +02:00
peter aa51644f5f fix(security): round 2 — the SEC-20 fix had four defects of its own
A second review round over the first round's fixes, which is where the remaining
real bugs usually are. All four are the same failure as the round it reviewed: a
claim stated in a comment that the code does not hold.

The middleware moved inside CORS, but the setup gate is registered LATER and is
therefore OUTSIDE it — so the comment's "the setup gate never runs for a body that
is already too large" was false the moment it was written. On an unconfigured
instance that gate opens a DB session per request, which is exactly the anonymous
pre-auth work SEC-20 exists to remove. The cap now sits between them, and the
order is asserted against `app.user_middleware` rather than described: prose about
a stack that `add_middleware` builds inside-out is how this got backwards.

The edge ceiling and the route's own cap were pinned EQUAL, but they measure
different things — the edge counts raw body bytes, the route counts the decoded
file. A file of exactly the advertised maximum was refused one layer before the
route that would have taken it. The edge now allows for the multipart envelope,
and the test asserts the relation and the gap instead of equality.

`sysconfig`'s new `max` was enforced on write only. That bound did not exist
before this epic, so a value that was legal when it was written can be sitting in
the database — served unclamped it produces exactly the drift the comment says is
impossible, with the admin page showing a number outside its own range. Int
overrides are clamped on read now, which also survives an env default being
lowered later.

And `DOWNLOAD_UPLOAD_MAX_BYTES=0` meant "no per-file cap" to the route but a hard
zero to the edge: every upload refused, including the admin page's own way back,
recoverable only by editing env and restarting. The setting takes `ge=1`.

Two test gaps from the same round: the header assertion that is the ONLY check of
the middleware's position was DB-gated and skipped on a host without Postgres, and
the headroom "ratio" the docblock calls load-bearing was held as `<=`, i.e. no
ratio at all. Both fixed, plus a sweep that every `capped_body` call site in the
routers is known and fits under its ceiling — the point of that list is the third
entry, not the two that exist today.
2026-09-21 23:59:45 +02:00
peter 378b7559e4 build(frontend): finish the React 19 move, and hold the versions that were left in prose
react-i18next was still on ^14.1.2. The E27 S1 plan named it as "the one peer that
actually needs moving — React 19 support is the 15 line", and then it did not
move. The 14 line's <Trans> reads `element.ref`, removed in React 19, and its
withTranslation types are built on the GLOBAL JSX namespace @types/react 19 no
longer declares — both latent only because this app imports neither. 15.7.4 fixes
both at the source (`c.props.ref ?? c.ref`, `React.JSX.LibraryManagedAttributes`).

`reactEngine.contract.test.ts` joins its Node and Python twins and holds what was
only a comment: the compiler's `target` against the installed React major, react
and react-dom together, the types against the runtime, and react-i18next's floor.
A stale target is not a corrupt bundle — the plugin validates the option — but it
is a silent one, since tsc never sees Babel's output.

pythonEngine gains the seventh spelling nobody held: `requirements.lock`'s own
regeneration recipe names a python image, and wheels resolve against THAT
interpreter, so a recipe left behind yields a lock that does not match the image
installing it.

The type lane now covers `alembic` too, for the reason the ruff lane was widened
to it — migrations run on every container start, so one that raises crash-loops
prod. It came back at zero new errors, so the baseline is untouched. `tests/` is
deliberately still out, and the config says why.

vitest.config's docblock claimed the React transform is "a no-op for the plain .ts
node tests". It is not: the compiler rewrites src/lib's hooks in that lane too,
which is the point of sharing one factory with the build — and worth knowing
before ruling the compiler out of a divergence.

Tests: useCardPager had none, in either the hook or its two callers, though its
page clamp moved to render in S2 — the transformation where a non-convergent
condition is a crash rather than a glitch. Four cases now, including termination
on an empty list. Combobox's and Avatar's close/failed-URL cases are relabelled
CHARACTERIZATION: both pass against the pre-sweep code, and the tests beside them
are the ones that discriminate.
2026-09-21 23:33:02 +02:00
peter 05121fbc99 fix(security): SEC-20's exemptions become raised ceilings, and the 413 is dressed
The module admitted its two opt-outs under a stated rule — "both enforce their own
limit on the bytes they actually read, which is what makes opting them out safe" —
and that rule is false for one of them. `POST /api/downloads/upload` declares
`file: UploadFile = File(...)`, so FastAPI parses and spools the whole multipart
part to a temp file BEFORE the first dependency runs, `require_human` included:
its in-handler cap measures bytes that have already landed. Measured on the
installed fastapi 0.141.1 / starlette 1.6.0 — starlette's max_part_size guards
non-file parts only, so an anonymous chunked upload was bounded by nothing. In the
shipped self-host compose that lands in a RAM-backed tmpfs.

The exposure predates SEC-20 (no route was capped before it); what this branch
added was the claim. So the list becomes a mapping of per-route CEILINGS instead
of a set of exemptions: every request now gets a finite number, the upload route's
being the same one the route itself enforces. `sysconfig`'s knob is bounded by it
so an admin cannot raise the handler's cap past the edge and start collecting 413s
from a layer the admin page never mentions — held by a test, not by prose.

Two more things the review caught in the same module:

The 413 was not byte-identical to `capped_body`'s after all. `json.dumps` puts
spaces where JSONResponse does not, and the test that promised to hold them
together compared PARSED JSON, so it could never have seen it. Built through the
same JSONResponse now, and compared as raw bytes.

And the middleware sat outside CORS and the security headers, so its 413 carried
neither — on the split-origin dev setup the browser drops it and the user gets a
generic network failure instead of the translated too_large, while the identical
refusal from capped_body one layer down renders fine. It moves to just inside
CORS: still ahead of the session, the setup gate and every auth dependency.

Also corrected: the docblock's "largest non-exempt body is 64 KiB, ~16x headroom"
(it is 512 KiB and 2x — the diagnostics sink, named eleven lines further down),
and CLAUDE.md's Request SIZE row, which forbade the one correct use of
capped_body that is actually in the tree.
2026-09-21 23:32:49 +02:00
peter bf518223e5 fix(frontend): the scheduler's two number editors stop swallowing what you typed
Both refused an out-of-range value by discarding it: the entry vanished, the box
snapped back to the server's number, and nothing said whether the save had
happened, been rejected, or been ignored. That is the house rule about honest
states, applied to a write instead of a read. They now keep the draft, mark the
field aria-invalid and name the range (HU+EN), and the Save control is disabled
rather than silently inert.

The interval editor also dropped a keyboard user's focus on every exit — Escape,
Cancel and Save all unmount the focused input — so it hands focus back to the
pencil that opened it, the pattern IssueComposer already uses.

Tests: the four interval-editor cases added in S2 are relabelled CHARACTERIZATION,
because they pass against the pre-sweep code too (the old effect was already
guarded by `editing`). The editor that genuinely lost typing to a poll is the
maintenance batch box, whose old effect had no guard at all, and it had no such
test — it does now, anchored on a value that follows the server so it cannot pass
on a page that simply has not polled yet.
2026-09-21 23:32:32 +02:00
peter 332d24a646 fix(frontend): four state defects the deep-review found in the E27 S2 sweep
Every one of them is the same shape: a value derived correctly, and a WRITE to
the thing it derives from left unattended. Deriving hides a stale value; it does
not consume it.

PlayerModal — the watch toggle inherited the previous video's auto-watch. Moving
the queue-advance sync into render was right for `currentVideoId`, but the same
block also synced `status`, and that one had an ordering constraint of its own:
useYtProgress's cleanup for the OUTGOING item can still call setWatched(true)
near the end of a clip, and cleanups run after the commit. As an effect the sync
landed last and corrected it; from render it lands first and the stray write
wins. Press Next in the final ten seconds and the NEXT video showed as watched —
one click on the toggle then sent onState(next, "new") for an item already new.
Fixed by keying the mirror to the id it describes, so a write for another item is
ignored rather than displayed, and no ordering has to be argued about at all.

ChannelsProvider — a focus-channel intent was masked by the page rather than
consumed, and nothing ever cleared it. The manager is a page chunk: it unmounts
on every navigation, and its seed effect runs again on each mount, so returning
to it re-applied a channel name from earlier in the session over whatever the
user had typed since. The consumed marker now lives in the provider, keyed on the
token — which also closes the case the old clear-on-leave effect never handled,
opening a channel page (that does not change `page` at all).

TagFilterChips + ChannelTags x2 — `open = openRaw && hidden > 0` hides the
overflow popover when the lane stops overflowing, but floating-ui only reports a
close for a real event on a mounted surface, so the raw flag stayed true and the
menu came back by itself on the next re-fit — a window resize or the text-size
slider is enough, with nobody near it. The flag is now reset in render against
the same condition that hides it.

Avatar + downloadRow — remembering a failed image BY URL fixed the recycled-row
frame but never forgot it, so a transient 429 pinned a channel to its initial for
the life of the instance, never re-requesting. The memory is now dropped once
`src` moves off it, which keeps both properties.

And playerModalHooks' last exhaustive-deps suppression claimed "nothing in the
body reads a render value" while reading five. The invariant that is actually
load-bearing is that each of them is a per-item snapshot — written down instead.

All four fixes are mutation-checked: each new test was seen red, against the
fixed line only, naming the real defect.
2026-09-21 23:32:22 +02:00
peter c5e022f056 refactor(frontend): the long tail — and eslint-suppressions.json is gone (E27 S2)
Closes the burn-down. Hook suppressions 113 -> 34, `exhaustive-deps` 45 -> 2 (the target was
<10), and every one of the 34 survivors is an inline directive NAMING the invariant it
protects. The suppressions file is deleted, because a bulk file hides the next violation
instead of answering it — and it cannot hold a warning-level rule at all.

The sprint plan's three verdicts missed the answer that turned out to be the most common one:
adjust the state DURING RENDER. React re-runs the component and discards that pass rather than
committing it, so the stale frame an effect-based reset paints does not exist. Applied to the
Plex player's up-next card, the playlist view reset, the feed's optimistic overrides, the
channel banner's retry count, the download center's tab guard, the Plex search highlight, the
comfort-pin latch, Welcome's per-view outcome and the card pager's page clamp. It is legal only
for a component's own state — PlexBrowse's language commit resets PlexProvider's filters, so
that one stays an effect and says why.

Mirrors of a prop or query became derived values or `draft ?? server`:
- `downloadRow` had ui/Avatar's defect exactly — a boolean "poster failed" that an effect reset,
  in rows recycled down a virtualized list, so one job's missing poster painted over the next
  one's for a frame. Keyed by URL now.
- Scheduler's two number editors held a mirror an effect re-synced, on a page that POLLS — the
  shape a poll can overwrite mid-edit. A draft while editing, the server's value otherwise; six
  tests where there were none, the one behavioural change (the save button stops being dirty at
  commit rather than at the echo) mutation-checked.
- SetupWizard's "no token" and WatchPage's "gone" are facts about the URL, so they are seeded and
  derived rather than pushed into state by an effect that had to commit first.

Elsewhere: PlexInfo's rating chip was a component declared during render — a new component type
every render, so React remounted the subtree and dropped the focus on its filter button. Hoisted.
Feed's undo action and PlexPlayer's session reclaim both re-entered their own useCallback, which
the checker reads as a forward reference; both go through the ref idiom this file already uses.
BootGate's dep list was load-bearing for correctness — a re-run meant `location.reload()` — so
the "already adopted THIS identity" comparison now enforces it in code, with a test that fails
without it (the existing reload tests all install their spy too late to catch it).

One correction to the batch before this: keying ChatThread's and Messages' decrypt passes on
`q.data` instead of `q.dataUpdatedAt` was wrong. react-query's structural sharing makes it safe
in production, but the component must not melt down when something hands back a fresh object per
render — and Messages.keyboard.dom.test does exactly that, so the effect span the suite. Both are
back on the fetch as the trigger with the payload behind a ref, which is honest AND loop-proof;
the replaced comment had warned about precisely this.

CLAUDE.md gains four canonical-patterns rows for the above, and its "React 18" line is now true.
2026-09-21 01:44:21 +02:00
peter 4065ac7e6a refactor(frontend): the two players — one real ordering bug avoided, five suppressions gone (E27 S2)
The interesting part of this batch is a change that was WRONG and the test suite said so.

PlayerModal synced its local watch-status mirror and `currentVideoId` from an effect on
`active.id`. Moving that into render is right — as an effect there was a frame per queue advance
showing the new episode's chrome over the previous item's watch state, with `navigated` still
true so the "you have navigated away" affordance flashed. But the same effect also advanced
`currentIdRef`, and that line must NOT move: useYtProgress's per-item cleanup persists the
OUTGOING item's position and guards it on `currentIdRef.current === id`. React runs every
cleanup before any setup, so in an effect the ref still names the item being left. Advanced to
render time the flip happens first, the checkpoint is skipped, and stepping next→prev back to an
item resumes from the frozen launch queue's stale position. "resumes an item played earlier in
the SAME session" caught it immediately (748 instead of 830); the ref write stays in its own
effect with the ordering written down.

The rest:
- PlayerModal's keyboard handler is a useEffectEvent, so the "bound once, reads live state"
  contract is enforced instead of asserted. `focusModal` is a useCallback now — it reads a ref
  and nothing else, and three hooks take it as a parameter where a fresh identity would spread.
- PlexPlayer's session reclaim called `loadSession` from inside loadSession's own useCallback,
  which the checker reads as a forward reference. Routed through a ref, the file's existing
  answer for exactly this (detachSessionRef, cancelAutoSkipRef).
- PlexPlayer's item-load effect excluded `prefs` by hand so a track pick from the player's own
  menu would not double-load the session. The two language prefs go through a ref instead —
  "latest value, not a trigger", said in code rather than in a comment.
- PlexPlayer closes its up-next card during the render that changes `id`, not a commit later.
- The one PlexPlayer directive that stays is the live session counter, whose cleanup must bump
  the CURRENT value: the rule's advice to copy it into a variable is precisely wrong there, and
  the comment now says why rather than pointing at another comment.

Suppressions 75 -> 70; exhaustive-deps 20 -> 17.
2026-09-21 00:27:15 +02:00
peter ce08fc2aec refactor(frontend): a stable useHistorySubview, and PlexBrowse can stop disabling the check (E27 S2)
useHistorySubview returned a fresh object with a fresh `open` on every render. Correct, but it
made "re-run when the sub-view moves" unwriteable: PlexBrowse had three effects that wanted
exactly that and had to disable the dependency check to get it, and a fourth that listed `sub`
honestly and therefore re-ran on EVERY parent render, saved only by a ref guard.

Fixed at the source, which is where the house rule says a principle belongs: `open` and `back`
are useCallbacks and the returned object is memoised. `url` is a module constant at every call
site (PLEX_SUB_URL, MODE_URL) and the body otherwise touches only module functions, so there is
nothing left to churn on. All three PlexBrowse dep lists are now the honest ones, and the
comment that claimed `open` was "behaviourally stable" is replaced by it actually being stable.

PlexBrowse's remaining four are escape hatches, and two of them are now ONE:
- `infoScrollRef` is written from four places, and the compiler objects to a ref being modified
  outside the effect that reads it. That objection is a conservative false positive — a scroll
  memory needs exactly that arrangement — but four bare mutations were worth concentrating
  anyway. One `rememberInfoScroll` writer now holds the "stash on the way out, reset on a fresh
  page" rule, and answers the checker once instead of three times.
- The language-commit effect cannot move into render: it resets PlexProvider's filters, and a
  render-phase setState may only touch the component's own state. Said so at the site.

Suppressions 78 -> 75; exhaustive-deps 23 -> 20.
2026-09-21 00:20:05 +02:00
peter ea6d829c29 refactor(frontend): the context providers adjust state in render, not after it (E27 S2)
Eleven suppressions across the five providers. The recurring shape here was "an effect that
undoes something when you navigate away", and every one of them ran a commit LATE — the new
page painted once while the old module's state was still live.

Adjusted during render instead (React discards that pass rather than committing it, so there
is no in-between frame):
- SavedViewsProvider reset the sticky view and closed its modal from an effect on `source`, so
  the first Plex render carried the feed's sticky id — long enough for a stale view chip. It
  also followed the active view into stickyId from a second effect; that one converges by
  construction, since preferSticky returns the sticky match when there is one.
- NavigationProvider ended a live YouTube search from an effect on `page`. Keyed on the page
  CHANGE rather than derived from it, because "ended" has to be permanent — deriving would have
  quietly brought the search back on return to the feed.
- ChannelsProvider's manager focus is now simply DERIVED from the page: it means nothing off
  the Channels page, and focusChannel sets it and navigates in one handler.

Honest deps by freezing what is genuinely mount-time:
- FeedFiltersProvider persisted a share link's filters with an empty dep list; listing `filters`
  honestly would have re-persisted on every edit and spun. The mount value is frozen into its
  own state, so all three deps are stable and the effect says what it does.
- ChannelsProvider's URL-seed layout effect and NavigationProvider's initial history stamp get
  the same treatment. The three slice setters they call were already stable useCallbacks
  (useFilterState / useSortState / useAccountPersistedState all return one).
- SavedViewsProvider reads `cfg` through a ref: it re-memoizes on every filter edit, and the
  per-source seed guard already enforces once. That also made the set-state finding go away —
  with the render value gone, the checker can see the seed for what it is.

WizardProvider keeps its one escape hatch, now inline with the invariant: auto-opening is a
one-way decision taken from server data as it arrives, and deriving it would reopen a wizard
the user had just closed.

Suppressions 89 -> 78; exhaustive-deps 27 -> 23.
2026-09-21 00:12:44 +02:00
peter 6a7ff330f6 refactor(frontend): the shared hooks in lib/ stop lying about their deps (E27 S2)
Ten more suppressions, and React 19 turns out to supply the missing primitive.

useEffectEvent (stable since React 19.2, which S1 brought in) is the canonical answer to the
"bind the listener once, but read the latest values" pattern these hooks were disabling the
checker for. Three of playerModalHooks' four escape hatches become honest dep lists: the
wheel-volume handler, the native-controls handover, and the fullscreen-change sync. The
contract is no longer prose — the compiler enforces it.

The fourth stays, deliberately, and now says why at the site: the YouTube player's lifecycle
effect must run once per video (re-running destroys and rebuilds the iframe), and YT.Player
takes its callbacks as an object literal, which an effect event may not be passed into.

Fixed, not suppressed:
- useDismiss captured onClose when the popover opened, so a caller that re-rendered with a
  different handler while it stayed open kept getting the OLD one called. The disabled dep list
  asserted the opposite ("read fresh from this run's closure"). Both the handler and the ref
  list are read through refs now; three tests, the two that matter mutation-checked.
- usePref wrapped its seed in a useCallback whose dep list had to be excused. useState calls a
  lazy initializer exactly once, so the memo was saving nothing; inlined.
- useCardPager clamped an out-of-range page from an effect. Adjusted during render instead —
  React discards that pass rather than committing it, so no out-of-range page is ever painted.
- plexPlayerHooks' auto-skip countdown reads its marker from the ref that already existed, so
  the dep list names its real inputs. That also made the set-state-in-effect finding go away:
  the progress is genuinely timer-driven, and the checker can now see that.
- useUrlMirror and useChipOverflowFit only needed their stable refs listed.

Two test-harness suppressions (a render-phase probe ref, a module-level setState handle) moved
inline with the reason they exist: both exist to observe a hook from outside without changing
what is being measured.

Suppressions 99 -> 89; exhaustive-deps 35 -> 27.
2026-09-21 00:07:12 +02:00
peter 7e1aeeb9fe refactor(frontend): honest effect deps in the app shell and the ui layer (E27 S2)
Burns the first 14 of the 113 hook suppressions. Three verdicts, applied per site:

A useEffect that should not exist (6):
- Avatar remembered "the image failed" as a boolean an effect reset on src change. The
  component is recycled across items in the virtualized feed, so after one avatar 429s the
  next item's slot committed the fallback initial over a perfectly good image and only the
  post-paint effect cleared it. The failure is now keyed BY URL; derived state cannot be stale.
- Combobox cleared the typed query from an effect on `open`. Moved into setOpenAnd, the single
  funnel every close goes through. That also stops a spurious onQueryChange("") AT MOUNT, which
  told a controlled caller its own initial query had been cleared before the user touched it.
- CommandPalette reset query + picked row from an effect. Closed now means UNMOUNTED, not
  hidden, so a fresh open starts blank because mounting already means that -- and the ranking
  memo over every channel, playlist and saved view stops running while the palette is shut.
- Pager's page box held a copy of the page that an effect re-synced; it now derives it and
  keeps a draft only while being edited.
- App's ?link= / ?oauth= / ?admin= one-shots asserted "once per mount" with an empty dep list.
  The ref guard the codebase already uses in 22 places ENFORCES it, so the honest deps can
  include `t` -- which a language switch legitimately changes, and which would otherwise have
  re-fired the toast.

Honest deps that also fix something (5):
- App's module preload was keyed on the account id, so a permission change that alters the
  module set without moving the id never re-preloaded. `preload()` is idempotent, so depending
  on `me` costs nothing.
- App's access-request nudge now re-issues on a language switch; it used to stay in the old
  language until the pending count changed.
- App's diagnostics effect: applyDiagnostics already no-ops on an unchanged effective level,
  so picking four fields out by hand was guarding nothing and missed any field added later.
- DataTable's two filter effects: a ref holds the latest onFiltersChange so commitFilters can
  be stable, which is what lets the effects name it instead of switching the check off.

Genuine escape hatches (7) moved OUT of eslint-suppressions.json into inline directives that
name the invariant at the site -- the toast handover's render-phase global, the live region's
deliberate arming tick, the docked one-slot component, the popover engine's floating-ui list
state, dnd-kit's per-drag `moved` flag, BackToTop's DOM read and the palette's debounced
announcement. A reader of the code now sees why; a JSON file no one opens does not.

Tests: Avatar (3, new file) and Pager (3) pin the derived-state frames with a layout-phase
probe -- layout effects run before any passive effect, so they read the frame the browser would
paint. Combobox (2) pins the close-clears-query contract and the no-callback-at-mount rule.
All three sets mutation-checked against the effect-based originals.

Suppressions 113 -> 99; exhaustive-deps 45 -> 35.
2026-09-20 23:57:08 +02:00
peter 7d682c90d5 build(frontend): React 18 -> 19 with the React Compiler (E27 S1)
The compiler auto-memoises, which is the point: it infers what the app's
112 `useCallback` and 48 `useMemo` call sites were hand-written for and
applies the equivalent itself. It is on repo-wide with no opt-out list —
it bails on anything it cannot prove safe, and says so through the lint
rules. Verified in the built bundle rather than assumed: 369
`react.memo_cache_sentinel` markers across 35 of the 76 chunks.

ONE transform, shared by vite.config.ts and vitest.config.ts through
reactPlugin.ts. Not two `react()` calls: if the compiler ran only in the
production build, every vitest run would exercise code the user never
receives and a compiler-induced change would ship green — the same
gate-vs-image split 0.93.2 was released to fix. The plugin is imported
rather than named by the usual string, because a string is invisible to
knip and, worse, a typo in it is a silent no-op that ships an uncompiled
bundle, while a bad import is a tsc error.

eslint-plugin-react-hooks 5 -> 7 brings the whole compiler rule set: 283
findings, triaged rather than bulk-suppressed, because the two halves
needed different answers.

  react-hooks/refs (217) is OFF. 153 of those 217 are this app's popover
  architecture, not its history: 79 are @floating-ui/react's documented
  `pop.refs.setReference` / `getReferenceProps` callback-ref setters, and
  74 more are passing the popover handle down as a prop, which is what
  components/ui's popover engine is for. Suppressing them would not help
  the next popover anyone writes. The remaining ~64 are genuine
  render-phase `ref.current = x` writes and are real, but the rule cannot
  tell them apart here; they are E27 S2's work.

  The other 66 go into eslint-suppressions.json as a ratchet — new
  violations fail the gate, and `--prune-suppressions` makes a fixed one
  fail too until it is removed. 45 of them are `set-state-in-effect`,
  which IS E27 S2's subject, so that sprint now has a measured worklist.

react-i18next deliberately stays on 14. It never declared a React-19
peer — every version from 14 to 17 asks only for `react >= 16.8.0` — and
what the newer lines actually require is i18next 25/26 against our 23.
That is an i18n runtime migration wearing a React hat, and it buys this
sprint nothing.

Two React 19 type changes: `useRef<T>(null)` is now `RefObject<T | null>`
(the honest shape — the element is null until it mounts), so
PlexPlayerSettings' prop follows; and reactPlugin.ts joins
tsconfig.node.json's include.

The compiler skips memoising VirtualGrid and reports it: TanStack
Virtual's `useVirtualizer()` returns functions that cannot be memoised
without going stale. That is the right trade for a component whose cost
is its row window, and it is the one such site in the app, so it carries
a targeted disable with the reason — a second incompatible library
anywhere else must still stop the gate.

Nine lanes green, vitest 1558. Browser UAT on the dev server against the
still-running React 18 build as a control: boot, feed (virtualized grid,
popover menu, a filter change that moved the list 195,187 -> 64 items
with the URL mirrored, so no stale memoised UI), command palette, player
modal — no console errors on any of it.
2026-09-20 23:21:08 +02:00
peter cc9fc3b544 build: pin one Python version across the app, the images and the gate
0.93.2 shipped a week's worth of confusion about which Node this repo
runs on; the Python side had the identical shape and nobody had looked.
Prod (Dockerfile), the test image (Dockerfile.test) and CI (check.yml)
each spelled 3.12 as an independent literal, there was no
`.python-version`, no `requires-python`, and ruff had no target-version
so it judged our syntax against some other interpreter. The dev machine
had no 3.12 installed at all — only 3.13 and 3.14 — so every ad-hoc host
script ran two majors ahead of production.

`.python-version` is now the single source of truth, and the contract is
held by tests rather than by prose. It is split in two because no single
runner can see every link: the pytest suite runs inside the test image,
which mounts backend/ alone and cannot read the repo-root Dockerfile or
the workflow.

  test_python_engine_contract.py : running interpreter == pyrightconfig
                                   == ruff.toml
  pythonEngine.contract.test.ts  : .python-version == pyrightconfig ==
                                   ruff.toml == Dockerfile ==
                                   Dockerfile.test == check.yml

Together they close the loop, and the backend half carries the assertion
that pays — that the interpreter running the suite is the pinned one. A
green suite on another minor does not mean the shipped image is green;
it means they have drifted and nobody has noticed yet.
2026-09-20 23:02:11 +02:00
peter c9101e3450 ci: type-check the backend — the gate's ninth lane (E27 S3)
Three tsc projects held the frontend and NOTHING held the backend: ~36k
lines of `app/`, annotated in 113 of 128 files, that no lane had ever
read. Asymmetric by accident, not by decision.

basedpyright, in basic mode. Measured against mypy on this tree with the
dependencies actually installed — without them basedpyright errors on
every import and mypy's --ignore-missing-imports quietly turns them into
Any and UNDER-reports. Both land ~190-200 errors with the same
distribution, so the choice fell to what differs: basedpyright is ~2x
faster (4.4s vs 7.4s, and this runs on every `siftlode lint`), resolves
our `app.*` imports from its own config with no MYPYPATH, and has a
first-class `--writebaseline`. That last one is the ratchet itself;
mypy would have needed a third-party tool for it.

198 pre-existing errors are baselined, so the lane fails on NEW errors
only and the list burns down epic by epic. A green lane means no new
damage, not a clean codebase.

It runs in the pytest test image rather than on the host, and that is
forced rather than chosen: `requirements.lock` cannot be installed on
Windows at all — uvloop refuses to build there — so a host venv could
only ever be the lock minus a package, and a type checker without the
real third-party packages reports nonsense rather than nothing. The
image installs the pin from requirements-dev.txt, so the lane's version
IS the pin and there is no host copy to drift. The cost is that
`siftlode lint` now needs Docker, which it did not before.
2026-09-20 23:02:02 +02:00
peter 51daf3e388 feat(security): cap every request body before auth runs (SEC-20)
FastAPI reads a declared body IN FULL before the first dependency, so
`POST /auth/password-login` drained an arbitrary upload from an anonymous
caller and only then answered 401. 70 of the 72 body-declaring routes
behaved the same way: one unauthenticated client could spend the
process's memory and time at will.

A raw ASGI middleware, not `@app.middleware("http")` — BaseHTTPMiddleware
cannot wrap `receive`, so it could only trust `Content-Length`, which a
client may omit (chunked) or understate. This refuses an oversized
declaration outright and counts the bytes of everything else as they
arrive, the same rule `capped_body` already applies per-route. Registered
directly inside the access log: a refusal is still logged with its
request id, but the session, CORS, the setup gate and every auth
dependency never run for a body that is already too large.

Two routes opt out, under one stated rule: an exemption is ONLY for a
route whose own cap is larger than the global one. That is why
`/api/diagnostics/events` is absent — it caps itself at 512 KiB, below
the ceiling, so exempting it could only widen what it accepts. The list
names route templates and a test resolves each against the live route
table, so a rename fails the gate instead of silently dropping an
exemption in prod.

The 413 is byte-identical to `capped_body`'s and the test compares the
two against each other rather than against a literal — two shapes for one
fact is how the `cancelled`/`canceled` bug happened.
2026-09-20 22:51:32 +02:00
peter 09d351a0e7 docs(ci): the two gate jobs queue, they do not run in parallel
Declared side by side, so they read as parallel; the lab runs a single
act_runner with one slot, so the backend job starts at the exact second the
frontend one ends. Measured across runs #86-#91: ~4 + ~6 = ~10 minutes from the
push, not ~6.

Written down because believing the comment cost something real — the ops
release read-back budgeted the max instead of the sum and timed out on its
first live run, reporting a still-running gate as a failing one.
2026-09-20 22:05:56 +02:00
peter b5078003ce release: 0.93.2 (bug/ci-node-version-split)
gate / frontend (push) Successful in 3m51s
gate / backend (push) Successful in 6m7s
2026-09-20 21:49:56 +02:00
peter f2c32cfb8e chore(release): 0.93.2 — a Plex amount reads the same in every browser 2026-09-20 21:49:36 +02:00
peter 1453fa4a33 ci: run the gate on branch pushes only, not on release tags
A release pushes main and then the v<ver> tag, and a bare `on: [push]` treated
the tag as a second push of the identical commit — every release ran the whole
gate twice to the same verdict (runs #81-#90 are five such pairs).

It has to be `branches`, not the `tags-ignore` this obviously wants to be:
Gitea's matchPushEvent requires every named filter to match and each one bails
on the wrong ref kind, so `tags-ignore` rejects every BRANCH push and silently
disables the gate. Measured in go-gitea v1.27.3, where it differs from GitHub.

A trigger filter that stops firing is invisible, so the ops `release` now reads
the verdict back for the sha it pushed and says so when there is no run for it.
2026-09-20 19:47:19 +02:00
peter efe106dfda build: pin one Node major across the dev, CI and build machines
The server gate ran node:22 while the dev gate ran node:24, and the two disagree about
Intl.NumberFormat's compact currency padding and about which realm's Blob a Response hands
back — so the frontend job had been red for 73 of its last 90 runs while the local gate was
green, and nothing said so.

Align everything on 24, with `.nvmrc` as the single source of truth: the CI container, the
prod build stage, `engines`, and `@types/node` (which was two majors ahead of the runtime, so
tsc was green over APIs the shipped engine does not have). The backend job's node only runs
the checkout action, so it tracks the pin as hygiene — node 20 is EOL — but deliberately
stays outside the contract.

The pin is held by a contract test that reads every one of those places AND the engine
running it, so the next drift is a red lane on its first run instead of a CI log nobody reads.

Verified on node:24-bookworm (the CI lane: 228 files / 1552 tests, typecheck, lint,
format:check, knip all exit 0) and on node:24-alpine (the prod build stage, v24.21.0, SPA
built). Each assertion of the new test was mutation-checked red.
2026-09-20 18:06:24 +02:00
peter ec63056bd7 test(api): assert the export body's brand, not which realm built its Blob
`expect(x).toBeInstanceOf(Blob)` asks which realm constructed the body, not
whether there is one. Under jsdom the ambient `Blob` is jsdom's class, while
`Response.blob()` hands back node's own on Node 22 and jsdom's on Node 24 — so
both assertions passed locally and failed on the server gate, for a reason that
says nothing about the code under test. `.text()` is the mirror image: jsdom's
Blob does not implement it, so reading the bytes back is green on 22 and red on
24.

Both now assert the brand (`[object Blob]`, agreed by both realms) plus size and
type. Mutation-checked in node:22-bookworm: dropping `{ raw: true }` from
`debugExport` still turns the test red, so the repaired assertion still catches
the defect it was written for.

core.test.ts also stops building its Response from a jsdom `Blob`: on Node 22
undici does not recognise it as a body and stringifies it, so the fixture's body
was the 13 bytes of "[object Blob]" rather than the zip header. A string body is
what the sibling test already used.
2026-09-20 16:33:26 +02:00
peter 09becb3d00 fix(plex): pin the compact currency format instead of inheriting the engine's
`{ notation: "compact", maximumFractionDigits: 1 }` leaves the MINIMUM to the
engine, and engines disagree: the same options render "$160.0M" on Node 22 and
"$160M" on Node 24 (measured 2026-09-20 in node:22-bookworm, the image the
server gate runs, against the local Node 24.17). Browsers split along the same
line, so a viewer on an older engine saw the padded form. `minimumFractionDigits:
0` makes both render "$160M"; the plain `compact` helper (vote counts) was
measured too and already agrees on both.

This is why the server gate's frontend job was red on PlexInfo since 09-17 while
the local gate was green: the two run different Node versions.
2026-09-20 16:33:26 +02:00
peter 8d2ec5e867 release: 0.93.1 (bug/preview-crashes-the-tracker)
gate / frontend (push) Failing after 4m10s
gate / backend (push) Successful in 5m50s
2026-09-20 07:55:37 +02:00
peter 0e9839e73d chore(release): 0.93.1 — the report preview works again 2026-09-20 07:51:16 +02:00
peter 05ab963951 fix(issues): the preview must not ask the tracker for a repository context
The composer's preview did not fail a call — it killed the Gitea process, and
with it the git server and the container registry. `POST /api/v1/markdown` with
`{"Mode": "gfm", "Context": "<repo>"}` makes Gitea resolve every hex-shaped word
as a commit link, and on the API's render path the commit checker it asks is nil:

  panic: runtime error: invalid memory address or nil pointer dereference
  renderhelper.(*commitChecker).IsCommitIDExisting(0x0)
   <- markup.hashCurrentPatternProcessor (html_commit.go:217)

Every report carries hex — a tab id, a request id — so this was not an edge case
but the only case: the preview answered "the tracker isn't answering" every time,
because the tracker had just died.

Measured on throwaway instances, not argued. Same body, same call:
  gfm + Context + "5645b1c0ae01"  -> 1.26.4 CRASH, 1.27.1 CRASH, 1.27.3 200
  gfm + Context, no hex           -> 200
  gfm, no Context                 -> 200, and the HTML keeps its tables, task
                                     lists and chroma-highlighted code blocks
Upstream fixed it in 1.27.2 ("fix: avoid markup render panic", #38698), but a
self-hoster runs whatever version they run, so the payload is what gets fixed.

Sending was never affected: the created issue's own page renders the same body
fine (measured) — a real repo context has a real commit checker. The only loss is
that a hash or a `#123` is not yet a link in the preview; the issue page links it.

The test pins the payload dict, because the defect is in what we send.
2026-09-20 07:51:08 +02:00
peter 8142484347 release: 0.93.0 (feature/e16-debug-issue)
gate / frontend (push) Failing after 4m11s
gate / backend (push) Successful in 5m58s
2026-09-20 05:54:26 +02:00
peter 0d91b82f86 chore(release): 0.93.0 — report a fault from inside the app
E16: a recorded debug session becomes a drafted issue with its own logs attached, sent to the private Gitea tracker or downloaded as a zip. Includes the four epic-end review rounds and the Plex player's focus trap.
2026-09-20 05:54:03 +02:00
peter 90fc86d437 fix(diagnostics): deep-review round 4 — a claim only its owner may write, and a player that keeps the Tab
Round 4 was targeted at the surfaces rounds 2 and 3 changed. 29 findings raised,
26 survived deduplication, 12 survived three adversarial readings each.

`set_delivery` renewed the claim but never checked it was still ours, while its
sibling `release_send` always did. `expire_on_commit` is off, so the token it
re-stamped came from the caller's own memory, not the row: a send that outlived
its 600 s lease came back, overwrote the take-over send's committed links with a
full-column write, and put its own dead token back. It now takes the token, reads
the row's claim with a targeted scalar (not a refresh, which would discard the
`issue_url` assigned one line earlier) and raises `SendClaimLost` rather than
writing. An issue created by a send that had already lost its claim is logged with
its number — that record is the only one that will exist.

The tracker's settings are read above the pre-call checkpoint instead of inside
`_client`. Reading them there was a SELECT one line after the commit that was
meant to end the transaction, and held it — idle, snapshot pinned — across the
repository check and the create. The comment and the module docstring said
otherwise; measuring it in pg_stat_activity is what settled it. Two more comments
were saying things the code had stopped doing: the `**state` spread does not carry
the claim (`set_delivery` strips and re-stamps it), and react-query does not
debounce the preview — the editor being unmounted is what paces it.

The composer stopped stranding the focus on `<body>`: a send that creates the
issue and then fails on an attachment unmounts the button that was just pressed,
and only the success path handed the focus on. Export no longer saves into a
refusal it does not need — the zip is the way out of a stuck send, and its
pre-save was the one step that could not survive one. The delivery banner
announces through a role instead of an `aria-live` created with its own message.

`historyState` returns a union, so `chipTone` matching on the same keys is checked
by tsc rather than by hope.

PlexPlayer was the only full-screen overlay in the tree with no focus containment
at all — Tab walked out of an opaque player into a page nobody can see. It now
carries the shared trap. Closing that hole exposed the one underneath it:
`useDismiss` registered as a layer that traps the Tab while installing no trap,
so the player stood its own down for its own menus. Measured in a real browser:
30 Tabs with the settings panel open, 9 escapes into the nav rail, and 0 after.
Escape ordering is unchanged — the panel first, then the player.

Three tests that did not hold their line: the advisory lock's call site (deleting
it left the suite green), the trail assertion whose `all()` was vacuous over an
empty filter, and a `layerStack` assertion the stand-in dialog already forced.
All eleven executable fixes mutation-checked red for the right reason; the two
comment corrections have nothing to pin.
2026-09-20 04:30:42 +02:00
peter 2fe58073e4 fix(diagnostics): deep-review round 3 — a keyboard owner that is not a Tab trap
Round 2 made the step prompt a blocking layer so that `j` and the `g` chords
would not reach the page behind it. The dialog it sits in gave up its Tab trap
because of that, while the prompt's own replacement trap covered only its own
two controls — so Tab from anywhere else in the dialog (the bar's Screenshot or
Stop, the dialog's own buttons) walked out behind the scrim. `layerStack` now
separates the two questions it had folded into one flag: a layer can own the
keyboard and still leave the Tab trap to the dialog below it. The mirror case,
a dialog opening over the prompt, falls out of the same split.

A send's claim is now a lease with an owner. Every progress report re-stamps
it, so a send slower than the lease cannot lose it while it is still running,
and only the send that took a claim can give it back. The issue_create audit
row is staged inside the send and rides the same checkpoint as `issue_url`, so
the trail cannot be missing for an issue that exists. The attachment-name
prefix has one definition, shared by the producer and both readers.

A running send is reported to the UI. A reload or a second tab used to render
it exactly like a send that STOPPED, and offered a retry the server was certain
to refuse; the composer now says the report is being sent, holds the draft, and
puts both the retry and Create issue out. The past-sessions list says the same
for the same reason — it is the list a user opens to check on a send, and both
"draft" and "half sent" describe one that died. `/preview` and the retry path
got a pace of their own, and the per-user byte cap took the user-scoped lock
that makes it a cap instead of a guess.

Verified in a real browser on localdev: Tab cycles the whole dialog with the
prompt open, the prompt still swallows the page's keys, and the issue link is
focusable where the composer puts it. All 24 regression tests mutation-checked.
2026-09-20 02:43:56 +02:00
peter bc03e82687 fix(issues): a send keeps its claim through every step, not just up to the issue
Creating the issue rebuilt the delivery state from scratch and dropped the
claim with it, so for a new issue the claim covered only the creation — a
draft, a screenshot or a second send could change the report under the uploads
and the body edit that follow, which is exactly what the claim is for. The
state is now built on what was read, and a test walks the send's checkpoints
to see the claim still held at each one.

Found by mutation-checking the round-2 tests: removing the claim's release
left them green, because for a new issue there was nothing left to release.
2026-09-20 01:03:23 +02:00
peter 3f983e6d76 fix(diagnostics): deep-review round 2 — a send that claims, not one that blocks
The send no longer holds the session's row lock while it talks to the tracker:
it claims the session, gives the lock back, and checkpoints each step. A Stop
is never held behind a send, while a draft, a screenshot or a second send are
refused at once instead of queueing. The claim ages out, so a request that
died mid-send does not strand the session.

What the composer says is what happened: an attachment that went is no longer
named as one that did not (the missing body edit is its own line), and the
issue's screenshot count comes from the server when the list cannot be read,
so a failed read no longer blocks saving the draft. The zip is fetched as a
request, so its pace is a message and not a saved error page. Screenshots of a
finished report can be removed again; only one still being sent is held.

A recording's log elevation can no longer outlive it. The step prompt keeps
the keyboard (the page's keys stay quiet) and traps Tab itself. An image
pasted into a dialog opened over the composer stays that dialog's.
2026-09-20 00:51:12 +02:00
peter 134ec9a68e fix(diagnostics): deep-review round 1 — a send that cannot race, lose or repeat itself
The send, the draft and the screenshots take the session's row lock, so an
autosave or a paste during a send waits for it and then sees it, and two sends
cannot open two issues. A save keeps the delivery's progress; a failed edit
that shows the screenshots is finished by the retry; an answer that is not the
tracker's API (a redirect, a proxy page) is refused, not a crash that loses the
issue's record. Only an attempt that opened an issue counts toward the pace.

Screenshots: deleting one after the send is refused, a user's storage is
capped across sessions, new sessions and zip exports are paced, images are
stored uncompressed in the zip. One liveness rule, one session view, one image
type table.

Logging: a session's elevation is tracked apart from the user's own, so the
browser's own requests (no header) no longer lower the floor a session needs;
Stop ends it at once.

Composer: an image pasted anywhere is taken; what is still unsaved is saved on
close, and "saved" is shown only when it is; the draft is held while it is
sent; a failed screenshot read says so instead of reading as none; focus moves
to what stays; the preview follows the text size. Pill: the step prompt and a
running Stop survive the pill moving between hosts, the prompt keeps the
dialog's Tab trap, the toast region lets clicks through, other tabs hear of a
Start or Stop, the last seconds are sent before the cap, a capture with no
frame is an error.
2026-09-20 00:09:52 +02:00
peter 9cf177b198 feat(diagnostics): send a bug report to a private Gitea issue tracker (E16 S4)
- admin config group "Bug reports": Gitea address, repository, token (secret),
  and "Check the connection" — the repository must answer, take issues and
  be PRIVATE (a report carries its reporter's log); checked on every send too
- POST /api/debug-sessions/{id}/issue sends the SAVED draft: create the issue,
  upload each file on its own request (the measured API), then show the
  screenshots inline; progress is kept in the draft, so a send that stops
  half-way keeps its issue and the retry sends only what is missing; 5 new
  issues an hour per user; audited as issue_create
- the composer: Create issue (only where a tracker is set up, naming the
  repository), a partial-send banner with retry, and a Preview — the tracker's
  own rendering in a sandboxed frame, or the zip's markdown without one
- the recorder's refusals answer in the localized family (debug_session_*,
  screenshot_*, issue(s)_*), not bare codes the UI could only call generic
- a tracker refusal logs the status only: its message can quote the title back
2026-09-19 22:48:20 +02:00
peter a5fae489e0 test: a request that raises is rolled back, as production rolls it back
The shared test session kept whatever a failing handler flushed and never
committed, so a later read in the test saw a write production loses when
get_db closes the session — and a handler whose failure path must commit
(E16: the tracker's issue link) looked right without the commit.
2026-09-19 22:48:20 +02:00
peter 8d0083d4e4 feat(diagnostics): the issue composer — draft, review, screenshots, zip export (E16 S3)
After Stop (or at the cap, or from Settings → Diagnosztika's past sessions)
one dialog drafts the issue: title, description, the steps drafted from the
session's markers and page changes (a marker without a label is named by its
time), the environment (version, browser, system, window, language). Beside
it, what goes with it: the browser and server logs — each reviewable line by
line, an unticked line is not sent — and the screenshots, removable, with
paste / drop / pick to add more.

- it saves as it goes (debug_sessions.draft, migration 0090): the rendered
  markdown in the user's language, the excluded rows, and its own fields to
  reopen as it was; a sent session is read-only
- the zip export (issue.md, client.log, server.log, screenshots) is built
  from the saved draft by app/issues/bundle.py — the one definition the Gitea
  delivery (S4) will share; .log, not .jsonl, which Gitea refuses
- the log line's level tags and clock move to components/logLineParts so the
  review list draws a level the way both log viewers do
2026-09-19 22:26:49 +02:00
peter 9f177557d2 docs(ui): the toast dock in the primitives catalog 2026-09-19 22:08:21 +02:00
peter 46fce4a767 feat(diagnostics): the debug recorder — start, the pill, steps, screenshots, stop (E16 S2)
- the start dialog is the consent: level (Részletes/Teljes), what is recorded,
  retention and the cap (now served as diagnostics.session_minutes)
- the pill docks under the toasts: elapsed time, Step (an optional label,
  its own Escape layer), Screenshot (tab capture where the browser has it),
  Stop; compact on a phone
- while recording, lib/log tags every event and every request path (req,
  beacon, uploadReq via one identityHeaders) names the session; applyDiagnostics
  raises the level for the session's run and never lowers a raised one, and
  lapses at whichever ends first
- Stop sends what is waiting before the server closes the session; the cap
  says so and opens the composer
- Settings → Diagnosztika and the command palette start/step/stop it
- the evidence leaves the log upload's own request lines out
- the step shortcut is deliberately palette-only: the keymap's D1 rule bars a
  new modifier chord, and a bare letter collides with the players'
2026-09-19 22:08:10 +02:00
peter 2d9595b0e5 feat(ui): a control can dock under the toasts and ride every toast host
The toast stack already has a host in the page and in every overlay (dialogs,
both players, the fullscreen player), the topmost one shows it, and inside an
overlay it sits above the player's control band. lib/toastDock lets one control
use exactly that placement; it renders outside the live region, so a ticking
control is not read out every second.
2026-09-19 22:08:10 +02:00
peter a7d7456279 feat(diagnostics): debug sessions — start, stop, the tag on every line, screenshots (E16 S1)
A session is one recording, capped at 60 minutes by the server. While it is
open, a request naming the caller's own session in X-Debug-Session is logged
at the session's level with every line tagged; a header naming anything else
is ignored. The browser's batches count a session as consent to its level and
keep the tag only on its own events. Screenshots are judged by their bytes,
capped in size and count, and served back to their owner only.

- debug_sessions / debug_screenshots tables, one open session per user
  (partial unique index), log_events.debug_session_id indexed
- /api/debug-sessions: start (joins a running one), stop, list, get, events,
  screenshots; /api/me reports the live session
- the hourly log job closes expired sessions and drops them with the logs
- logsink.serialize and routes._common.capped_body shared by their readers
2026-09-19 21:38:53 +02:00
peter 44a792c34f test(i18n): a 404 target the scan cannot read fails the backend-id contract
what=SOME_CONSTANT reached no locale check at all; only _common.py's own
pass-through may name the target by variable.
2026-09-19 21:38:53 +02:00
peter 3987cd4260 release: 0.92.0 (improvement/debt-burn)
gate / frontend (push) Failing after 4m30s
gate / backend (push) Successful in 5m25s
2026-09-19 19:53:37 +02:00
peter 95d9ed4aa6 chore(release): 0.92.0 2026-09-19 19:53:32 +02:00
peter 8d0063d514 fix(ui): the reader's own use of a stand-in stops the restore; the pill's name carries its cap
Round 2 of the deep review:
- A click on, or a key typed into, the focus keeper's stand-in fires no focusin, so a restore still
  polling for its list could take the focus from under the reader. A pointerdown or keydown now
  makes it the reader's own focus.
- The Jump-to pill hid its "Ctrl K" from its name (the voice command "click Ctrl K" still matched
  nothing); the key cap's tooltip now equals its name, so it is not read twice as a description.
- listNav's comment still cited DataTable's two layouts as the reason for its visibility filter.
2026-09-19 19:30:27 +02:00
peter 0ca0cac534 fix(api): a waiting reorder deals from the fresh order; the commit check sees staged writes
Round 2 of the deep review, on round 1's own fixes:
- `ORDER BY position … FOR UPDATE` sorts before it locks, so the reorder that waited got the
  fresh rows in its STALE order and dealt the pins it left out (a tab with Plex off) back to their
  old slots. It locks in id order — one global order, so two reorders cannot deadlock — and sorts
  the fresh rows itself. The race test now sends that partial list and fails unless tab B really
  blocked.
- The request check saw only writes the handler flushed: a `db.delete` left staged, or a flush
  followed by a released savepoint, passed. Staged changes count at the end of the request, and
  only the ROOT transaction's end clears it (a savepoint test drives the dependency directly).
- The `ok` test exempted every call as a serializer, hiding four domain REPORTS (YouTube sync,
  revert, push; Plex sync) without `ok`. They are spread into `{"ok": True, …}`; the exemption is a
  named list of real serializers.
2026-09-19 19:30:27 +02:00
peter cf0c55ba80 fix(a11y): names that carry what they show; the unused-key check stops covering whole files
- The palette launcher's name replaced its visible "Jump to…" (and the key cap's its "Ctrl K"):
  a voice-control "click Jump to" matched nothing (WCAG 2.5.3). Each name contains its text now.
- SegmentedToggle's `title` was only a tooltip on a non-focusable box — the live search's "how
  many per batch" hint reached no keyboard or screen-reader user. It is the group's accessible
  description, standalone and captioned alike.
- The unused-key contract let a template like `card.${status}` cover its whole namespace, which hid
  five dead keys (deleted). Dynamic keys are matched by shape (`editor.*.hint`); a bare namespace
  shape covers nothing, and the audit/quota namespaces are now held both ways by backendIds.
- CLAUDE.md's "FE control" row matches the lint rule it cites; each no-restricted-syntax entry has
  its reason directly above it.
2026-09-19 19:19:43 +02:00
peter fa4f823fc8 fix(ui): Back finds the item again past the focus keeper; nested drags stay in their list
Deep review of the debt burn, each measured in Chrome:
- Back from a sub-view whose focused control the swap removed: the focus keeper put the focus on a
  stand-in (the returning list's first item) and `restoreNavPosition`, finding the focus "already
  somewhere", gave up — Plex show → Back landed on tile 0, not the tile it was opened from. A
  stand-in the keeper placed now counts as untouched, like <body>. The keeper's walk up also stops
  at its container: a slot left with nothing focusable no longer sends the focus to the first
  control of the whole document.
- DataTable enabled both layouts' keyboard lists with the same flag, and `useListNav` reads its ref
  only when the flag changes: after a resize across `md`, the layout that mounted was never
  registered (no j/k) and the one that left stayed registered. Each is enabled with its layout.
- A Plex playlist nests blocks and each show's episodes in one DndContext; closestCenter and the
  keyboard coordinates weighed every droppable, so a movie landed "over" an episode — announced as
  a place in the other list, dropped somewhere else or nowhere. `sameListCollision` and
  `sameListKeyboardCoordinates` keep a drag among its siblings; onDragEnd's episode → block
  resolution is gone with the case.
- dnd-kit reports the item over itself in the pick-up's own tick; announced, "moved to position 1"
  replaced "picked up … use the arrow keys" before a screen reader read it.
2026-09-19 19:19:43 +02:00
peter 884060891e test: a handler that flushes a write and never commits it fails its test
Helpers flush and handlers commit — but every test request shared the test's session, so a
follow-up GET or an assertion read the flushed write and the suite stayed green over a handler
whose commit was missing (production's get_db closes the session: the write is rolled back behind
a 200). The db fixture now hands out `db.info["request"]`: the same session, recording each request
that ends normally with a flushed, uncommitted write, and failing the test on it at teardown. All
28 get_db overrides use it (a convention test keeps it so); the full suite passes under it, and
dropping the commit from add_item turns a playlist test red.
2026-09-19 19:19:30 +02:00
peter 955c8c3445 fix(api): a token hiccup is not "reconnect"; reorders and approvals are one unit each
Deep review of the debt burn:
- `refusal_reason` told the user to reconnect on ANY token-endpoint failure — a 5xx, a 429, a
  misconfigured client — while the grant stayed and the account still read as connected, with no
  connect button. Only `invalid_grant` (the one that drops the grant) means reconnect now.
- `pins.reorder` writes only the rows that change, so two overlapping reorders each skipped rows
  that matched in their own stale snapshot and interleaved into two pins on one slot. It locks the
  user's pins first; the second waits and deals from the first's result.
- Adding a whitelist entry committed the invite and the activation it implies separately, and its
  audit row was written before a new invite had an id (target NULL). One transaction, flushed for
  the id.
- The mutation convention test skipped async routes and dicts returned by name: request-access,
  purge-discovery and the subscription sync answered without `ok`. They do now, and the test
  covers both. The helper-commit check looked only at `_`-names; it reads every non-route
  function, which listed two deliberate committers (upsert_pending_invite, ensure_welcome) — now
  allowlisted with their reasons.
2026-09-19 19:19:29 +02:00
peter 99df44d7fb fix(api): mutations answer ok; route helpers flush and leave the commit to the handler
Two rows of the canonical-patterns table had drifted, found while closing the roadmap audit:

- 63 mutation responses carried no `ok` — `{"unsubscribed": true}`, `{"status": "ok"}`,
  `{"added": 3, …}` and so on. Each now leads with `"ok": true` (additive: every field a client
  reads is still there).
- Twelve route helpers committed on their own. Six were just the end of their handler's work and now
  flush: the playlist item add/remove and Watch later get-or-create, the Plex series/season bulk
  mark, and the invite decision + account activation — which an approval used to land in separate
  transactions. The other six commit on purpose (a token durable before its link is mailed, a
  single-use token burned as its own unit, the Google callback's tail, a background registration
  with its own session, the staged search ingest around YouTube calls), each listed with its reason.

`tests/test_route_conventions.py` holds both rules in code; CLAUDE.md's two rows point at it.
2026-09-19 18:21:06 +02:00
peter 30fbdc52b2 fix(ui): a focused, disabled FocusSafeButton no longer lights up on hover
A FocusSafeButton that holds the focus is disabled by aria-disabled, not the native attribute, so
every hover gated on `:disabled` / `enabled:` alone still fired on it — the glass lift, the primary /
danger / warn dim, and eight `enabled:hover:` chains (saved views, the player's list buttons, the
undo toolbar). Measured on localdev: the Pager's Next, focused at the last page, now keeps its resting
border and shadow under the pointer. The CSS selectors and the chains also exclude
`[aria-disabled="true"]`, and a lint rule refuses a bare `enabled:hover:` from now on.
2026-09-19 18:14:18 +02:00
peter 046ff735cf test: close the last coverage debt the roadmap still listed
- `overflow-clip` gets a safelist anchor in index.css (its only occurrence is a `.ts` constant),
  and the chip-lane contract keeps it there.
- PlexPlayer's Backspace ownership rule is pinned on all three branches: nothing above, a foreign
  layer above (it defers), and its own panel open under a layer (it still closes the panel).
- Removing a playlist item bumps the list's `updated_at`, like the three other item-only writes.
- Seen in a real browser on localdev: an exhausted comfort-shelf tile (an empty pinned playlist) is
  focusable, aria-disabled, and its Shift+F10 menu offers Play off and Remove from shelf.
2026-09-19 18:14:18 +02:00
peter 7afaba0547 feat(playlists): say which lists YouTube's API cannot give us
The module shows fewer lists than youtube.com, and without a word about it that read as a Siftlode
bug (the owner's ask, 2026-09-03). Owner's pick from the S7 mockups (2026-09-19): one quiet line at
the foot of the rail, "Missing a list? Why?", whose Why? opens three facts — YouTube's own Watch
later and History are closed to its API (the Watch later list here is Siftlode's own), only lists
you own sync, and an account with no YouTube channel has no playlists at all.
2026-09-19 17:58:40 +02:00
peter db8ed5a004 feat(ui): a visible way into the command palette
Ctrl/Cmd+K and the shortcut sheet had no entry point but a palette row, behind the other
undiscoverable chord. Owner's pick from the S7 mockups (2026-09-19): a key cap at the end of the
header's search field opens the palette, and a page with no search field shows a small "Jump to…"
pill with the same key cap in its place, so the entry is on every page. The cap reads the chord the
palette row binds from the keymap table (Ctrl K, or ⌘ K on a Mac) and stays out of a phone's
header, where there is no keyboard to press it on.
2026-09-19 17:58:39 +02:00
peter 0ef25a8037 feat(ui): the six native selects become the house controls
Owner's pick from the S7 mockups (2026-09-19): a short fixed set is a SegmentedToggle — the live
search's result count, the table's rows per page, a download profile's mode (on its own row, so the
three worded choices fit a phone) and audio format — and a longer set or the user's own list is a
FilterSelect: the profile's quality (seven heights) and the download dialog's profile.

A lint rule refuses a native <select> from now on (the canonical "FE control" row). The px-frozen
size rules also caught up with a fraction: `text-[9.5px]` had slipped past the integer-only pattern
in the Glass tuner, which is rem now.
2026-09-19 17:53:22 +02:00
peter 17a26894b0 fix(a11y): every sortable list speaks the user's language to a screen reader
dnd-kit's screen-reader text is hard-coded English and names an item by its id ("Draggable item
10357 was dropped", a role of "sortable"). Only the YouTube playlist replaced it; the Plex playlist,
the saved-views list and the side-panel groups read it out under a Hungarian UI — found while
checking the Plex playlist's keyboard reorder on localdev.

`ui/sortableA11y` is now the one place: `useSortableA11y` gives a DndContext its instructions and
announcements (HU/EN, `common.dnd.*`), `useSortableItem` announces the role. The YouTube playlist's
own announcements move onto it, which also fixes them: its lists reorder on drop, so reading the
dragged row's own index said "moved to position 1" at every step and "dropped at position 1"
wherever the row went. The place announced is now the slot the row would land in.

A `local/sortable-a11y` lint rule (with its contract test) refuses a DndContext without
`accessibility` and a direct `useSortable`.
2026-09-19 17:37:21 +02:00
peter 946c8f26db test(plex): drive the playlist's reorder and rollback through a hook
The Plex playlist's reorder is a dnd-kit drag, and neither of its sensors activates under jsdom,
so `commit()`'s failure path could not be reached from a test. The query, the optimistic copy and
the two writes (reorder, remove) move into `usePlexPlaylistEdits`, unchanged, and are driven
directly: a failed reorder is said and the rows go back; an accepted order lands in the detail
cache; a failed removal comes back.

It also settles the `putBack` question that could not be driven before: with the refreshed list
taken in BEFORE the reorder fails, a rollback bound to the render that started the reorder puts a
removed row back, and the deep-equal refetch never clears it. The live cache read the code has now
is what keeps it out; the test fails if it goes back to the closure's `q.data`.
2026-09-19 17:30:31 +02:00
peter 8759b5860d test(be): every dependency override goes through override_dep
22 test files set `app.dependency_overrides[...]` by hand and tore down with a blanket `.clear()`
in a `finally`, which wipes every override rather than the test's own and stops fixtures composing.
They now take the `override_dep` fixture (helpers get it passed in), whose teardown restores each
key's prior value; no clear() carried meaning between two phases of a test. A convention test fails
any test file that writes `dependency_overrides` directly.
2026-09-19 17:26:07 +02:00
peter dc902d2c0e test(fe): cover the palette's content, AltGr, the banner's name, share colours, playlist headers
- AppCommandPalette had no test: modules gated by the account, hidden channels left out, the
  playlist row selecting before it navigates, and the surprise riding setPage's `then`.
- keymap: an Alt/AltGr-modified key (Ctrl+Alt on Windows, a constant on the HU layout) fires no
  bare-key or Mod row.
- Banner: the dismiss button falls back to "Close" rather than going unnamed.
- ShareDialog: a protected link's lit lock and Revoke's danger colour stay on the icon.
- Both playlist headers are rendered now: the YouTube one needed the toolbar slot it portals into.
  Meta line sums known runtimes and treats an ABSENT duration as nothing (null never exercised the
  guard: `3000 + null` is 3000); a one-row list offers no view controls.
2026-09-19 17:23:55 +02:00
peter cedcbb1434 test(i18n): every locale key must be read by the app; 75 dead keys removed
A contract test scans the source (tests excluded) for each key as a whole literal, or under a
dynamic prefix like `plex.sort.${k}`, with plural forms counted through their base key. It found 75
keys that had outlived the code showing them (the old sidebar sort vocabulary, the old login page, a
pager the DataTable replaced, …); they are gone from EN and HU. A key read in a way the scan cannot
see goes in an explicit ALLOWED map with its reason.
2026-09-19 17:15:18 +02:00
peter 4f72b193aa fix(i18n): every label a screen reader or tooltip says is translated
- The Plex playlist rows' "reorder"/"remove", the Plex filter chip's "remove" and the player
  slider's "reset" were English literals; each now names its row or setting in HU/EN.
- The playlist show header's chevron had no accessible name; its decorative thumbnail button left
  the tab order.
- The opt-in Glass tuner panel was English only: every word is translated, and each slider's label
  now names its number box and range.
- New `local/literal-a11y-text` lint rule (with its contract test): a literal word in aria-label,
  title, placeholder, alt and the other spoken/shown attributes fails lint.
2026-09-19 17:12:47 +02:00
peter 7b94e9a651 fix(home): shelf honours the Plex gate; hero unpins its origin; palette asks for a surprise
- A13: with Plex off, the Home shelf and the hero's comfort tier go YouTube-only like the rails
  beside them (owner's call: hide the tiles). The pins are kept; GET /pins still lists them.
- Reorder deals the listed pins into the slots they already hold, so a partial list (hidden Plex
  pins, a stale tab) no longer puts two pins on one position; repeats are ignored.
- A21: a comfort hero carries `pinned_from`, the shelf entry it was resolved from; its menu acts on
  that pin, so a pinned show's next episode offers "Unpin" instead of creating a second pin.
- B12: the palette's "Meglepetés" asks for the surprise tier (the next press leading with it); the
  rotation carries on from there.
- B13: the surprise token names the history entry it was requested for (minted by setPage's new
  `then`, after the push) and is dropped on any other entry, so a navigation that never mounted
  Home cannot replay it later.
2026-09-19 17:07:11 +02:00
peter c6bf84a578 fix(home): a shelf move acts on the order shown, one write at a time
- During a budget switch the rail shows the previous budget's payload
  (keepPreviousData) while the new key is empty; the move read that empty
  cache, found no tile and silently did nothing. It now acts on the order
  the rail rendered.
- A refused move was undone only by a re-read; when that read failed too,
  the shelf kept showing the order the user had just been told was not
  saved. Moves are now written one at a time (the move rows wait, with a
  "saving" note), which makes restoring the shown order correct — the
  overlapping-moves problem that ruled a rollback out cannot occur.
2026-09-19 08:13:07 +02:00
peter 444f9b8166 fix(ui): a control removed by its own action leaves the focus in its slot
The second half of the pending-button sweep: where the pressed control does
not merely disable but LEAVES the page — an invite row gone after Approve,
the inbox's Clear all gone with the last notification, a section gone once
its last row is handled — the browser dropped the focus to <body> and the
next Tab started over at the top.

useFocusKeeper, on the page scroller: when the element holding the focus is
removed and the focus fell to <body>, the first focusable thing in its slot
takes it (what followed, else what preceded, else the nearest ancestor still
on the page). Portaled layers (menus, dialogs) sit outside the scroller and
return focus themselves; listNav's keyboard lists are left alone — their `j`
already goes on from the slot, and a row scrolled out of a virtualized list
must not drag the focus after it.
2026-09-19 08:10:03 +02:00
peter 7a47223a58 fix(ui): no button drops the keyboard focus when its own action disables it
A focused button that turns natively disabled loses the focus to <body> in
Chrome. The app disables the very button that was pressed on ~60 sites — a
mutation pending, a pager at its last page, undo with nothing left, a player
at the end of its queue, a form gone clean — so a keyboard user was thrown
out of the page on every such press.

One primitive instead of a fix per site: ui/FocusSafeButton renders a
focused-and-disabled (or busy) button as aria-disabled with a click guard,
and applies the native attribute again once the focus is elsewhere. It asks
the document at render time as well as listening for focus events, because
a window without focus fires none (measured). ui/Button and ui/form's Switch
are built on it; 57 hand-rolled buttons swapped their tag for it keeping
their classes, and three hand-copies of the rule (the channel page's Block
and Hide, ChannelDiscovery's Subscribe) now pass `busy`. The soft state's
dim look is one rule in index.css.

`local/focus-safe-disabled` refuses a new raw <button disabled>, pinned by
focusSafe.contract.test.ts. lib/plexDetailUi moved to components/: it is
components, and lib may not import the primitive.

Measured on localdev in Chrome: the audit log's Next button keeps the focus
on reaching the last page, reads aria-disabled, and the guarded click does
nothing.
2026-09-19 08:07:25 +02:00
peter bc138e3b37 fix(ui): a disabled menu row says why it is off, under its label
The reason for a disabled menu row lived in a `title` on a natively disabled
button: no pointer events, so no tooltip, and roving focus skips the row —
four translated sentences reached nobody (the Watch later rename/delete
lock, "no other list to move to", the reorder gate). An aria-disabled row was
tried and reverted in E10's review. ActionMenu items now take a `note`,
rendered as a muted line under a disabled row's label: seen, and readable in
a screen reader's browse mode. Only the icon and label dim — row-level
opacity would dim the explanation too. The reorder group states its reason
once, under its first row.
2026-09-19 07:57:20 +02:00
peter df7c3db085 test(ui): the px-box probe does not trip its own rule; tsc lib-safe last()
The pxBox contract test held the literal w-[150px] it probes, which the new
lint rule refuses, and stickyTop's test used Array.at, outside the tsconfig
lib. Both landed because a local lane check piped its exit code into tail.
2026-09-19 07:53:01 +02:00
peter 6fda49369f fix(ui): a sticky table header turns the scroller's top fade off itself; DataTable renders one layout
- "A page with a sticky ColumnHead must declare fadeTop: false" lived in
  prose: a registry flag on three pages plus an App special case for the
  Channels card layout, and the next table page would have re-acquired the
  fade-on-the-header defect with every lane green. ColumnHead now registers
  itself with the enclosing PageScroller (ui/stickyTop) while mounted; the
  registry flag, the App branch and the prop plumbing are gone. Measured on
  localdev: the top mask is 0px under Channels' and Audit's headers, 20px on
  Home and the Feed.
- ui/DataTable mounted its table AND its card list at every width, one
  hidden by CSS: double DOM and render cost, duplicate ids, two keyboard
  copies of every row. It now mounts the one the `md` media query calls for
  (lib/useMediaQuery); the keyboard test asserts one layout at each width.
- The Plex module's content width went px → rem with the rest of the page.
2026-09-19 07:51:33 +02:00
peter 8018cd4685 fix(ui): the platform layer speaks platform words; boxes scale with the text
- PriorityStepper labelled itself with channels.row.* and told every caller
  it orders the Feed; Combobox said plex.loading / plex.filter.moreRefine /
  sidebar.match on the feed sidebar. The generic strings moved to common.*,
  and what a priority MEANS arrives as the stepper's `hint` prop.
  uiI18n.contract.test.ts pins components/ui/ to the platform namespaces —
  the import boundary could never see an i18n key.
- 32 px-frozen box utilities (w-[150px], max-w-[1600px], left-[18px], …)
  became spacing-scale steps (w-37.5, max-w-400, left-4.5): identical at the
  default text size, and now they grow with the Settings slider instead of
  clipping or re-wrapping their own labels. An eslint rule refuses the px
  form beside the older text-[Npx] one, pinned by pxBox.contract.test.ts;
  measured in the browser that every new step is generated at its old px.
- chipLane's import-strip regex ran past a side-effect import into the next
  statement's `from`, hiding a lane between them; it now stops at the
  import's own `;`.
2026-09-19 07:46:33 +02:00
peter 29875b7367 fix(ui): stable shelf-card menu triggers, a named Button by type, true layout comments
- Home's rail handed each card a fresh `() => menu.triggerProps(c)` per
  render, defeating memo(VideoCard) on every tile whenever the menu's own
  state changed. `useItemTriggers` (ui/useContextMenu) keeps one trigger per
  item, index-aligned; tested for identity across renders.
- The Button accessible-name union had one hole: `<Button icon={…} />` with
  no children and no size typechecked into a nameless button. A textful
  button now needs children or a name; pinned with @ts-expect-error.
- comfortPin's header said the pin belongs "on every card and row", which
  the code never meant (an episode, a season, a person are not pin kinds).
  It now says what is pinnable — and the show page's Related strip, whose
  shows ARE pinnable, gets the pin menu it was missing (and a rem width).
- gridColumns / feedView / VirtualGrid claimed layouts 1024–2560 are
  "byte-identical" under the column cap; the small cards are capped at 2560
  with the filter rail collapsed. The comments say so.
2026-09-19 07:40:48 +02:00
peter 7ae5c57b6b fix(i18n): every refusal the API answers is structured and said in the reader's language
~150 HTTPExceptions carried a hand-written English sentence as their detail,
shown verbatim under the Hungarian UI — in the blocking error dialog and on
every inline surface that printed HttpError.detail (sign-in, setup wizard,
change password, the passphrase gate, live search). Four also quoted an
upstream exception's text back to the user.

- app/refusal.py: `refused(status, reason, **params)` → {code: "refused",
  reason, …params}; localizeDetail translates errors.refused.<reason> with the
  params interpolated. "Unknown X" 404s use the existing not_found(what).
- sysconfig.set_value raises InvalidSetting (reason + bounds) instead of an
  English ValueError; the three writers answer it through setting_refused.
- The Plex connection/write/image failures log what the upstream said and
  answer a reason; the search scraper's failure is logged and answered as
  youtube/unavailable (it crashed refusal_reason before).
- HttpError.detail now only ever holds a translated reason: a string detail
  (only the framework's own can remain) is left out, so reading .detail is
  safe everywhere — including destructured, the hole the raw-detail source
  scan could not see. That scan and its five-file exception list are gone.
- Enforced: test_http_details.py fails on any HTTPException with a sentence
  for a detail; the contract test requires every refused reason and every
  not_found "what" in both languages, and no key for one that is gone.
- local/jsx-comment-text: a // inside JSX children renders as text; the rule
  is ten lines in eslint.config.js rather than eslint-plugin-react's ~1800
  lockfile lines.
2026-09-19 07:34:17 +02:00
peter dc4e8b9fcd fix(plex): playback sessions per viewer, bounded in total; a fair share of subtitle passes
HLS sessions were keyed by the item, so a second account opening the same
film killed the first one's session; the displaced player reclaimed at its
playhead and evicted the other in turn. The registry is now keyed by
(viewer, rating_key), segment requests resolve by both, and the session
directory carries the viewer (the orphan sweep still recognises the older
shapes).

Only transcodes counted toward a cap, so remuxes — each a whole segmented
copy of the film on the scratch volume — were bounded by the library size.
Three bounds now apply after every start, each dropping the least recently
used: two sessions per account, the transcode cap, and a new admin knob,
plex_max_sessions (default 8), over every session. A superseded session
request returns at once, since it waits on the killed ffmpeg, so an account
cannot park more request workers than it may hold sessions.

The subtitle extractor's full-pass slots were global and minutes long each;
one account opening subtitles on film after film could hold them all. An
account may now hold three of the six, and still gets windows past that.
2026-09-19 07:16:17 +02:00
peter 6f9ba428d5 fix(playlists): the move's conflict fallback says why it is silent
The house guard requires a silent handler to carry a silent-ok note; the
savepoint fallback added in e74986cc did not, and the gate's pytest lane
caught it.
2026-09-19 07:10:43 +02:00
peter c60359bfed fix(plex): subtitles — a retried full pass resumes, a stalled track is polled slowly
A full pass that ran out of time kept what it reached, but its retry an hour
later started again at 0, so a file too big to read inside the extraction
timeout never completed. The pass now starts where every embedded track's
coverage from the head ends (less the window pre-roll, so a cue across the
join is read whole). The seek needs -avoid_negative_ts on this read as on a
window, measured on a real file: without it the cue running at the start
shifts the whole output.

A partial track whose full pass had failed never reached a settled state, so
the player polled it every 2 s for the rest of the film. The status now says
whether the track is still growing, and a stalled one is polled every 15 s —
still inside the 60 s lookahead that reads the next window before an edge.
2026-09-19 07:07:53 +02:00
peter e6b76e156a refactor(discover): one shelf query per kind, one RAIL_LIMIT, a true rollup comment
The comfort shelf resolved each pinned channel with three queries (the
row, its unwatched count, its newest video) and each playlist with two.
channels_summary gets the count and the newest row per channel as window
functions over one filtered set, playlists_summary reads every pinned list
in one query, and the rows come in one IN() each — so the shelf's cost no
longer grows with its length. A test counts the statements.

RAIL_LIMIT had two definitions held in step by a comment; the Plex Discover
route now imports the domain's. And rollup's "continue the latest-started
episode" was false: it takes the last in-progress one in broadcast order,
which is also what Home's next_episode picks.
2026-09-19 07:03:35 +02:00
peter e74986cc9a fix(playlists): a racing move is not a 500; ownership through owned_or_404 everywhere
move_item checked "already in the target?" and then re-parented without a
lock, so a second tab's add or the scheduler's YouTube merge committing the
same video in between hit uq_playlist_video and answered a raw 500. The
re-parent now runs in a savepoint and a conflict falls back to the
already-present outcome. The target is touched only when the row landed in
it (an untouched list was promoted in the rail's "recently changed" sort),
and _touch stamps the database clock like the mixin's own onupdate.

The canonical-patterns table names owned_or_404 against ad-hoc ownership
helpers; six were left (_own_playlist, _own_playlist_or_404, _own_view,
_own_tag, notifications._owned, downloads._own_job), two of them
hand-rolled with a bare English 404. All call sites now use owned_or_404,
and the download/link/notification 404s are structured like the rest. The
contract test reads every "what is gone" the routes name, and the YouTube
refusal reasons, and requires both languages to name each.
2026-09-19 07:01:18 +02:00
peter 2df45b27d3 fix(plex): a refused finish restores no resume point at the media's own end
The rollback of a refused watched mark guarded against Plex's REPORTED
duration. On a file shorter than its metadata the end sits outside the
finish margin, so playing to the end and having the mark refused stored the
very end as a resume point on an unwatched row: a ~99% bar that reopened in
the last seconds. The media's own ended flag now decides.
2026-09-19 06:58:11 +02:00
peter bacd6aec4f fix(plex): one scrobble per finish, and no resume push lands after it
Two near-end checkpoints arrive together routinely (the 10 s tick and the
debounced seeked flush); both read "not watched" before writing and both
scrobbled, bumping Plex's viewCount twice. The transition is now decided by
the write: an insert that did not conflict, or an update conditional on the
old status.

The finish also popped the push-ordering entry that _resume_push_superseded
reads, so an older resume push still in flight could land at Plex after the
scrobble and revive a resume point on a watched item. It now stamps the
entry instead; the existing eviction bounds the map.
2026-09-19 06:57:20 +02:00
peter fad301e7a4 fix(youtube): a refused YouTube action answers a 422 that says what to do
The two manual syncs (subscriptions import, playlist pull) had no YouTubeError
handler, so a quota, token or transport failure was a raw 500 and the blocking
"Server error" dialog (E9 F1). The other YouTube routes answered either the
upstream's English or a playlist-scoped "youtube_failed".

One helper, routes/_common.youtube_refusal: 422 {code: "youtube", reason} with
the reason sorted by what the user can do — quota (wait for the reset),
reconnect (the token step refused), unavailable (retry). Used by every route
that surfaces a YouTube failure; localizeDetail translates it.

"Reconnect" had no path: a grant Google revoked still read as granted, so
Settings and the wizard offered no connect button while every call failed.
An invalid_grant from the token endpoint now clears the stored grant.
2026-09-19 06:55:40 +02:00
peter 1cfb19bdc7 release: 0.91.0 (bug/prod-uat-2026-09-19)
gate / frontend (push) Failing after 4m0s
gate / backend (push) Successful in 5m23s
2026-09-19 05:56:19 +02:00
peter 90c30d596f chore(release): 0.91.0 2026-09-19 05:56:12 +02:00
peter 7ed6ccf171 fix(feed): the card menu's reset uses the card's own reset string
Two keys for one action had already drifted once; the menu now reads card.resetState like the
card's button.
2026-09-19 05:54:19 +02:00
peter ce30b99327 docs(nav): restoreNavPosition's doc back on restoreNavPosition 2026-09-19 05:54:19 +02:00
peter a7891d03c5 fix(sync): only a refusal about the channel spends its fallback window
A failed connection or token step answered nothing about the channel, yet stamped it: one proxy blip
during an RSS blackout delayed every due channel by a whole window. YouTubeError now names the
endpoint that answered; a failure no endpoint gave, or a spent quota, ends the round and stamps
nothing.
2026-09-19 05:54:19 +02:00
peter 038da5374f test: the boundary stays quiet while reloading for a new build; Back restores Home's whole question 2026-09-19 05:47:17 +02:00
peter 92979fa920 refactor(ui): one rule for resettable, role key and display name; an honest reset label
The card menu's reset read 'Clear watch progress' but also un-hides and un-watches; it now says so.
The resettable rule, the demo-first role key and the display-name fallback each lived in several
components tied by comments; they are shared helpers now, and the menu reuses the card's own
strings instead of duplicate keys.
2026-09-19 05:47:17 +02:00
peter 2d5bd275fa fix(diagnostics): a pivot or Back applies the search text at once
The query read the typing-debounced search, so for 300 ms after a jump it mixed the new view's
filters with the old view's text and fetched rows neither view shows. useDebounced takes a settle key
whose change applies the value immediately.
2026-09-19 05:47:16 +02:00
peter 49fe56b6e7 fix(nav): Back's focus keeps its place among items of the same name
Restoring by name picked the first item of that name, so Back on the log console (repeated
http.request lines) or two same-titled downloads landed on the wrong one. The saved place wins while
it still holds the name; otherwise the nearest item of that name.
2026-09-19 05:47:16 +02:00
peter 1fa6a0bea8 fix(sync): a failed feed-fallback read waits its window out, and spent quota ends the round
A channel whose uploads playlist also failed was never stamped, so a terminated channel was charged
again on every 20-minute poll instead of once per rss_fallback_hours. An upstream quotaExceeded now
stops the round instead of charging every remaining channel the same refusal.
2026-09-19 05:47:16 +02:00
peter 582f20fab6 fix(sync): the stored RSS warning says what the API fallback read
An INFO line outside a request never reaches log_events, so the fallback's
own summary was invisible in the Server log. Its counts ride on the
sync.rss.failed warning, the one line of a poll the console keeps.
Measured on localdev: 375 of 375 dark channels read, 403 new videos.
2026-09-19 05:05:13 +02:00
peter 6cb8231a9f fix(logs): the browser's copy of a polled read is quiet too
The server now logs polled reads at DEBUG, and the browser's copy of each
request line promises the server's level: at INFO the unread-count polls
filled My log within minutes and, at Detailed, the shipped log. The server
marks a line it logged at DEBUG with X-Log-Quiet, so the polled-route list
stays in one place.
2026-09-19 05:03:31 +02:00
peter 6bbf532b70 fix(diagnostics): count the 24 hours from the moment the level was raised
The countdown's clock was the moment Settings mounted, so a level raised
two minutes after opening the page read "24 h 2 m" until the first tick
(measured on prod). The clock is read at render; the interval only
re-renders.
2026-09-19 05:03:31 +02:00
peter 2fa5c8c394 fix(ui): nothing past the right edge of a phone
The Feed's chips + Sort + View row did not fit one line at 375px and pushed
the page 21px wide; the toast stack capped its width by the viewport while
it sits right of the sidebar. The toolbar rows (Feed, Plex browse) wrap,
and the toasts are capped by their host.
2026-09-19 04:54:18 +02:00
peter 373415a2eb feat(account): show the account its own last sign-in
Settings → Account names the latest sign-in on any device (only the
admin's user list did, though the 0.88 notes said otherwise), and the role
in the reader's language instead of the raw `user`.
2026-09-19 04:51:43 +02:00
peter 582c033e1c feat(feed): the card menu carries the card's own actions
Right-click, long-press or Shift+F10 on a feed card offered only the
comfort-shelf pin; watched, saved, hidden and the progress reset were
reachable only by hovering or Tabbing through every card.
2026-09-19 04:51:43 +02:00
peter ef652ee5d8 fix(ui): honest tab titles and a download field that names every site
Signed out, the tab read Home's question over the sign-in form; the legal
pages read just "Siftlode". The download field asked only for a YouTube
link although Facebook and the rest download too.
2026-09-19 04:51:43 +02:00
peter 250e3a9f90 fix(home): Back keeps the question, and the focus finds its card by name
Opening a card left Home, and Back came back to "Doesn't matter" with a
different hero, whose card then dropped out of the Continue rail and moved
the keyboard onto its neighbour. The budget, reroll seed and seen list live
on the history entry now, and a returning list focuses the item by its
name before its place.
2026-09-19 04:51:43 +02:00
peter 6096dd6927 fix(diagnostics): Back from a request's lines returns to the view it came from
Every filter change replaced the history entry, so Back after a request or
job pivot left the console altogether (the 0.90 notes promised otherwise).
A pivot follows a link and is its own entry now; the console reads the
landed entry back on popstate. The other filters still adjust in place.
2026-09-19 04:51:42 +02:00
peter 51e98bfc38 fix(app): reload onto a new build instead of crashing on a missing chunk
A tab left open across a release asked for a lazy chunk the server no
longer has (prod: DownloadDialog 404 an hour after 0.89) and fell to the
crash screen. `vite:preloadError` now reloads the tab once a minute at
most; a second failure is let through to the error boundary, which now
wraps every entry — the public /watch, /privacy and /terms pages crashed
to a blank page.
2026-09-19 04:51:31 +02:00
peter 3ee00fb051 fix(logs): keep polling reads and a channel-less account out of the warnings
The reads the frontend polls (unread counts, /api/me, the download index,
sync status, subtitle progress) are logged at DEBUG: at INFO they were ~95%
of the stored log on prod. A scheduler job failure and a playlist sync
failure carry event codes, and a Google account without a YouTube channel
has an empty playlist list instead of a warning every six hours.
2026-09-19 04:51:31 +02:00
peter 29094f5f51 fix(sync): read a dark RSS feed through the API instead
YouTube's RSS goes dark per channel for hours (prod 2026-09-18/19: 393 of
396 feeds answered 404 for four hours, sticky per channel, so a retry does
not help), and nothing else read a followed channel's new uploads. A
channel whose feed has not answered for `rss_fallback_hours` (admin knob,
default 2, 0 = off) now gets one page of its uploads playlist through the
Data API (1 unit), at most once per window, yielding to the backfill
reserve. Spend is attributed to its own quota action.
2026-09-19 04:51:31 +02:00
peter 16a44dec79 release: 0.90.0 (feature/e22_6-diagnostics-console)
gate / frontend (push) Failing after 4m16s
gate / backend (push) Successful in 5m19s
2026-09-19 03:57:44 +02:00
peter 4e3e68d6e6 chore(release): 0.90.0 2026-09-19 03:57:29 +02:00
peter eb29f22eb7 fix(diagnostics): say which read failed, and let Live restart
- "Load older" is not offered off a loading filter's placeholder rows,
  whose cursor belongs to the previous filter.
- A failed older page, stale rows and missing counts each get their own
  message (HU/EN).
- Live shows as off once a failed read stopped the polling; pressing it
  again reads now.
2026-09-19 03:48:01 +02:00
peter 8f6f89832f fix(diagnostics): honest live console after review
- Live stops polling on a failed read and marks the list as possibly
  stale (with Retry), instead of re-raising the error dialog every tick.
- While live, a refetch reads the newest page only, whatever the cache
  holds (a late "Load older", a revisited filter set).
- Copy is disabled and the list dimmed while a filter change loads.
- Dates follow the UI language; the pivot chip uses the shared .chip.
- Tests: query-string wire test, Copy, every facet's skip key.
2026-09-19 03:42:26 +02:00
peter 1c49870a7a fix(sync): don't try a timed-out feed again
A request that hung for the full timeout says the way out is hanging, not
that the endpoint is flaky; three of them per channel stretched a poll over
a dead egress to about three times its interval.
2026-09-19 03:42:26 +02:00
peter 142593083f feat(diagnostics): walk the log lines from the keyboard; name the console "Server log"
Both log viewers put their lines in a NavGrid: j/k (and the arrows) step
from line to line, and Enter opens one exactly as a click does (the line's
button is its `data-nav-open`). A focused line is named by its code and
message.

The console's nav entry is "Szervernapló" / "Server log", its page title:
"Diagnosztika" was already the Settings tab that holds the user's own log,
and two places with one name read as one place.
2026-09-19 02:02:45 +02:00
peter 8cfab498cf fix(sync): try a channel's feed again before counting it as failed
YouTube's feed endpoint answers the same channel with 404, 500 and 200 in
turn: on prod 182, then 296 of 396 channels failed a poll, and a probe of
24 channels saw 15 fail a first request while 16 had succeeded by the
fifth. A channel that failed once waited for the next poll, 20 minutes
later, so new uploads showed up late.

A feed is now tried up to three times (after 1 s and 2 s) on a transport
error, a 404, a 429 or a 5xx; a refusal such as 403 is not retried. The
last failure is what gets counted. A feed that is really gone costs two
more requests per poll.
2026-09-19 01:58:40 +02:00
peter b5996a0e56 feat(diagnostics): the admin log console
Admin → Diagnostics reads the server's stored log: one list, newest first,
where a line opens in place to its full record (time, source and logger,
account, request and job ids, version, message, attributes, stack). A
request or job id narrows the list to that request or job with every other
filter off, shown as a removable chip. Filters: level group, source,
account, event, period and a free-text search; all of them live in the URL
(`l*` keys, which setPage clears like the Plex and Channels keys), so a
filtered view is a link and Back restores it. "Live" re-reads the newest
page every few seconds while the tab is visible; paging into older rows
turns it off. Copy puts the loaded lines on the clipboard as JSON lines.

The facet counts now honour every filter but their own, as the feed's do:
a request pivot narrows them, and a picked level does not zero its
siblings. A value in use stays offered even when the counts drop it.

"Saját napló" and the console draw a line with one shared component,
`LogLine`, whose phone layout gives the message its own full-width line
(sharing the code's line left it a few characters). The level groups
moved to lib/log.ts for both.

The console's own reads are logged at DEBUG on success, like the
healthcheck: at INFO its live tail filled the log with itself.
2026-09-19 01:54:48 +02:00
peter 5e208353e3 feat(diagnostics): admin query API over the stored log
GET /api/admin/logs pages `log_events` newest first on a (ts, id) keyset:
a browser row's ts is clamped into its consent window, so ts alone is not
unique. Filters AND together: level, source, account (or `none`), event
code (or `none`), request id, job id, a free-text search over the
message, code, request id, stack and attributes (typed text is a literal),
and a time window. Each page names the accounts it mentions.

GET /api/admin/logs/facets counts what the filter chips offer over the
window alone, so a chip row does not shrink as filters are picked.

Admin-only and read-only; reading is not audited (the console's live tail
polls). There is deliberately no "ids after N" tail: several processes
write this table and a batch takes its ids at insert, not at commit, so a
tail keyed on ids could pass a row that commits late.
2026-09-19 01:40:46 +02:00
peter 66dbd0af81 fix(search): typed text is a literal in every LIKE pattern
The feed's channel-name match and the live search's term fold built their
LIKE pattern from the typed text as it was, so `_` and `%` acted as
wildcards: `build_log` also matched `buildXlog`. The term fold also tested
a substring, so a term already present inside a longer word (`cat` in
`category`) was never folded into search_terms. It now compares whole
terms, escaped.

`like_escape` moves from the Plex query module to `app.utils`, beside
`to_tsquery_str`, and every pattern built from typed text goes through it.
2026-09-19 01:33:32 +02:00
peter 59fd068711 release: 0.89.0 (improvement/e22_5-backend-logging)
gate / frontend (push) Failing after 3m28s
gate / backend (push) Successful in 5m1s
2026-09-19 00:05:53 +02:00
peter 8e12eee996 chore(release): 0.89.0 2026-09-19 00:05:47 +02:00
peter 89c77ae6b5 fix(logs): deep-review round 3 — a failure nobody could record is one scrubbed line
- The download and edit failure lines go back after their DB write: a job the
  write could not fail is not a failed job, and the worker's child_exit line
  already says it failed.
- The download child's runner is the last stop before Python's hook: an
  escaping exception is one download.child.failed line with its chain scrubbed,
  then exit 1, instead of a raw stderr traceback carrying the URL.
- A crash the database will not record carries its stack once, and the
  fail-write line has an event code.
- The job's own generic-extractor name is taken out as a literal (formats.hiding),
  so brackets and colons in a direct link's file name no longer leak; the
  pattern stays as the fallback. `Downloading just the video <id>` is covered.
- Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths,
  the image steps' failure kinds, the watchdog error through the real
  _fail_running, exact poster/storyboard messages, ffmpeg skip marker.
2026-09-18 23:44:42 +02:00
peter 0b672ed6e7 fix(logs): deep-review round 2 — a stack names code, not data, and survives a failed write
- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a
  message that is the whole story). A plain RuntimeError, RecursionError or OSError
  is also what a bug of ours raises, and its stack is the only lead.
- The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError
  under a DB error no longer prints yt-dlp's text unscrubbed.
- Engine hide_parameters: a DB error's message no longer carries the title or path
  it was writing.
- The download and edit failure lines go out even when their DB write fails.
- The worker's child_exit/crash/supervisor lines say download.job.failed only when
  the job failed there; a crash the DB will not record no longer ends the thread.
  Stopping a paused child is download.child.stopped, not a second abort.
- log_safe also covers generic ids with spaces, playlist titles and bare ids.
- The ffmpeg image steps log the failure kind, not the argv with its path.
- The consent contract accepts only the body of a plain consented_trace() check,
  and any level that may be TRACE.
- Tests for each; every one mutation-checked.
2026-09-18 22:50:37 +02:00
peter 54d19cc386 fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
  `formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
  file paths, and the job-failed, refused, edit-failed and backfill lines use
  it. Done lines log the asset id, not the title-built path. The PII contract
  now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
  failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
  is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
  set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
  a process's last batch closes the repeat window, so a short-lived download
  child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
  `download.job.failed` with a cause; they left no line at all before. The
  crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
  and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
  not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
  carry reason `transport`. Subscribe recognises a duplicate by reason code
  only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
  directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
  were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
2026-09-18 03:16:15 +02:00
peter 04b58e5c6f feat(logs): give the log the events that no request line already carries
The gap this sprint was scoped against turned out to be smaller than the survey
said, and in a way worth writing down: `log_requests` already records every HTTP
call with its route template, status, duration, request id and user id, and the
audit log already records every admin act on someone else's account. Putting a
line in each of routes/downloads.py's twenty mutating endpoints would have added
a second copy of the first and nothing else. So this instruments what neither of
them can see.

yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting:
with no `logger` its whole account of a download went to the child's stdout, so
"Requested format is not available", a signature-extraction failure or a POT
refusal reached us only as whatever text ended up in the raised exception — the
real reason a download failed was routinely one line the app never kept. Measured
against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls
`debug` for extraction and progress lines and `warning`/`error` for the rest, and
neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being
several lines a second, so the noisy half is the half the user's own consent
gates. The full source URL is trimmed to its host on the way in — it is a row of
someone's download history, and `logs.set_job` already puts the job id on every
one of these lines. The option is set AFTER the adapter merge, like `cookiefile`
and the size cap: a source adapter must not be able to take away the one account
of what went wrong that a failed job leaves behind.

An HLS session now says which pipeline it got (transcode/remux, hw, tonemap,
variants, offset) — everything downstream is about that choice, and it was only
inferable from an ffmpeg command line nobody keeps. A session culled by the
concurrent-transcode cap says so: somebody's playback stops there, the route turns
it into a retryable 404, and "my film keeps rebuffering" had no other explanation
anywhere. The HW→SW fallback keeps its line and gains an event code.

A live search refused by a quota cap says WHICH cap: the 429 is in the request
line either way, but the per-user daily cap is answered by tomorrow and the shared
budget by the admin.

The download and edit job lifecycle lines gain event codes, which is what E22.6's
console filters on — the job id was already a structured field.

tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth
(a `%` in a progress line) survived its mutation — stdlib only applies `%`
formatting to a record that has args — so it is gone rather than kept as a case
nothing can break; the reasoning moved into the comment on `_emit`.
2026-09-18 01:28:52 +02:00
peter 6b0401b922 fix(logs): rule on every swallowed exception in the backend
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.

What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
  degraded to a software transcode with nothing saying why; a failed audio probe
  collapsed a multi-audio file to one track and the player showed no language
  picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
  complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
  out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
  therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
  and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
  stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
  the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
  fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
  own root, leaving every crashed session's segments on disk forever.

Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.

ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.

logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.

tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
2026-09-18 01:20:42 +02:00
peter 0cff964ff2 refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.

`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.

app/scheduler.py called its logger `logger`; every other module calls it `log`.

ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
2026-09-18 00:57:57 +02:00
peter dc221338be fix(logs): keep what a person typed out of the backend log lines
The E22 redactor is a safety net over addresses and credential params, and every
one of these call sites was leaning on it to do the whole job. What it does not
cover went in whole, and what it does cover came out as a constant.

Not covered, so a real leak:
- routes/search.py logged the raw search term at ERROR. The privacy policy shipped
  in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own
  consent. The ERROR line now carries the shape of the failure (source, candidate
  and collected counts) and the term follows on a TRACE line, which nothing emits
  unless that consent is in force.
- downloads/backfill.py logged the full source URL at INFO — a row of the user's
  download history. Now the asset id and the host.
- youtube/client.py logged the upstream error body, which is prose that quotes the
  request back. Every failing YouTube call now goes through one _upstream_failure()
  that logs the API's `reason` code, keeps the body for the same consented TRACE
  line, and raises a YouTubeError carrying `reason` as a field — callers put that
  error in log lines of their own (routes/channels.py) and in a 422 answered to the
  user, so the body must not travel in it either.

Covered, so merely useless: routes/auth.py named the account by an address that
reached both sinks as `[email]`, and app/email.py printed recipient lists that read
as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is
also the first half of E22.5 S4's failed-sign-in coverage.

routes/channels.py branched on the text of that error message to recognise an
already-existing subscription. It now branches on `exc.reason`, with the substring
check kept beside it for a body that carried no reason at all.

tests/test_log_pii.py pins the call sites and the family: a source-level contract
test walks every log call in app/ and fails on an argument naming user-typed
content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites
carry that marker today, each with its reason. Gate 8/8 green; all five cases
mutation-checked.
2026-09-18 00:49:43 +02:00
peter 8c10e57208 release: 0.88.0 (feature/e22-activity-trail)
gate / frontend (push) Failing after 3m31s
gate / backend (push) Successful in 4m44s
2026-09-18 00:05:50 +02:00
peter dd040adc96 chore: 0.88.0 release notes
E22 (activity trail): a consented, self-expiring diagnostics level with the user's own log visible
before anything is sent, last sign-in, bilingual privacy policy and terms, an admin diagnostics
section, and pickers for every closed-set setting. Plus what the epic's measurements turned up on
the way: doubled chat delivery, Settings on a phone, locale-ignoring dates and numbers, and the
feed poll narrowed to followed channels.
2026-09-18 00:05:46 +02:00
peter 4ab128ccab Merge branch 'main' into feature/e22-activity-trail 2026-09-18 00:02:12 +02:00
peter a28193a00c test: pin the focus restore, the choice labels and the phone layout to their fix
Three round-1 fixes whose tests passed against the broken version too.

The focus restore keeps watching until the element that closed the box is GONE, rather than spending
itself on the first render after the change — the distinction Chrome exposed and jsdom did not,
because jsdom commits the save in one render where Chrome takes two. Driven now through the general
case instead: the user moved the focus elsewhere while the request was out, renders keep coming
while that element is still there, and the restore happens only once it disappears.

The choice test's i18n mock returned an option's bare value, which is exactly what dropping the
`config.choices.*` lookup renders — a Hungarian admin would have seen `on`/`off` instead of
`GPU`/`Szoftver` with the gate green. Marked labels now, still short enough not to move the
segmented-bar budget the same file measures.

The admin config row's stacking had no test at all, only its sibling fix did — and stacking is the
only fix for a text or number key, whose fixed-width input is the wide control. Class contract, as
SearchBar.dom.test.tsx already does for the narrow-header rules, since jsdom does no layout.

Also: the time-zone cleanup deleted TZ, and Node re-reads it on assignment only, so the process kept
the last zone the file set. Harmless until someone appends a test below it.
2026-09-17 23:45:41 +02:00
peter 986bc8d01e fix: a batch the user themselves cancelled is not called a lapse
Picking "Alap" revokes the consent, and a batch still in flight is then answered by the server with
the same 409 a real expiry gets. The 409 arm only checked that the consent session had not changed,
not that this tab was still streaming — so the user's own log gained a "diagnostics level lapsed"
line for a level they had just turned off themselves, plus a pointless `me` refetch. The same guard
also stops a lapse this tab has already announced being announced a second time.

Measured in Chrome by holding the POST at the network layer, switching to Alap and only then
answering 409: the line appears without the guard and not with it.

Three tests that named the session counter never reached it — they settled the refusal while the
stream was already stopped, where `unsubscribe` does the work, so the counter could be deleted from
both arms with the suite green. The refusal now settles after the re-raise, which is the ordering it
exists for. Same for the queue trim on the 429 path: the old test unshifted a batch into an empty
queue, so the trim was a no-op and only the subscribe path was ever measured.
2026-09-17 23:45:29 +02:00
peter 07e3ea3737 docs: the deep-backfill card counts the backlog, not the next pass
Two reviewers in a row read the admin card's selection against `run_deep_backfill`'s, saw it was
missing the job's `recent_synced_at` filter, and called it a card that counts channels the job will
never touch. It is the opposite: the card is the same backlog figure /api/sync/my-status shows the
user, and a channel still missing its recent pull is one the same job tick fetches first. Narrowing
it would hide genuinely queued work. Written down where the next reader will look.
2026-09-17 23:45:29 +02:00
peter b2eaa9e00b fix: a credential, an erased account and a stale level stay out of the log
Round 2 of the E22 review, on what round 1 changed.

`redact_query` decodes a parameter before the text rules see it, which is what lets `%40` be
recognised as an address — but `_SECRET_PARAM_RE` was written for the raw, encoded form, where a
value holds no space, `&` or quote. On decoded text its value class stops at the first one, so
`?state=a%20b` kept its tail and `?code=%20abc` was not matched at all. The parameter-NAME rule now
decides, on the key, before any text rule runs.

`forget_user` dropped the account's level elevation without recomputing the root level, so a process
whose only elevated account had just deleted itself kept building TRACE records to throw away, until
some later request happened to recompute it. Every other path that drops an elevation recomputes.

The sink's writer took the erasure lock and THEN waited for a pooled connection, while every request
path already holds a connection when it reaches that lock — so under pool pressure the writer held a
process-global lock against the very requests that could hand a connection back. Same order on both
sides now; the lock still spans the check, the insert and the commit.

The storage-budget notice said the rest of the hour was not kept, and then smaller batches kept
landing underneath it: the window slides and a refused batch is not charged to it. Says what it does.

Tests: the route's half of the erasure rule had no test — only the sink's did — and a source-level
contract test now covers the third insert nobody has written yet. `logs._erased` is process-global
with an hour's memory while the DB fixture hands serial ids back out from 1, so it is swept before
each test like the rate limiters are.
2026-09-17 23:45:16 +02:00
peter 791ba5f507 fix: a calendar date is the same day everywhere; legal pages use the shared language picker
formatDate read a date-only value as UTC midnight and showed it in the viewer's zone, so the legal pages' last update read a day early west of UTC. Date-only values are shown in UTC. The legal pages' hand-rolled language buttons are the landing page's LanguageSwitcher. formatDateTime and formatDate get the two-language test formatNumber already had.
2026-09-17 03:53:40 +02:00
peter 7789db8240 fix: admin config pickers fit a phone
A segmented bar cannot shrink or wrap, and the five log levels (~300 px) are wider than a phone leaves beside the navigation rail. A choice is a bar only when its labels are short; the rows stack below sm.
2026-09-17 03:53:40 +02:00
peter f92cd9017f fix: the diagnostics tab keeps its focus, its level and its consent
Deep-review round 1 of E22.

- After a 429 the stream sent one POST per new event while a full batch
  waited; it now stays quiet for 15 s. Past the queue cap the oldest
  events go, as the comment always said, a refused batch included.
- A batch refused after the level was lowered came back into the queue
  of a later consent in the same tab; each consent is its own session.
- A `me` fetch that left before a level or preference change could put the
  old value back for up to a minute. `updateMeCache` writes the cache and
  cancels that fetch; usePref goes through it too.
- The consent box's buttons and "revert now" took the focus with them to
  <body>; it returns to the level picker. Watched until the button is
  gone — in Chrome the saved level arrives in two renders, and a one-shot
  check spent itself on the first (jsdom never showed it).
- The countdown line is no longer a live region re-read every minute.
- "Saját napló" rows were keyed by index on a newest-first list, so every
  new event remounted every row and dropped the focus off a row button.
2026-09-17 03:53:40 +02:00
peter 4bfba4aa86 refactor: one definition of a followed channel and of a choice spelling
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional.
2026-09-17 03:53:29 +02:00
peter 86bc008a81 fix: a crashed job's crash line carries its job id
The bind sat inside the try, so it was already reset when the crash line and the cleanup were logged — the one line most worth finding by job id had none.
2026-09-17 03:53:29 +02:00
peter fac5c4465a fix: stored logs keep bad rows, erased accounts and unconsented queries out
Deep-review round 1 of E22.

- A NUL character (`GET /assets/%00` logs the decoded path) made Postgres
  refuse the row and with it the sink's whole batch, every user's rows
  included. `to_row` replaces NUL in every text column and in attrs.
- An erased account's lines kept arriving after purge_user's DELETE: the
  erasing request's own access line, a queued row, a background task.
  `logs.forget_user` runs first; later rows are stored without the account,
  and the check and the insert share a lock with the erasure.
- An admin setting the whole process to TRACE logged every user's query
  string. The query is now logged only under the user's own Teljes consent,
  and decoded before redaction, so `%40` no longer hides an address.
- The browser log ingest has an hourly storage budget per account (5 MiB,
  one notice when spent) and reads only a JSON Content-Type, so a
  same-site text/plain form post cannot write into a user's log.
  RateLimiter takes a per-attempt cost for the byte budget.
- The root level is applied under the elevation lock: two concurrent
  changes could leave it at the base while an elevation was active.
- Tests for the running sink (level gate, drain on stop) and for the
  admin log level applied at runtime and at boot.
2026-09-17 03:53:29 +02:00
peter 9c9b1cf6d5 test: the gate's output carries no noise to hide a real warning in
- Home tests mocked the api without downloadIndex, so every card's download button logged a
  missing-queryFn error (64 stderr blocks per run).
- QueryState rendered without an i18n instance; Modal.fullscreen awaited a synchronous act,
  which React reports as an act environment it does not support.
- pytest: Starlette 1.x's TestClient is built on httpx2 and warned on every run without it
  (pinned in requirements-dev); alembic.ini declares its path separator.
2026-09-17 03:03:37 +02:00
peter c3d88d9b20 refactor: component files export only components; lint fails on a warning
The 43 react-refresh/only-export-components warnings the gate let through meant an edit to a
provider (or AnchoredPopover, Toaster, plexDetailUi, …) could not be hot-swapped: Fast Refresh
reloads a module that also exports hooks or helpers.

- Each provider's contexts, the types its hooks return and the hooks themselves live in a sibling
  context module (navigationContext.ts, channelsContext.ts, …); the provider file keeps the
  component. loadAccountFilters moved next to the feed filters' defaults (feedFiltersInit.ts),
  following navigationInit/playlistsInit.
- Helpers that sat in component files have their own modules: welcomeMode, downloadTitle,
  plexPlayable, channelSync, ui/anchoredPopoverHooks, ui/pageToolbarSlot, ui/pagerContent,
  ui/toastLevels, ui/overlayOrder, ui/tableAlign, lib/plexDetail; the cells of the channel columns
  and the registry's page wrappers are components files of their own (channelCells, modulePages).
- usePersistedTab, an alias of useAccountPersistedState, is gone; its callers use the original.

npm run lint is now eslint --max-warnings 0, so a warning fails the gate like an error does.
2026-09-17 02:59:09 +02:00
peter 569190480d refactor: hook dependencies name everything the hooks read
The 12 react-hooks/exhaustive-deps warnings the gate let through:
- ShareDialog and YearFilter built a fresh [] per render while their query loaded, so their
  memos recomputed on every render; they share one empty list now.
- VideoEditor's drag effect called a seek/moveBoundary recreated each render; both are stable
  callbacks in its deps.
- WizardProvider re-checks onboarding on exactly the fields it reads (not the me object, whose
  60 s refetch would reopen a closed wizard).
- Channels, Feed and PlexPlayer list the stable values they already used.
- The player's keydown effect is bound once on purpose (its helpers read refs): said so, with a
  disable, as the codebase's other deliberate sites do.
- dialogFocus and the YouTube player's cleanup read refs React may already have let go of; they
  use what they captured when the effect ran.
2026-09-17 02:45:19 +02:00
peter 5883ea509d feat: Settings → Diagnostics, a consented log level and the tab's own log
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a
new Settings tab, after a consent box that says what is recorded, that the server keeps it and for
how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent.

- users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which
  computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force.
- The server logs an elevated user's requests at that level and stores every line of them
  (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation
  is active). Teljes adds the redacted query string to the request line.
- POST /api/diagnostics/events stores the browser's log as source=web rows: account from the
  session, body capped while streaming, events/attrs capped, re-redacted, events under the
  consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed.
- lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in
  batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its
  own timer), on withdrawal and on 409.
- Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user
  text, input values or a link's query), visibility/bfcache/resize, filter changes
  (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are
  warnings at every level). Offline/online at the base level.
- "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy
  the filtered events as JSON lines, clear.
- Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to
  the same sections); the policy now describes the server logs and diagnostic logging, self-service
  deletion and password sign-in.

Fixed on the way:
- A sync auth dependency ran logs.set_user on a copied context, so the account never reached a
  handler's own log lines (measured); a per-request mutable scope carries it now.
- The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is
  a scrolling row above the content.
- PUT /api/me/preferences took a bare dict body; it is a Pydantic model now.
2026-09-17 02:42:12 +02:00
peter a210992bfc feat: client log with request ids, joined to the server's
The browser now keeps its half of the one event shape logs.py defines:
lib/log.ts is a levelled, redacted, bounded (2000 events) in-memory ring
with the server's level names and severity numbers. Nothing leaves the
browser yet; sending is the consented diagnostics level (S5).

- lib/http.ts `send` is the one fetch: every request carries an
  X-Request-Id and becomes one http.request event. The server echoes the
  matched route template in X-Route (only for a matched route, so a raw
  path with a value in it is never sent back); without one the client
  templates the path itself. req, beacon, the upload XHR, the public
  watch page and the legal pages all go through it.
- Hooks: app start snapshot, uncaught errors and rejections, History
  push/replace/popstate (a scroll-only replace is not an event; parameter
  values only at DEBUG, typed search only at TRACE), notifications, the
  error dialog, the crash boundary, failed queries and mutations (key
  head only), and the messages socket (lifecycle, frame types at DEBUG).
- Redaction is one contract: backend/tests/fixtures/log_redaction_cases.json
  runs against both redact() implementations.
- Lint: fetch, XMLHttpRequest and WebSocket each belong to one logging
  module, console to none; a contract test proves the rules fire.

Fixes a socket takeover: the close of a socket released by the last
subscriber cleared the newer socket a quick re-subscribe had opened and
scheduled a third, leaving two live sockets that delivered every message
twice (measured in Chromium: 3 created / 2 live before, 2 / 1 after).
2026-09-17 01:44:29 +02:00
peter 0517de1f59 fix: poll and backfill only channels somebody follows
The scheduled RSS poll and recent backfill covered every channel row,
but rows also come from search results, playlist sync and dropped
subscriptions, and nothing deletes them. On prod 582 of 979 polled
channels had no follower, three of them dead feeds answering 404 on
every poll. Both jobs now take subscribed or explored channels only,
and the scheduler card counts the same set. As a deliberate trade-off,
the shared Library no longer receives new uploads from channels nobody
follows.
2026-09-17 00:38:46 +02:00
peter b59fd38a9c feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.

Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.

Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
2026-09-17 00:20:14 +02:00
peter 88a166e4aa fix: call the audit log "Auditnapló" in Hungarian
"Naplózás" (logging) named the audit log in the nav, its page and its
Configuration tab, right beside the new "Naplózási szint" (log level)
under Diagnosztika — two different things under one word. The scheduler
hint also pointed at a "Beállítások → Naplózás" path that does not
exist; it is Konfiguráció → Auditnapló, as the English already says.
2026-09-17 00:04:01 +02:00
peter 07766c43e8 feat: closed-set admin knobs are pickers, not free text
log_level, search_source, download_layout and the transcode hw/preset
were typed by hand, and every consumer carried its own guess at what a
typo meant. The value set is now a Literal in config.py: pydantic
rejects a bad env value at boot (any letter case accepted), sysconfig
refuses a bad override and derives the admin picker's options from the
same Literal, and the consumers read the value as-is.

The Configuration page renders a choice key as a segmented bar, or a
dropdown past six options, with translated option names. Its tabs now
follow the registry's order instead of a hand-kept copy that had
already dropped the explore group.

A single-choice FilterSelect now opens on the option in use: in the
browser it could land on the first row instead.
2026-09-17 00:03:56 +02:00
peter bb3c6dec45 release: 0.87.1 (bug/youtube-cookie-host)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m22s
2026-09-16 04:41:40 +02:00
peter 3cf8f04d73 chore: 0.87.1 release notes 2026-09-16 04:41:10 +02:00
peter 2b56c5c2eb fix: name the error yt-dlp actually raises for a cookie-emptied client set
A review round read yt-dlp's source and found the comment behind the previous commit's safety-net
marker did not match it. Measured in the worker with a real account cookie and
`player_client=android_vr`: yt-dlp logs `Skipping client "android_vr" since it does not support
cookies`, drops it, and the request left with no client fails with "Requested format is not
available" — not "No player clients have been requested". That raise sits EARLIER in yt-dlp, before
the cookie-capability removal, so the cookie path can never reach it; and an unknown or renamed
client is skipped with a warning and replaced by yt-dlp's defaults, so the other scenario the
comment named cannot reach it either.

So the marker bought nothing: the symptom that does occur was already retriable here before this
branch. It is dropped, the comment now states what was measured, and the test asserts the real
symptom instead of a message the code cannot produce. The cookie-capability filter itself is
vindicated by the same measurement — without it every signed-in download would burn its first
attempt on a client yt-dlp discards.

Also from the round: an enqueue test for a `source_kind="youtube"` job, the one join nothing else
made (a YouTube job stores a bare 11-char ref and the URL the cookie gate keys on is rebuilt by the
adapter — both ends were pinned apart, the join was not, and a break there would stop every YouTube
cookie silently while signed-out downloads kept working). And copy corrections: the group's only
unhedged claim is hedged, "described the video" loses the pipeline vocabulary, and four Hungarian
phrasings that read as translated English are rewritten.
2026-09-16 04:29:13 +02:00
peter c06a456a9a fix: a YouTube stream refused after extraction is the gate, not a network fault
Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was
answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN
then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden".
Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in
full. So the wall moves between the player response and the stream, and the text it wears there was
costing the download twice.

retriable: the format URLs belong to the client that minted them, so the next client set is a real
second chance — but this text classifies as `network` and nothing in the YouTube marker list matched
it, so the ladder ended the job on attempt 1 and never tried the fallback.

catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network
error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly
as narrowly as the measurement: another host keeps the generic `network` category, and so does a
download that already carried a cookie — telling a signed-in user to sign in is the failure the
catalog exists to avoid. EN + HU copy.
2026-09-16 03:43:29 +02:00
peter 864f673ef5 fix: a YouTube video behind the bot wall is downloadable with cookies
A YouTube link can fail with "Sign in to confirm you're not a bot" while every
other YouTube download works — YouTube gates INDIVIDUAL videos behind a
signed-in session. Measured on prod job 273: all twelve of yt-dlp's player
clients fail on that one video while its neighbours download fine, the bgutil
PO-token sidecar is live and answering, and the newest yt-dlp fails identically.
So the download is not broken by our stack or a stale yt-dlp, and no retry,
client switch or upgrade gets past it. A cookie is the only way in.

The failure copy already sends the user to Settings -> Account -> Download
authentication, but YouTube was not in the cookie host registry, so that
dropdown could not help them: the app asked for something it did not offer.

credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be,
youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the
private-asset cache salt all derive from the row, so that is the whole wiring.

adapters: a signed-in YouTube download runs only cookie-capable player clients.
Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies
attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set
it empties that way raises "No player clients have been requested" — which is
not a per-client flake it retries past. Our primary set is android_vr, one of
those, so without this the cookie would have failed the download on attempt 1
and never reached the POT-backed web fallback it exists for. A test pins the
client list against the installed yt-dlp so a version bump turns the gate red
instead of silently un-authing every YouTube download, and the empty-set message
joins the retriable markers as the second line of defence.

errors: catalog entries for the wall with and without a cookie, so a user who
has already added cookies is told they were rejected (YouTube rotates exported
ones) rather than to add some. The marker is quote-agnostic: YouTube writes the
apostrophe as U+2019, so "you're not a bot" with a straight quote never matches
the real text. EN + HU copy.
2026-09-16 02:03:00 +02:00
peter 6c63df6704 feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.

Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.

The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.

`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.

Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
2026-09-16 01:37:02 +02:00
peter a5bcd9fca2 fix: numbers follow the UI language, not the browser's
`n.toLocaleString()` groups in the BROWSER's locale, so an English UI in a
Hungarian browser printed "1 234 567" while a Hungarian UI in an English
browser printed "1,234,567" — the number half of the date defect fixed in
618ef812. One `formatNumber` in lib/format.ts (a per-language cached
Intl.NumberFormat, like the relative-time formatters beside it) now serves
every grouped figure: Stats, Scheduler, the channel columns and the stats
popover, VideoCard's view tooltip, the ItemCount pill, and formatTotalHours.

ReleaseNotes' `toLocaleDateString(undefined, …)` was the one DATE site the
previous sweep missed; it moves onto formatDate.

The new cases assert both languages and that the two differ, instead of
comparing against the runtime's own default: that default is Hungarian on the
dev box and English on a CI image, so a one-language assertion would pass
under the old bug on half the machines. Verified in Chromium on :5173 with the
browser forced to hu-HU and the UI in English — Stats, Scheduler and the
channel table all keep comma grouping.
2026-09-16 01:05:53 +02:00
peter 618ef812e6 fix: dates follow the UI language, not the browser's
Seven sites formatted with a bare toLocaleString()/toLocaleDateString(), so an
English UI on a Hungarian machine printed Hungarian-shaped dates (and the reverse).
They now go through formatDateTime/formatDate, which take the i18n language.
2026-09-16 00:49:23 +02:00
peter e299c66121 feat: show each account's last sign-in in Users
A right-aligned column at the end of every Users & roles row says how long ago the
account signed in, with the exact moment as the hint and as a line in the account
panel; a dash where nothing is recorded, and the demo account says it is not recorded.
formatDateTime formats that moment in the UI language rather than the browser's.
2026-09-16 00:45:03 +02:00
peter 11aef4ab7a feat: record when an account last signed in
users.last_login_at (migration 0085, no backfill) is stamped by _establish_session,
which the Google callback now shares with password login instead of repeating its
session setup inline. The demo entry does not stamp its shared account. The admin
user list carries the field.
2026-09-16 00:34:50 +02:00
peter 4e52bdd4ca fix: a signed-out tab no longer asks for saved views
The providers mount for the public landing too, and the saved-views query had no
enabled gate, so every signed-out load logged a 401 for /api/saved-views. The query
now waits for a signed-in account in BootGate's me cache, like WizardProvider.
2026-09-15 23:45:13 +02:00
peter 595490cb98 release: 0.87.0 (improvement/watch-background-audio)
gate / frontend (push) Failing after 3m19s
gate / backend (push) Successful in 4m25s
2026-09-15 03:02:19 +02:00
peter 7537882235 chore(release): 0.87.0 2026-09-15 03:01:59 +02:00
peter 47d968eeea fix: review round 4 — an account switch unloads as the old account, a share link's lockout costs five networks
Deep-review round 4, targeted at round 3's changes (f569b19c..e7f38396).

Accounts
- Switching, adding or signing out an account moved the tab's pin before
  the page unloaded, so the players' resume beacons fired on the way out
  under the NEW account: one account's playback position saved into the
  other. The pin is now handed to the next page (pinOnNextPage) and taken
  over at import; clearActiveAccount is gone.
- Switching to a suspended account was refused but left it in the
  switcher, refused on every click; a deactivated one was not refused at
  all, and made the default it signed every account in the browser out on
  the next request. Every account _session_user would refuse — gone,
  deactivated, suspended, revoked — is refused and dropped from the list.

Share page (/watch)
- An IPv6 client is limited by its /64 (client_ip becomes client_key, for
  every limiter): keyed on the full address, one subscriber could rotate
  through endless identities. The comments now state what the per-link
  ceiling costs: guessing from five networks can keep a link shut for its
  window, and guesses already running when it fills are still checked.
- The link is looked up before the limiter records the attempt, so made-up
  tokens no longer grow it by an entry each.

Forms
- Failing the same way twice is announced again: the sign-in, passphrase
  and setup errors are keyed on the attempt (a clear and a set in one
  handler left the alert node untouched).
- The install wizard's own validation showed the generic error — it threw
  plain objects its catch did not read. It sets the step's message.

Tests: a re-pin leaving the previous account's notices behind; the pin
hand-over; switching to an ended account; the /64 viewer; made-up tokens;
the repeated alerts. Each mutation-checked RED for its own reason.
2026-09-15 02:53:22 +02:00
peter e7f38396a2 fix: review round 3 — a new tab keeps its account's notices, a revoked tab account lands on the default
Deep-review round 3 over main..improvement/watch-background-audio.

Account boot
- The notification history and settings, and the hints toggle, are read
  when their modules load, which in a new tab (bookmark, typed URL) is
  before BootGate pins the account: they read as nobody's, and the tab's
  first notice replaced the account's stored history (up to 100 entries)
  with a one-item list. They now read again when the tab is pinned
  (onAccountPinned, run by setActiveAccount); a notice raised before the
  pin stays, after the history.
- A tab pinned to an account still in this browser's wallet but whose
  session ended (suspended, deleted, or revoked by "log out everywhere")
  got "signed out" from /api/me while the default account was still
  signed in, and signing in again as anyone else left it there. A refused
  per-tab account now leaves the wallet, and the probe answers as the
  default, so the tab reloads onto it.

Share page (/watch)
- The password unlock is limited per viewer (link + client IP), under a
  per-link ceiling that counts only wrong passwords. Keyed on the link
  alone, ten wrong guesses every five minutes locked out every viewer,
  the ones with the password included. client_ip moves to app.ratelimit
  so the public routes can key on it.
- Its failure messages and the load-failure screen are announced
  (role=alert, the password input described by its error). The sign-in,
  passphrase, password-change and setup form errors get role=alert too.

Comments that still described removed machinery are corrected: App's
collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig.

Tests: the account's history and hints across the pin; a revoked and a
suspended pinned account; the per-viewer limit and the ceiling; the
limiter's exhausted(); the share page's announced failures; a share
link's feed filters persisted. Each mutation-checked RED for its own
reason.
2026-09-15 02:25:00 +02:00
peter f569b19c49 fix: review round 2 — a signed-out tab acts as nobody, honest share-page failures
Deep-review round 2 over main..improvement/watch-background-audio.

Account resolution (server)
- A request naming an account that is no longer signed in to this browser
  (X-Siftlode-Account or ?account=) now acts as nobody instead of the
  session's default account. A tab still pinned to an account signed out in
  another tab went on acting as the default: its player checkpoints and
  preference saves landed in that account, the messages socket and Plex
  playback ran as it, and its Sign out signed THAT account out. A hidden tab
  never polls /api/me, so a tab playing in the background kept doing it.
  The /api/me bootstrap probe alone keeps the fallback, which is how such a
  tab learns who it now is: the app's 401 handler reloads, and BootGate
  re-pins the tab.

BootGate
- On an identity change the tab reloads without moving its pin first. The
  tree being torn down still runs its unmount saves before the page goes,
  and with the pin already moved they wrote into the new account's local
  keys (a player volume still pending). The reloaded page corrects the pin.

Share page (/watch)
- Only a 404 says the link is gone. A 5xx, a dropped connection, and the
  unlock's own rate limit all rendered "no longer available ... expired or
  revoked" to a viewer with a valid link, with no way to try again. A load
  failure now offers Retry; too many password tries says so and keeps the
  form; an unlock that gets no answer says so instead of doing nothing.
  Strings in en/hu/de.

Downloads
- A cookie-authed asset is recognised under the current signature version
  and the one before it, so a job queued before a deploy that bumps the
  version still downloads with its cookie instead of failing on the login
  wall with an error that says no cookie was used.

The app chunk is preloaded through lazyWithPreload, so a failed fetch no
longer leaves an unhandled rejection beside the ErrorBoundary. The panel
layout seed is one helper (seedLayout).

Tests: the stale-tab refusal, its harmless Sign out and the probe fallback;
the pre-bump authed asset; BootGate keeping the pin on reload and staying
mounted through a same-account change; the reveal clock's later-commit
hand-off; the prefs, panel layout and demo-scope seeds from the me cache;
the share page's failure states. Each mutation-checked RED for its own
reason.
2026-09-15 01:35:25 +02:00
peter e795d191c5 fix(boot): review round 1 — reconcile the tab pin, one loading clock, honest probe failures
Deep-review round 1 over main..improvement/watch-background-audio.

BootGate
- The tab's account pin is corrected to the account the server answered as.
  The server honours the pin only for an account still signed in to this
  browser and otherwise answers as the default, so a tab pinned to an account
  signed out in another tab ran as the default on the server while reading
  and writing the gone account's local state.
- A mounted tab whose identity changes (its session ended, or the case above)
  reloads instead of rebuilding the tree in place, like every account switch;
  that also drops the previous account's query cache. The account key on the
  tree goes: the readyFor gate already unmounted it on every change.
- An unanswered setup probe shows the error with Retry, like an unanswered
  account probe, instead of the signed-out landing to a signed-in user.
- Comments: /api/me never 401s; the ended-session path is identity → nobody.

Loading states
- LoadingState's reveal delay is one clock across loading states that hand
  over to each other (boot screen → page-chunk fallback → the page's own
  fetching state), instead of a CSS delay restarted by each, which blanked a
  visible spinner for another 300 ms at every hand-off. BootScreen is now the
  shared loading visual, full screen, counting from navigation start.

Plex shared link
- Persists only the dimensions the link carries again. Writing the default
  filters for an omitted `pf` created the stored-filters key, which is what
  tells SavedViewsProvider the account picked filters, so the account's
  default Plex saved view was never applied again.

Watch page
- The Repeat toggle uses the shared pressed-pill paint (a solid fill when on)
  instead of Button variants that differ only in colour, and its flag goes
  through storage.ts readJSON/writeJSON.
- The actions wrap under the title when the row is too narrow, instead of
  squeezing the title into a column a word wide on a phone; the title breaks
  long words.

Comments that still justified code by the removed adopt effects (usePref,
PrefsProvider, FeedViewProvider) are rewritten against the seed-once model.

Tests: BootGate error/retry for both probes, setup mode, a failed background
refetch keeping the app, the pin correction and the reload on an identity
change; the shared reveal clock; the Plex link persist; the audio signature
actually moved off version 4; a video download (av and v) still remuxed to
an mp4. Each mutation-checked RED for its own reason.
2026-09-15 00:53:15 +02:00
peter 4ad7183102 feat(players): keep a video playing in the background on Brave iOS; repeat on the watch page
On iPad Brave, a shared video stopped the moment the browser left the
foreground, even with Brave's "Enable Background Audio" on, while an mp3
kept playing. Brave iOS keeps a web <video> going by re-playing one the OS
paused (brave-core ios/browser/web/media/resources/media_backgrounding.ts),
but it only hooks a <video> present at document start or added as a
MutationObserver addedNode that IS a video. React inserts a player's
<video> inside its freshly mounted wrapper in one go, so Brave never saw it.

lib/browserBackgroundPlay re-inserts a mounted <video> at its own position,
once per element: it becomes an added node, the DOM React manages does not
change, and playback is not interrupted. The watch page, the downloads
library player and the Plex player hand their <video> to it. Measured on the
owner's iPad (Brave) with a prototype of the same idea: the video kept
playing in another app and with the screen locked. The YouTube modal is not
ours to change: its <video> lives in YouTube's cross-origin iframe, and
nothing of ours pauses it on backgrounding.

The prototype's two audio-element variants (a muted video following an
<audio>, and an <audio> taking over on hide) went silent in the background on
the iPad and are removed.

The watch page gets a Repeat toggle, on by default and remembered per
browser (LS.watchRepeat; the page has no account): it sets `loop` on the
<video> or the audio-only <audio>.
2026-09-14 23:58:30 +02:00
peter 2c3aaec62d feat(watch): prototype background sound for a shared video on iPad (dev only)
On iPad Brave a shared mp3 keeps playing when the browser leaves the
foreground, but a shared h264 video stops at once, even with Brave's own
"Enable Background Audio" on. Three variants behind a `?bg=` param on the
/watch page, for a real-iPad comparison; without the param the page is
unchanged.

  a: the sound comes from a hidden <audio> on the same file, the <video> runs
     muted and follows it (drift correction, user seeks and mute moved onto
     the audio; our own catch-up seeks are not copied back, which re-seeked
     the sound on every correction and sent both back seconds in WebKit).
  b: the <video> plays with sound; an <audio> unlocked by the start tap takes
     over while the page is hidden and hands back on return.
  c: no audio trick: the <video> is inserted as a node of its own into an
     already-mounted wrapper. Brave iOS's background play only hooks a video
     present at document start or added as a MutationObserver addedNode that
     IS a video (brave-core ios/browser/web/media/resources/
     media_backgrounding.ts); the page inserted the video inside a new wrapper
     in one go, so Brave never saw it. A re-implementation of that detection
     rule reports the default page's video unhooked and mode c's hooked.

Modes a and b were driven in Chromium and WebKit with a simulated backgrounding
(document.hidden forced plus the OS's pause): sound continues, the picture
catches up on return, a user pause stops both. iPadOS's own media policy can
only be judged on the iPad.
2026-09-14 23:40:51 +02:00
peter 70598d47a3 fix(dev): proxy the messages WebSocket through Vite, which stops a reset killing the dev server
The Vite dev proxy had no `ws: true` on /api, so the in-app messages socket
(/api/messages/ws) was an upgrade nobody handled: it never connected on :5173,
and the first reset on it (a closed tab, a dropped connection) threw an
unhandled ECONNRESET that took the whole dev server down. Reproduced with a raw
upgrade + RST (dev server dead); with `ws: true` the backend answers the
upgrade, the dev server survives the reset and a closed tab, and the signed-in
socket is accepted through the proxy.
2026-09-14 23:08:41 +02:00
peter 5858e7b513 test(e2e): run the net on WebKit and an iPad profile; opt the account out of Home
The Playwright net now runs every flow in three projects: chromium (Desktop
Chrome), webkit (Desktop Safari) and ipad (iPad gen 7 landscape: touch, mobile
viewport). WebKit catches the engine-level differences every iOS browser
inherits; the config and the README say plainly what it does NOT catch: iPad
media refusals. Playwright's WebKit decodes through the host OS, and on Windows
it played an mp3-in-mp4 and an AV1 mp4 that a real iPad refuses, so media
formats are checked on a real iPad against the dev stack over the LAN.

global.setup opts the e2e account out of starting on Home (the Startup setting),
which the feed, channel and player specs need since E24. It only takes effect
with the new-tab boot fix: before it, all twelve of those specs landed on Home
in every project. 49/49 green on :8080 across the three projects.
2026-09-14 22:29:33 +02:00
peter 5346efcd10 fix(boot): a new tab knows its account before anything renders
With Settings -> Appearance -> Startup turned off, every NEW tab or window still
opened on Home; only a same-tab reload honoured it. The provider tree mounted
before /api/me answered, so NavigationProvider's initializer could not read the
per-account opt-out, landed on Home and mirrored ?page=home into the address bar.
When App pinned the account a moment later, the re-hydrate re-read that mirror
and stayed on Home. The same mount-then-patch shape made a new tab start on
default filters, the default theme and the browser language, and correct each
after the first paint.

BootGate (main.tsx, above the providers) now owns the setup-status and /api/me
queries and mounts the tree only once the identity is settled: it pins the tab's
account and adopts the account's language in a layout effect, so the first frame
of the app is already right. The tree is keyed on the account id, which makes
"per-account state is read once, for this account" a structural guarantee.

With that, the whole re-hydrate machinery is gone: useAccountRehydrate, the
reactive active-account store, the markDirty guards, the pin-deferred saved-view
seed and the present-dimension bookkeeping of the shared Channels/Plex links.
PrefsProvider, FeedFiltersProvider, usePref and App's panel layouts seed from the
me cache instead of adopting it in an effect; the theme is applied in layout
effects. The app chunk is fetched in parallel with the probe, and the boot screen
only fades its label in after 400 ms, on one clock across the gate and the
Suspense fallback.

Module switches no longer flash a placeholder either. Measured frame by frame
in Chromium and WebKit, every first visit to a module showed "Loading..." for
17-70 ms: the lazy chunk's Suspense fallback, then often the first fetch's
LoadingState. Pages are now lazyWithPreload components, and App fetches the
chunks of every module the account can reach (plus the channel page) once the
first page is idle; a preloaded page renders synchronously and never suspends.
LoadingState, the one shared loading visual, fades in only after 300 ms, so a
fast load shows nothing in between and a slow one still gets its spinner.
2026-09-14 22:29:33 +02:00
peter 6169e32ca5 fix(downloads): a signed-in Facebook reel keeps its title, and an mp3 stays an mp3
Since a Facebook cookie was added on prod, every reel came down titled
"Video": measured on dev, a signed-in read gives yt-dlp the placeholder
"Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text
is still in the description. A new SourceAdapter.title hook takes the whole
first line of the description in that case, and both title writes (the early
fill and the finished download) go through it, so the row, the file name and
the share preview carry the post's text again. The line is not cut with an
ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to
".", which read as a finished sentence (seen in dev UAT).

The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the
browser-safe step (the audio spec is clamped to container mp4 too) and served
as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now
skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4.
Audio downloads version their cache identity on their own (v5), so a cached
remuxed audio re-derives without re-downloading every cached video.
2026-09-14 17:46:46 +02:00
peter 90523284ab release: 0.86.0 (feature/e11.5-download-failures)
gate / frontend (push) Failing after 3m17s
gate / backend (push) Successful in 4m17s
2026-09-14 01:53:06 +02:00
peter 17bfea8754 release: 0.86.0 notes 2026-09-14 01:52:34 +02:00
peter 2a9ddb78ac fix(downloads): E11.5 epic-end review round 2
Pause now re-reads the job under the same asset-then-job row lock as resume,
cancel, delete and redownload. Its unlocked write waited for a concurrent
cancel's lock and then set the released job back to paused, a holding status,
so a later delete released the same hold again and could free a file another
user's job still held (measured: the overlapping-request test ends 'paused'
instead of 'canceled' with the old read). The lock test now calls the route
handlers, so it also pins which helper each route uses.

Comments follow the code: resolve_url's contract names both verdicts and that a
failed page read is raised as itself; the inherited-failure diagnosis asks the
cookie gate as it stands when the job is claimed, so its advice fits the cookie
that job's Retry would use; the undecryptable-cookie warning no longer claims a
download goes ahead, since the worker parent reaches it only to diagnose.
2026-09-14 01:42:57 +02:00
peter e887233210 fix(downloads): E11.5 epic-end review round 1
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.

Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.

UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.

Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
2026-09-14 00:50:05 +02:00
peter b17d92d7c2 fix(downloads): confirm dialogs name the action on their confirm button
The download center's confirm button repeated the dialog's question
("Re-download this file?"). It now reads the action: Stop download,
Re-download, Remove, Remove from my list.
2026-09-13 23:38:11 +02:00
peter 90ee34a1b4 feat(downloads): a Facebook group post downloads its own video, or says it has none
Given a group post's link, yt-dlp either found no formats or took another
video from the page's feed data, so a "successful" retry of such a post was a
wrong download. With the user's cookie the Facebook adapter now reads the post
page, takes the single video node in the script blocks that carry the post id,
and downloads that video's watch link (the Chrome retry does the rest). No
single video in the post fails with its own message and the group-post copy,
reworded to say the post has no video; the post link is never handed to yt-dlp
as a fallback. Without a cookie nothing changes.
2026-09-13 23:38:11 +02:00
peter 118fc70e08 fix(downloads): take the duration from the file when yt-dlp gives none
A Facebook or Instagram reel extracts with no duration, so the asset kept a
NULL duration_s although the file plays for minutes, and the Library row had
no trim action. The finalize probe now returns the duration with the codecs
(the file wins, info fills a gap), and runs before the sidecars so the .nfo
runtime gets it too.
2026-09-13 23:12:16 +02:00
peter 636372511b chore: remove the unreferenced backend/Dockerfile
Nothing builds from it: both compose files build the root Dockerfile, and the
test image has its own Dockerfile.test. It installed the unpinned
requirements.txt, contradicting the lock header that every image installs
requirements.lock.
2026-09-13 23:12:16 +02:00
peter 1354f5b503 feat(downloads): a Facebook cookie download stuck at "Cannot parse data" is retried as Chrome
Measured on dev: with the user's cookie some reels still answer "Cannot parse
data" and only extract when the request impersonates Chrome. Impersonation is
a second attempt behind that one error, only when a cookie is attached — links
that already download with the cookie keep their plain path, and without a
cookie the same error is the login wall.

curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it
every target silently reports "unavailable"); the lock only gains the new
packages. A test fails the gate if the locked pair can no longer impersonate.
2026-09-13 22:48:21 +02:00
peter 98ec1ffcab fix(downloads): Retry and Re-download take a cookie added after the failure
The login-wall copy tells the user to add their cookies and press Retry, but
Retry kept the job on the shared asset the worker never attaches a cookie to,
so it failed on the same wall. Resume now moves a pending/errored job onto the
asset enqueue would pick with the cookie (never off a file or a download in
flight).

Re-download salted its fresh asset with the job's previous asset id, which the
cookie gate cannot recompute, so even a cookie download rebuilt without the
cookie. The rebuild salt is now the new asset's own id (plus the user's cookie
salt when one applies), and is_authed_asset recognises it.
2026-09-13 22:31:49 +02:00
peter 18c11ca437 feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do
A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.

- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
  (all-of markers, cookie state, link fragment) and falls back to the category.
  First entries: facebook.login_wall, facebook.cookie_rejected,
  facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
  failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
  with it; resume and redownload clear code and detail (redownload never
  cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
  "Download failed" where the first job said why. It is diagnosed again, with
  the cookie state from credentials.is_authed_asset (now shared with the
  cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
  copy, else the category. Copy in EN and HU; a contract test pins every
  backend entry to both locales.
2026-09-13 21:53:25 +02:00
peter ca5e446fbd release: 0.85.0 (feature/e11-power-navigation)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m36s
2026-09-13 21:02:22 +02:00
peter 4c7ab12a3e release: 0.85.0 — E11 Power Navigation
Back, dialogs and menus keep your place, and the keyboard walks every list: five sprints plus three epic-end review rounds (19 → 14, 18 → 12, 14 → 14). Minor: j/k list navigation, s and g-chords, and Back-to-item are features.
2026-09-13 21:02:22 +02:00
peter c6b552e75f fix(keys): epic-end review round 3 — the part-way clip count shows, busy buttons keep focus, cards say their names
Targeted round over 3cd0c525's fix surfaces: 14 raised, 14 fixed.

- VideoEditor: the "only X of Y clips" toast went silent exactly on the job-cap/quota refusal it was
  written for (req had already reported the error); it now always shows. The cut list holds still while
  Create queues, and a retry continues the "(i/N)" series.
- ChannelPage: Subscribe/Unsubscribe use Button `loading`, "Load more" and ChannelDiscovery's Subscribe
  block the click with aria-disabled instead of native disabled; busy Block/Hide no longer light up on
  hover.
- listNav: an item can name itself (`navItemAttrs(i, name)`) — Plex posters, seasons, episodes, message
  rows and download rows pass their title; focus on a toast's Undo keeps the removed card's slot.
- AnchoredPopover: the hand-back only for focus-managed surfaces (a hover card no longer pulls focus
  back); the dead panel-ref check is gone.
- AccountPanel: no confirmation mail offered to the demo account; send_failed copy no longer blames
  the message for an unreachable server. Pager comment points at scrollParent.
2026-09-13 20:47:13 +02:00
peter 3cd0c525e2 fix(keys): epic-end review round 2 — focus comes back to the card, the right rail and the reason
Whole-epic deep-review of E11 (18 raised, 12 kept; B and G refuted, four latent test-hardening notes dropped),
plus the admin refusal copy gap round 1 surfaced.

- A context menu opened from a card the keyboard is on gives the focus back to the card. floating-ui returns
  focus through the opener's first tabbable descendant even when handed a ref, so for a non-tabbable opener the
  surface restores it itself (measured in Chrome: j, Shift+F10, Escape left it on the thumbnail link).
- Back finds a named list (a rail) by its name: a rail that appeared above it after the cache expired sent the
  focus into the wrong rail. An unnamed list is matched only when the page has the same number of lists.
- The slot kept for a removed item is forgotten once focus moves on to page content (search, a chip), so j after
  new results starts at the first visible item; a menu or dialog opened from the item keeps it.
- The list keys reach the cheat sheet only while a list has an item.
- A j-focused card root is a named group while it holds the focus.
- The channel page's Block and Hide toggles keep the focus while their request runs.
- A split export that fails part-way says how many clips were added and deselects them, so a retry adds only
  the rest.
- Config save, the one-click share link and "log out other sessions" report failures with their reason.
- VirtualGrid uses the one scroll-parent walk listNav measures against.
- Admin refusals (activate, resend, invite, set-password link) say why in HU and EN instead of "try again";
  the email-off special cases collapse into that one reason. The contract reads the codes from localizeDetail.
  HU copy points to Konfiguráció → E-mail / SMTP; HU "unreachable" matches the EN change of round 1.
2026-09-13 19:18:04 +02:00
peter f130a1c05a fix(keys): epic-end review round 1 — Back, menus and dialogs keep the focus where it belongs
Deep-review of E11 (19 raised, 14 confirmed; 2 refuted, 3 latent test-hardening notes dropped).

- Back from a sub-view (a Plex show page, a Messages thread) restores the item it was opened from
  even when focus was still inside the view: the first attempt now waits one task, past the
  sub-view's own popstate listener (measured in Chrome: it left focus on <body>).
- The restore listener is installed at load, so the first Back after a reload finds it; a push
  cancels a restore still waiting for the last landing's lists.
- Navigating over an overlay asks it again, a render apart, while its entry stays: the Plex
  player peels one panel per Back step, so Shift+W over info + stats closed one panel and dropped
  the navigation.
- j/k: a native <button> item keeps its own Tab stop; landing twice on the same item no longer
  leaks its tabindex; after the focused item is removed (a hidden card) the next j goes on from
  its slot when that slot is on screen.
- An abandoned `g` no longer swallows a modifier chord (g, then Mod+K opened the browser search).
- A hover card above a dialog no longer switches its Tab trap off (player, Modal, wizard,
  Lightbox): new layerStack `anyBlockingLayerAbove`.
- "Manage views…" closes its menu before opening the dialog, through a shared `closeThenAct`
  that ActionMenu now uses too.
- Copy: the EN welcome says "Feed"; the EN unreachable reason no longer claims a change;
  stale comments about PlayerModal's scroll lock and dialog contract.
2026-09-13 17:17:49 +02:00
peter bbd31c7143 feat(keys): Back returns focus to the item a view was opened from; "Feed" everywhere; every backend id named
Owner answers to the E11 S5 raises (2026-09-13): 1 → A, 2 → "Feed" in every language, 3 → translate the
proxy label and fix every config key missing in BOTH languages ("maintenance", "tmdb" — old bugs).

Focus across a view change (1-A). Opening a Plex title from Home's rail, a channel from the Channels
table or a title from a Plex grid switches the page or the sub-view, which unmounts the list; Back used
to hand focus to <body>, so the next `j` started over at the top. The YouTube player (an overlay)
already restored onto the card.
- `lib/listNav`: `navPosition()` (list's place in reading order, the page's list count, item index)
  and `restoreNavPosition()`: focuses the item once the page has as many lists on screen as when it was
  left, polls on a timer (a hidden tab runs no animation frames), gives up after 2 s or at once when
  something already holds focus, focuses without scrolling (the scroll restore owns the position).
- `lib/history`: `rememberNavPosition()` stamps `sfNav` onto the entry being left and returns the state
  without it (a new entry must not inherit it); one popstate listener restores and spends the stamp,
  so a later landing on the same entry (an overlay closing over it) cannot pull focus to an old item.
- Called before every view push: `setPage`, `openChannel`, `enterYtSearch`, `useHistorySubview.open`,
  the Channels tab push. `navPosition.wiring.contract.test` fails on any `history.pushState(` without
  it (the overlay stack's own push is the one exemption).

"Feed" (2). The module is "Feed" in both languages: every "hírfolyam" in the Hungarian UI (≈45 strings,
suffixed: Feedből, Feeded, Feedben…) and the prose "feed" in English that means the module ("your
Feed", "Hide from Feed"); "RSS feed" keeps its meaning. The welcome email says it too.

Backend ids without a name (3). The Configuration page showed `maintenance`, `tmdb`,
`maintenance_revalidate_batch`, `download_upload_max_bytes`, `audit_retention_days`, `plex_overlay_map`,
`tmdb_api_key` and `tmdb_language` as raw identifiers — the lookup falls back to the id, and the i18n
parity test stayed green because both languages missed the same keys. Added labels + hints (EN/HU),
the two groups (and their tab order), the missing `scheduler.jobDesc.demo_reset`, removed the stale
`plex_sync_interval_min` field, and "YouTube API kimenő proxy". `i18n/backendIds.contract.test` reads
`sysconfig.SPECS`, `scheduler.JOB_INTERVALS` and the audit/quota action enums from the backend sources
and fails on an id without its name in either language, or a name left for an id the backend dropped.
2026-09-13 07:36:41 +02:00
peter 8731508645 feat(keys): the conversation list walks with j/k; the cheat sheet names keys in the reader's language
E11 S5 — the HU/EN sweep and the keyboard-only pass over every module.

- The keyboard cheat sheet printed "Space" and "Menu" on a Hungarian sheet: a keycap that carries a
  WORD (not a legend every layout prints alike, like Esc, Ctrl or Enter) is now named by the caller in
  the reader's language (`formatChord(chord, words)`, `keys.cap.*`: Szóköz / Menü).
- The Plex player's ⌫/Esc line said "Stop & back to feed", but the player goes back to wherever it
  was opened from (Home, a Plex page): "Stop & go back" / "Leállítás és vissza".
- Messages: the conversation list is a keyboard list (NavGrid + navItemAttrs, the name button is the
  open element; the pop-out button keeps its own Enter). D1 covers every list whose rows open one
  thing; this one was missed in S4. Notification cards have no single open action and stay out, like
  the download queue.

Sweep, measured: every static `t("…")` key used in the source exists in both languages (the only
misses are template prefixes); the HU values equal to EN are proper nouns and loanwords (Admin, Plex,
Trakt, Shorts). Palette in Hungarian: "téma" makes "Téma: Midnight" the active row ahead of the channel
"Topik - mindig témában", live region "7 találat".

Keyboard-only pass in Chrome, Hungarian UI — 1024×768: every `g` chord lands on its module, `j` rings
the first item (solid 2px accent) on Home, Feed, Channels, Playlists, Plex and Downloads, `s` focuses
the search on Feed, Channels, Playlists and Plex, no page scrolls sideways. 3440×1440: `j` → → ↓
walks Home, Feed (8 columns: 0 → 1 → 9), Channels, Playlists, Plex and Downloads, no sideways scroll.
The help sheet lists every line translated; inside the Plex player "Szóköz / K" and "⌫ / Esc
Leállítás és vissza". Messages: a real `j` rings the row, Enter opens the thread.
2026-09-13 07:00:51 +02:00
peter b5d37ca142 feat(keys): j/k and the arrows walk the channel, playlist and download lists
Third and last slice of the E11 keyboard layer (S4c). S4a/S4b covered every card grid and rail; the
table-shaped pages — the channel manager, a playlist, the download center — were still Tab-stop
country: a channel row carries a priority stepper, four action buttons and a tag picker before the next
row's name.

- `ui/DataTable keyboardNav`: the rows are the items, Enter clicks the element a column marks
  `data-nav-open`. DataTable renders its rows twice (a table, and a card list below `md`); both register
  and `lib/listNav` now walks only lists on screen (`checkVisibility`), so `j` past the last row never
  carries on into the hidden copy.
- `lib/listNav` leaves a key something before it already handled (`defaultPrevented`): during a keyboard
  drag a playlist row's grip moves the row with the arrows, and dnd-kit takes the key on the document
  before listNav's window listener. Without the guard ↓ moved the focus off the grip and dropped the drag.
- `ui/NavGrid as="tbody"` for a table whose body is the list (the playlist, under dnd-kit's
  SortableContext).
- Wired: Subscriptions and Discovery tables, the card layout (`ChannelLayoutGrid` → VirtualGrid
  `keyboardNav`), Hidden/Blocked cards (the ChannelLink name is a channel's open element); a playlist's
  rows (the title plays; the grip keeps Space/Enter); the download queue, library and shared lists (the
  play button opens, `IconBtn open`; a row without one opens nothing).
- `listNav.wiring.contract.test`: `VirtualGrid` and `DataTable` must say `keyboardNav` like `Carousel`,
  `DownloadRow` is a card; the audit log's table is the one exemption (its rows open nothing), and an
  exemption that no longer matches a tag fails the test.

Measured in Chrome: Channels table j → row, ring round the whole row; a row walked down keeps 12px above
the viewport edge, walked up clears the sticky header by 35px; Enter opens the channel. Cards 3 columns:
→ 0→1, ↓ 1→4. Hidden and Discovery: j + Enter open the channel. Playlist: j + Enter plays, Escape returns
focus to the row; a keyboard drag (Enter on the grip, ↓) moves the row with focus kept on the grip, and
Escape cancels with the order unchanged. Downloads library: Enter plays a video row, does nothing on a
plain file row.
2026-09-13 06:08:05 +02:00
peter eb0507f942 fix(keys): a card reached by Tab rings the whole card, not just its poster
Owner report: on the Plex browse grid, Tab onto the first card ringed only the poster, and the first
arrow key then moved the ring round the whole card. The poster is the card's own role=button Tab stop,
so Tab gave it the app-wide button ring, while listNav's arrows focus the card root (D2: the ring goes
round the whole item).

The card's "open" element (`data-nav-open` — the Plex poster, a video card's title link) now stands
for the card: when it holds keyboard focus the card root wears the ring and the element's own ring is
transparent. Controls inside the card (watched toggle, actions) keep their own ring.

Measured in Chrome (jsdom evaluates neither :focus-visible nor :has): Plex browse, Tab from the sort
control → poster focused, card outline 2px accent, poster outline transparent; → moves to card 1 with
the same ring. Feed, Tab onto a card's title link → the card is ringed, the link is not.
2026-09-13 05:47:43 +02:00
peter 6d09b94bbb feat(keys): j/k and the arrows walk Home's rails and every Plex grid and rail
Second slice of the E11 keyboard layer (S4b). S4a gave the feed, the library and the channel page
`j`/`k`, the arrows and Enter; Home's shelves and the whole Plex module were still Tab-stop country —
a Plex poster is its own Tab stop, and a rail's cards sat behind every control of the cards before.

- `lib/listNav`: a list that mounts every item no longer declares a count or columns — both are
  measured off the DOM when a key is pressed, so an auto-fill grid's columns follow its width and a
  rail is one row. ↑ on a list's top row and ↓ on its bottom row go on to the neighbouring list, onto
  the item horizontally nearest the one left (owner decision D4: ←→ within a rail, ↑↓ to the rail
  above or below). The first `j` skips cards a rail has scrolled out to the left.
- `Carousel keyboardNav` registers the scroller; the scroller gets a 4px inset (against a -4px margin,
  layout unchanged) so the focus ring is not clipped, and `[data-nav-index]` a 2.5rem inline scroll
  margin so a card stepped into view clears the rail's edge fade and paging arrow.
- `ui/NavGrid`: the plain, fully mounted grid that registers itself.
- `navIndex` on PlexPosterCard / SeasonCard / EpisodeCard (the card root is the item, the poster the
  open element — the ring goes round poster and title, D2), PosterTile and Home's ShelfTile (the button
  is the item); cast, related and collection-strip buttons carry the attrs directly.
- Wired: Home's comfort and resume rails; Plex Browse titles + episodes, Discover rails + genres,
  collection view, playlists/collections browse, person known-for + filmography, show seasons, season
  episodes, cast/related strips, info-page cast + collection strips.
- `listNav.wiring.contract.test`: every `Carousel` says `keyboardNav`, every card says `navIndex`, and a
  card with `navIndex` sits inside a NavGrid / Carousel / VirtualGrid — both halves are opt-in, and
  either one missing leaves a list the keyboard silently skips.
2026-09-13 05:38:36 +02:00
peter 5274216981 feat(ui): PageUp/PageDown page through every dropdown list, not just the palette
The palette got PageUp/PageDown, and the same crawl stayed everywhere else a list opens under a
control: the sidebar's channel filter, the Plex genre/studio/collection pickers, the filter and sort
menus, submenus of every playlist you own. floating-ui's list navigation has only the arrows and
Home/End.

- `lib/listPaging`: the one definition of a page (the rows wholly in view minus one, a fixed 10 with
  no layout) and of where it lands (stops at both ends, never on a row it may not take, looking back
  toward the start rather than past it). The palette now uses it instead of its own copy, and
  `listNav` takes its scroll-parent helper from here.
- `AnchoredPopover`: one keydown handler on the reference AND the surface, so a combobox (keys on its
  input, `aria-activedescendant`) and a roving menu (keys on the focused row) both page. It runs after
  floating-ui's own, which has by then switched to keyboard modality and scrolls the new row in. A
  disabled row is never the target; a submenu's key does not also page the menu around it; with no
  mounted rows (closed, a hover card, a dialog) the key keeps its native meaning.

Measured in the browser: the channel filter shows 8 rows and pages by 7 with the active row in view
after the scroll; a menu pages to its last row and back, the page behind it does not scroll; the
palette still pages by 11 of its 12 visible rows.
2026-09-13 05:20:53 +02:00
peter 75d89613d0 feat(palette): PageUp/PageDown move a page of results at a time
The jump-to palette lists every module, channel, playlist, view and action — 461 rows on the
owner's account, ~11 of them in view — and the arrows were the only way through it: forty presses to
cross the channel group.

- PageDown/PageUp step by the rows that fit in the list right now, minus one, so the row you paged
  from stays in sight at the new page's edge. Measured per press (the list is 50vh and group headings
  eat into it); a fixed 10 when there is no layout to measure.
- Unlike the arrows they stop at the ends instead of wrapping: a page-sized jump that lands back at
  the top reads as having lost your place.
- Home/End stay the text field's own.

Measured in the browser: 0 → 10 → 20 rows, stops on row 461, PageUp from the end steps 9 (a group
heading is in view), the selected row stays visible throughout.
2026-09-13 05:03:35 +02:00
peter 8fb276aa90 feat(keys): j/k walk the feed, s finds the search box, g jumps to a module
A feed card cost ~10 Tab stops, so the tenth card sat ~135 Tabs from the top of the page, and the
only keyboard focus ring (a 60% accent outline on the thumbnail) all but vanished on the dark glass.
First slice of the E11 keyboard layer (S4a); Home, Plex and the tables follow in S4b/S4c.

- `lib/listNav`: one window listener over every registered list, walked in document order. `j`/`k`
  step item to item (on into the next list — owner decision D4), the arrows move around a grid, Enter
  clicks the item's own `data-nav-open` element, so it opens exactly like a click. An item root is
  never a Tab stop: it gets `tabindex=-1` only while the keyboard is on it, and keeps it until the next
  item takes it, so the player's focus restore lands back on the card. The first `j` takes the first
  card you can see, not the top of the list.
- `VirtualGrid` takes part with `keyboardNav` (column count + scrolling an unmounted row in);
  `VirtualFeed` turns it on, so the feed, the library and the channel page all have it.
- Keymap: two-step sequences (`g h`) whose pending step belongs to the scope that started it; the
  page-level scopes (`page`, `search`, `list`) are quiet while a layer that owns the keyboard is open,
  enforced in `dispatch`. A test pins the owner's rule that none of them needs a modifier on the
  Hungarian layout.
- `g h/f/c/p/x/n/m/d/s/,` jump to a module this account can reach; `s` focuses the header search box
  and selects the old query; Escape in it hands the keyboard back to the page (Plex's typeahead spends
  its first Escape on its panel).
- Hover popovers are passive layers: an open tooltip still takes Escape first but no longer silences
  the page keys.
- Focus ring (owner decision D2 = B): solid 2px accent around the whole item; the app-wide button/
  link/row ring goes from 60% to solid too. The item keeps a transparent resting outline because the
  video card transitions `all` — without it every `j` flashed a 3px white ring (measured in Chrome).
- Help sheet: the new keys, in a fixed order after the surface's own; a `g` chord only for a reachable
  module; opened over a player or dialog it leaves out the page-level keys and the context-menu key,
  which nothing can reach there (the latter pre-existing since E10).
- CLAUDE.md: a canonical-patterns row for list/grid keys (`useListNav` + `navItemAttrs` +
  `data-nav-open`, not a private arrow-key handler).
2026-09-13 04:16:20 +02:00
peter 78a9324381 fix(ui): the palette ranks what you typed, and lib stops leaning on components
Typing "theme" put the channel "The Meltzone Podcast" (matched only through its handle
`@themeltzone`) above every "Theme: …" row, and Enter opened the channel — the palette listed
matches in pure group order and always highlighted the first one.

- CommandPalette ranks each word: label starts with it > elsewhere in the label > only the hint,
  keywords or group name; an item ranks by its worst word, then the sum. The group order stays
  fixed, rows are ranked inside each group, and the best row overall starts highlighted. The
  highlight is held by id and follows the best match until the arrows or the pointer pick a row,
  so a channel list landing mid-typing no longer shifts it.
- A11y: result groups are `role="group"` named by their heading; `aria-expanded` is false with
  nothing to choose; the listbox stays mounted (hidden) so `aria-controls` never names a missing
  element, and "Nothing matches" sits outside it; a polite status announces the count once the
  typing pauses.
- `Column`/`FilterMap`/`filterRows`/`isActive` move from `ui/tableHeader` to `lib/tableColumns`,
  and the player's auto/loop modes from `playerModalControls` to `lib/playerModes` — three lib
  helpers imported the column contract from a ui component, and `playerModalHooks` its types from a
  feature component.
- eslint: `lib/` may not import from `components/` at any depth, type-only and re-exports included;
  the boundary contract test proves it fires. CLAUDE.md and the ui README say so.
- keymap: `dispatch`'s docblock sat stacked on `matchesShortcut`'s; moved onto `dispatch`.
2026-09-13 02:14:07 +02:00
peter 8a50538275 fix(ui): a write that fails says what failed and why, wherever it fails
`req` opens its blocking dialog for 400/409/422/5xx only. A 403/404/429 was left to the caller and a
502-504 or lost connection got just the self-clearing connection notice, so 34 mutations without a
reporting onError and 15 direct api writes with a bare or missing catch failed in silence: a hidden
channel that snapped back, a share link that never revoked, a confirmed account delete that ended
with nothing said.

- lib/queryClient: one MutationCache reporter. react-query awaits it BEFORE the mutation's own
  onError, so it waits a task and skips a failure the site already reported; the toast title comes
  from `meta.failTitle`, `meta.errorShownInline` opts out.
- notifyActionError: title names WHAT failed, the line under it says WHY (`failureReason`: a
  structured localized detail, else the status). It marks the error reported (`markReported`), so
  nothing reports one failure twice; `notifyKnownFailure` for a site's own specific sentence.
- Every silent site named (45 new HU/EN titles, 4 reasons). Stale "the global dialog shows it"
  comments corrected on the job actions, channel block, Plex sync, cookies, download enqueue,
  account delete and usePref.
- Channels passed the tag hook's channel id to the reporter as the error; the tag and priority hooks
  now report themselves, with the real error.
- A contract test keeps every useMutation naming its failure; CLAUDE.md gains the pattern row.
- history.dom.test: the "navigates only once the entries are back" assertion raced an async act's
  macrotask flush (1 in 5 runs); it now asserts from a later popstate listener.
2026-09-13 01:45:28 +02:00
peter b4aea7fa5b fix(ui): dialogs, menus and busy buttons keep the keyboard focus where it belongs
Measured in Chrome before each change:

- A dialog opened from a menu lost its focus: the menu row ran its action
  and closed in one batch, and floating-ui's focus return (a microtask)
  moved the focus to the row BEHIND the confirm dialog's scrim 7 ms after
  it opened, where Enter activated the row. MenuItemList now closes first
  and runs the action a task later; the dialog also captures the row, not
  a dying menu item, as its restore target.
- A side panel opened as a flyout collapsed the moment a dialog opened
  from inside it, unmounting the row the dialog hands the focus back to.
  useHoverFocusWithin holds while the focus is inside an aria-modal dialog.
- ui/Button's loading state set native disabled; Chrome moved the focus of
  the pressed button to <body> and never gave it back. Loading is now
  aria-disabled with a click guard that also cancels implicit submission.
- Escape/Enter in the playlist rail's rename field dropped the focus on
  <body>. The row refocuses from its ref callback when it re-attaches (a
  post-commit focus sticks; the earlier attempt ran before the button
  existed).
- PlayerModal was the one aria-modal shell without the dialog contract:
  it is now named by its title, traps Tab, restores the focus to its
  opener and shares one counted body scroll lock with every dialog.
2026-09-13 01:04:11 +02:00
peter 92c305bab9 fix(nav): a navigation closes every overlay before it pushes
The overlay history stack let a navigation land ABOVE open overlays and
"buried" their entries with a counter. Measured in Chrome: Shift+W over
About navigated under the still-open dialog, the next Back both undid the
navigation and closed About, and the Back after that did nothing; Shift+W
on Home with the player open did nothing at all. The counter could also be
spent by a different overlay closing first, and callers that could not
report whether they pushed were assumed to have.

setPage and openChannel now close every open overlay and push only once
the history is back in sync (one entry per live overlay, no programmatic
back() in flight, no reconcile queued), so nothing is ever buried and the
whole mechanism goes: navigateOverOverlays, closeThenNavigate, buryUnder,
setPage's boolean, and the three test files that pinned them. The palette
closes only itself before running a row, so a theme picked over the player
keeps the player. Shift+W on Home with the player open now closes it
(owner decision 2026-09-09).

The wait is on state, not on "a popstate": the old 250 ms fallback could
run with back()s still in flight. An overlay that refuses to close within
1.5 s drops the command instead of being navigated out from under.
2026-09-13 00:17:09 +02:00
peter 32fe0768b6 release: 0.84.0 (bug/hunt-2026-09)
gate / frontend (push) Failing after 2m45s
gate / backend (push) Successful in 4m5s
2026-09-12 16:44:01 +02:00
peter 415a6c16f9 release: 0.84.0 — the owner's sixteen-item bug hunt
Sixteen items written from real use: eleven fixed, one measured and closed without a code change
(the micro-stutter is a 23.976 fps film on a 120.0148 Hz display, reproduced identically outside
Siftlode), two parked for a reproducible case, one scoped out to its own sprint. Two of the eleven
were destroying user state — a YouTube video that rewound itself on reopen, and a "clear resume
point" that told Plex nothing about an in-progress item.

Then seven deep-review rounds over the fixes and over each other's fixes: 32 → 24 → 43 → 31 → 22 →
26 → 18 raised. Minor, not patch: item 14 added an admin knob, item 10 a sort control, and items
2/4/5/7 are features.
2026-09-12 16:43:54 +02:00
peter 8c51062e5b fix(api,ui): the demo refusal says why at every site, and the contract guard cannot fail open
Round 6 converted three of four `is_demo` 403s to a structured detail and wrote a docstring saying it
had converted all the sites there are. The account-delete route in `routes/me.py` still answered a
hand-written English sentence, which `notifyActionError` discards in favour of the caller's
translated fallback — so the demo user gets no hint that the demo account is the reason. It is
UI-gated today, so the string is unreachable; what was reachable was the false completeness claim,
which is precisely what stops the next reader from looking. The site is converted, and the list is
now DERIVED from the source: a new test walks `app/` for every `is_demo`-guarded 403, asserts each
raises a structured detail, and fails when the count moves — so this file cannot fall behind the code
the way it just did.

The same round's frontend contract guard was failing open. Its pattern required a bare identifier
before `.detail` and excluded `.detail` used as a last call argument, so `setErr(e.detail)`,
`(e as HttpError).detail`, `String(e.detail)` and four more idiomatic forms were invisible — and
because the file set is compared with `toEqual`, a file the pattern cannot see is simply never listed
and the gate stays green over a fresh copy of the contract. Widened to cover cast receivers, optional
chains and call-argument reads; the pinned set is unchanged, so the narrowing had removed no false
positive. A destructured `detail` stays out of reach (a pattern loose enough for it fires on
react-query's own `const { data: detail }`), and that hole is written into the test rather than
papered over with two innocent files.

Its anti-vacuity guard also did not guard anything: the comment says the glob must have matched
something, but the assertion under it read a sibling file by its own URL, which resolves whether or
not the glob enumerated anything. That only mattered while the pinned list was non-empty — the moment
the deferral sweep lands and it becomes [], the whole file would pass over a glob matching zero
files. The scan size is asserted now.
2026-09-12 16:22:27 +02:00
peter 63bf1bca78 fix(plex): an accepted reorder refetches the detail it just wrote
`commit`'s success handler writes the dragged list into the detail cache — round 6's fix, so a
re-entry inside the 30s staleTime renders the accepted order without a request — but then invalidated
only the LIST key, while the YouTube sibling it cites invalidates both. `next` is the membership of
the render that started the drag, and `reorder_playlist` only touches the rating_keys it is handed,
so it never ratifies that list: anything that landed meanwhile was clobbered by the write with
nothing scheduled to correct it. `commit` sets no busy flag either, so a removal really can complete
while a reorder is in flight, and with staleTime 30s and refetchOnWindowFocus off the phantom row
stayed on screen for as long as the view was mounted — and the next drag persisted an order built
from it.

Round 6's test had to be re-pointed, because it was written in a world with no refetch: its mock
answered the pre-reorder order forever, which a server that just accepted the write cannot do. It now
holds the refetch open instead, so it still pins what round 6 actually fixed — the cache is right
WITHOUT waiting for a request — and would fail if the write were deleted. Both halves are
mutation-checked against each other.
2026-09-12 16:22:27 +02:00
peter c7a377396b fix(ui): a closing dialog leaves the focus to the dialog standing above it
Round 6 widened dialog focus ownership to count focus inside ANY `aria-modal` subtree as ours, so a
dialog and the dialog stacked on it closing together would still hand the focus back. That clause has
no way to tell a layer closing WITH us from one that stays open, so a dialog closing on its own — an
in-flight mutation's `onSuccess`, not a click, since a scrim covers it — under a modal raised above it
pulled the focus onto its own pre-open trigger, behind that scrim, where Enter still activated it.
Three verifiers confirmed the mechanism and two corrected the severity down: it self-heals on the
next Tab, which is why this is a containment violation rather than a broken end state.

`lib/layerStack` cannot refine it. Measured against the real hook: at the instant the lower dialog's
focus cleanup runs, the both-closing case and the sibling-stays-open case give identical readings —
`anyLayerAbove` true, the upper layer topmost — because passive cleanups run parent-first and
`useDialogChrome` is declared before the effect that calls `removeLayer`. Gating on the stack turns
round 6's own test red. What separates them is the DOM: React detaches a deleted subtree before
passive cleanups, so a layer closing with us is already disconnected while one that stays open is
not. Both tests are green, and each was mutation-checked against the other.

Verified in Chrome on localdev: opening the cheat sheet from a sidebar button and closing it restores
that button; with the command palette stacked over the cheat sheet, closing the top one leaves the
focus neither on the background trigger nor on <body>, and closing both returns it to the trigger.

Welcome's Lightbox got the trap and the restore in round 6 but not the layer membership the other
shells have, so its Escape and Tab fired from a raw window listener whatever was on top — the shape
the wizard's comment records being bitten by, unreachable today only by a mount ordering nothing
enforces. It joins the stack. Its comment also read as though the four-shell sweep were finished:
`PlayerModal` is still an `aria-modal` surface with no name, no trap and no restore, which is on
ROADMAP.md now rather than implied to be done.
2026-09-12 16:22:06 +02:00
peter 8036251589 fix(plex): the reclaim window rolls, and a refused finish is not performed through a third route
Two ways round 6's own fixes did not bound what their comments said they bounded.

The reclaim budget's window anchor was written only where the window RESET, so it marked the window's
start rather than the last loss — a tumbling window, not the rolling one the constant is named for.
Any loss cadence slower than half the window satisfied the reset on every second loss, so the budget
never reached its bound: measured over 200 loss cycles, 61/70/90/119s apart all gave 200 reclaims and
never an error, while 5s and 30s apart correctly gave two and then the honest answer. The failing
band is exactly the case the ERROR handler names — a transcode that falls behind realtime — and both
committed tests spaced their losses 0s and 10min apart, i.e. outside it. The anchor now moves with
every loss.

And `finish`'s refusal handler lowered the play-through latch, which is the only gate on the 10s
interval, the page-hide beacon and `settle`. All three post `absRef`, which on the play-to-the-end
path IS the end of the media, and the backend reads a position within `_FINISH_MARGIN_S` of the
duration as a finish — so within ten seconds of the toast saying the mark had failed, the item was
marked watched, its position cleared and a scrobble sent to Plex. Round 6 closed the explicit
rollback write and left the other three open; the test it added set `upNext: false`, which tears the
effect down before any of them can fire. The refusal now takes back the CLAIM rather than the latch:
a second flag stops `settle` re-asserting watched, and playing again clears both.

`FINISH_MARGIN_S` duplicated the backend's constant in a second language with nothing relating them,
and its comment had the asymmetry backwards — if the BACKEND's margin grows, this guard starts
admitting a rollback the backend reads as a finish, i.e. re-opens the defect above. Pinned by a
contract test that reads the Python, and the comment corrected.
2026-09-12 16:21:51 +02:00
peter 71d7646bf1 fix(plex): a name this session cannot write fails at once, and the layout has one spelling
Round 6 turned `artifact_may_arrive` into a positive rule, but keyed it to the HLS vocabulary in
general rather than to what THIS session produces: the playlist regex accepted master/index/stream_*
for every session and the `seq == 0` branch accepted any variant token unconditionally. The two argv
branches are disjoint — a single-audio session is given `seg_%d.ts` + `index.m3u8` and no `%v` at
all, so `master.m3u8`, `stream_*.m3u8` and every `seg_<variant>_<n>.ts` are names it can never write,
and vice versa — so each of those was read as "an HLS name, so wait and see" and held a threadpool
worker, plus its pooled DB connection, for the full timeout. That is verbatim the round-5 outage the
route's own comment claims it prevents.

Measured through the real route against a live single-audio session, before: master.m3u8 404 after
30.0s, stream_q7z.m3u8 404 after 30.0s, seg_a0_0.ts / seg_zzzz_0.ts / seg_0_0.ts 404 after 20.1s
each. After, on the running localdev API: 404 in 44-316ms for all of them, while index.m3u8,
seg_0.ts and seg_1.ts still answer 200 — nothing is over-refused.

The session now carries the rendition names its ffmpeg was told to write, and `entry` is derived from
them instead of passed. And "one place" now means the argv BUILDS its patterns from the same
functions the parser reads back (`segment_name("%d", "%v")`, `variant_playlist_name("%v")`): the
layout was still spelled out independently in six places, with `_VIDEO_VARIANT` read by no consumer,
which is the drift that caused round 6's HIGH. A test asserts the argv's own patterns are what those
functions produce, and that every name they expand to is one the guard accepts.

The load-bearing MEASURED comment cited ffmpeg 8.1.1 — the Windows host's. The binary that writes
these files is the API container's 7.1.5; re-measured there, the conclusion holds.
2026-09-12 16:21:37 +02:00
peter 3eb0f41962 fix(plex): a Plex reorder the server accepted is the order the cache holds
The YouTube list got this rule in round 5 — one accepted write, one cache entry
that matches it — and the Plex list got only the rollback half of that fix. Its
success path invalidated `qk.plexPlaylists()`, a disjoint key root, and never
touched `qk.plexPlaylist(id)`: the key `putBack` rolls back to, and the one a
re-entry renders from without refetching.

So after an accepted drag the detail cache still held the pre-reorder order.
Leaving the playlist and coming back inside the 30 s staleTime showed the
reorder undone, and the next drag from there persisted that — silently undoing
an order the server had taken. Measured in jsdom: server 2,1,3, cache 1,2,3, no
refetch.

1 test, mutation-checked.
2026-09-12 05:50:40 +02:00
peter a0114deff1 fix(api,ui): a refused write says why, wherever the refusal happens
Round 5 taught `notifyActionError` to discard a plain-string `detail` in favour
of the caller's translated fallback, because `req` leaves 403/404/429 to the
caller and those routes hand-write English. Three places did not get the
message.

**The demo account lost its explanation.** `require_human` refused with a plain
string, so from that commit on the shared demo account got its caller's generic
"couldn't do that" and no hint that the demo account was the reason — and the
gate is not backed by a UI gate everywhere: the comfort shelf has no `is_demo`
check, so the pin button is live and simply fails. A structured `{code: "demo"}`
and an `errors.demo` string in both locales. The two auth routes that hand-roll
the same check were swept with it.

**`ChatThread` carried a second copy of the contract.** It filtered 404/429
itself and then did `e.detail || t(…)`, the pre-fix order, so the central fix
could not reach it and "That member hasn't set up messaging yet." stayed English
under a Hungarian UI. It goes through the shared helper now, and a contract test
pins the remaining raw-`detail` consumers as an explicit SET — never a count —
so a new copy fails the gate instead of shipping. The five on that list are
inline surfaces (a form field's error line, a setup banner) and are deferred
with the ~60 backend routes whose details they print.

**The Plex "Add to playlist" dialog was still silent.** Same shape as the
collection editor round 5 fixed and the YouTube twin that already had the fix:
no catch, so a playlist deleted on another device answered a structured 404 that
nothing reported — the tick never flipped, the gone row stayed, and the floating
promise became an unhandled rejection.

Also: the `ABORTED` sentinel's VALUE was asserted nowhere. The consumer test
hand-built the error it expected, so replacing `xhr.onabort`'s rejection with a
friendlier message left the lane green — and every upload the user cancels would
then raise a red "the upload failed" toast, which is what the sentinel prevents.
Both producer tests assert through `isAborted` now.

And two backend tests were pinning fewer sites than their own docstrings claimed:
the channel-404 loop said "all five" and issued three requests (explore and
subscribe unexercised, the excuse for subscribe naming a different call site
with a different reason slug), and the recipient/job test named three refusals
and requested two — `share_download`'s `not_found("recipient")` had no test in
the suite at all, nor did the locale keys added beside it. Every site named now
has a request behind it.

6 tests, mutation-checked against 7 mutations: the demo detail at both the gate
and an auth route, a re-hand-rolled ChatThread, the silent Plex dialog, the
abort sentinel, and each unpinned 404 — all red on the test that names them.
2026-09-12 05:50:31 +02:00
peter d89d887a9b fix(plex): the reclaim budget bounds what it says it bounds, and a refused finish is not performed anyway
`MAX_SESSION_RECLAIMS` was cleared by the first fragment of any session, and a
reclaimed session essentially always delivers one — `start_session` does not
return until its first segment exists. So the budget only ever bit where a
session parsed its manifest and then died before a single fragment, which is not
either case its own comment names. Measured: 40 loss cycles produced 41 sessions
and never an error. A transcode falling behind realtime, or two viewers of one
rating_key taking turns, respawned ffmpeg indefinitely — the round-5 behaviour,
still there behind the guard added to stop it.

It counts within a rolling window now. Two losses inside two minutes is a
session that keeps being lost; one an hour apart is not. And the window is reset
per item, because the player stays mounted across a queue advance — otherwise a
budget spent on one episode would deny the next its first legitimate reclaim.

Separately: `finish`'s rollback re-performed the write it was taking back. On
the play-to-the-end path the position it restores IS the end of the media, and
the backend reads a position within its finish margin as a finish — so the
branch named "take the finish back" set watched, cleared the position and
scrobbled to Plex through a second route, while the caller's error toast was
already on screen saying the mark had failed. There is no resume point at the
end of the media to restore; the cache drop is the whole of the rollback there.
The credits-skip caller captures a mid-episode position and still restores it,
which is the case that needs it.

7 tests, mutation-checked: the FRAG_LOADED reset, the missing window, and the
unguarded rollback each red on the test that names them.
2026-09-12 05:50:11 +02:00
peter f8a31d48cf fix(ui): a dialog keeps the focus it was given, and hands it back where it took it from
Round 5 extracted `ui/dialogFocus` so no shell would hand-roll the focus
contract, then introduced four ways to lose the focus anyway — all four measured
in jsdom against the real components.

**Ownership was narrower than the trap.** `ui/Modal` traps Tab over the whole
dialog on purpose, so a toast's Undo or ✕ stays reachable; ownership was tracked
over the panel alone. Tabbing to that ✕ therefore read as "the user left the
dialog", and since removing a toast fires no `focusin` to re-arm, closing
afterwards skipped the restore and left the focus on <body>. Before this hook
existed the restore was unconditional, so it was a regression.

**A dialog stacked above us is still ours.** A nested `ui/Modal` portals into its
own host, so it is outside the outer dialog's root however wide that scope is
drawn — widening alone could not fix it. Both closing in one commit left the
focus on <body>: the outer skipped its restore, and the child's own target (a
control inside the outer panel) was already detached, because passive cleanups
run parent-first. Ownership now means "inside my dialog, or inside a modal layer
above it", which is what the rule always intended.

**The wizard disagreed with `ui/Modal` on its first line.** `trapTab(e, panel)`,
with neither the dialog-root scope (so its own toast host was outside the cycle
— the connection notice never auto-dismisses, and its ✕ was unreachable from the
keyboard) nor the `container` argument the JSDoc exists for: `panel.contains(
panel)` is true, so focus sitting ON the tabIndex=-1 panel matched no branch and
a Shift+Tab walked into the app behind the scrim. Both call sites now pass the
same three arguments.

**The toast handover was a render-phase read-modify-write.** A `useMemo` read a
module global that the next line overwrote; React re-invokes a component body
and every `useMemo` factory in it on a StrictMode double render, and the LAST
invocation commits — so the second pass re-read what the first had just written,
the takeover host classified its OWN new toast as inherited, and the dev build
behaved exactly like the code round 5 replaced. Snapshotted through a ref keyed
on `isTop`, so a repeated invocation of one render is a no-op.

And the fourth `aria-modal` shell, `Welcome`'s screenshot lightbox, is finally
swept into the same contract: it declared itself a modal dialog while leaving
the focus on the thumbnail behind an 80%-black scrim and letting Tab walk the
landing page underneath.

The headline fix of the previous commit — StrictMode's re-mount taking the focus
off an `autoFocus` field — had no test at all; deleting it left all 1009 green.
It has one now.

9 tests, mutation-checked against 8 mutations, each red on the test that names it.
2026-09-12 05:49:58 +02:00
peter 7f17b17c13 fix(plex): an HLS session's file names are one vocabulary, and a name it cannot produce fails at once
Round 5 added a guard so a request for a segment beyond the produced region
answers instead of polling out the full timeout — the route is a SYNC one, so
every such wait parks a threadpool worker and, through `current_user`, a pooled
DB connection. Measured against real ffmpeg, the guard could not fire where it
mattered.

`-hls_segment_filename seg_%v_%d.ts` substitutes the VARIANT first: a dual-audio
session writes seg_0_0.ts (video) and seg_a0_0.ts / seg_a1_0.ts (the `name:aN`
renditions). The regex read those as sequence-first, so every video segment
parsed as segment 0 and every audio segment did not parse at all — the guard was
inert on every multi-audio item, which is most of this library. `_seg()` carried
the same inversion: the warm-up waited 8 s for `seg_1_0.ts`, a name ffmpeg never
writes, on every multi-audio session start AND every seek-restart.

The layout is now one constant plus one parser (`segment_name` /
`_SEG_NAME_RE`), with a test pinning them to each other, instead of three
comments that could drift — and they had drifted, all three ways.

The guard is also a positive rule now. It was a blocklist, so anything that was
not a single-audio segment name was waved through: `GET …/anything.m3u8` still
parked a worker for 30 s, while the route's comment claimed a flood of hopeless
names was what it prevented. A session produces its playlists and its segments
and nothing else. `.vtt` leaves `_HLS_CT` with it — ffmpeg runs with `-sn` and
subtitles are a separate route, so that extension could only ever name a file
that will not arrive.

Also here:
- A 5000-digit segment index reached the client as a 500: `int()` on an
  unbounded digit run raises past 4300 digits and only OSError was caught. The
  name is bounded on the route and the index in the regex — the same bug class
  the subtitle routes were bounded for.
- A session torn down between the wait and the read is a 404, not a 500. `_kill`
  rmtrees the directory and no lock spans those two steps; the player's recovery
  is keyed on the STATUS, so a 500 skipped the reclaim and ended playback with
  "couldn't play" instead of restarting at the playhead. The trigger is ordinary
  — `loadSession` awaits the new session before destroying the old hls instance.
- Two comments claimed an open player checkpoints every 10 s "whether or not it
  is playing", which is the stated reason the 30 s finished-idle reap is safe.
  `flush` sends nothing once the finish latch is up, or before a position is
  known. The states it does cover — playing or paused mid-item — are the ones
  the rule needs; the comments now say that instead of more.
- The route's tests leave the DB lane: the route reads nothing from the database
  since the auto-start was removed, and there those four regressions SKIP on a
  machine with no Postgres while pytest prints green.

14 tests, mutation-checked: the old parse, the old warm-up name, the blocklist,
both bounds off the absurd name, the unguarded read, and the route dropping
`proc` — each red on the test that names it.
2026-09-12 05:49:38 +02:00
peter 1f81684b6e fix(ui,plex): a dialog keeps the focus its child asked for, and a hopeless segment fails at once
Both found by UAT in a real browser, after the tests for them were already green — which is the
whole reason this repo settles focus and timing claims by running them.

- **The autoFocus fix worked, and then the dialog took the focus back.** React 18's StrictMode runs
  every effect mount→cleanup→mount in development: the cleanup handed focus back to the trigger, and
  the second mount, finding the focus outside the dialog, moved it to the first focusable — off the
  `autoFocus` field a beat after the browser had put it there. Chrome measured it: INPUT (commitMount)
  → BODY (cleanup) → ✕ (re-mount). The effect is idempotent now — it remembers what the dialog was
  focused on and restores THAT on a re-mount — and the close-time restore is conditional on the
  dialog still holding the focus, which it tracks as it changes rather than computing at cleanup,
  where the panel is already out of the document. A dialog no longer yanks focus back from wherever
  the user has since moved it.

- **Passing ffmpeg's handle to the segment wait only shortened the FINISHED case.** Measured against
  the running localdev API: a request for `seg_99999.ts` on a live session still took 20 195 ms,
  because that ffmpeg has not exited and the wait has no other reason to stop. ffmpeg writes segments
  in order, so a segment whose predecessor does not exist is not the next one and cannot arrive —
  and that is knowable at once. Same request now: 45 ms.

Verified on localdev against the real library (Westworld S01E06, remux, multi-audio): Mod+K opens the
command palette with the focus in its search field and typing reaches it; playback runs on the new
session-scoped HLS URL; a request naming a stale session id is a 404; and a viewer paused for longer
than the finished-session rule keeps their 2.4 GB session across two reaper passes — the progress
checkpoint touches it — and resumes cleanly.

vitest 1009, pytest 1127.
2026-09-12 04:31:51 +02:00
peter 33fddacee2 test(plex): the over-long-name guard is asserted the same way on every OS
Deep-review round 5. The assertion behind `subtitles._exists` was only red where pathlib re-raises
ENAMETOOLONG: on Linux it does, on Windows the same stat fails with ERROR_INVALID_NAME, which
pathlib's `exists()` already swallows — so the assertion passed with the fix reverted. It forces the
errno now instead of relying on the filesystem. (The gate's pytest lane is the Linux container, so
the branch was never actually unguarded; the test was just dishonest about why it passed.)
2026-09-12 04:18:35 +02:00
peter 12121255dd fix(playlists): a rejected reorder goes back to the order the server accepted
Deep-review round 5, over round 4's own fixes.

- Round 4 made a failed reorder roll the rows back to the CACHED detail, because a resync cannot do
  it (the server's list is unchanged, so the refetch is deep-equal, react-query keeps the same
  `data` object and the effect keyed on it never runs). But nothing wrote an ACCEPTED order into
  that cache — only the refetch that follows a successful reorder did. So a second reorder rejected
  while that refetch was still in flight rolled the rows back past the first one, which the server
  had accepted; the membership guard cannot correct an order (its key is the sorted id set), so the
  wrong order stayed on screen for good and the next drag persisted it. `onApply` now writes the
  accepted order into the cache before invalidating.

- `PlexPlaylistView.rename` had a `catch` no test could reach, and `commit` (reorder) was not gated
  by `busy` the way its siblings are — a reorder in flight while a removal completes is what makes a
  rollback ambiguous in the first place. The rename failure path is now covered; `commit` defers.

- `putBack` read `q.data` from the render that bound the failing handler instead of the cache at
  rollback time, unlike the sibling `restoreFromCache`. Aligned. Stated plainly: I could NOT drive
  that staleness in jsdom — a mutation that restores the old shape leaves every candidate test green
  — so this one ships as an alignment with the proven sibling, not as a fix with a red test behind
  it. The scenario and the harness limit are on ROADMAP.md.

Two regression tests, both mutation-checked red (the cache write, the rename catch). The
drag-reorder failure path still has no test: dnd-kit's pointer and keyboard sensors do not activate
under jsdom even with geometry and scrollIntoView stubbed, so a test there would have pinned the
harness, not the code. Logged, not faked.

vitest 1009.
2026-09-12 04:15:15 +02:00
peter 4a7b65d187 fix(api): a refused write is named, in the reader's language
Deep-review round 5, over round 4's own fixes.

- `notifyActionError` preferred the server's `detail` over the caller's translated fallback, and
  `reportFailure` deliberately leaves 403/404/429 to the caller — so for exactly those statuses the
  dozen call sites round 4 converted started printing whatever English a route hand-wrote into an
  otherwise Hungarian UI ("No user with that email.", "Not subscribed to this channel", "Unknown
  job"). A STRUCTURED detail is still preferred: `req` has already run it through `localizeDetail`,
  so it is both localized and more specific. A plain string is not.

- The other half of the same defect, on the backend: round 4's `get_or_404(what="channel")` sweep
  reached exactly ONE of the five identical sites in channels.py and shipped the HU/EN string with
  it, so the Block button said it in Hungarian and the four controls beside it in English. All five
  are converted, and the subscription / share-recipient / scheduler-job 404s with them (two new
  reasons in both locales).

- `PlexCollectionEditor`'s add/remove, take-over and delete were try/finally with no catch — in the
  very file round 4 edited to give `createNew` one. 403 and 404 are the caller's to report, so the
  tick simply never appeared and a CONFIRMED "delete collection" did nothing, silently.

- `UploadDialog` is the only `uploadReq` caller and kept an empty catch whose comment still claimed
  the global error modal had it. It had not since `reportFailure` became shared: a proxy 502 raised
  only the self-clearing connection notice and a 413/403/404 nothing at all, so the dialog snapped
  back to its pre-upload state with no error named. The user's own Cancel stays silent — and since
  a cancel and a network drop both carry status 0, that difference is now a named `isAborted`
  rather than a string literal at each end.

Regression tests for each (5 vitest, 2 pytest, plus two existing assertions corrected), all
mutation-checked: 6 mutations, each red on the assertion it is named after. vitest 1007,
pytest 1127.
2026-09-12 04:07:25 +02:00
peter 47af4d5c05 fix(plex): an HLS artifact names the session it belongs to, and a paused viewer says so
Deep-review round 5, over round 4's own fixes.

- Sessions are keyed by rating_key alone while a session's DIRECTORY is per (key, offset, tag), and
  the artifact URL carried no session identity — so once a seek-restart or a second viewer of the
  same item took the registry slot, the old viewer's request for `seg_12.ts` was answered 200 with
  the new session's twelfth segment: bytes from a different `-ss` offset under `-copyts`, on a
  timeline the client is not on. Wrong content with a success status is the one failure a player
  cannot notice, and round 4's reclaim-on-404 could never fire on it. The id is in the path now (so
  the playlist's relative names resolve back into the same session), a mismatch is a 404, and the
  player reclaims at the playhead. The entry playlist's auto-start is gone with it: it could only
  ever begin at 0, silently discarding the viewer's position.

- Round 4 exempted paused viewers from the reaper by keeping EVERY finished session for the full
  ten-minute idle window — one whole segmented copy of a film per item, on the volume the download
  center shares. The signal it needed already existed: an open player checkpoints its position every
  10 s whether or not it is playing, so that route now touches the session and the periodic pass is
  back on the short rule.

- POST /session's opportunistic reap was never scoped to the key being started: it walked the whole
  registry with the 30 s finished rule, so pressing play on ANY item — even a direct-play one, the
  call ran before that early return — killed another viewer's paused session. It is gone; the
  periodic pass owns reaping.

- A segment that can never arrive no longer polls out the full 20 s: the producer's handle is passed
  to `wait_for`, so a finished ffmpeg answers at once. This is a sync route, so each such wait parked
  a threadpool worker AND a pooled DB connection, and an ordinary seek past the produced region was
  enough to do it.

- The DataError handler's re-raise branch — the half that keeps a genuine DataError a 500 — is
  pinned, so simplifying it to an unconditional 422 cannot pass in silence.

Six regression tests (two route-level, in the DB lane), each mutation-checked red on the assertion
it is named after. pytest 1124, vitest 999.
2026-09-12 03:57:52 +02:00
peter 8a83bc0e9f fix(plex): a finished item is finished when the server says so, and a lost session is reclaimed a bounded number of times
Deep-review round 5, over round 4's own fixes.

- Round 4 added a `finish` latch so a played-through item records no checkpoint at the credits.
  On a NATURAL play-through it was raised a beat too late: per the HTML spec the browser sets
  `paused` and fires 'pause' BEFORE 'ended', so the pause checkpoint went out while the latch was
  still down and raced the watched mark — both requests read "not watched yet" from their own
  snapshot, so Plex counted one watch twice, and where Plex's reported duration runs more than the
  finish margin past the real media the checkpoint wrote a resume position AFTER the scrobble. The
  pause that ends the media no longer flushes. The regression test that was meant to cover this
  modelled the end as a bare 'ended' event — a browser that does not exist — and is now the real
  order.

- The watched mark was fired and forgotten (`.catch(() => {})`) while the latch patched the cached
  detail to watched / position 0 regardless, and round 4's once-per-id start guard meant the
  corrected detail arriving later was never used: an episode whose mark failed reopened at 0:00
  with a Watched badge the server disagreed with. `finish` now takes the in-flight write and takes
  itself back if the server refuses it — the real position is reported, the lying cache entry is
  dropped, and the failure is said out loud instead of swallowed. Both callers (the natural end
  and the credits auto-skip) hand it over.

- The reclaim of a 404'd HLS session had no bound. `parsed` cannot provide one — it belongs to one
  hls instance and every restart builds a fresh one — and the server answers 404 both for "the
  session is gone" and for "that segment is not there yet", so a transcode falling behind realtime,
  or two viewers of one item taking turns, respawned ffmpeg forever behind a spinner that never
  became an error. Two consecutive reclaims, reset by the first fragment that actually loads, then
  the honest failure.

Five regression tests, each mutation-checked red on the assertion it is named after.
2026-09-12 03:47:42 +02:00
peter 41b652f8fd fix(ui): a dialog opens focused where it said it would, and the wizard is a dialog
Deep-review round 5, over round 4's own fixes.

- Round 4 gave `ui/Modal` a persistent portal host so a fullscreen transition would stop tearing
  the dialog down. The host was attached in a `useLayoutEffect` — one commit phase AFTER React
  runs `autoFocus` in `commitMount`, and `focus()` on a disconnected node is a no-op. Every
  autoFocus in every dialog was dead: Mod+K opened the command palette with the focus on the ✕,
  so typing searched nothing and Enter closed it; the same for the download rename field and
  every non-danger confirm. The host is attached in a `useInsertionEffect` now, a whole phase
  earlier, and the whole re-parent — including the focus save/restore across the move, which
  `appendChild` needs because it is remove-then-insert — lives in one `useFullscreenPortalHost`.
  `ui/AnchoredPopover` had a second copy of that re-parent WITHOUT the focus restore, and a
  comment claiming focus survived it; both now come from the hook.

- `ui/Toaster` classified a toast raised in the SAME commit as a host takeover among the ones it
  inherits, because it read the store live at the takeover render. A dialog that notifies and
  closes in one handler (UploadDialog) therefore put its toast up with no role, into a region
  still `aria-live="off"` — announced by neither path. "Inherited" is now what the outgoing host
  last had on screen, and a toast that is NOT inherited is announced even inside the unarmed
  window.

- The onboarding wizard renders its own shell, and Escape ordering was all of the dialog contract
  it had: no role, no label, no initial focus, no trap, no restore. A keyboard user tabbed out of
  the first-login wizard into the app behind the scrim. The focus half of the contract is now one
  shared `ui/dialogFocus` that both `ui/Modal` and the wizard use.

Every fix has a regression test, each mutation-checked (6 mutations, all red on the assertion
they are named after).
2026-09-12 03:41:37 +02:00
peter 545996f807 fix(ui): a toast host ranks by when its overlay opened, and an overlay keeps its Escape, its dialog and its navigation straight
Deep-review round 4, the overlays.

- Toast hosts were ranked by mount time, and PlayerModal swaps its host on every fullscreen change:
  leaving true fullscreen with the download dialog open put the player's host back on top, so toasts
  were drawn under that dialog's scrim and a click on one closed the dialog. Each overlay takes a
  rank once, when it opens (`useOverlayOrder`), and hands it to every host it renders.
- A dialog open while fullscreen is left keeps a stable portal container (moved, not re-created):
  React tore the portal down when `fsRoot` changed, resetting a half-filled form and dropping focus
  to <body>. Focus is put back after the move.
- The onboarding wizard is a layer, not a raw document listener: one Escape used to dismiss the
  error dialog over it AND end onboarding, skipping the write-scope step.
- `navigateTo` (a toast's "View", clicked inside the player's own host) goes over the overlays it
  unmounts, so the player's Back-to-close cleanup no longer pops the page it just opened — the same
  fix Shift+W already had. The access-requests toast action too.
- The live region arms itself a task after it takes over inherited toasts: React flushes effects in
  the same task as that render, so the "silent inheritance" arming was a no-op.
- The overlay-host guard matches class tokens in any order (PlexPlayer's root was never checked),
  globs `ui/` too (ui/Modal was not checked either), and requires the rank.
2026-09-11 23:43:23 +02:00
peter 267fd7f70f fix(playlists): a refused edit is taken back on screen, and a gone list or tag is said, not swallowed
Deep-review round 4, the playlist surfaces.

- A failed move or reorder relied on the resync to put the rows back — but the server's list is
  unchanged, so the refetch comes back deep-equal, react-query keeps the same `data` object, and the
  re-seed effect keyed on it never ran: the moved row stayed gone, the rejected order stayed on
  screen. Both restore from the cached detail now.
- A failed removal puts its row back itself: when the server is unreachable the resync fails too,
  and round 3's "keep the list on a failed refetch" guard then kept showing it as removed.
- The Plex playlist page (remove / rename / delete / reorder), the add-to-playlist popover and the
  tag manager had no failure handling at all: a list or tag deleted elsewhere (a 404, now a
  structured `not_found` the request layer leaves to the caller) failed in silence and an
  optimistic removal stayed removed. Each says it and resyncs; deleting a Plex playlist that is
  already gone just leaves the page.
- Playlists' own loading/error branches use the shared `LoadingState` / `StateMessage` (the loading
  line had no live region).
- The "player stays mounted" tests count mounts: a remount ended with the same flag as staying up.
2026-09-11 23:34:33 +02:00
peter 6d49da17fe fix(api): a write that hit a connection failure is still reported, and every failure is reported once
Deep-review round 4, the "one report per failure" contract round 3 introduced.

- `req` marked gateway (502/503/504) and network failures "already reported" because they raise
  the connection notice. But that notice is about connectivity and removes itself on the very next
  answer — which every caller's own resync refetch fetches at once — so a delete that hit a
  keepalive reset flashed the notice and was then reported nowhere, the list still there. Only the
  durable reports (the error dialog, the 401 sign-out) count now; the caller says the action failed.
- A connection notice the user closed swallowed every later one for the rest of the visit; it is
  raised again.
- `req` and `uploadReq` share one failure policy (`reportFailure`). The upload copy had drifted: a
  502 opened the blocking "Server error (502)" dialog and cleared a notice that was still true.
- A taken Plex collection name is a structured 409, localized, and the create/rename calls are
  quiet: it was an English dialog plus a Hungarian toast. `notifyCollectionEditError` is gone.
- The mutation onErrors that toasted unconditionally after a non-quiet request (invite approve /
  deny / add, demo whitelist, audit clear, scheduler, share, full history, edit, config tests, tag
  attach) go through `notifyActionError`; the playlist delete's hand-rolled copy of it too (it takes
  a level now). The user role / suspend / delete actions had no onError at all: a user deleted in
  another tab failed silently and stayed listed — now said, and the list refetched.
2026-09-11 23:27:48 +02:00
peter 6a92585098 fix(plex): the player restarts a reaped session, finishes a played-through item as watched, and starts each item once
Deep-review round 4, the Plex player.

- A 404 on an HLS session that WAS playing means the server reaped it (a long pause, a deploy);
  hls.js never retries a 4xx, so playback ended with "couldn't play". A new session at the playhead
  now takes its place (the frontend half of the paused-viewer reap fix). A session that 404s before
  its manifest is still a plain failure, so it cannot restart itself in a loop.
- Leaving an item that ended, or that the credits auto-skip left, sent a final checkpoint at the
  credits (racing the watched mark, and pushing a resume point to Plex after the scrobble) and
  patched "new, at 49:55" into the cache — Prev reopened it at its last seconds. `finish` settles it
  as watched at 0, said before the pause whose own checkpoint could land first.
- An audio switch, an A/V offset change and every seek control read `abs`, which is 0 until
  playback ticks: a resume at 20:00 restarted from 0:00. They use the shared `playhead`.
- A seek while a restart is pending always restarts (the old media's `seekable` described the old
  session, and the pending answer overrode the newer seek).
- A superseded direct-play session's loadedmetadata/canplay are generation-gated and detached (they
  seeked the next item to the previous one's resume point).
- The session starts once per item: a reconnect refetch, the info panel's watched toggle or a
  language switch no longer restarts the film at the last checkpoint.
- Tests for the session generation's failure paths and the item-change bump, which had none.
2026-09-11 23:18:33 +02:00
peter 31dc4184f5 fix(api): a paused viewer keeps its HLS session, a refused subtitle publish backs off, and a NUL is a 422 everywhere
Deep-review round 4, backend half.

- The periodic HLS reaper (round 3) applied the "finished and unfetched for 30 s" rule that only a
  NEW session request may apply. A remux finishes long before the film, and a paused player with a
  full buffer fetches nothing, so a one-minute pause killed the session and playback died ~30 s after
  resuming. The periodic pass is `reap_abandoned` now (the long idle limit, finished or not).
- A subtitle `publish` that could not keep its output (full / read-only cache volume) returned
  quietly, so every poll queued the window and the whole-container pass again. It reports failure
  and the pass is backed off like a failed read; a cached VTT that cannot be READ is answered
  `unavailable` (`cache_unreadable`) instead of scheduling reads that can never land.
- The subtitle routes bound `rating_key` / `sub_ord` (they become a cache file name: a 250-digit
  one was ENAMETOOLONG, which pathlib's exists() re-raises -> a 500), and the cache helpers no longer
  throw on such a name.
- One app-wide handler turns psycopg's "text fields cannot contain NUL" DataError into a 422: round 3
  guarded only the by-rating_key lookup, and `contains=`, `contains_group=`, reorder/bulk key lists
  and the collection search still 500'd.
- The hand-written `db.get(...) is None -> 404 "Unknown ..."` siblings use `get_or_404` with a
  `what` (channel / video / user), so the Hungarian UI no longer shows English for a gone row.
2026-09-11 23:08:11 +02:00
peter 86971c574d fix(api): progress checkpoints past the int4 column are a 422, and idle HLS sessions are reaped on a timer
Deep-review round 3 of bug hunt 2026-09.

- The YouTube and Plex progress bodies are bounded by POSITION_MAX_S (the int4 position columns): a
  finite position of 2^31 or more was a DataError, i.e. a 500 — the sibling of round 2's non-finite
  fix.
- plex/stream.py promised "a periodic reaper", but reap_idle ran only inside the NEXT session
  request: after the last playback an abandoned session (its ffmpeg still working, its directory a
  whole segmented copy of the film) lived until someone pressed play again. A lifespan task reaps
  every 60 s.
- Tests for what round 2 left untested: the 422's body keeps the list shape the UI localizes, and
  stream_session refuses non-finite start/aoff.
2026-09-11 18:05:30 +02:00
peter 785080f3da fix(plex): a Surprise card whose watched toggle failed is put back
Deep-review round 3 of bug hunt 2026-09 — the sibling of the collection page's fix (d349bc90).
Discover's toggle flips the Surprise draw too, but its resync refetches only the rails (refetching the
draw would reshuffle it under the viewer), so a refused toggle stayed flipped there. toggleWatchState
takes an optional `revert`, called on a failure only; the queryKeys comment that claimed the watched
toggle invalidates the Surprise key now says what the code does.
2026-09-11 18:05:30 +02:00
peter 76b5f7c93e fix(playlists): a failed background refetch keeps the list on screen, and the player outlives every branch
Deep-review round 3 of bug hunt 2026-09.

- The detail's error branch applies only when there is nothing to show (`isError && <no data>`, the
  QueryState convention): one failed refetch — on focus, after a move or a removal — replaced a
  loaded list with "couldn't load". The error line is announced (role=alert).
- Tests for the two branches of "the player survives a change of selection" that round 2 left
  untested (a list that fails to load, no selection).
2026-09-11 18:05:19 +02:00
peter 1cfe6a7f59 fix(api): a gone row and a refused playlist action are said in the reader's language, and a failure is reported once
Deep-review round 3 of bug hunt 2026-09.

- get_or_404 / owned_or_404 answer a structured {code: "not_found", reason: <what>}
  (errors.notFound.* in HU+EN, generic fallback). The structured-404 fix of round 2 had gone into the
  ad-hoc `_own_playlist` (CLAUDE.md's "Not" pattern) while the shared helpers stayed string-only, so
  views, tags, pins, Plex playlists, invites and users still 404'd in English. `_own_playlist` now
  goes through owned_or_404; admin's hand-rolled user lookups use get_or_404.
- A move into a list that is gone names the TARGET (reason "move_target") — "this playlist doesn't
  exist any more" was read as the list on screen, which still exists.
- Every refusal in the playlists router is a {code: "playlist", reason} (playlists.errors.* HU+EN)
  instead of a bare English string shown verbatim in the Hungarian UI.
- `req` / `uploadReq` mark the errors they already put on screen (the error dialog, the connection
  notice, the 401 sign-out), and notifyActionError skips those. The watched toggles, the saved views
  and a playlist delete toasted a fault the dialog was already showing; Playlists' two hand-written
  status filters now go through the same check.
2026-09-11 18:05:19 +02:00
peter 33fe895539 fix(ui): every full-screen overlay hosts its own toasts, above the players' controls, without re-announcing them
Deep-review round 3 of bug hunt 2026-09 (the round-2 overlay toast host, finished).

- Inside an overlay the toast stack sits above the bottom band (bottom-24, the UpNextCard rung) and
  lets clicks between cards through: at the app's own spot it covered the Plex transport row and
  PlayerModal's docked bar and took their clicks while a toast was up (measured: play/pause, ±10,
  stop, prev/next, mute all hit the toast).
- ui/Modal and the onboarding wizard mount an overlay host too, so a toast raised under any dialog
  is readable and a click on it no longer closes the dialog; the Modal's Tab trap spans the whole
  dialog, so a toast's action and dismiss are keyboard-reachable. A test now enforces the
  convention: a component that covers the viewport at z-overlay must mount <Toaster overlay />.
- PlayerModal's host moves into the stage in true fullscreen (outside the element holding the lock it
  was painted under it).
- A host that takes over shows the toasts it inherits silently (no second role=alert, its region
  aria-live=off until they are in), and each countdown bar starts as far in as its toast already is
  instead of restarting while the real dismiss timer does not.
- The overlay-host registry uses lib/store instead of a hand-rolled listener triad.
2026-09-11 18:04:54 +02:00
peter 5c5cabbe5a fix(plex): the player drops a session answer that lands late, and Next neither loses nor lends a position
Deep-review round 3 of bug hunt 2026-09.

- loadSession checks, after its await, that it is still the newest call and the player is still
  mounted. A late answer used to replace what was playing (Next twice quickly: the slower transcode
  landed last and played the middle item under the last one's title, recording its progress there)
  or attach a stream to the detached <video> after Back, never destroyed.
- The <video>'s clock is trusted only once the new session has answered. go()'s pause() fires a
  timeupdate AFTER the reset to 0 (measured in Chrome), so the previous item's position was sent as
  the next item's subtitle position and saved as its progress (final, pushed to Plex).
- Next settles the item being left (final checkpoint + reopen-cache patch) before switching: the
  progress cleanup ran with the position already reset and skipped, so every Next dropped the last
  stretch watched.
- A seek-restart puts its target into the position state too; the ref alone was re-synced back to
  the pre-seek spot by the restart's own render.
- The unmount teardown captures the <video> at mount: React detaches refs before an unmount's
  cleanups, so the stop/detach never ran.
- The tracks panel exposes the selected audio/subtitle track via aria-pressed.

Verified in real Chrome on localdev (Westworld S1E6 → Next): the item left was settled at 783 s, the
next asked for subtitles at its own start (0), and no progress was written for it.
2026-09-11 18:04:41 +02:00
peter 4f37c41021 fix(plex): a subtitle failure about one position no longer ends the track, and windows are read ahead and never wait behind a full pass
Deep-review round 3 of bug hunt 2026-09.

- `failed` is now final only for a track that can never be text (bitmap, no item/track/file). A Plex
  hiccup or a stretch ffmpeg could not read answers `unavailable`: said, polled again every 5 s, and
  the <track> stays mounted when cues exist elsewhere (gen > 0). Re-picking the selected track asks
  the server again at once.
- Within LOOKAHEAD_S of a covered stretch's end the next window is read ahead, instead of only once
  the playhead has fallen off it (subtitles dropped out at every edge while the full pass was backed
  off or slower than a window lasts).
- Windows and full passes run on separate pools with separate in-flight caps, so a window never
  queues behind (or is refused by) minute-long full passes — the pool comment promised that and
  nothing enforced it.
- A pass that times out is TERMinated and keeps what it reached (published as partial) instead of
  being killed and discarded.
- Lapsed failure memos are swept on every write (they were only pruned when the same key was read
  again, so polling fresh keys grew the process without bound).
- `publish` no longer merges into a sidecar with no VTT behind it, or claims a region whose VTT write
  failed, or overwrites an unreadable track with one window.
- A NUL in a rating_key names no row (it was a DataError, a 500, on every Plex route) and no longer
  500s the status poll.

Tests: 16 new, each mutation-checked (18 mutations, all red on the test that names them).
2026-09-11 18:04:30 +02:00
peter f7faa19ee2 fix(plex): the player starts, and resumes after a session-restart seek, when React mounts it twice
The player's "still alive" flag — every play() is gated on it, so a late one cannot start a detached
<video> after Back — was only ever LOWERED, in an effect's cleanup. React may run a cleanup and mount
again (StrictMode does exactly that in development), and then the flag stayed false for the life of
the player: it never started by itself, and a seek back past the loaded region, which restarts the
session, left it paused. Found on localdev (Westworld S01E06: playing → seek back → the picture
settles, paused). Production does not double-mount, which is why only dev showed it — and why
several earlier localdev measurements "stayed paused".

The effect now raises the flag in its setup. A new test renders the real player under StrictMode and
checks it plays once the media is ready; red without the fix. Verified on localdev: it starts by
itself, and keeps playing through a backward session-restart seek.
2026-09-11 03:47:36 +02:00
peter 0b4b268678 test(feed): pin the sort options' default directions as literals
The feedSortOptions test compared each option's default direction against SORT_DEFAULT_DIR — the
implementation's own table, so a wrong entry passed — and that import was the export's only user
outside its module, which kept knip from seeing it had become dead after F3. The test now states the
directions, the table is private again.
2026-09-11 03:19:17 +02:00
peter d349bc9084 fix(plex): a collection's failed watched toggle is put back the way it was
C1 moved the optimistic flip onto the key the page reads — and so made visible that nothing corrects
it: the resync only invalidated `plex-library`, so a failed toggle stayed "watched" beside its own
error toast, and the collection's other cached sort orders stayed stale after a good one. The
resync now takes this page's `detail` prefix too, as the Discover and person pages do for theirs.
The C1 test holds the request open, so it still pins the optimistic write on its own.
2026-09-11 03:19:17 +02:00
peter dcd8a9566a fix(playlists): the player's Remove follows the list it was launched from, the player survives a change of selection, and a gone list is said in the reader's language
- D3 snapshotted the removal's TARGET, but whether the player offered Remove still followed the
  selected list: launched from Watch later (no Remove, by design), selecting another list under it
  made Remove appear — and delete from Watch later. The snapshot now carries the launch list's
  editability too.
- Selecting another list under an open player (the command palette works over it) put the page into
  "Loading…", and the player was rendered only in the loaded branch — it vanished mid-playback and
  came back at its launch video. It now sits at one place in the tree whatever the page is doing.
- A removal takes the video out of the CACHED list as well as the rows: the player's progress
  checkpoint rewrote that entry before the refetch landed, handing the membership guard a list that
  still held the video — the row came back and the undo history went.
- The watched-state callback refreshed the selected list, not the launch list (D3's sibling).
- `_own_playlist`'s 404 is structured (`{"code": "playlist", "reason": "not_found"}`) and localised:
  the bare "Unknown playlist" reached the Hungarian UI in English, which also made round 1's new
  `removeFailed` string unreachable. Both Remove and Move report through `notifyActionError` now.
- The "viewer moved on under the confirm" test stepped back only once, which lands on the same video
  whether or not the removed one left the queue; it now takes the second step that tells them apart.
2026-09-11 03:19:07 +02:00
peter 6de2cd950d fix(ui): toasts raised while a full-screen player is open show inside it, not under its scrim
The app's Toaster lives in the content column, which is a stacking context (its `@container` implies
layout containment), so both full-screen players — portalled to <body> at the same z-index — painted
over every toast. Round 1's "a failed Remove is reported" fix was therefore still silent inside the
player: its toast sat under an 80% black, blurred scrim (elementFromPoint returned the scrim), and a
click on it closed the player.

An overlay now mounts its own `<Toaster overlay />`, and exactly one host shows the toasts: the most
recently mounted overlay, else the app's. Same component, same look — and inside the aria-modal
dialog, where a screen reader can still reach it. A click on a toast inside an overlay stays there.
Both players get one (the Plex player's also works in native fullscreen, where nothing outside the
fullscreen element is drawn). Verified on localdev: the removal failure reads above the scrim,
bottom-left, and clicking it leaves the player open.
2026-09-11 03:18:55 +02:00
peter 645d14f45d fix(plex): a subtitle track's failed load is heard and retried with a pause, and the poll asks about the right item, position and failure
- The <track>'s `error` never reached `onSubTrackError`: it does not bubble, and React 18 attaches no
  listener of its own to a <track> (measured in Chrome and jsdom — `onError` 0 calls, `onErrorCapture`
  1). Round 1's remount-and-give-up was dead code behind it. The hook now hands the player
  `onErrorCapture`, and the test fires a real non-bubbling `error` at a rendered <track>.
- Once reachable, the retry burned its whole budget within milliseconds of an outage (a remount
  fetches at once) and the give-up outlived turning subtitles off and on. Retries now wait (2 s,
  doubling), running out is reported at once, and picking the track again starts a fresh budget.
- Until the first timeupdate the poll sent `sessionK` — still 0 while the session request was out,
  and the PREVIOUS item's start after a move to the next episode. It now sends where the session was
  asked to start, and the track choice belongs to the item it was made for, so the next item is not
  asked about the old ordinal at the old position.
- A failing poll over a stale `partial` answer was silent: react-query keeps the last good data, the
  track stayed "showable", and the notice only rendered for an unmounted track. The hook now decides
  what the player says (`subNotice`), independently of whether a window is mounted.
2026-09-11 03:18:42 +02:00
peter 7360339039 fix(api): a non-finite number in a JSON body is refused with a 422, not a 500
The JSON parser under FastAPI accepts `Infinity`/`NaN`, and both progress checkpoints truncate the
position with `int()` — a hand-written checkpoint came back as a 500 with a traceback. Their bodies
now refuse non-finite floats, and the app's validation handler — FastAPI's own shape — echoes a
non-finite input by name: the default one failed to serialise its own error (JSON has no NaN), which
was a 500 by another route. Sibling of the subtitle routes' non-finite query parameters.
2026-09-11 03:18:17 +02:00
peter 473d9f02f2 fix(plex): subtitle windows land where the full pass does, and the extractor stops re-reading, respawning and outliving the process
Deep-review round 2 of the 2026-09 bug hunt, over the round-1 fix diff.

- A window was LATE by the distance to the keyframe its input seek landed on, and then doubled for
  good once the full pass merged in: a cue already running at the window's start came out at a
  negative time, and the webvtt muxer's default shifted the whole output so it sat at 0. Measured on
  the real Westworld S01E06 at 25:00: every window cue +5.005 s, 682 cues merged instead of 608.
  `-avoid_negative_ts disabled` keeps the times exact (the parser reads ffmpeg's signed-field
  spelling of a negative time); now 608/608 on that file, and a real-ffmpeg test pins it.
- A full pass that failed was re-armed every minute while the viewer watched what it did publish —
  a whole-container read per minute. It now backs off for an hour; windows keep running meanwhile.
- A failed WINDOW set the track's failure memo, so any uncovered position read "failed" — a final
  answer that ends the player's poll, even with the full pass about to deliver, and one any account
  could plant with an absurd `at`. It now marks only the span it failed to read; "failed" is given
  once the full pass AND a window there have both failed.
- Windows join by what they COVER, not a bucketed identity: a backward seek no longer waits behind a
  window that cannot cover it, and a playhead crossing a bucket edge no longer queues a second read.
  The queue is bounded per track and overall.
- shutdown() sets a flag a worker checks before it spawns ffmpeg (one still in its Plex lookup used to
  start a read after the kill sweep); queued-job cancellation and the lifespan call are now tested.
- A window starts where ffmpeg is told to seek (rounded to the ms it gets), a rounded-up millisecond
  carries into the minute (no more `00:00:60.000`), a key's first publish claims nothing before its
  VTT lands, and `at`/`offset`/`start`/`aoff` refuse non-finite values (`at` also an absurd one).
2026-09-11 03:18:08 +02:00
peter d2e225666b test(playlists): let react-query's notify tick run inside act before the reopen click
The D1 regression test patched the cache and clicked at once; react-query delivers a cache write to
its observers on a setTimeout(0), so under the gate's parallel load the click could beat the page's
re-render and read the old row (748, not 830). Test-only race — a human reopen comes seconds later.
Stable across three full-suite runs; the D1 mutation still turns it red.
2026-09-11 01:45:53 +02:00
peter efd95fca01 fix(ui): the card's channel name is a real link too; call sites the bug hunt left untested now have tests
Deep-review R6 — the small items:

- The video card's channel name was still a <button> (pre-existing): ctrl/middle-click did nothing
  there while the title above it and the player's channel name honoured them. It renders through
  ChannelNameLink now, which grew a forwarded ref (the clamp measurement), pass-through attributes,
  a `tooltip`, and an opt-in `stopPropagation` so no click reaches the card's row.
- ChannelNameLink rendered an empty, still-tabbable <a> for a channel with no title yet (a stub first
  seen through a live search; pre-existing at this site). It falls back to the translated "Channel".
- The feed's SortKey→SortOption mapping existed twice in Feed.tsx; one `feedSortOptions` in
  feedSort.ts serves both modes (DIRECTIONLESS is module-private again).
- The backfill refresh's comment claimed a one-page cost. It refetches every loaded page of the
  grid, and that is the right price: backfilled videos are older, so they land below the head.

Tests for call sites that could be deleted with every lane green: the channel page running the
backfill refresh (knip had been blinded by the hook test's import), Feed handing the grid the live
results in the chosen key AND direction, the card using the link. Plus the alt-click arm of the shared
click predicate, and a plex filter-chip test that claimed it made no request while firing a real one.
Verified on localdev: the feed's channel names are anchors (same 14 px muted look), a plain click
opens the channel in-app. 7 mutations, each red on its assertion.
2026-09-11 01:38:31 +02:00
peter 060deb89d2 fix(playlists): "remove" inside the player targets the list it came from, reports failure, and keeps the cursor and focus honest
Deep-review R5 — five defects in the Remove button f76a2afb put inside the YT player:

- It removed from whichever playlist was selected by the time it was pressed, not the one the player
  was launched from; the selection can change under an open player (the command palette opens over
  it). The launch list is now part of the player snapshot, and the optimistic row removal only
  touches the list actually on screen.
- A 403/404 failed in total silence: the catch claimed the global dialog had reported it, but `req`
  deliberately leaves those to the caller. It now notifies, as moveItemTo already did (the silent
  catch predates this round; the full-screen consumer did not). New key playlists.removeFailed, HU+EN.
- It computed the next item from the cursor as it was when the button was pressed, but prev/next and
  auto-advance keep working under the confirm dialog: the viewer was thrown back onto a video they had
  moved past. It reads the live cursor, and if they moved on it just drops the item from the queue.
- The button was disabled for the confirm's lifetime, so the dialog's focus restore hit a disabled
  element and a keyboard user who cancelled was left on <body>, outside the aria-modal player. A ref
  guards the double press instead.
- Its accessible name was a bare "Remove"; it now names its object, like the row control.

The call-site wiring (step on, drop from the queue, close on the last item, leave everything alone
when the removal did not happen) finally has tests. Verified on localdev in the real YT player: a row
reopened at the 136 s it had just stored; Cancel returned focus to the Remove button; a confirmed
removal stepped 1/3 -> 1/2. 10 mutations, each red on its assertion. Scratch list and positions reset.
2026-09-11 01:28:27 +02:00
peter 4869d6c108 fix(player): the resume position reaches the playlist page and live search, not just the feed
Deep-review R4. 1630a3df wrote the YT player's stored position into the react-query cache so a
reopen could not rewind it — but only the feed actually read it back:

- Playlists renders from its own undoable copy of the list, re-seeded only when the SET of ids
  changes, so the patched position never reached it: close a video at 830 s, reopen it from the
  playlist, and it resumed at the old spot and the next checkpoint wrote that over the 830. A detail
  with the same membership now rebases its fresh rows into every snapshot, keeping order and undo
  history. (The ids-only guard is older than this round; the write that depended on it was new.)
- Live YouTube search results carry the same per-user position and open the same player, but their
  cache was never patched. It is now — patched only, never invalidated, since refetching an
  API-sourced search page spends 100 quota units.

Tests for what had none: a Playlists harness (PlayerModal stubbed to record its launch props) and a
PlayerModal harness with a fake IFrame API that records where each item is started — the latter pins
that stepping back to an item watched earlier in the same session resumes at THIS session's position.
3 mutations, each red on its assertion.
2026-09-11 01:19:14 +02:00
peter d20317e0f1 fix(plex): the collection page's watched toggle and Sort control survive the sort joining the query key
Deep-review R3. When 889d2acf put the sort into the collection detail's query key, two things on
the page did not follow it:

- The watched toggle's optimistic write kept the key's old spelling. setQueryData matches exactly,
  so it landed on an entry nothing reads, and its resync only invalidates the library root — the
  click succeeded server-side and changed nothing on screen (badge, icon, aria-label) until a
  reload; a second click toggled it back. One `detailKey` now serves the read and the write.
- Every sort change was a key with no data yet, so the whole page — hero, grid, and the Sort control
  with its open panel — was swapped for a spinner, and focus fell to <body>: key and direction could
  not be set in one visit. The previous page now stays on screen while the new order loads, for the
  SAME collection only (stepping to another must not show this one's hero meanwhile).

Verified on localdev (Rocky Collection): the toggle flips the poster; IMDb rating then Descending set
in one open panel, the hero never blanked, order reversed. 3 mutations, each red on its test.
2026-09-11 01:12:51 +02:00
peter d2ba65e383 fix(plex): the subtitle poll backs off on errors, a failed <track> is replaced, and a resume asks for the right window
Deep-review R2, plus one defect found while verifying it on localdev.

- The status poll re-armed every second after a FAILED request: `data` stays empty on an error,
  so it read as "still preparing" for as long as the player was open — and the request was not
  quiet, so a persistent 500 re-opened the global error dialog every second. It now backs off to
  10 s (a blip still heals on its own) and is quiet; the player already shows the failure inline.
- A <track> whose own VTT fetch failed never fetched again: with the server's answer unchanged its
  key was unchanged, so React kept the dead element — silently blank subtitles. The key (now built
  in the hook, where it is tested) carries a failed-load count; after two remounts the track is
  reported as unavailable instead.
- Found in UAT: until the first timeupdate `abs` is 0, so after a resume every poll said at=0 and
  the first window was cut around the head of the film (a resume at 20:00 polled at=0 for as long
  as it stayed paused). The poll reports the session's start until playback ticks. And the window's
  dedup identity is the track plus a window-length bucket of its start — keyed on the track alone
  (R1), the right window queued behind that wrong one.

The poll's start/stop rules finally have tests (every earlier one resolved a single fixed answer).
Verified on localdev against Westworld S1E6 resumed at 20:00: the polls carried at=1197, one
subtitle line on screen at 21:02, the poll stopped at `ready`; the real file's window (56 cues) +
full pass (608) merged to 608 — the old merge would have served 664. 11 mutations, all red.
2026-09-11 01:08:27 +02:00
peter cf7ec5e9ec fix(plex): the subtitle extractor's seams — doubled cues, a truncated track cached as whole, a poll that respawned ffmpeg
Deep-review of the bug-hunt round (R1). The background extractor from ed4ab4e1 held, but its seams
did not:

- Window + full pass doubled every cue in the first hour. merge_vtt de-duplicated on the raw timing
  text, and the two passes spell a timestamp differently (shift_vtt writes HH:MM:SS.mmm, ffmpeg's
  webvtt muxer drops the hour below 1 h; shift_vtt(-0.0) never normalised the full pass). Timing is
  now parsed and re-emitted canonically, keyed with the cue text so two cues sharing a time survive.
- A full pass that exited non-zero was published as complete: "ready" forever, subtitles ending
  where ffmpeg died, and no eviction to ever retry. It now covers only up to its last cue, is never
  complete, and records the failure memo.
- The window's dedup identity held the live playhead, so every 1 s poll queued another read of the
  file. It is keyed on the track now, and a queued job re-checks coverage before reading anything.
- A sidecar track waited behind an unrelated embedded full pass (the full job is per item). Its
  window job now fetches it whole — one small request.
- The <track> fetch ran the scheduler at position 0: an unrequested window over the head of the file,
  and a refusal of a window that was on disk. It is a pure read of the cache now.
- publish() is serialised (a lost update became permanent once complete was set); the pool is shut
  down and running ffmpeg children killed at app shutdown (they held interpreter exit for minutes);
  ?at=inf is a 422, not a 500; and both subtitle routes use current_user instead of the cookie-only
  poster gate, which let a suspended or epoch-revoked session keep scheduling reads (pre-existing).

_extract_and_publish finally has tests of its own — every earlier test stubbed it out. 19 mutations,
each red on the assertion that names it.
2026-09-11 00:52:59 +02:00
peter f76a2afbf1 feat(ui): the four small gaps — player delete, repeat-move, a real channel link, YT search sorting
Items 2, 4, 5 and 7 of the bug hunt, one round. They are independent, but they are all the same
kind of thing: an action that exists somewhere else in the app and is missing where you actually
want it.

**5 — the channel name in the YT player is a link again.** It was a `<button>`, so middle-click and
Ctrl/Cmd-click did nothing on a thing that looks exactly like a link (the YouTube icon beside it IS
an anchor and opens a tab). It is now a real `<a>` whose href is the same `?chan=<id>` URL
`openChannel` pushes, with a plain left click still taking the in-app path — that path does more
than navigate, since the player must close and reclaim its own Back-to-close history entry first.
The "is this click ours or the browser's" test is now ONE predicate (`isPlainLeftClick`), which
VideoCard's card link also uses, so a second such link cannot forget the middle-click arm.

**2 — "Remove" from inside the player.** The parent owns the removal (the SAME `removeItem` the row
menu and the inline ✕ use: one confirm, one optimistic update, one undo-stack rebase), and the
player owns what happens next, because it is the only side that knows which item is playing — its
launch queue is frozen at mount. It drops the item, plays what followed, and closes when the list
runs out. That forced a real fix in Playlists: the player's mount was derived from `shown` every
render, so anything that dropped the launch row yanked the player off the screen mid-playback. That
was a latent edge (change the view and it happens) and the normal case for this feature. The parent
now snapshots what it launched with, and only onClose takes the player down.

**4 — "Move to <the last list you moved to>".** The submenu is two levels deep and moving a run of
videos into the same list is the shape this gets used in. The remembered id is a per-account pref;
the row is derived from `moveTargets`, which already excludes the list you are looking at, so it
cannot offer a move to where the video already is. Remembered only on a SUCCESSFUL move — a
shortcut to a move that failed is a shortcut to failing again. Absent, not disabled, when there is
nothing to repeat: there is nothing to explain.

**7 — the live-YouTube results can be sorted.** Client-side, over the page already loaded. Not a
new `order=` param, and this is the decision worth recording: the owner's case is "even one page of
results", i.e. reorder what is on screen; a re-query would REPLACE the results, which is not what
"sort these" means; and the zero-quota scrape source has no order parameter at all, so a
server-side control would silently do nothing (or quietly cost 100 quota units) depending on which
source served the page. Only keys a search hit carries are offered — `subscribers`/`priority`
describe a subscription, which a search hit is not, and `shuffle` on a ranked list is noise. Its
own pref, so sorting a search page never re-sorts the library behind it.

Verified in the browser on localdev against the real account: the channel name renders as
`<a href="…&chan=…">` and a ctrl-click / middle-click is left to the browser while a plain click
still opens the channel in-app; removing from the player stepped 1/3 → 1/2 onto the next video and
closed on the last one; the row menu grew "Move to “…”" only after a move and only on a list that
is not the target; and the results sort reordered by views 41.7M → 29.8M → 24.2M → 15.8M. (The
pane's own ctrl-click arrives without the modifier — a harness limit, so the modifier cases were
driven as real MouseEvents.) Scratch playlists and the touched prefs were cleaned up afterwards.

22 vitest, mutation-checked against 14 mutations (no href at all · href without the rest of the
query · every click hijacked in-app · the middle-click arm dropped from the predicate · plain click
no longer in-app · always jumping to the top of the list · the removed item left in the queue · no
clamp when the last item goes · an empty queue not reported as "nothing left" · sort-then-reverse
flipping ties · unknown values first · titles compared numerically · shuffle offered on search
results · direction ignored) — each red on exactly the test that names it. Item 4 has no test of
its own: its rule is enforced by construction, not by a check.
2026-09-10 22:34:12 +02:00
peter 889d2acf52 feat(plex): the collection page sorts, and a linked collection has a name
Two halves of the same report ("Collectionben nem működik a rendezés?"). The pasted browse URL
sorts correctly — that part was working. What did not:

The Collection INFO page had no sort control at all and a hardcoded release-year-ascending order.
It now shows the browse grid's Sort control, with the browse grid's vocabulary, because a second
sort vocabulary on a neighbouring surface is exactly the kind of drift this repo keeps paying for:
`PLEX_SORT_KEYS` is now one tuple in `plex/query.py`, and a test pins BOTH implementations to it —
browse's key→column map (extracted from the handler to be reachable) and the collection page's
key→member map. A key added to the vocabulary without teaching one of them fails the gate instead
of putting a dead option in the menu.

Ordering a collection cannot be a single ORDER BY: its two legs are different tables, so the merged
member list is sorted in Python. Members with no value for the key sink to the END in both
directions (a movie with no rating is not "the worst rated" — the browse grid's `nullslast()`), and
ties always resolve title-ascending, so flipping the direction reverses the ranking rather than the
alphabet inside it. `lastwatched` needed the show leg's `resumed_at`, so the page now takes ONE
rollup pass that yields both the badge status and the recency instead of the status alone — without
it that menu entry would have ordered every show as never-watched. The default stays release-year
ascending: a franchise reads in release order, and that is what the page did before.

The other half: a shared Plex link carries the collection filter as bare rating_keys, so a link
opened cold had no name for them — the sidebar token rendered `17045` and the chip could only say
"Collection". `GET /api/plex/collections` takes a `keys=` CSV now, and a display-only hook resolves
just the missing ids. It deliberately does NOT write them into `collectionTitles`: that map is
hashed into the Plex query keys, so storing a resolved label there would split the cache from the
identical state and refetch the grid for a piece of text.

Verified in the browser on localdev against the real Plex library: the movie collection's menu
offers Length and the show collection's does not (the same scope rule as browse); rating-ascending
reorders 8.1/7.4/7.2/6.6/6.5 correctly; `lastwatched` on a show collection brings the in-progress
shows up; the choice survives a reload (it is a server pref, on its own key so opening a collection
never re-sorts the library behind it); and the reported URL now reads "Pirates of the Caribbean
Collection" in both the sidebar and the chip.

12 pytest + 8 vitest, mutation-checked against 14 mutations (direction ignored, nulls first, no
stable tiebreak, unknown key untouched, wrong default, show recency dropped, empty `keys=` falling
through to everything, a key missing from either sort map, every id refetched, ids unsorted,
fetched titles unmerged, the chip ignoring them, the hook's result never reaching the builder) —
each red on exactly the test that names it.
2026-09-10 22:05:39 +02:00
peter 782362b808 fix(channels): the video grid follows a running backfill instead of freezing until F5
"Get full history" on a channel page left the list on the cards the channel already had. The
count above it climbed into the thousands, the grid did not move, and only a browser F5 brought
the rest in.

The mechanism: the backfill runs in a BackgroundTask, so the PATCH that starts it returns before
a single row is written — and the mutation invalidates `qk.feed()` on that RESPONSE, which just
refetches the OLD catalogue. Nothing afterwards refetches it: the feed's infinite query has no
interval, and the app-wide defaults are `staleTime: 30s` + `refetchOnWindowFocus: false`. The job
emits no event either (the realtime layer only carries messages today), so the channel detail's
4s live poll is the only thing in the app that sees the job progress — and it refreshed only
`qk.channel`.

So the poll now drives the grid too: on a tick where the stored count moved (or the backfill
finished), it invalidates `qk.feed()` and the `qk.feedSummary()` behind the count pill. The window
is the poll's own (`deep_requested && !backfill_done` on the previous reading), so a detail
refetch caused by some other mutation — each of which invalidates the feed itself — cannot come
through here twice.

Measured on localdev against the real account, same instrument, `document.visibilityState`
forced visible (the poll deliberately pauses on a hidden tab, and a headless pane reads hidden —
without that the first measurement was about the harness, not the fix):

  before — CASCADA 19 -> 176 stored: the header updated live to "162 normal", the grid and its
           count pill sat on "5 items" for 50s.
  after  — fern 54 -> 139 stored: "54 items" -> "139 items" by ~12s, no reload.

6 vitest, mutation-checked against six mutations (no invalidation at all; grid only, count pill
forgotten; predicate ignored; no channel-identity guard; completion with an unchanged count
dropped; poll-window guard dropped) — each one red on exactly the test that names it.
2026-09-10 20:56:16 +02:00
peter 5a42d76e8a feat(plex): an off-by-default loudness normalisation toggle for playback
Bug hunt 2026-09, item 14 — "how does the multichannel downmix work, and is it
normalised?" It isn't: `plex/stream.py` has always encoded `-c:a aac -ac 2 -b:a
192k`, i.e. fixed stereo with ffmpeg's default downmix matrix and no loudness
handling at all, which is why a film mastered for a cinema arrives with whispered
dialogue and deafening action.

New admin knob `plex_audio_loudnorm` (Configuration → Plex), DEFAULT OFF: it makes
a film markedly louder and narrows its dynamic range, which is a taste call rather
than a fix. It sits at the end of the Plex group, after the software-transcode
knobs — it is an audio concern, not one of the libx264 ones.

  -af aformat=channel_layouts=stereo,loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000

Three things about that chain are load-bearing and each has a test:

  * the downmix is FORCED FIRST rather than left to `-ac 2` — measuring 5.1 and
    folding it down afterwards would normalise something other than what is heard;
  * the trailing resample is not decoration: `loudnorm` runs its own resampler at
    192 kHz internally, which the AAC encoder will not take;
  * one `-af`, no stream index, so EVERY audio rendition of a multi-audio file is
    normalised — not just the first.

The knob is also part of the session directory name (`…_multi-ln_+0.00`), because
it changes the audio the segments contain, and a session left over from before the
admin flipped it must not be mistaken for one that honours it. The suffix is
appended with a DASH: `_SESSION_DIR_RE` treats the tag as one underscore-free
field, and that regex is the only thing keeping the orphan sweep off what may be a
shared scratch path. A test pins both halves — the name changes, and the sweep
still recognises it.

Measured end-to-end on the app's own HLS segments (a real 5.1 AC3 file, same item,
same 160 s offset, same ten segments): OFF -35.2 LUFS, ON -17.2 LUFS. On a 60 s
bench: I -27.6 → -16.6 LUFS with LRA 13.3 → 9.8 LU. A/V sync is untouched — under
this pipeline's own `-copyts` the audio start PTS is identical with and without the
filter, so the 3 s lookahead desyncs nothing.

Direct-played files are served untouched and stay so; a 5.1 mix is always `remux`,
so the reported case is fully covered.
2026-09-10 20:02:06 +02:00
peter ed4ab4e13c fix(plex): subtitles stop blocking the request, and stop re-reading the file
Bug hunt 2026-09, items 8 and 12 — the two things the Plex player was measurably
lying about.

Item 8: picking a subtitle track showed nothing for a long time, and the reported
workaround was to stop/start until it suddenly worked. Measured on prod against a
17.06 GB MKV with two embedded SubRip tracks:

  full pass, one track                154 s
  full pass, both tracks in ONE run   111 s
  -c:s copy (no WebVTT conversion)    143 s
  from 1:00:00 to the end of file      66 s
  a 630 s window at 1:00:00, both       8 s

So the conversion was never the cost: ffmpeg streams every byte of the container
between its seek point and its stop point, and the array delivers ~140 MB/s. The
only lever is reading less. Three changes follow from that:

  * a window around the playhead is extracted and published FIRST (seconds), and
    the full pass completes behind it;
  * every embedded text track rides that same single read — picking Hungarian used
    to cost a full pass and then switching to English cost another one;
  * none of it happens inside a request any more. The endpoint answers 202 while a
    background worker extracts, the player polls /status (carrying its position, so
    the window follows the viewer), and one ffmpeg run finally has a timeout.

-copyts is deliberately absent from a windowed pass: with it ffmpeg ignores -t and
reads the whole container anyway (measured — a [0, 600 s] window still took the
full 145 s), so a window comes back on a zero-based clock and is shifted to
absolute time here. Cached tracks written by older builds have no coverage sidecar
and count as complete, so nothing is re-extracted after the upgrade.

The <track> is no longer mounted before the server has something to put in it; a
"Preparing subtitles: …" notice stands in, and a track that can never be shown says
why instead of leaving the viewer to conclude the picker does nothing.

Item 12: "Bitrate" read a constant 211 kbps on every remux, because it was the
master playlist's BANDWIDTH — which ffmpeg derives from the one stream whose rate
it knows, the -b:a 192k AAC track, plus 10%. It is a measurement now: bytes per
second of MEDIA from hls.js FRAG_LOADED, accumulated per fragment type and summed
(a multi-audio session's video and audio renditions cover the same media seconds,
so pooling them would report half the truth), and an honest dash until the first
fragment lands. The panel also finally names the audio and subtitle track in play.

Verified in the browser against the real Plex library, and every new test
mutation-checked.
2026-09-10 19:31:05 +02:00
peter b8bff504d8 fix(plex): "Clear resume point" now tells Plex about an in-progress item
The button deletes the local row and mirrors the un-mark to Plex, but it decided whether
to push from the previous STATUS alone (prev_status != "new") — and an in-progress item's
row is status="new" carrying a position, which is exactly how the importer maps a bare
viewOffset. So for the single case the button exists for, nothing was ever sent to Plex:
the offset stayed, and the next pull imported it straight back. The user cleared Protector
twice and it returned to Continue watching at 92s both times.

Push now when Plex holds anything to clear — a view count, or a resume point. "hidden"
stays Siftlode-only and pushes nothing, which is what this endpoint's docstring always
claimed and the status-only guard contradicted.

One unscrobble clears BOTH viewCount and viewOffset — measured against the live server from
a watched+offset state and an offset-only one, so the push itself needed no change. The
client docstring that claimed "clears viewCount" is what sent the first diagnosis down a
wrong path (a timeline write with time=0, which does not clear an offset at all); it now
records what the server actually does.

Verified end to end on the real Plex server: the button leaves viewCount=None viewOffset=None,
and a full 44,885-item re-import brings back no row for it.
2026-09-09 23:55:51 +02:00
peter 1630a3df3c fix(player): the YT progress checkpoint reaches the cache, so a reopen cannot rewind it
The player checkpointed every 5s and the server stored it, but nothing wrote the new
position back into react-query: the close path only marked qk.feed() stale with
refetchType "none", and nothing refetches the feed until it remounts (staleTime 30s,
refetchOnWindowFocus off). So the resume read the position the cache had been FETCHED
with, and the next checkpoint then persisted that older value over the newer one.

Measured on localdev: played to 830s, closed (server had 830), reopened the same card
without a reload -> the row was back to 751s. 79 seconds of progress destroyed by a
reopen, and the card's progress bar never moved either.

The Plex player already solved this for its detail query, with a comment describing the
identical failure; this is the feed-side twin that was missed in that sweep. patchCachedProgress
writes the position the SERVER stored (the endpoint clears trivially-early and near-finished
checkpoints to 0, so mirroring our own reading would disagree with the row) into every cached
feed page and playlist detail, keeping array identity where nothing changed. A session map
covers the second half: the launch queue is frozen at mount, so stepping next->prev back to an
item played earlier in the same session resumed from that snapshot.

Verified in the browser: 811 -> close 816 -> reopen 825, and the card now shows Continue/Restart.
2026-09-09 23:55:51 +02:00
peter 14b5fe2e31 release: 0.83.0 (feature/e24-anti-paralysis)
gate / frontend (push) Failing after 2m5s
gate / backend (push) Successful in 7m0s
2026-09-09 22:04:14 +02:00
peter d260aabfd7 chore(release): 0.83.0 — the Home surface 2026-09-09 22:01:33 +02:00
peter d832bb19ce fix(e24): review round 4 — a test that pinned nothing, and prose still ahead of the code
Round 4 reviewed round 3's commit. Nothing in production behaviour was wrong; what was wrong
was the round's own test and its comments, which is the failure class this loop keeps
finding.

- The new NavigationProvider case named after the `&& !channelViewRef.current` clause never
  reached it. With storage cleared and a bare URL the provider lands on "home" (E24's own
  default), not "feed", so asking for "feed" was an ordinary page change and the guard's
  first conjunct short-circuited. Deleting the clause left all 828 tests green — a dead nav
  control (the rail's own module stops closing an open channel page) with no red anywhere.
  The test now asserts the precondition and calls `setPage` with the page that IS current.
- Both new Home cases rejected a promise and then asserted after one fixed macrotask.
  React-query notifies through `notifyManager`'s own `setTimeout(0)`, registered after any
  timer the test schedules, so the assertion raced it; two reviewers measured intermittent
  reds. They poll for a signal that the rejection LANDED instead — the reroll button
  releasing, which is also the honest one for the empty-answer case, where the empty state is
  already on screen while the press is in flight and asserting it early would pass either way.
- `buryUnder`'s comment still opened with "one rule for both entry points" ten lines above the
  paragraph saying the rule is complete only for one of them, listed the non-reporting palette
  rows without `openChannel` — which has its own `replaceState` no-push branch and is reached
  from PlayerModal too — and pointed at a ROADMAP.md this repository does not contain.
- `history.dom.test.ts`'s header claimed its layers close by genuine Backs only, which two of
  its own tests contradict; and `CommandPalette.ordering.dom.test.tsx` still claimed to be
  "the only assertion in the suite" doing what the file corrected in round 3 also does. Round
  3's message said that pair was handled and only one side had been.

The provider test was re-mutation-checked (dropping the clause now goes red), and both
hardened Home tests still go red against their own reverts. Seven consecutive full runs green.
Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 828, pytest 1005.
2026-09-09 01:02:22 +02:00
peter 59efc8f2e8 fix(e24): review round 3 — the holes round 2's own fixes left
Round 3 reviewed the round-2 commit across six dimensions. Fifteen findings, five survived
three adversarial refuters each; three died on 2+ refutations (a keyboard-focus claim that
is a property of `ui/Button` at all six of its `loading` call sites, a contract-test
false-pass that needs a deliberate revert plus a self-incriminating comment, and a
MAX_EXCLUDE coupling whose slice is unreachable from the real client).

- `rerolling` was still wrong for a FAILED press. `data === undefined` is true forever for a
  key whose request failed (`retry: false`), so the next unrelated mutation re-disabled the
  reroll button — the same defect round 2 reported, one branch over. `!isFetched` is the
  flag that remembers a key settled; `!isError` would not have worked, because query-core
  resets a data-less query to "pending" for the duration of a refetch, hiding the error
  exactly when it matters.
- The hero fallback used `??`, which cannot tell "this budget's last pick resolved to NO
  hero" from "nothing remembered". So a press that failed after an empty one resurrected the
  card the page had already excluded as seen and just said it had nothing beyond. The tagged
  slot is now authoritative for its own budget.
- `setPage`'s boolean — the input the whole burial rule reads — was asserted by nothing.
  The type fences the SHAPE, so deleting a `return` fails tsc, but swapping the two values
  passed typecheck and all 822 tests. NavigationProvider gets its first render test, on the
  value AND on the effects the early return skips.
- `closeThenNavigate`'s door into `buryUnder` was equally unpinned: reverting that one call
  site to the old unconditional burial left the suite green. It has the mirror case now.
- The comment claiming one rule serves both entry points was overstated. `buryUnder` fails
  OPEN, and `CommandItem.run` is typed `() => void`, so only the palette's bare
  `() => setPage(page)` rows forward the boolean at all. Nothing regressed — that path buried
  unconditionally before — but the prose said more than the code does, so it now says what is
  true and points at the deferred fix. Same for a "the only assertion in the suite" claim
  that the file it cites as precedent also makes.

Every new test mutation-checked red for the right reason, both edges where there are two.
Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 828, pytest 1005.
2026-09-09 00:39:37 +02:00
peter e4ffd59ca7 fix(e24): review round 2 — a buried history entry, a stale hero, and a spinner that was not ours
Nine of the round's ten surviving findings. The tenth (a failed shelf reorder leaving an
order the server rejected on screen) is on ROADMAP.md: its obvious remedy reintroduces the
overlapping-move defect the code already documents, so it wants a design, not a review edit.

- `navigateOverOverlays` buried the live overlay entries before knowing whether the
  navigation pushed one. `setPage` no-ops when the page is already current and Home is the
  default landing page, so Shift+W over a player ON Home marked its entry buried under a
  push that never happened: the unmount consumed the mark instead of reclaiming the entry,
  and the next browser Back was a press that did nothing. `setPage` now reports whether it
  navigated, and one `buryUnder` rule serves both entry points — `closeThenNavigate` takes
  the same no-op navigation from a palette command for the page you are already on.
- The hero could answer a new time budget with the previous one's pick. The chip row resets
  seed/seen/press but cannot reset an answer that already resolved, and `keepPreviousData`
  chained through the disabled seed-0 query — so a reroll under "10 perc" showed the film
  picked for "Bármennyi" while it loaded, and kept it if the request failed. `lastPick` now
  carries the server's own budget echo, and the placeholder is scoped to the budget.
- The reroll button spun and swallowed clicks after mark-watched, bookmark, pin and shelf
  moves: the pick key sits under the invalidated `discover` prefix on purpose, so raw
  `isFetching` was true for background refetches. It now means "a press of mine is
  outstanding" — which still covers the first press, where there is no placeholder.
- Failed writes on Home toasted the page's LOAD-failure string over a fully loaded page;
  they get their own key (HU+EN). The empty-state reroll had no `loading`, so the press was
  unacknowledged and a second click skipped a tier. `danger` moved into the shared unpin row
  so the same command is not red on the shelf and neutral in the rail below it.
- Two comments claimed things the code does not do: the `exclude` cap guards no "unbounded
  scan" (it never reaches SQL), and the route test that named the cap sends three keys.

Tests: the burial rule gets its own file (it is the one assertion that lets a reconcile
issue a real programmatic back()); App's keymap wiring gets a source contract, because
dropping it left all 818 tests green; `add_pin`'s IntegrityError arm was reached by nothing.
Every new test mutation-checked red, both edges where the predicate has two.

Gate: ruff, ruff format, 3x tsc, eslint (0 errors), prettier, knip, vitest 822, pytest 1005.
2026-09-09 00:12:53 +02:00
peter 3709b71a94 fix(e24): the review's low findings — dead constants, honest labels, a 500 that should be a 404
`add_pin`'s IntegrityError handler assumed the unique constraint was the only way in and
re-raised otherwise, so a target row pruned between `resolve_ref` confirming it and the
INSERT (a Plex sync dropping a removed item) tripped the FK and handed the user a 500.
Semantically that is the same "no such thing" the pre-check already returns. Untested: the
race needs two concurrent transactions.

The five per-kind unique constraints had no test at all — the route's own dedupe returns
before reaching the DB, and the migration drift check filters constraint diffs out as noise,
so they could go missing with every lane green.

Frontend, all from the same review: the shelf's unpin row is now `danger` like every other
destructive row in the app; the hero's artwork button says what it will actually do and
names the item instead of always announcing "Play"; the empty-budget message offers a reroll
rather than `StateMessage`'s "Retry", which promised a re-run of something that had not
failed; `setState` and the watch-later toggle catch their rejections instead of leaving an
unhandled promise and a click that silently does nothing.

`StateMessage` gained a `compact` rung, which retires the frontend's only two `!important`
overrides — they existed to beat the primitive's own padding from the call site, and were
the copyable idiom for the next surface that wanted a compact empty state.

Dead vocabulary removed or made load-bearing: `DISCOVER_BUDGETS` was exported, re-exported
and consumed by nothing while `Home.tsx` re-typed the same four bands (a `readonly
DiscoverBudget[]` accepts any subset, so a fifth band would have typechecked green and
rendered no chip); `PIN_KINDS`' value export had no consumer at all; `useComfortPin` no
longer publishes `pinIdOf`/`toggle`, which nothing outside the module calls.

Three comments corrected against the code they describe: a pointer to a `resolve.py` that
was never written, the file header's claim that Home defines no card component of its own,
and the `seen` window still quoting the 8 that was measured and abandoned for 32.

The rest of the review's survivors are logged on ROADMAP.md under "Deferred — E24 epic-end
deep-review (2026-09-08)" — the ones whose dissent was substantive, and the ones whose fix
is wider than the finding.
2026-09-08 03:23:59 +02:00
peter cedebe0ffb test(e24): close the review's coverage gaps, and correct a comment that was untrue
`test_only_a_show_row_pays_for_a_resolution` seeded no watch state, so every rail query
inner-joined its way to zero rows and the loop body — with it the assertion — never ran. It
now seeds a video, a movie and an episode, and asserts all three kinds are present before
the loop, so it cannot go vacuous again.

The Plex half of the surprise rail had no test for either of its promises: every test that
reached `plex_surprise` passed budget="any" against a library with no state rows, so the
duration bounds and the already-started exclusion could both be deleted with the suite
green. The YouTube half has had tests for the same two rules since S1 — the asymmetric half
of one principle.

`plex_item` and `plex_show` — two of the five pinnable kinds — were pinned by no test at
all, leaving the rating_key → internal-id translation and both shelf arms uncovered,
including the self-reference cycle that once 500'd the shelf. And `GET /api/discover/pick`
was exercised only through its domain function, so the route's own query-string translation
could regress silently; the new test excludes all three candidates, which is what proves the
whole list parsed rather than one unsplit string.

`PIN_COLUMNS`/`PIN_KINDS` claimed in a comment to be the one place the pin vocabulary is
spoken and had no readers anywhere: `resolve_ref` re-spelled the column names and the route
re-typed the kind list. Both now drive off the constants.

The `_NO_TS` sentinel was introduced as a crash fix and is not one. The key it replaced,
`(ts is not None, ts)`, cannot raise — tuple comparison finds the first UNEQUAL element via
`==`, so two null keys compare equal and `None < None` is never reached; three independent
runs confirmed it over every permutation. The rationale now says what the sentinel actually
buys, lives once in `plex.rollup`, and the third rail that was still on the older spelling
has been swept to match. Its test is renamed and re-scoped to the mutation it does catch.

Frontend: the reroll's second press asserted `expect.any(Array)`, which cannot tell an
accumulating exclusion window from a replacing one — both that and SEEN_WINDOW = 1 (each
reinstating the measured nine-card cycle) left the suite green. And `onInfo`'s jump type was
untested while `onPlay`'s was pinned twice, so pointing the ⓘ affordance at the player
typechecked and shipped green.

All eight fixes mutation-checked; every mutant died on its intended test.
2026-09-08 03:14:21 +02:00
peter fdc1eabbe0 fix(e24): the review's seven frontend defects on the Home surface
Shift+W was dead whenever anything was open. The global handler called `setPage` directly,
so the page change unmounted the player (it lives inside the module column) and that
unmount's reconcile reclaimed the entry the navigation had just pushed — the player closed
and the page snapped back. `closeThenNavigate` cannot serve here: a global key does not
know which overlay is open, or whether one is. `navigateOverOverlays` is the same fix one
step shorter — bury the live overlay entries first, so the unmount consumes the marks
instead of issuing `history.back()`.

`q.isError` had no `&& !data` guard, alone among 33 call sites and against the convention
`ui/QueryState` documents: react-query flips status to error on a failed REFETCH while
keeping the payload, and every mutation here invalidates the discover prefix, so one blip
while marking a card watched blanked a fully populated landing page.

The reroll's key rooted at "discover-pick", outside the prefix every mutation invalidates,
so a rerolled hero outlived the page: play it, finish it, and it was still the answer with
a zeroed progress bar. It now sits under the same root. A FAILED reroll also rewound the
hero to the card the page loaded with, several presses back, because a new key per press
means `keepPreviousData` (pending-only) cannot cover an error; the last resolved pick is
held instead.

The bookmark on a rail tile never updated — `VideoCard` reads `video.saved` from this
payload and keeps no optimistic state — so the star stayed empty and the next click
re-sent the same call.

The shelf reorder wrote a captured snapshot back on failure rather than re-asking the
server, which its own comment promised: two overlapping moves each reverted to their own
snapshot and settled on an order neither side held, and since `before` is the whole
payload a rollback also discarded any rail refresh that had landed meanwhile. It now
invalidates the discover PREFIX on settle, which also fixes the sibling budgets keeping
the pre-move order.

The hero/rail de-duplication matched one key, so for a Plex series — the rail row is the
show, the hero is the episode it resolves to — the same series printed twice; it now
matches both keys, as `hero.pick`'s own exclusion already did. And when the hero empties
the resume rail, the rail is hidden rather than announcing "nothing half-finished right
now" under a hero captioned "where you left off".

Ten regression tests, all mutation-checked. Three of them were rewritten after the first
mutation round showed they could not go red: two asserted before the state under test had
settled, one matched a title the Plex tile does not render.
2026-09-08 03:03:45 +02:00
peter 84f4b20e5a fix(e24): the review's three backend defects — blocklist, shelf order, next episode
The resume rail built its own SELECT instead of going through `filtered_query` and copied
only three of its predicates, so it dropped the per-user channel blocklist the module
docstring claims it inherits. Blocking a channel never touches `video_states`, so a
half-watched video from it kept its resume position and headed the rail — and, at the
default budget with the resume tier leading, became the landing page's hero: the app
answering "what should I watch" with the thing the user had just banned. The clause is now
`feedquery.not_blocked_cond`, shared with `filtered_query` rather than copied a second
time; the feed's SCOPE clauses stay deliberately unapplied, for the reason `yt_video_row`
already gives.

`next_position` returned the shelf's row COUNT while nothing ever renumbers positions —
`remove_pin` deletes, and every FK cascade removes a pin when its target disappears. Past
the first hole the count under-counts, so a new pin landed in the middle of the shelf and,
past the second, on a position another pin already held. `max(position) + 1`, the idiom
`routes/playlists` already uses.

`next_episode` took the FIRST in-progress episode in broadcast order while `plex.rollup`,
which drives the show page's Resume button, takes the last. One show therefore had two
different "continue watching" answers, with the Home tile ranking it by its most recent
touch and then offering the episode abandoned weeks earlier. The new test asserts the two
implementations against each other rather than against a literal.

All three mutation-checked: each new test goes red for the defect it names.
2026-09-08 02:32:08 +02:00
peter 828dad015c feat(e24): reach the Home surface — the landing toggle, a palette command, Shift+W
S4's reachability half. Home already led the nav order; what was missing was every
way IN other than the rail.

- Settings → Appearance → Startup: "Start on Home", the documented escape hatch for
  the landing override. Deliberately still localStorage and not a server pref —
  `loadInitialPage` answers before /api/me lands, and a landing module decided after
  the first paint is a visible jump. Its two stored values are now ONE exported
  constant, so the writer and the boot-time reader cannot drift on a magic string.
- The command palette gets "Meglepetés" / "Surprise me": it navigates to Home and
  rerolls the hero on arrival. No second "Mit nézzek?" row beside it — the Home
  module row already matches that phrase through its descriptive title.
- A global Shift+W jumps to Home. A chord, not a bare `w`: the Plex player binds bare
  letters and `w` sits on `s`, so a blind reach for the subtitle key would have thrown
  a viewer out of the film.

The palette's request crosses a lazy module boundary, so it is a token that waits
(lib/homeIntent) rather than an event fired into an empty room. Browser-measured:
taken with a compare-and-clear, because StrictMode's double-invoked mount effect
otherwise rerolled twice and landed on the page two picks deep.

HU/EN parity kept; UAT on localdev in both languages, including the empty shelf.
2026-09-08 00:06:34 +02:00
peter 82914b56ca fix(e24): the Home surface's three UAT findings — Back, the reroll, and a show card
All three were measured before they were touched, and the diagnosis moved twice.

**Back from a Plex title played off Home landed on the Plex module's root.** One
click pushed three history entries: the page navigation, the sub-view opened by
the `jumpTo` hand-off, and — under StrictMode, whose effect body runs twice
before `setJumpTo(null)` re-renders — that same sub-view a second time. So the
first Back visibly did nothing and the second stranded the user on a Discover
page they had never visited.

A cross-module hand-off is the CONTINUATION of the navigation that brought you
there, not a second navigation, so it now REPLACES the page entry instead of
stacking on it (`useHistorySubview.open(view, {replace})`, asked for by the jump
target). Replacing is also idempotent, which is what disarms the StrictMode
double-run. The intent rides on `PlexJumpTarget` rather than being inferred,
because the header typeahead jumps from INSIDE the module, where the entry being
replaced would be a sub-view the user really was looking at.

**"Mást kérek" cycled nine cards and never left the resume tier.** Measured:
twelve presses, nine distinct titles, period nine, every one `tier: "resume"` —
the shelf and the surprise tier were unreachable by pressing the button. Two
causes multiplied: the tiers were tried in a fixed priority order over a 20-row
resume rail, so the button just walked down it; and the client's rolling
exclusion window of 8 guaranteed the wrap.

The press count now rotates which tier answers first (`start`), so each press
offers a different KIND of answer — the user's call, given the choice between
that, a wider window, and a surprise-only reroll. The fallthrough is kept, so a
press does not come back empty because the tier it led with had nothing. The
window is matched to the server's own cap rather than a smaller number of ours.

**A half-watched series on Home opened the show page.** A movie already went
straight to the player; only shows did not — and the comfort shelf and the hero
both resolved a show to its next episode while this rail alone did not, handing
back six seasons to choose between. The rail row now carries `next`, and the
client already played `next ?? card`, so this is a backend fix.

Affording it needed `queries.next_episode` to stop loading every episode of the
show into Python to pick one of them (236 rows for Friends, 227 for JAG) — three
ordered LIMIT 1 probes instead. The show→episode rule now lives in ONE function
(`rails.playable`), which is what the shelf, the hero and the rail all call; two
implementations are how they came to disagree.

Ten mutations run, all ten red for the right test — including a first pass where
the press count reached `api.discoverPick` and stopped there, invisible because
every Home test mocks that module. The slice now has its own URL test.

Verified in a clean browser tab against the live account: one Back returns to
Home, the rotation gives resume/comfort/surprise across nine presses with no
repeat, and Westworld opens at S1E1 rather than at its show page.
2026-09-06 05:38:28 +02:00
peter ccd5f3739a feat(e24): pin to the comfort shelf from every card and row
S3 wires the shelf's editor. The gesture is the E10 right-click menu, so the
decision "this is comfort media" can be made where the thing is — the feed, a
Plex grid, a channel row, a playlist — rather than only on the page that lists
the shelf.

One shared piece, `components/comfortPin.tsx`: the five pin-kind constructors,
the shelf query, and ONE `ActionMenuItem` that switches Pin/Unpin honestly (and
says "not known yet" with a disabled row until the shelf has answered, since a
confident "Pin" over something already pinned is exactly the quiet lie the house
rules forbid). Surfaces that already have a row menu compose the item in
(`Playlists`, `PlaylistsRail`); the rest mount `ComfortPinMenu`.

Two structural notes:

- The feed's menu lives in `VirtualFeed`, not in `Feed` — that is the list, and
  it is also what gives the channel page the gesture for free, since it renders
  `<Feed channelScoped>` all the way down.
- Cards take `menuProps` as a FUNCTION of the row, not the built props object.
  `triggerProps` is stable across renders while its return value is not, and a
  fresh object prop would blow past `VideoCard`'s memo for every card in the
  virtualized feed on every parent render.

`DataTable` and `ChannelLayoutGrid` grow a generic `rowProps` so a channel row
answers the same gesture in both of the manager's layouts; the table stays
domain-free and never learns what a menu is.

On Home the shelf tile's menu is the whole editor: four move rows (front / left
/ right / end, reusing the playlist menu's ordering idiom) plus the unpin, over
menu rows rather than drag & drop — dragging inside a horizontal, scrolling
Carousel fights the scroll, and menu rows are keyboard-reachable for free. The
reorder is optimistic on the cached payload and re-asks the server on failure
rather than showing an order that was never accepted.

The exhausted shelf tile (a pinned channel with nothing unwatched left) is
`aria-disabled`, not `disabled`: measured in Chromium, a disabled button still
receives contextmenu and pointerdown, but it loses focus — and with it Shift+F10,
the only keyboard route to the menu it most needs.

Verified end to end in the browser against the live account: all five pin kinds,
the reorder (persisted across a reload), the unpin, and the honest Unpin label on
a resume tile that is already on the shelf.
2026-09-06 05:07:06 +02:00
peter f3c0b7be23 feat(e24): the Home surface — one question, one answer, both sources
E24 S2. A new `home` module in layout C+ (the mockup the owner picked): the
time-budget chips on top, one large hero that answers them, then the comfort
shelf and a cross-source resume rail. Home is the landing — deliberately OVER
the remembered last module, since landing on whatever you happened to close the
app on is the habit this epic exists to replace; `LS.startOnHome = "0"` opts an
account back out (Settings toggle lands in S4).

No third card component: each unified card carries the native feed / Plex
payload, so the rails render through the existing VideoCard (`cardsSmall`) and
PlexPosterCard, and `aspect` decides the tile shape without branching on source.
Plex playback hands off to the Plex module via a new `jumpTo` type — the player
lives there with its lazy hls.js and its provider tree, and mounting a second one
on the landing page to save one hand-off would be the expensive way round.

Four defects found by driving it in a real browser against the live account, each
now pinned by a mutation-checked test:

- **The reroll gave the same card back.** A new seed only re-draws the SURPRISE
  tier; the resume tier deterministically returns the most recently progressed
  thing that fits, so anyone with something half-finished pressed a dead button.
  `/pick` now takes an `exclude` list of `<source>:<id>` keys and the client keeps
  a rolling window of 8. Excluding a SHOW also excludes the episode it resolves
  to — otherwise the same suggestion comes back wearing a hat.
- **"Nothing to suggest yet" flashed over a full library** whenever a reroll was
  in flight: the new query key has no data for a beat. An empty state standing in
  for a loading one is the dishonesty the house rules name outright. The hero now
  falls back to the loaded payload while a pick is pending, and only trusts a null
  once the pick has RESOLVED.
- **A budget change replaced the page — chips included — with a spinner**, so the
  control you just clicked vanished under your cursor. `keepPreviousData`, the
  same reason the feed uses it.
- **The hero printed twice**, once as itself and once as the first resume tile.

Two smaller UI calls made from the same session: the budget caption sits NEXT to
its chips rather than in SegmentedToggle's justify-between mode, which on an 80rem
page throws the answers to the far edge; and the feed takes the YouTube mark its
own toolbar already uses, since Home now owns the house icon.

Gate green: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 780,
pytest 989. Verified end to end on localdev against the real account: budget
chips re-answer (matched on REMAINING time — a 51-minute video with 2:46 left is
a ten-minute answer), the reroll walks three distinct heroes, and a Plex show
tile opens its drill-down in the Plex module.
2026-09-06 04:14:14 +02:00
peter 53b50231ff feat(e24): the Home surface read-model — hero, resume rail, comfort shelf
E24 S1. The anti-paralysis answer layer, served from our own Postgres with no
upstream call: one hero suggestion, one CROSS-SOURCE resume rail, and the user's
own comfort shelf.

Survey first, because the roadmap line was wrong about the starting point: Plex
already ships the whole trio (E5's Discover rails, including a reshuffling
Surprise me), the YouTube side ships none of it, and nothing anywhere answers
the question across both. The data already existed on both sides — VideoState
and PlexState are structurally identical, the feed has an `in_progress` filter
and a seeded shuffle, and the YT player already resumes position.

- app/discover/budget.py — the time bands. "How much time do you have"
  (10 perc / 30 perc / egy este / mindegy) replaces 15/D's mood questionnaire:
  deterministic, answerable from a column we have, and it picks the source for
  free. Matched against REMAINING time, so a film with 8 minutes left is a
  ten-minute answer. An unknown runtime fits only the unbounded band.
- app/discover/cards.py — a thin unified card that carries the NATIVE feed /
  Plex payload, so the page renders with the existing card components instead of
  a third one. `aspect` is the one new field: a rail needs to know poster from
  16:9, and that is a property of the artwork, not of the module.
- app/discover/queries.py — the SQL, reusing feedquery and plex.rollup rather
  than the Plex Discover ROUTE, whose LibraryFilters narrowing belongs to a
  surface that has a sidebar. One seed drives both halves so a reroll moves
  them together.
- app/discover/hero.py — resume -> comfort -> surprise, with the chosen tier in
  the payload so the client labels it honestly instead of guessing.
- comfort_pins (0083): five nullable CASCADE-ing FKs under num_nonnulls(...) = 1
  rather than a flag or a polymorphic string ref. A flag cannot order a shelf or
  hold a channel; a string ref would let a deleted library item leave a dead tile
  on the one surface whose whole job is to be reliable.

Two honest-state rules the tests pin: the surprise draw excludes anything
already started (the resume tier may have skipped it for not fitting, and
offering it back as "Meglepetés" would contradict the question), and which
source leads the draw follows the seed, so the Plex library can actually answer
"egy este".

Gate green: ruff, ruff format, 3x tsc, eslint, prettier, knip, vitest 770,
pytest 987.
2026-09-06 03:31:26 +02:00
peter 5703c4d713 fix(plex): a null progress timestamp must sort a rail row last, not raise
The Continue-watching and Surprise rails sorted on `(ts is not None, ts)`. That
key LOOKS null-safe and is not: two rows that both lack a timestamp compare
equal on the flag and then compare `None < None`, which raises — so a show whose
rollup has no `resumed_at`, or a movie state written before
`progress_updated_at` existed, could 500 the rail instead of sorting last.

Replaced with an explicit epoch sentinel, and named it beside the other rail
constants so the sibling rail E24 adds cannot re-derive the same broken key.
2026-09-06 03:31:09 +02:00
peter 013f19faca refactor(plex): move content_libs down into the query layer
The browse scope is a pure DB query with no request state, but it lived in
routes/plex/_shared.py. E24's Home surface needs the same scope from a domain
package, and a domain package importing from routes/ inverts the one-way
dependency CLAUDE.md draws.

Moved to app/plex/query.py and renamed to content_libs (public — it is a domain
helper now). Swept all five call sites and both tests in one go rather than
leaving a re-export shim: the canonical-patterns table already names the
routes.admin re-export as the anti-pattern.
2026-09-06 03:31:02 +02:00
peter 8d9a99ce54 release: 0.82.0 (feature/e10-platform-ergonomics)
gate / frontend (push) Failing after 2m7s
gate / backend (push) Successful in 3m36s
2026-09-06 02:37:10 +02:00
peter 45dd4bdda7 chore(release): 0.82.0 — E10 Platform & Ergonomics 2026-09-06 02:37:07 +02:00
peter dac391a890 fix(e10): review round 3 — settle the focus race in a browser instead of guessing again
Round 3 reviewed round 2's diff. Trend across the loop: 32 findings over
the whole epic, 15 over round 1's fixes, 11 over round 2's. What it caught
is mostly the same shape as before — a guard narrowed until it was wrong
in the other direction — plus one item this branch had now got wrong
three different ways.

Backspace ownership over the player, third and final answer. Round 1
deferred to nothing (the cheat sheet took the key and the player stopped
underneath it). Round 1's fix deferred to `anyLayerAbove` (the player's
own gear/tracks popovers push layers too, so it stopped closing them).
Round 2's fix named the sheet and palette (the app-wide ErrorDialog is
neither, so the player tore down under it). The rule that actually holds
is not a list of overlays: the player's own panels answer first, because
handleBack's cascade is what closes them, and anything else layered above
takes the key.

The rail's inline rename is settled by measurement rather than argument.
Two rounds reasoned about whether a React passive effect lands before or
after floating-ui's return-focus microtask and reached opposite
conclusions; a jsdom harness built to decide it reproduced no failure for
ANY shape, because focus never enters the menu there — so it was deleted
rather than kept as a test that discriminates nothing. Checked in a real
browser instead: the field opens and holds focus. The shipped fix works,
and the comment now says what was measured rather than what was inferred.

The same check found that round 1's OTHER rail fix never worked at all.
Escape does run its own handler (typing then Escape leaves the name
unchanged, so it cancels rather than blur-commits) and the row button is
focusable when focused from outside — yet focus still lands on <body>.
Three explanations, none of which survived. So the refocus machinery is
reverted to the shape that shipped before this epic and the gap is
documented where the next person will hit it, with the evidence and an
instruction to instrument before theorising. A fourth guess would have
been worse than a known bug.

Also: the two comments in Playlists.tsx that still claimed the disabled
menu rows explain themselves — untrue since round 2 reverted `inert`, and
exactly the comment-vs-code drift CLAUDE.md calls the source of the worst
bugs here. And the move-failure toast printed HttpError.detail verbatim,
which for the reachable 404 is the backend's bare English "Unknown
playlist"; only a structured detail has been through localizeDetail, so
only a structured detail reaches the toast now.
2026-09-06 02:22:23 +02:00
peter 4b33036e7c fix(e10): review round 2 — revert what round 1 got wrong, narrow what it got too broad
Round 2 reviewed round 1's own fix diff, which is where a fix round does
its damage. It found 15 issues in 792 lines — a much higher density than
round 1 found in the whole epic — and most were regressions round 1
introduced. So this follows the rule rather than fighting it: revert the
fixes whose mechanism was wrong, narrow the ones whose guard was too
broad, and stop reaching for a third variant.

REVERTED, because the mechanism does not work:

- The `inert` menu item. Round 1 swapped a native `disabled` for
  `aria-disabled` so a row's "why is this off" tooltip could be reached.
  floating-ui's list navigation skips `aria-disabled="true"` exactly as it
  skips `disabled` (isListIndexDisabled) — and AnchoredPopover's own
  comment, written before this branch, already said so. The swap fixed
  nothing and made the mouse worse: without native `disabled` the row took
  pointer events, the full active highlight and DOM focus, then no-opped
  on click. The underlying finding is logged where the prop was, unfixed
  and honest: the reason has to go somewhere a disabled row is not.

- `useContextMenu` owning the focus restore. Turning floating-ui's
  returnFocus off platform-wide traded one bug for two: Playlists spreads
  triggerProps on a bare <tr>, so restoring to it is a spec no-op and
  Escape stranded focus on <body>; and with dismissOnScroll now on, the
  restore's plain focus() scrolled the row back into view, undoing the
  scroll that closed the menu. Patching that needs three more guards on
  one fix. The original bug — the rail's rename field dying on arrival —
  is fixed at the field instead: it focuses from an effect, which is
  scheduled after the microtask that was blurring it.

NARROWED, because the fix was right and the guard was not:

- PlexPlayer's Backspace deferred to `anyLayerAbove`, but the player's own
  gear and tracks menus register layers via useDismiss — so Backspace
  stopped closing them and did nothing at all. Only the cheat sheet and
  the palette own that key, so it names them.
- `historyDepth = 0` also desynced the counter from overlays the
  navigation does NOT unmount (About and ReleaseNotes live outside the
  page column), leaving one Back to both navigate and close them. The
  entries are marked buried and consumed silently by reconcile instead.
- The move's error toast fired for every status, so a transport fault
  surfaced twice on top of req's own dialog. Scoped to 403/404.
- The palette's moved-pointer guard could not judge the FIRST move of a
  session and went stale across a close. `e.movementX/Y` asks the browser
  directly and needs no state.
- `useKeyScope("contextMenu")` runs once per hook, and the Playlists page
  mounts two — so the cheat sheet rendered the chords twice under a
  duplicated React key. The scope stack dedupes.
- The `enabled:hover:` sweep missed the half that lives in CSS, so a
  disabled glass button still lifted. `.glass-hover:not(:disabled):hover`.
- ShareDialog's migrated buttons passed a colour className that the
  variant's own colour class outranks under clsx, so Revoke hovered
  neutral and a password-protected link looked unprotected. The colour
  goes on the icon, as the Copy button beside them already did.
- `matchesShortcut` took an untyped id and ids are only unique within a
  scope, so `back` would have answered for the wrong player. Keyed on
  scope + id, both literal.

And the systemic one: round 2 reverted eight of round 1's behaviour fixes
as a single mutation and all 763 tests still passed. Round 1 tested the
three fixes that were easy to test and none of the rest. So the guards
with the widest blast radius are pinned now, each mutation-checked to
fail for its own reason: the global keymap's repeat/allowInInput/disabled
handling (a new dom test), the cheat sheet bound to the real
useContextMenu rather than a scope stub, the scope dedupe, the history
burial, and the Button accessible-name union — whose @ts-expect-error
fixtures turn a widened type into a red typecheck lane.

The chipLane contract test that round 1 rewrote to count per lane counted
the import line as a lane, giving every file one free mirror — so the
exact hole it was rewritten to close was still open. Mutation-checked
both ways this time.
2026-09-06 01:57:32 +02:00
peter 80dea4f307 fix(e10): review round 1, part 2 — the docs the epic invalidated, and three contracts made real
Finishes round 1. Part 1 took the runtime bugs; this takes the prose that
had gone false and the three "the comment says so" gaps behind it.

CLAUDE.md is auto-loaded into every agent session and ui/README.md is
not, so an architecture paragraph that still described a flat components
bag was the most expensive stale sentence in the repo: the next agent
adding a popover reads it, sees no platform layer, and writes the generic
component into components/ — the invisible-layer failure this epic exists
to end. It now names components/ui/ and says the boundary is lint-enforced.
The canonical-patterns table gained the three rows the epic established:
ui/Button over a hand-rolled <button>, a keymap row over a private keydown
listener, and disabled-when-busy vs inert-when-inapplicable.

ui/README.md's inventory was written in this branch's first commit and
never learned about CommandPalette, ContextMenu/useContextMenu,
KeyboardHelp or gridColumns — in a section whose own closing rule is that
a second component painting the same thing is how the last inconsistency
happened. It also claimed "everything is rem-derived" while VirtualGrid's
minCol/GRID_GAP are deliberately px; that is now stated as the exception
it is, and a new section records which promises here are tests rather
than sentences.

Three contracts stopped being prose:

- A text-less Button must carry a name. "Required for iconOnly" was a doc
  comment, which is why the migration shipped two nameless icon buttons
  through eight green lanes. It is a union now, covering both spellings
  (iconOnly, and size="icon"/"icon-sm"), satisfied by label, aria-label or
  title. It caught one more on the way in: Banner's dismiss took an
  optional dismissTitle, so it could render unnamed — it falls back to
  common.close now.
- gridColumns' gutter arithmetic. Every existing assertion survived
  replacing floor((w + GAP) / (minCol + GAP)) with the naive
  floor(w / (minCol + GAP)): the fixed widths agree, the capped cases are
  cap-bound, and the loops compare columnsFor against itself. Pinned at
  the exact-fit boundaries where the two first disagree (536 and 828).
- ShareDialog's link row. Only Copy was migrated in S1, leaving one 2rem
  button between two 1.75rem ones — and on a coarse pointer .touch-target
  grew only the migrated one to 2.75rem while the destructive Revoke
  stayed small.

Also: the maxCols docstring said 8 columns of 384px where the test it
shipped with asserts 369.
2026-09-06 01:30:40 +02:00
peter 20e17c0ccb fix(e10): review round 1 — the boundary that did not bind, and 14 more
The epic-end review found 32 verified issues. This is the first fix round:
the three highs, the user-facing mediums, and the tests that were passing
for the wrong reason. Docs and the remaining low-severity tail are next.

The boundary rule was the worst of them. Both `no-restricted-imports`
regexes were written in plain "..." literals, where `\.` is an unknown
escape that collapses to the any-char `.` — so the rule read like a
boundary and matched far less than one. `../../lib/api` (the barrel every
feature component imports) and `../../components/Feed` (the same file
`../Feed` names) both sailed through a green gate. String.raw, a third
pattern for the second spelling, and a vitest over ESLint.lintText so a
rule that silently matches nothing can never look like a clean codebase
again.

closeThenNavigate waited for exactly one reclaim popstate. With a second
Back-closable layer still mounted — a palette opened over the player —
the navigation unmounted that layer too, and its reconcile backed over
the entry the navigation had just pushed: "go to Channels" closed the
player and landed on the feed. It now waits for the LAST programmatic
back, then adopts the entries the navigation buried, and falls back after
250ms rather than hanging if no popstate is ever coming.

The rail's inline rename died on arrival. With no domReference, floating-
ui's returnFocus falls through to a module-global stack and restores some
unrelated element from an earlier popover — blurring the field into its
own commit-and-close, from the second popover of a session onward. The
context menu now owns the restore: back to the row, but only if the row
is still there and nothing else has claimed focus.

Also:
- Backspace under the cheat sheet stopped Plex playback and dropped to
  the grid; `back` now defers to any layer above, like `escape` beside it.
- A move invalidated only the source list, so the target kept serving
  pre-move contents for 30s while the rail already counted N+1.
- Every move error a user can reach is a 404, which `req` leaves to the
  caller — so the row snapped back in silence and the shipped moveErrors
  strings were unreachable.
- swallowClick armed on long-press but was disarmed only by the click it
  was armed for; any press that produced no click ate the next tap on the
  whole list.
- The context menu stayed parked over whatever row scrolled under it.
- overflow-clip on the chip lanes ate the focus ring's top and bottom.
- "Why is this off" rode on disabled rows, which show no tooltip and take
  no focus: a new `inert` item keeps the reason readable.
- Shift+F10 was advertised app-wide but wired only on Playlists; it has
  its own scope now, and useContextMenu matches through the shared parser
  instead of its own literals.
- Mod+K could not close the palette, `h` machine-gunned on auto-repeat,
  and both were bound above the branch that mounts their overlays.
- Palette options were in the tab order under aria-activedescendant, and
  a scroll-synthesised mousemove stole the keyboard selection.
- The OSD live region was created together with its own text.
- Ending a rename stranded focus on <body>.
- The Button variants' hover halves are `enabled:` — the one migrated
  site that had guarded its own lost the guard on the way in.
- HU builtinLocked named the list "Watch later" in a Hungarian sentence.

Tests: the chip-lane contract counted per FILE, so a second hand-rolled
lane in an already-compliant file was invisible; the closeThenNavigate
ordering test only passed on a popstate left behind by its neighbours and
failed alone, so it moved to its own file.
2026-09-05 23:52:29 +02:00
peter 7d302f71b7 feat(e10): let an ultrawide buy bigger cards instead of more of them
The viewport contract has two ends, and only the floor was enforced. An auto-fill grid
answers "more room" with more columns, never bigger ones, so it settles just above its
minimum column at any width: measured at 3440px, the feed laid out 11 columns of 264px.
A feed diluted across an ultrawide fails the contract as surely as a broken one at the
floor.

Multi-column feed views now carry a `maxCols`; past it the leftover width flows into
the items. At 3440: cards 11x264 -> 8x370, cardsSmall 15 -> 11x264 (still visibly the
dense card, and never as wide as cards gets), tiles 7 -> 6x498. Each cap is set so the
item tops out near 1.5x its own minCol, which is what keeps a view recognisably itself.

A cap can only bind ABOVE the width at which that many columns already fit, so it is a
pure ultrawide affordance — 1024, 1920 and 2560 lay out exactly as before. Verified in
the browser at all three widths and across all five views.

The Channels grid needs no cap: the page's own 96rem --content-mw already bounds it.

Also renames VirtualGrid's `maxCol` to `colMaxWidth`. It was never a column count — it
is the single-column reading-width cap — and `maxCol`/`maxCols` one letter apart is the
kind of drift that has bitten this repo before.

The column maths moves to ui/gridColumns.ts as a pure function so both ends of the
contract are unit-testable without a layout engine; the tests assert the cap's numbers,
that it never binds at the floor, and that the dense card cannot grow into the standard
one.
2026-09-05 22:22:28 +02:00
peter f99240d8ca fix(e10): stop the hidden chip measurer scrolling the whole app sideways
The four chip lanes render an intentionally unbounded hidden "mirror" so the shared
fit core can read each chip's natural width. An absolutely positioned box still
contributes to its ancestors' SCROLLABLE overflow, and `visibility: hidden` does not
exempt it — and nothing between the toolbar band and <html> clipped it.

At the 1024x768 floor that meant the channel manager's document scroll width was
1258px: the whole app scrolled 249px sideways into dead space and dragged the fixed
header to x=-177. The same mirror was also 228px wide inside the Tags column's 144px
cell, which pushed the table to 990px inside its 906px container — i.e. it broke
DataTable's "the table never exceeds its container" contract and truncated channel
names to "The...". One cause, two symptoms.

The lane now carries a shared `chipLaneClass` = `relative overflow-clip`, so the
contract lives in one constant instead of four hand-rolled class strings. `clip`
rather than `hidden` because it creates no scrollport, so nothing can ever scroll the
mirror into view. Measurement is untouched: offsetLeft/offsetWidth are layout values
and a clip does not change layout.

Measured after: document 1258 -> 1024, sideways scroll 249 -> 0, table 990 -> 921
with no cell overflow, and the "+N" collapse still fits exactly as before.

A source-level test asserts every component rendering a mirror uses the lane class —
the drift a fifth hand-rolled lane would introduce, which no DOM test can catch since
jsdom has no layout engine to overflow.
2026-09-05 22:22:14 +02:00
peter cc8cf96c19 feat(e10): a jump-to palette on Ctrl/Cmd+K
Reaching a channel three modules away cost a rail, a page and a scroll.
Now it costs a keystroke and half a word: Mod+K searches modules,
channels, playlists, the saved views applicable where you stand, and the
app-wide actions.

Scope agreed with the product owner: navigation + actions, but only the
reversible local ones (the shortcuts sheet, the theme). "Sync from
YouTube" is deliberately absent — it is a long-running job against a
quota, it wants the progress and error surface the Channels page already
gives it, and one mistyped Enter is a worse failure than the walk it
would have saved.

Matching is word-wise and accent-folded, because this is a bilingual app
and nobody reaches for the accent keys in a hurry: "pod thevr" finds
TheVR Podcast and "penzugy" finds pénzügy. The arrows walk one flat
result list so crossing a group heading is invisible, and the field
carries the combobox/listbox aria tree that virtual selection needs.

ui/CommandPalette is domain-free (items are a prop, per the ui/ import
rule); AppCommandPalette supplies them. The shell is Modal again, with a
new `top` placement so a search-first dialog grows downwards instead of
jumping as results change.

Also fixes a bug the palette hit on its first run, and that PlayerModal
had already hit once: closing a dialog reclaims its Back-to-close
history entry with a programmatic history.back(), which undoes any
pushState made before it lands — so a command that navigates only closed
the palette. The rule is now closeThenNavigate() in history.ts with a
test, and PlayerModal's inline copy is gone.

The global `h` moves out of KeyboardHelp into useGlobalKeymap beside
Mod+K, so one listener owns the whole `global` scope and dispatch's
exhaustiveness keeps meaning what it says.
2026-09-05 21:33:29 +02:00
peter 86163958d6 feat(e10): one keyboard cheat sheet for the whole app
The Plex player had a shortcuts overlay nobody else did, so the YouTube
player's Space/F/arrows and the Plex browser's Backspace were documented
nowhere. ui/KeyboardHelp is that sheet for every surface, opened by a
global `h` - not the conventional `?`, which is a Shift chord on a
Hungarian layout, while `h` is the key this app's reader already has in
their fingers from the player.

It renders the shared table filtered to the scopes that are LIVE: a
surface registers its scope with useKeyScope while mounted, and only
while its keys really do something - PlexBrowse deactivates its own
scope inside the player, where its Backspace no-ops, so the sheet cannot
promise a "Back one level" the handler declines to perform. Same rule
for the `when` rows: ⇧←/⇧→ are listed only inside a series, an answer
the player publishes because the sheet cannot ask.

No new overlay primitive, after checking: Modal already portals into
document.fullscreenElement ?? document.body (so this works over a
fullscreen player), registers an Escape layer, traps focus and handles
browser Back. Third time this epic that the thing was already built.

Shift+F10 is in the table as `handledLocally` - documented and sharing
the chord parser, but matched by useContextMenu on the focused row.
dispatch neither routes it nor demands a handler for it, so the
exhaustiveness check stays honest about what it actually covers.

Also: A/S have cycled the Plex audio and subtitle tracks since E5, but
silently - the only way to see where you landed was to open the tracks
menu, the exact round-trip the shortcut exists to avoid. cycleAudio and
cycleSubtitle now return the track they moved to, and the player flashes
it over the video.
2026-09-05 21:15:42 +02:00
peter 12822a9e04 feat(e10): put every keyboard shortcut in one table
Six surfaces each owned a private keydown listener with its own key
literals and its own idea of "am I typing?", and they had drifted:
PlexPlayer matched `case "a"` without checking modifiers, so Ctrl+A
(select all) cycled the audio track, and its typing guard missed
<select> and contentEditable where the other two surfaces covered both.

lib/keymap.ts is a TABLE, not a dispatcher. Each surface keeps its own
listener and closures - PlexPlayer's refs are not worth re-plumbing -
but the chords, the modifier rules and the typing guard live in one
place, and dispatch() binds a scope's rows to handlers through a mapped
type, so a row without a handler (or a handler without a row) is a tsc
error rather than a comment someone forgets.

Matching is now exact on modifiers, which is what kills the Ctrl+A bug
and lets Shift+arrow be its own row beside the bare arrow. Escape opts
out of the typing guard (allowInInput) so it stays the way out of a
player's own subtitle-offset field.

The player's shortcut sheet renders helpLines() instead of a hand-copied
array beside the handler - the two used to be edited separately, which
is how a sheet ends up documenting a key that no longer exists. Labels
move to the new `keys` i18n namespace (EN + HU), and a test asserts
every row is named in both.
2026-09-05 20:58:17 +02:00
peter 928eac4c89 feat(e10): open the app's menu with a right-click, starting with playlist rows
S2's headline was never a new Menu: `ActionMenu` already sits on `AnchoredPopover`
(arrows, typeahead, Escape, anchor-flip). What was missing is an INPUT PATH into it, so
this adds `useContextMenu` + `<ContextMenu>` over the same `ActionMenuItem[]` and the same
`MenuItemList` — a row cannot grow a right-click menu that looks or behaves unlike its own
⋯ menu. Three ways in: right-click anywhere on the row, long-press on touch/pen, and
Shift+F10 / the Menu key. The LEFT button is untouched and keeps the row's own action.

One popover per list, anchored to a virtual element at the pointer (native behaviour, and
a table of 200 rows would otherwise mount 200 floating contexts to keep 199 shut).

Platform bits it needed: `hoverAlso` on AnchoredPopover (a submenu falls open under a
mouse and is tapped by a finger — mouse-only, so a tap's synthetic mouseenter can't race
the click toggle) and `elementRef` on PopoverItem (the submenu anchors to the very
menuitem that opens it, where aria-haspopup/expanded belong). `ActionMenuItem` becomes a
discriminated union — a row either fires or opens a submenu — plus `danger` and
`separatorBefore`.

First consumers, both in Playlists:
- item rows: play · move to another list (a submenu of every other playlist, one gesture)
  · to top / up / down / bottom · remove. Reordering goes through the same undoable store
  as the drag handle and is disabled under a view for the same reason, with the same
  explanation rather than silence.
- rail rows: rename (in the row that was right-clicked, keeping its cover so the rail
  doesn't jump) · delete. Both now run through `usePlaylistActions`, which also takes over
  the detail header's pencil and trash — the delete dialog's YouTube branch and the
  "what gets selected next" rule exist once, not twice.
- the rail gains a "recently changed" sort key off the new `updated_at`. NOT the default:
  a rail that reshuffles itself while you look at it costs more than it saves.
2026-09-05 19:59:56 +02:00
peter b28be682fe feat(e10): make "move a video to another list" one atomic request
The one-step move S2 needs has no honest client-side shape: "add to the target, then
remove from the source" leaves the video in BOTH lists whenever the second call fails,
and the reverse order loses it entirely. So it is one endpoint in one transaction —
POST /api/playlists/{id}/items/{video_id}/move — which re-parents the existing item row
(so `added_at` survives), recomputes `dirty` on both sides, and is idempotent when the
target already holds the video.

Also publishes `updated_at` on the playlist summary, and bumps it explicitly for the
edits that only write `playlist_items` (add / remove / reorder / move). UpdatedAtMixin's
`onupdate` fires on an UPDATE of the playlist row, so without `_touch` the rail's coming
"recently changed" sort would have seen renames and nothing else.
2026-09-05 19:59:41 +02:00
peter 7e97559615 feat(e10): put the 1024x768 size contract inside the primitives
S1c, the last part of S1. The contract now lives in components/ui/ and
in one CSS recipe instead of in ~160 call sites:

- Minimum target 2.75rem, on coarse pointers only (.touch-target). A
  32px icon button is fine with a mouse -- precise pointer, hover to
  confirm the aim -- and not fine with a thumb, so it grows exactly where
  there is no precise pointer, which on this app's real second device is
  the tablet. It grows the real box rather than painting an invisible
  ::after hit area, because two expanded hit areas on icon buttons 4px
  apart would overlap and the overlap fires the WRONG button. `link` is
  excluded: it is text inside a sentence.
- 2.75rem rather than 44px so the target obeys the text-size slider like
  everything else; verified that the type/geometry ladder was already
  rem-derived off --font-scale, so nothing else was needed there.
- The ladder is documented as a closed set in Button.tsx, with the site
  counts it was derived from, and ui/README.md states the contract.

Measured in the browser: fine pointer 36x36 and 34px-tall buttons
(unchanged), coarse pointer the same buttons at 44x44 and 44px tall.
1024x768 renders identically. 3x tsc, eslint, knip, prettier and 690
vitest green.
2026-09-05 18:49:48 +02:00
peter 721467ef69 feat(e10): finish the mappable Button migration, including the ui/ layer itself
Third sweep, 20 more sites: the rest of AdminUsers (the access-request
and invite clusters), Scheduler, SettingsPanel, SetupWizard, About, Feed,
KeyGate, OnboardingWizard -- and, importantly, four platform components
that were still hand-rolling their own buttons: Modal's close, Banner's
dismiss, SidePanel's two toggles and QueryState's retry. A platform layer
that bypasses its own primitive is the same defect one level up.

Hand-rolled buttons with their own geometry: 110 -> 39; Button call sites
37 -> 106. What is left is deliberate, and it is the list Button's own
docstring excludes plus two families this pass refused to force: video
overlay controls painted on white/N, drag handles, chip removers,
accent-solid pills, the bg-card family on the signed-out screen, and
sites whose paint is conditional (`suspended ? warn : neutral`) or
otherwise has no variant yet. Forcing those through a className override
would be unsafe without tailwind-merge, which is exactly the hazard this
sprint documented.

3x tsc, eslint, knip, prettier, 689 vitest green; UAT on Settings and the
About dialog (its close button and full-width CTA are both migrated),
clean console.
2026-09-05 18:41:41 +02:00
peter 47e2293de3 feat(e10): migrate the notification, message, auth and Plex action buttons
Second sweep, 34 more sites: NotificationsPanel (14 -- eight identical
accent text links, three ghost icon buttons, two ghost text actions),
Welcome's five auth links, KeyGate, Messages, NavSidebar, ChatThread,
ShareDialog, ChannelPage, PlexPlaylistView, PlexPlaylistAdd,
PlexCollectionEditor and plexViews.

Moves the font-weight out of Button's BASE and into each variant, so the
`link` variant can be semibold (which is what every text link in this app
looks like) while the action variants stay medium, and no element can end
up with two font-* utilities.

Welcome is the one signed-out surface this session cannot see, so only
its byte-identical link swaps were taken; its primary submit and the
bg-card button family are left for a pass that can UAT the screen.

3x tsc, eslint, knip, prettier, 689 vitest green; UAT on Notifications,
Messages and the nav rail, clean console in a fresh tab.
2026-09-05 18:39:22 +02:00
peter 29bb914a95 feat(e10): re-aim the Button variants at the paint the app actually uses, and migrate the first 15 sites
Counting the un-migrated call sites showed the primitive and the app had
drifted into two different looks: the majority of hand-rolled buttons
hover to accent (outline) and fill with card (ghost), while Button
hovered to fg and filled with surface. That split is the inconsistency
this epic exists to remove, so the primitive follows the majority rather
than the other way round -- 16 existing call sites shift slightly, ~90
hand-rolled ones stop being special.

Adds the variants the call sites prove are missing (outline-accent,
outline-danger, outline-warn, link), an icon-sm size for the p-1.5 icon
family, and a size=none so a link owns no button padding; renames
danger-outline to danger-soft so it cannot be confused with
outline-danger (red at rest vs red on hover). Two new tests lock the
link-size default and the rule that a variant never adds a second
font-weight utility -- without tailwind-merge, two font-* utilities are
decided by stylesheet order, not class order, so that is a real hazard.

Migrates AdminUsers, AuditLog, Channels, TagManager, Playlists,
PlaylistsRail and SettingsPanel (Button call sites 37 -> 52). Left alone
deliberately: conditional-paint sites, video-overlay white buttons, drag
handles, chip removers and card-click targets -- the families Button's
own docstring excludes.

3x tsc, eslint, knip, prettier and 689 vitest green; visual UAT on Users,
Access requests and Playlists at 1280x900, no console errors.
2026-09-05 18:31:00 +02:00
peter 5bfcf5d83a refactor(e10): move the platform components into components/ui/ and enforce the boundary
The app HAS a platform layer -- AnchoredPopover (22 importers, with
arrows/typeahead/Escape/anchor-flip), QueryState (21), Modal (17),
PageShell, DataTable, Tabs, Tooltip and ~26 more -- but it sat flat among
~160 feature components with nothing marking it as one. Several of those
files describe themselves as 'the one place' or 'canonical' and were
still bypassed by hand-rolled markup, because a layer nobody can see is a
layer nobody uses; three separate inventory passes of this codebase
misread it the same way.

Moves the 33 domain-free modules into components/ui/, rewrites the
imports in 89 files, and makes the boundary a lint rule rather than a
convention: inside ui/ a one-level-up import (a feature component) and
any lib/api slice are errors. The rule was verified to fire on both and
to leave generic lib/ helpers alone. README.md states which side of the
line a component belongs on and why.

No behaviour change: 3x tsc, eslint, knip, prettier and 687 vitest tests
all green; no backend file touched.
2026-09-05 16:58:29 +02:00
peter 31fa554b87 release: 0.81.0 (improvement/e12-security-ops)
gate / frontend (push) Failing after 1m57s
gate / backend (push) Successful in 3m25s
2026-09-05 04:54:26 +02:00
peter 6a0efba477 chore(release): 0.81.0 — E12 Security & Ops Hardening 2026-09-05 04:54:15 +02:00
peter 9f28f46839 fix(e12): unbreak the gate, and stop printing a command that can strand the restore
Round five. 47 -> 34 -> 15 -> 7 -> 6 raw findings, and this round's sharpest was the
loop's own blind spot: **the branch did not pass the gate.** `prettier --check` is one of
its eight lanes and I had been running the other seven, so an unformatted `onError` from
round one sat on the branch for four rounds — `siftlode publish` would have refused it,
with no --skip flag to get past.

**The restart hint could still strand the restored data.** Round four said the scripts no
longer print a command they cannot resolve; they did. The COMPOSE_FILE fallback only fires
when `docker-compose.yml` is ABSENT, so in a source checkout it never fires and the hint
expanded to a bare, pasteable `docker compose up -d` — which reads the other file, whose
volumes are `pgdata`/`downloads` rather than `siftlode_pgdata`/`siftlode_downloads`, and
recreates the database on an empty one. When the file is unresolved the scripts now print
`-f <the compose file you started with>`: not runnable, which is the point.

**A comment naming the wrong owner.** `formats.py` still said a zero budget is "a refusal
for enqueue to make" — the premise round one disproved, three files away from the two
docstrings that now say the opposite. Anyone acting on it would delete the worker's refusal
and re-open the original inversion, so it now says where the refusal actually lives.

Also: the reset-flag rule was rewritten in three consecutive rounds and pinned by nothing —
two tests now cover a fragment losing its token, both by a pasted verify link and by
navigating away entirely; `backup.ps1` deletes a partially-copied dump before throwing, the
guard round four gave the docs but not the script it mirrors; and one comment that read as
if the asset were untouched by a line four below it.

Gate green, all eight lanes: ruff, ruff format, 3x tsc, eslint, prettier, knip,
vitest 687, pytest 950.
2026-09-05 04:41:21 +02:00
peter 14f1c97c0a fix(e12): a bare up -d picks the wrong compose file, and recreates the database
Round four, on round three's diff. 47 -> 34 -> 15 -> 7 raw findings.

**The printed restart command could strand the restored data.** Round three replaced
`start` with `up -d` because `start` ignores a newly pinned tag — but printed it bare. In
the git-clone layout BOTH compose files exist, so the scripts' COMPOSE_FILE fallback never
fires and a bare `docker compose up -d` reads `docker-compose.yml`: different volume names,
locally built images, no IMAGE_TAG at all. `stop` and `exec` had been fine there (they
resolve running containers by project and service), but `up -d` RECREATES — so the operator
pastes the hint straight after a successful restore and gets `db` on a new empty volume and
a first-run setup wizard, with their data in an orphaned volume. Verified: the two files
resolve `siftlode_pgdata` vs `pgdata` and different images.

The scripts no longer print a command that recreates from a file they did not resolve. They
name the compose file when they resolved one, and otherwise say to start it the way the
operator normally does — with the caveats on their own lines rather than in a parenthetical
at the end of a very long one.

**The reset flag is now decided where the live hash is already parsed.** Round three cleared
it in the token effect, which only runs when a NEW verify token arrives — so the back
button, or re-pasting an already-consumed verify link, still left the banner promising a
recovery whose token was gone. Setting it from the hash on every fragment change is both
smaller and closes those.

Also: the doc said `start` "still carries the old tag" four lines after "pin the older
IMAGE_TAG", where "old" then reads as the tag being rolled back TO — it now reuses the
scripts' unambiguous wording; the Windows recipe removes a partially-copied dump before
throwing, so a mid-stream `cp` failure cannot leave a truncated file under the final name;
and `$WarningPreference` is pinned beside `$ErrorActionPreference` so an inherited
SilentlyContinue cannot drop the half-restored warning, with the comment corrected to say
what the warning stream actually is.

Gate green: vitest 685, pytest 950.
2026-09-05 04:31:13 +02:00
peter fc56f514b3 fix(e12): the restart hint pointed at the one command that undoes a rollback
Round three, on round two's diff. The trend across the loop is 47 -> 34 -> 15 raw
findings; the ops half kept producing the sharpest ones because the gate cannot run it.

**`docker compose start` ignores a newly pinned IMAGE_TAG.** Round two removed the
automatic restart because it re-ran `alembic upgrade head` — and then told the operator to
run that exact command by hand, in both scripts and the docs. `start` relaunches the
container that already exists, which still carries the tag being rolled back FROM, so
pinning 0.79.0 and following the printed instruction brings 0.80.0 back up and re-applies
its migrations. Verified against a real daemon: after changing the pinned tag, `start` kept
the old image and only `up -d` picked up the new one. Both scripts and the doc now say
`up -d`, with the pin-first caveat where the operator reads it.

**The reset flag outlived its token.** Reverting round one's "each link clears the other"
went one step too far: a `#verify=` link pasted over a standing reset notice replaces the
whole fragment, so the reset token is gone — but `resetPending` stayed true, and being
checked first it hid the confirmation while promising a recovery that signing out could no
longer complete. The flag is now cleared when the fragment it describes is gone, which is
narrower than what was reverted and does not blank the signed-out card.

**A test that could not fail.** The shared-asset regression test seeded the asset as
`pending`, and `_reset_asset_pending` only acts on `downloading` — so the release half was a
no-op and deleting it kept the suite green. Seeded as the worker actually leaves it now.

Also: the malformed-URL guard answers with urlguard's structured detail rather than a bare
English string, so both halves of one typo class get the same localized copy, and the third
site in that module handing user input to `urlparse` was swept with it; the asset is
released after the job is terminal, not before; the Stats 403 toast gets the wizard handle
its own comment claimed it had; the pasted Windows recipe sets `$ErrorActionPreference`
inside the block (a scriptblock inherits the caller's, so a `SilentlyContinue` profile
swallowed every throw) and cleans up the container-side dump on the failure path too; the
restore warning moved to the warning stream so a `> log` redirect cannot file it away; and
two comments narrowed to what the code does.

Gate green: vitest 685, pytest 950.
2026-09-05 04:18:33 +02:00
peter 358d722ce9 fix(e12): close what round one broke, and revert two fixes that overreached
Round two of the review loop, on round one's own diff. Two of its findings were
regressions the fixes themselves introduced, which is exactly what the loop is for:

**A per-user refusal was writing onto a shared row.** MediaAssets are deduped across
users, and the generic failure handler puts the message on the asset — right for an
upstream failure, wrong for "your storage is full". Two users queuing the same video share
one asset, so the first user hitting their cap failed the second user's job with the first
one's private reason, telling someone with an empty library to free up space. The refusal
now raises `_OwnerRefused`, caught separately: the job fails, the asset goes back to
`pending` for whoever else holds it. Pinned by a test that drives `_download` directly —
the refusal happens before extraction, so nothing needs stubbing.

**The restore scripts restarted the app, undoing the restore.** `docker compose start`
re-runs the container command, and that runs `alembic upgrade head` — so on the rollback
path this documents as recommended (restore the dump, THEN pin the older tag) the newer
image immediately re-applied the migrations the restore had just undone, and on a FAILED
restore it put the app back on a half-restored schema. Both are the opposite of why the
stop was added. Nothing is restarted automatically now; the scripts say what to run and
when, and the docs' snippets say the same.

**Reverted rather than patched:** the OnboardingWizard toast (the wizard already renders
that failure inline, next to a Retry button — three recoveries for one failure), and
`useVerifyEntry`'s "each new link clears the other". The latter blanked the signed-out
card's only banner, where nothing renders the reset flag to replace it. The two states are
independent again; what changes is that `dismiss` peels ONE layer, which is what the
original finding was actually about.

Also: `Stats` uses the YouTube helper rather than the bare one, so the route's English 403
detail no longer beats the translated fallback under a Hungarian UI; `urlparse` guarded
alongside `.port` (an unclosed bracket raised just as late), the reason renamed `malformed`
since "bad port" misdescribes it; three docstrings narrowed to what the code actually does
(the worker's byte check is per-job, not an aggregate ledger, and not on the edit path;
the redirect residual leaks a reachability oracle, which the note had denied; the
invalidation helper does not cover suspend); the pasted Windows recipe wrapped so `throw`
can stop it; the bash one-liner's failure branch no longer exits 0; and the tests that were
missing for the round-one fixes — the account panel's error paths, the invalidation set,
and the shared-asset case above.

Gate green: vitest 685, pytest 950.
2026-09-05 04:02:57 +02:00
peter fe38dfbcca fix(e12): close the eighteen findings the E12 review round left standing
The deep review of the branch raised 47 findings, 18 of which survived adversarial
verification. Grouped by what they actually were:

**The size ceiling inverted at its boundary.** `remaining_bytes` clamped to 0 for a user
already at their storage cap, and `build_ydl_opts` reads 0 as "no limit" — so the one
user over quota was the only one downloading with no ceiling at all. `check_enqueue`
could not catch it: it ran while the job was queued and the user was still under the cap.
The arithmetic moves into `quota.remaining_bytes` (where every other max_bytes decision
lives, and where it can be tested), and the worker now fails the job outright rather than
proceeding unbounded — as a non-retryable `quota_exceeded`, since retrying cannot help
until space is freed.

**Silent failures in the new account panel.** All three actions went through `quiet: true`
endpoints, which suppress the global error dialog on the promise that the caller reports
inline — with no `onError`, that promise was broken and a refusal rendered as nothing. On
the broken-SMTP instance this panel exists for, an admin would read silence as "the mail
went out". The co-invalidation set and the email-off predicate move to `lib/adminUsers`,
shared with the pending-access queue, so the two copies of these mutations cannot drift
apart again — which is how half of this was lost in the first place.

**A rollback recipe that destroyed data without working.** `alembic downgrade -1` steps
back one migration, but a release routinely ships several (0.80.0 shipped four): it drops
the newest one's columns and leaves the schema at a revision the older image also cannot
resolve, i.e. the crash loop it was meant to escape, minus the data. The dump restore is
now the recommended path, and the alembic route asks the target version for its own head
revision instead of counting backwards.

Also: redirects named in the urlguard docstring beside the DNS-rebinding residual they
share a fix with (SEC-17); a malformed port refused as `bad_port` instead of escaping as
a 500; the pull cooldown extended to the sibling route spending the same QuotaAction;
restore stops the app before dropping its tables; the PowerShell cleanup can no longer
decide the script's verdict on 5.1, nor report the wrong exit code; the documented
one-liners hardened to match the scripts; the registry's retention window stated where
pinning is advertised; the two banners made mutually exclusive so neither can shadow the
other; `hover:bg-hover` (a utility this project never generated) replaced by the
`selectable` recipe; the clipboard helper reused; and the sync cooldown's copy swept to
the two callers that discarded it.

Verified: gate green (vitest 678, pytest 949), and the panel, the row hover and the reset
banner UAT'd on localdev.
2026-09-05 03:41:58 +02:00
peter f98b060e2b feat(admin): an account panel, and a set-password link that needs no email
Design option C, chosen by the user 2026-09-05. Nothing in the admin UI
ever showed an account's GATE STATE: an admin could see a "Pending
approval" chip but not whether the address was confirmed, whether SEC-10
had cleared the password, or which Google account was attached — so
every support conversation started by guessing. Clicking a row now opens
a drawer that says which gate the person is actually stuck behind, with
the recovery actions under that readout rather than in the row.

The drawer is `Modal placement="drawer"`, not a second dialog: the layer
stack, ESC, the focus trap, Back-to-close and the scroll lock are the
hard parts, and duplicating them is how two shells drift apart. It
slides in, and honours prefers-reduced-motion like the rest.

The link itself (SEC-15, SEC-16, and the everyday "no SMTP and someone
forgot theirs") reuses the existing token machinery and the SAME
`#reset=` entry the mailed link uses, so the signed-out card consumes it
with no new flow. It is a distinct token KIND for one reason: a mailed
reset arrives in the mailbox, so completing it proves ownership and
confirms the address; an admin-issued link proves nothing about the
mailbox — the admin hands it over by whatever means they choose — so it
must not satisfy the email gate. Claiming otherwise would be the same
false attestation SEC-15 is about.

Issuing does not revoke the current password: a misclick must not lock
anyone out. USING it rewrites the hash and bumps the session epoch,
which is what evicts a squatter — SEC-16's actual remedy. Spending
either kind burns the siblings of both, so a completed reset cannot
leave a live admin link behind. The non-consuming probe learned the new
kind too, or every admin link would have opened onto "this link is
spent". The audit log records that a link was issued, by whom, for whom
— never the link, which is a live credential-setting secret and the last
thing that belongs in a long-lived log a human reads.

Not gated on account state on purpose: it is useful for a pre-empted
address, a revoked password, a Google-only account that wants one, and a
plain forgotten password. Narrowing it would only make it refuse exactly
when the admin needs it. The shared demo account is the one exception.

Verified on localdev, both languages, end to end: issue from the panel
-> the signed-in tab shows the honest "reset links open in a signed-out
window" banner with the token still in the address bar -> signing out
draws the "Set a new password" form, which is only possible because the
probe recognises the new kind. Gate: pytest 925 -> 942, vitest 676.
2026-09-05 01:46:58 +02:00
peter bc9e34d0ae fix(auth): close the three leftovers E9.5 logged but kept out of its diff
DX-17, three one-liners the epic's canonical sweep did not reach:

`routes/setup.py::setup_admin` still resolved the wizard's admin by
literal `User.email ==`. The wizard can be re-run, and E9.5's premise is
that two spellings of one Gmail mailbox are one person — so a second run
under the other spelling would have created exactly the duplicate admin
row the epic exists to prevent.

`routes/admin.py::_decide` hand-rolled `db.get` + 404 where the file's
newer endpoints use `get_or_404`. There were TWO such sites, not the one
the review named; both are converted, per the house rule that a
principle gets fixed everywhere in one sweep.

`useVerifyEntry`'s `hashchange` listener recognised `#verify=` but not
`#reset=`, so pasting a reset link into a tab already running the app
was silently ignored — the same silence that made the verify half a bug.
It now notices, and says out loud that reset links need a signed-out
window; the fragment is still deliberately left in the address bar
(review round 4), so signing out from that tab recovers it.

DX-16: `.catch(() => setOutcome("invalid"))` could not tell a spent
token from a 502 mid-deploy or a dropped connection, and by then the
fragment is cleared — so a network blip was reported as "that link is no
longer valid", pushing the user to request a fresh one on the strength
of a false statement. Only a 400 carrying {code:"verify"} is the
server's verdict now; everything else gets its own outcome, with no
resend CTA, because the token is untouched and retrying it works. The
rule was already written one file away for the reset-link probe: "a
network blip must never be cached as 'this link is dead'".

Two existing tests asserted the old conflation by rejecting with a bare
Error("400"); they now reject with the real HttpError. The errors are
built inside the rejection, not hoisted: mountAt calls vi.resetModules(),
so an HttpError imported earlier is a different class identity and
`instanceof` would quietly miss.

Gate: vitest 671 -> 676, pytest 924 -> 925.
2026-09-05 01:35:09 +02:00
peter 0cc450d1c9 fix(downloads,sync): bound what one user can make the server fetch, download and spend
SEC-12: the download box accepts an arbitrary http(s) URL and yt-dlp fetches
it from the WORKER, so any approved user could aim the server at the compose
network, the LAN or a cloud metadata endpoint — and read the result off
job.error, where "connection refused" vs "HTTP Error 401" vs "timed out" is a
port scanner. downloads/urlguard.py refuses a URL whose host resolves to a
loopback/private/link-local/reserved address, at enqueue AND again in the
worker (DNS can move in between, and the worker is where the socket opens).
A network-class failure on a `url` job now reports one fixed sentence instead
of the raw transport text; the error CODE is still classified from the
original, so the localized explanation is unchanged. DNS rebinding is
explicitly NOT closed — that needs the connection gated inside yt-dlp — and
the module says so rather than implying more than it does.

SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to that video,
so a URL that is purely a playlist or channel still expanded into every
entry — one enqueue, unbounded downloads. `playlist_items` pins it to one.
Size was unbounded too: `check_enqueue` compares the CURRENT footprint against
the cap, and a download's size is unknown until it lands, so one job could
overshoot by any amount; the owner's remaining budget is now handed to yt-dlp
as `max_filesize`, applied AFTER the adapter merge so no source adapter can
raise a limit the quota layer set. A join's cut-list is capped at 100: an
accurate join emits two filters per segment into one ffmpeg filter_complex.

SEC-14: the YouTube budget is instance-wide, so one user holding down a
manual sync spent the day for everyone. Per-user (not per-IP — the cost
belongs to the account) 3-per-10-minutes limiters on the two bulk pulls,
subscriptions and playlists, with separate budgets so one does not close the
other. The review also listed /rss, /backfill and /enrich; E3 deleted them,
and search stays unlimited on purpose — it is interactive, not a bulk pull.

Gate: pytest 896 -> 924 (none skipped), vitest 671.
2026-09-05 01:03:24 +02:00
peter 688efb9ed6 feat(selfhost): license the repo, and make Plex actually usable on the public compose files
DX-10: the repo invited people to clone, build and self-host while carrying
no LICENSE at all — which legally means all rights reserved, so none of that
was permitted. Adds the verbatim GNU AGPL-3.0 text, the SPDX id in
package.json and a README section. AGPL rather than MIT deliberately: this is
a server application, so the case worth covering is someone running a
modified copy as a network service.

DX-12: .env.example documented PLEX_MEDIA_HOST_PATH -> /plex-media and the
README advertised the Plex library, but only the localdev and home compose
files carried the mount — on both PUBLIC files the module could read Plex's
catalogue and then fail to open a single file, because it streams the media
itself rather than proxying Plex. Both api and worker now mount it read-only
with an empty named volume as the fallback, and self-hosting.md gains the
section that explains the server URL/token, the path map and the mount
together (it had never mentioned Plex).

DX-15: the worker had no init: true on either public file, though localdev
and home both carry tini with the reason written out — without PID 1 reaping,
a watchdog hard-kill of a download's process group orphans its ffmpeg/yt-dlp
grandchildren.

Verified: 'compose config' resolves on both files, the Plex mount comes out
read_only: true on api and worker, and init lands on worker (not just the
bgutil-pot sidecar). Gate green: vitest 671, pytest 896.
2026-09-05 00:53:26 +02:00
peter ca4e559cc7 fix(ops): make backup/restore work, and give upgrades a pin and a rollback
DX-13: both shell scripts ran bare 'docker compose', which has no config
file to find in the layout docs/self-hosting.md tells people to use (only
docker-compose.selfhost.yml is downloaded) — they now fall back to it, while
a source checkout keeps resolving docker-compose.yml as before. The two
PowerShell scripts moved the dump through the PS pipeline, which corrupts
binary on 5.1 and cannot express -Encoding Byte at all on 7 ('Byte' is not a
supported encoding name), so they simply failed there; they now write inside
the container and 'docker compose cp' it across. backup.sh also wrote its
output straight into backups/, leaving a truncated file that looks like a
real backup when pg_dump fails — it writes a .part and promotes on success.
Both restores report a non-zero pg_restore rather than printing 'Restored'
over it. Docs now state that a backup is the dump PLUS .env: without
TOKEN_ENCRYPTION_KEY every encrypted secret in the restored database is
unreadable, which the old one-line 'back up the volume' never said.

DX-14: 'latest' moves with every release and migrations run on every boot,
so self-hosters had no way to choose when they move and no way back. Docs
gain IMAGE_TAG pinning, a rollback recipe (image-only vs schema-bearing) and
the migration-failed-at-boot case; CHANGELOG.md is generated from
releaseNotes.ts so the file cannot drift from what the app shows, and
'siftlode release' refuses on a stale one rather than regenerating it into
the clean tree it just demanded.

Verified: dump + restore round-trip on both platforms against localdev into
a scratch database (8 users, 262822 videos, schema 0082_invite_lang), and
IMAGE_TAG=0.80.0 resolves on both services in 'compose config'.
2026-09-05 00:50:07 +02:00
peter e23128b4a1 release: 0.80.0 (improvement/e9.5-onboarding-access)
gate / frontend (push) Failing after 2m26s
gate / backend (push) Successful in 3m42s
2026-09-04 23:28:30 +02:00
peter 0d5f8f0708 chore(release): 0.80.0 — E9.5 Onboarding & Access flow
Release notes for the epic: bilingual mail, one-mail-one-click approval, order-independent gates
with named states, canonical Gmail identity, the admin lifeline for sign-ups stuck outside, and the
four-round closing review that converged 12 -> 4 -> 3 -> 2.
2026-09-04 23:28:23 +02:00
peter a03bf8d371 fix(auth): stop erasing a reset link the signed-in tree never reads (E9.5 review round 4)
Round 4 came back converged — nothing above low, and both findings in one family. This is the half
worth fixing, because the branch introduced it.

Only the signed-out auth card consumes a `#reset=` token, and S2 moved the address-bar cleanup up to
a hook App mounts unconditionally. So opening a password-reset link in a browser that already holds a
session erased the token from the URL while nothing had read it: the page loads, nothing is said, and
clicking the link again does exactly the same, because the token is untouched and the session is
still there. That is the dead end this epic exists to remove, and `main` did not have it — the strip
used to live inside the card, which a signed-in visitor never renders.

An unspent reset fragment now stays, which restores the recovery that shape always had: signing out
reloads with the fragment intact and the form is waiting. Once the card has spent the token a later
reload meets S4's named dead-link screen, which is true. The affirmative version — a banner telling a
signed-in visitor that reset links open in a signed-out window — is logged with DX-15, along with the
`hashchange` half that still only recognises `#verify=`.
2026-09-04 23:14:46 +02:00
peter 3dc14402b0 test(auth): pin what rounds 1 and 2 left unpinned (E9.5 review round 3)
Three low findings, all in what the review rounds themselves wrote.

The comment on `test_approving_an_unverified_account_keeps_its_password` still asserted the argument
rounds 1 and 2 refuted — that dropping the approval-time revoke was safe because `password_login`
reads `email_verified` first — while the docstrings on the two functions it describes now say the
opposite. Two contradictory "why" records for one rule is how the round-1 takeover primitive got
reasoned into existence in the first place, so the test comment now carries the corrected account
and names SEC-16.

`test_a_later_registration_leaves_a_link_already_in_flight_alive` asserted a subset relation that is
vacuously true on an empty set, and the guard that made it non-empty had gone out with the revert.
It now pins both halves: the old link survives, and exactly one new one appears.

Round 2's own fix — activation moved to after a successful send — was unpinned; putting it back
above the send left the suite green. The failed-invitation test now also asserts the account was not
activated by a request the UI reported as a failure.
2026-09-04 23:04:09 +02:00
peter 540ca244c8 revert(auth): the newest registration does not own an unverified account (E9.5 review round 2)
Round 1 fixed the address pre-emption chain by letting a later registration replace the password on
an unverified account and delete its outstanding verify tokens. Round 2 read that code and showed it
was worse than what it fixed, so it goes back out.

It closes nothing. The rule is symmetric and an attacker can simply register last — on a timer,
inside the five-per-five-minutes budget — while the owner registers once. What it does add is two
primitives an unauthenticated caller did not have: overwrite a real registrant's chosen password,
and delete the confirmation link already sitting in their inbox (the approval mail's 7-day token is
an ordinary verify token, so the blanket delete took that too). Worst of all it fired on the rows
SEC-10 left stripped — password_hash NULL, password_revoked_at set, active — installing an
attacker's password where the entire state means "there is none", and clearing the stamp that is the
only reason password-reset still accepts them. That is a straight takeover which did not exist
before, in the code written to prevent one.

The owner's recourse needed no new primitive and already worked: password-reset accepts a
pre-empted account, proves the mailbox, rewrites the hash and bumps the session epoch. What remains
is the bounded window between a confirmation click and that reset, which is SEC-16 — logged with
this attempt written down, so the next person does not reach for it again.

Two smaller round-2 findings go with it. `upsert_pending_invite`'s new exact-spelling tie-break ran
before any status preference, so it could deterministically pick a DENIED sibling over a pending one
and reopen it — the very outcome its comment claimed to prevent; the status preference now comes
first. And `send_invitation` activated the account before the send it might have to report as
failed, so a request the UI called a failure had quietly let someone in; activation now follows a
mail that actually left. The verify token minted to render that mail still commits either way — it
is single-use, expires on its own and was mailed nowhere, and the note says so rather than leaving
the next reader to wonder whether it was missed.
2026-09-04 22:54:52 +02:00
peter 563fa0513d fix(auth): close what the epic's own review found (E9.5 review round 1)
Eleven findings survived adversarial verification; these are the fixes.

**Address pre-emption.** S1 dropped the approval-time password revoke, arguing `password_login`
refuses an unverified account before it reads `is_active`. Two verifiers showed that holds only
while the address stays unverified — and S6's approval mail exists to get the real owner to end that
state, at which point the squatter's password works. While an address is unverified the NEWEST
registration now owns it: the password is replaced and every outstanding verify token deleted, so
the owner takes the row back simply by registering, and a link already in flight cannot confirm it
for anyone else. The residual (an owner who confirms and neither registers nor resets) is logged as
SEC-16 beside SEC-15, where the re-attestation path it needs already lives. The comments that
asserted the old, false invariant now say what actually guards this.

**The entry URL.** S2 moved `clearEntryUrl()` out of the signed-out auth card into a hook App mounts
unconditionally, and it blanket-stripped the whole query string, the fragment and `history.state` on
the FIRST commit. That broke first-run setup outright — the wizard mounts only after its query
resolves and then finds no `?token=` — and silently ate `?link=`/`?oauth=` (no toast, no refetch
after connecting Google), `?v=`/`?chan=` deep links, and scroll restoration. It now clears only the
fragment and the two one-time flags it owns, and preserves `history.state`.

**The env allow-list.** S3 canonicalised the Invite branch of `is_allowed` and left the
ALLOWED_EMAILS/ADMIN_EMAILS fallback matching literally — the 2026-09-03 lockout with a different
source of truth, plus a shadowing trick where an unauthenticated request under a sibling spelling
permanently masked an env grant. Canonical on both sides now, and the same for the delete dialog's
warning and the ADMIN_EMAILS role bootstrap.

**Honest states.** `send_invitation` stamped "I already wrote to them" before a fail-soft background
send; it sends inline and records only what left. The admin-side resend answered `{ok: true}` for a
dropped mail; it now says so. Both also activate/serialize consistently with the other decide paths.

**Correctness sweep.** The 7-day approval link said "expires in 168 hours" — durations render in
days once there is more than one. `_register_account`, `upsert_pending_invite` and `add_invite`
picked among canonical siblings with no ORDER BY and no exact-spelling tie-break, so which account
got the re-send, and whether a denied row was reopened over a pending one, was up to Postgres.
`purge_user` deleted every canonically-equal invite, revoking a surviving sibling account's access.
`ConfirmProvider` wrapped its ReactNode message in a `<p>`. Approving no longer leaves the
stuck-sign-ups list stale. The refusal panels announce and take focus.

**Tests.** Three mailers rewritten this epic never rendered in any test; every `send_*` now renders
in both languages. The orphan-key guard could not fail (it scanned the catalog that defines the
keys) and had no reverse direction. Newly pinned: canonical activation, the surviving sibling's
grant, a failed invitation and resend, an expired reset link, a throttled login's silence, the
per-language admin fan-out, and that the entry URL keeps what it does not own.
2026-09-04 22:42:15 +02:00
peter 10c0ca4cf7 feat(email): write to people in their own language (E9.5 S7)
Twelve mails carried hardcoded English while the app itself has enforced HU/EN parity since E5.
A mail is the one user-facing surface nobody can re-render by switching the UI language: it arrives
once, in whatever language we picked, and stays that way. That made the epic's own headline mail —
"one mail, one click" — English for a Hungarian reader.

The text moved to a catalog (`mailtext.py`) with a `text(key, lang, **vars)` renderer, and the parity
rule is now a test rather than a habit: every key must carry both languages AND the same
`{placeholders}`. A translation that drops `{link}` sends a mail with no link in it; one that invents
a placeholder raises inside a background task nobody is watching. Both are silent in production.

Every `send_*` takes the language as its second argument, resolved by the REQUEST handler and never
inside the mail — these all run as background tasks, after the request's session is gone, which is
the same reason the approval mail computes its link in the handler.

Where the language comes from, in order: the account's own preference; failing that, the language the
person asked us in; failing that, English. The middle one is new — `invites.lang`, migration `0082` —
because the mails an access request produces go to an address that usually has no account yet, and
the signal does exist: the page their form was submitted from, or the locale Google reported for a
sign-in we refused. Captured rather than derived, for the third time in this epic (`google_email` in
S3, `preferences.language` in S6 taught the same lesson): a signal that exists only in one moment
cannot be backfilled later. So the admin is asked in exactly one place — the invitation to an address
they typed in themselves, where nobody has ever told us anything.

That send is now a dialog rather than a bare click, which is also where the "you already wrote to
them" warning belongs — it used to live only in a tooltip the admin may never hover, guarding the one
action whose failure mode is mailing a real person twice.

S6's registration `lang` field turned out to be dead on arrival: the backend accepted it, the
frontend never sent it, so every account has been storing nothing. The request builders now stamp the
UI language themselves — a signal that has to be passed in by every caller is one every caller can
forget, which is exactly how it shipped unset.

Admins are notified one mail per language group instead of one blast to everyone. They are people
with their own UI language; a single message would have to pick one of them and be wrong for the rest.

Also removes a duplicated `invited_at` column declaration on `Invite` (two identical definitions
landed in S6).
2026-09-04 20:29:33 +02:00
peter a76bab9b34 feat(auth): one mail, one click — the approval mail carries the confirmation link (E9.5 S6)
The funnel's common ordering made a person open TWO messages: one saying their access was approved,
another asking them to confirm their address. The approval mail now branches on what they actually
still have to do — carry the confirmation link, point at sign-in, or say a confirmation mail follows
once they register — so the usual case is one mail and one click. A mail that guesses wrong is worse
than a plain one: telling someone to sign in while their address is unconfirmed contradicts the
screen that will refuse them.

That embedded link lives 7 days rather than the usual 24 hours. Nobody sits waiting for an approval —
it can easily be read on Monday after a weekend, and an expired link spends the one-mail promise and
puts them back on two. The risk is small: the token only ever confirms an address; it signs nobody in
and sets no password.

The most likely root cause of the 2026-09-03 report that a confirmation mail "never arrived" is here
too, and it was never a delivery failure: re-registering an address that already had an unconfirmed
account returned in SILENCE — same cheerful "check your inbox", no mail sent at all. It now re-sends
the link. Nothing new leaks: the response and its timing are unchanged, and the public verify/resend
already does exactly this for the same input.

Whitelisting stays silent, and that is a decision, not an omission. The operator opens a door as a
possibility — usually already agreed with the person — and a row can sit approved for weeks with
nobody registering; a mail fired automatically there would be nudging someone who never asked.
Telling them is its own act on its own button, `invited_at` records it so a second click is a
decision rather than an accident, and the button only appears where nobody is behind the address yet:
"the door is open, come in" is meaningless to someone already inside.

Two things UAT caught that no test would have. Every fresh registration appeared in BOTH admin lists
at once, because the stuck-sign-ups filter keyed on "unconfirmed" — the rule is now "the address may
sign in, and this account still can't", which leaves an undecided request with the queue that owns
it. And the invitation button was offered on every approved row, half of which are existing users.

Also: the confirmation mail finally says what its link's rules are (once, 24 hours) — the reset mail
always has, while this one left the reader to discover both by meeting a dead link. The card's
"didn't get the email?" moves up under the form, and the three waiting messages name the spam folder,
which is where a first message to a fresh recipient sometimes lands.

Registration now records the language the sign-up page was in. Nothing reads it yet — the bilingual
templates are S7 — but it is the only moment that choice exists anywhere the server can see, and a
preference nobody recorded cannot be backfilled later.

UAT on localdev: re-registering re-sent the link (one token became two, one account); approval issued
a 7-day token and moved the account into the stuck list with only a re-send; whitelisting mailed
nobody; the invitation recorded itself and the row then warned that sending again would write twice.
Test data removed.
2026-09-04 04:05:28 +02:00
peter 7acdf5d577 feat(admin): surface the sign-ups that are stuck outside, and let an admin finish letting them in (E9.5 S5)
Approve an address BEFORE the person registers and their sign-up produces no queue row at all —
`_decide` only takes a still-pending invite — so the Access-requests screen sat empty, the tab badge
read zero, and the only working move was re-typing the address into the whitelist box. Nobody would
guess that, and on 2026-09-03 nobody did.

This is roadmap UX-25, where two earlier attempts were reverted after review. Both moved AUTHORITY:
auto-activating on registration turns a standing whitelist entry into a bearer grant for whoever
registers the address first, and reopening the approved invite to `pending` lets an anonymous sign-up
REVOKE a standing approval. What was actually missing was visibility, so this changes nothing about
what an unauthenticated caller can cause — the admin could already do exactly this by re-entering the
address; they just had no way to know they had to.

The section lists every account that cannot sign in yet and says WHICH gate is shut, because the two
dead ends look the same from the admin's chair: waiting for activation, or waiting on a confirmation
mail. Naming the unconfirmed ones is not decoration — on a no-SMTP instance that is the difference
between the owner and whoever registered the address first (SEC-15), and an admin about to let
someone in should see it. A person the queue above already covers is deliberately NOT listed here;
two buttons for one person is worse than one.

Activate is gated on `is_allowed` — the same function the sign-in paths consult, in core, so the
button that appears and the badge that counts cannot disagree. Without that gate it would
manufacture an inconsistent account: password sign-in only checks `is_active` and would let them in,
while their Google sign-in would be turned away and silently reopened as a fresh access request.
Letting in someone with no approved invite stays the whitelist box's job, which says so out loud.

The badge counts activations SEPARATELY from pending invites rather than widening an existing
field's meaning, and only counts what genuinely waits on the admin: an unconfirmed account waits on
its owner, so it is listed with a re-send but does not nag. The admin-side resend says plainly what
happened, including refusing when the instance has no working SMTP — mail is fail-soft everywhere
else, and that silence is exactly what left the 0.78.5 sign-ups with nothing to fall back on.

`send_verification_link` is now one function with three callers (registration, the public resend,
this one), so the link's shape and TTL cannot drift by who triggered it.

UAT on localdev, both languages: the UX-25 shape reproduced end to end (approve the address, then
register), the badge counting it, Activate letting them in and dropping the badge, the row losing
its Activate but keeping the re-send, a fresh token issued by the re-send, and both actions in the
audit log. Test data removed.
2026-09-04 03:27:48 +02:00
peter b1086e05dc fix(auth): stop drawing a reset form the link can no longer fill (E9.5 S4)
Reported from UAT: delete an account, then open its password-reset link. The card drew the full "set
a new password" form — because a `#reset=` fragment was present, and nothing more — and only after
the user had typed a password did the server answer 400. Measured first: no security hole. A user's
tokens cascade away with the row, so nothing can be set on a deleted or a re-created account. What
was left is a dead end of exactly the kind this epic exists to remove, and it is not rare: a reset
link lives one hour, mail is slow or lands in spam, and whoever got two mails clicks the older one.

password-reset/check answers whether a link is still usable WITHOUT spending it, so asking can never
cost the user their one real attempt — the property the tests pin hardest. It gets its own limiter,
because opening a link must not eat the budget for requesting one, and a throttled caller is told
"ok", never "dead": this check may only ever save someone a wasted form, so when it cannot answer it
degrades to the old behaviour instead of declaring a live link finished. The confirm endpoint stays
the authority, and its failure raises the same named screen, since the token can die between the two
calls or the probe itself may have failed.

A too-short password is deliberately NOT that screen. The length check runs before the token is
spent, so a typo costs the link nothing, and the card tells the two 400s apart by `reason` so only
one of them takes the form down.

Both email-link confirmations (verify, password-reset/confirm) now answer in the {code, reason}
shape S4 introduced. They were the last refusals a user meets by simply clicking an old link while
still being answered in hardcoded English under a bilingual UI. The seven remaining bare strings in
this module (register, request-access, the demo guards, set-password) sit behind client-side
validation the UI does first; they are logged as a follow-up rather than swept in silently here.

One bug found by the UAT itself, in the new code: the check went out FOUR times per opened link —
StrictMode double-invokes effects and the shell remounts the card as the session probe settles.
Harmless per request, but against a per-IP limit it would exhaust the budget in two reloads and then
silently fall back to drawing the form unchecked: nothing errors, the feature just stops working.
Deduped in a module-level map holding the in-flight promise (a ref dies with the remount that causes
it); a failed probe drops itself so a blip is never cached as "dead". The test renders the card
twice, because with one render the guard would rot undetected.

UAT on localdev: the reported scenario end to end (delete the account, open the link) shows the named
screen and no form; a live link still opens the form and completes; the CTA lands on the request-a-
link form; one probe per opened link. Throwaway accounts removed.
2026-09-04 02:38:01 +02:00
peter ed91c83dd7 fix(auth): name the state and the address behind a refused sign-in (E9.5 S4)
"Your account is awaiting admin approval." was true and useless. It never said WHICH mailbox the
account was on, and on 2026-09-03 that silence hid a Gmail alias mismatch for a day: a registration
as np.temu@gmail.com met a Google claim of nptemu@gmail.com, and every screen in the funnel agreed
to say nothing about it.

Every refusal password-login can answer with now uses the codebase's structured {code, reason}
detail, built in one helper. The three that arrive AFTER the password is proven carry the address on
the account plus both gate flags; the two that answer before it (invalid, rate_limited) carry
nothing, so a wrong password and an unknown account stay byte-identical. The frontend renders the
first three as a named screen in place of the form — there is nothing to retype once the password is
correct — with an icon, the two-gate checklist, the address in monospace, and a resend CTA where one
helps. Reporting each gate from its own flag is the payoff of S1: approved-but-unverified is a real
state now, and a screen that assumed a sequence would send that user off to wait for an admin who
has already decided.

The Google access-request notice names the address too, and says so when the request was already on
file under ANOTHER spelling of the same mailbox — the honest version of "we didn't open a second
one", stated only when it applies. It moved from the query string to the fragment: the notice now
carries an address, and a query string lands in every reverse-proxy access log. Same reasoning that
put the verify/reset tokens there, and the same owner already clears it. urlencode, not an f-string,
because URLSearchParams reads a bare + in a fragment as a space and would mangle exactly the
+tag addresses this epic exists to protect.

upsert_pending_invite returns the row it settled on beside the is-new flag, since the row's stored
address is what the notice has to name and what mail will go to.

Deviation from the PendingScreen mockup, deliberately: no "Sign out" button and no separate "Not
your address?" link. The mockup drew the panel in the app shell; the accepted decision (a) keeps the
pending user session-less, so there is no session to end, and the address is named directly above a
"Back to sign in" that answers the same question.

UAT on localdev, both languages: all three refusals with the typed spelling deliberately differing
from the stored one (the panel named the stored one), the resend CTA carrying the typed address into
the resend form, and both access-request banners with a +tag address surviving the fragment round
trip. Throwaway accounts and the language switch reverted.
2026-09-04 01:56:55 +02:00
peter a614ee2435 feat(auth): show the Google address beside the one the account registered with (E9.5 S3)
An account can now hold two true addresses: the one its owner typed, which a Google
sign-in no longer overwrites, and the one Google answers with. Only the first was
stored, so Settings could say 'Connected' without saying connected AS WHOM -- exactly
what someone who registered with a +tag needs to know.

users.google_email (migration 0080) records Google's answer at sign-in and at an
in-app link. /api/me exposes it ONLY when it differs from the account address, so the
comparison lives in one place and the majority, whose two addresses are the same
string, see no change at all. Rendered in the account menu as a muted third line under
the identity, and in the Settings sign-in row under the existing hint.

A short 'Google' label rather than a brand glyph: the app draws one icon set (lucide),
and a generic mail icon next to an email address says nothing. It also reads to a
screen reader, which an unlabelled logo would not.

No backfill is possible -- Google's answer is only known at sign-in -- so accounts
linked earlier stay NULL until their owner next signs in with Google.
2026-09-04 01:23:11 +02:00
peter c74487fbb3 fix(auth): keep the address the user registered with, and let it sign them in (E9.5 S3)
Two halves of one rule: the address someone typed is theirs to keep, and it must be
the one that gets them back in.

A plus-tagged or dotted Gmail is often a deliberate choice -- registering as
me+siftlode@gmail.com is how you find out later which site leaked your address. The
Google callback overwrote it with the account's primary, which can never contain a
'+', so the tag vanished on the first Google sign-in and, because sign-in matched the
stored address, took their chosen address with it. The rewrite line is older than
this epic, but S3 made it reachable here by matching adoption across spellings. It
now runs only when the mailbox GENUINELY changed, and lives in a named helper rather
than inline, so the rule has one home.

The three lookups that resolve a typed address -- password login, password reset,
verify resend -- go through find_user_by_email: exact spelling first, then the unique
canonical candidate, and None when several accounts share a mailbox rather than
guessing which one a password should open. Reset and resend previously answered a
different spelling with silence (they reply uniformly by design), which was a dead end
with no way out. Their mail now goes to the address on the account, never the one
typed into the form.
2026-09-04 01:07:54 +02:00
peter 876cece881 fix(auth): match identity on a canonical email, so one Gmail mailbox is one account (E9.5 S3)
Gmail delivers x.y@, xy@ and xy+tag@ to one inbox; the invite gate only lowercased.
On 2026-09-03 that turned an approved user's Google sign-in into a silent SECOND
access request -- OAuth returns the account's stored primary, which need not be the
spelling they typed -- and left prod with two user rows for one mailbox (ids 13, 14).
The +tag case fails every time: a Google account cannot contain +, so OAuth can only
ever answer with the base address.

Mail still goes to the address AS TYPED; only identity matching uses the canonical
form. utils.canonical_email is the single definition, and a model validator derives
users.email_canonical / invites.email_canonical on every write, so no call site can
forget it. Scope is narrow on purpose: dots are significant on most providers and the
Gmail rule does not hold for Workspace custom domains.

Swept every sibling of the same principle, not just the reported one: is_allowed,
upsert_pending_invite, the Google-callback adoption, _register_account (so no new
duplicate pair can be created), _activate_user_by_email, admin _accounts_for (an
invite backing a live user under another spelling used to look deletable) and
add_invite, plus purge_user, which now erases every canonically-equal invite instead
of leaving one behind that still grants access.

Migration 0079 backfills both columns and indexes them NON-uniquely: prod already
carries canonical duplicates (approved invite pairs since 2026-06-16), and merging
accounts is a judgement call about people's data, not something a migration does on
its own. A test pins the backfill SQL against the Python helper.
2026-09-04 00:14:12 +02:00
peter e5997ca567 fix(auth): consume a verification token that arrives without a page load (E9.5 S2)
Found in UAT. Navigating a tab that already runs the app to a URL differing only in
the fragment is a same-document navigation: nothing reloads, so the import-time entry
snapshot never sees the token. It sat unconsumed in the address bar and the previous
outcome stayed on screen -- the same silence this sprint set out to remove, wearing a
different hat. Pasting the emailed link into a running tab does exactly this, and our
verification mails were plain text until 0.78.8, which makes pasting the normal way to
use one.

A hashchange listener now picks up the new token, the guard tracks consumed tokens
rather than a single fired flag (still exactly one POST per token), and a fresh link
clears the standing outcome instead of leaving the old banner up.
2026-09-03 23:57:18 +02:00
peter c134b514c1 fix(auth): confirm an emailed verification link wherever it is opened (E9.5 S2)
The token was only ever consumed inside Welcome.tsx, which App renders ONLY when
there is no session. Opening the emailed link in a browser that already held a
Siftlode session posted nothing, the fragment was then stripped, and the user got
no signal at all -- the link simply did nothing. Incognito worked, which is why it
survived. Confirmation now happens in App, above that branch: signed out the outcome
still drives the auth card's banners, signed in it drives a top-of-app bar.

The entry URL is snapshotted once at import time (lib/entryUrl) instead of per
component, so no future URL writer can strip a token before whoever needs it has
read it -- that ordering race is what broke this in the first place. POST /auth/verify
now names the address it confirmed, which matters when the browser holds a session
for a different account; the same reasoning as password_reset_required already
covers the disclosure.

Also here, from the 2026-09-02 review tail: the dead GET /auth/verify goes (both
senders emit the fragment form, and the last query-token link expired 24h after the
0.78.7 deploy), a spent #verify=/#reset= fragment is cleared while signed in, and
the auth-card banners get role=status. The verifyOkReset copy no longer blames
admin approval, which E9.5 S1 made false.
2026-09-03 23:45:31 +02:00
peter 243869401a fix(auth): stop discarding a legitimate password on admin approval (E9.5 S1)
SEC-10 revoked any password set while an account's email was unverified, on both
the Google-adoption and the admin-approval path. On the approval path that was
redundant: password_login refuses an unverified account before it ever reads
is_active, so approval hands a pre-emptor's password nothing it could use. All the
revoke achieved there was destroying the password a legitimate registrant had just
chosen -- and since the admin normally decides before the user reads their mail,
'your password was cleared for security' was the DEFAULT experience, not an edge
case. The guard stays on the Google-adoption path, where the real owner does attest
ownership.

password_reset_required and the ok_reset banner stay: accounts stripped on prod
before this change have no other way back in.

Two existing tests pinned the old rule, so they are rewritten rather than deleted,
and the SEC-10 recovery pack now builds the revoked state through the surviving
producer.
2026-09-03 23:32:34 +02:00
peter 7339a0b84b release: 0.79.0 (improvement/e9-playlist)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m48s
2026-09-03 21:59:31 +02:00
peter c960c07a6a release: 0.79.0 notes + version bump
E9 Playlist module: total playtime in both playlist headers, the header
compacted to two rows, and the sticky column header sitting flush. Plus the
six defects the epic-end deep review confirmed.
2026-09-03 21:59:09 +02:00
peter 3cfdb36f06 test(format): pin the sub-minute truncation (E9 R2)
Round 2 mutation-checked round 1's own fix and found the one line it left
uncovered: every table row was an integer, so `Math.floor(sec)` in the new
seconds tier could be deleted or swapped for round/ceil with the suite still
green — exactly the "later cleanup removes it as redundant" failure the fix's
comment claims to prevent. Two fractional rows kill all three mutants.
2026-09-03 21:51:14 +02:00
peter ada95d04a4 fix(playlists): one rendering for a playlist total, honest below a minute (E9 R1)
Epic-end deep review, round 1. Six findings survived verification; five of them
were one defect wearing different hats.

formatRuntime floored 1-59s to "0m". Harmless while it only ever formatted a
Plex movie's length, but 12D repurposed it for an aggregate, and a total is the
only input that can land under a minute: a playlist of Shorts reported "0m"
while the rail beside it read "0:45". Fixed in the function, not behind a
`>= 60` guard at each of the seven call sites — the invariant belongs where it
cannot be forgotten. Its contract comment claimed "Plex movie/episode lengths"
and was two epics stale.

The rail rendered the same playlist total as formatDuration (1:23:45) while the
new header rendered formatRuntime (1h 23m) — two shapes of one number, side by
side on one screen. The rail now uses formatRuntime too; formatTotalHours had
already recorded that H:MM:SS is wrong for an aggregate.

The rename input kept min-w-0 without flex-1 in the compacted row, so it was
the only shrinkable child beside a shrink-0 action cluster. The Plex half of
this same epic already had the fix; this restores the symmetry.

plex.browse.items had no plural forms, so a one-item Plex playlist read
"1 items". Pre-existing, but the new header is the call site where count=1 is
ordinary. Both languages get both forms, no key drift.

formatRuntime had no unit tests at all across seven call sites. Added a tier
table plus a sweep asserting no non-zero second ever renders as "0m".

Dropped after 3/3 adversarial refutations: the meta-line segment ordering
(the filtered count is labelled, and count-then-runtime is the convention) and
the null-duration lower bound (the server's own SUM has skipped NULLs since
long before this branch; changing only the client would split them).
2026-09-03 21:43:04 +02:00
peter 1b004e6ccf feat(plex): total playtime in the playlist header (E9 S3)
Parity with the YouTube playlist header (E9 12D): the Plex playlist detail
now carries a meta line under the title — "N items · Xh Ym", summed from the
items' duration_seconds. No backend change: leaf_card already serializes
duration_seconds for both movies and episodes, and no new i18n key either —
the count reuses plex.browse.items, the same key the collection header uses.

The title/rename input moved into a min-w-0 flex-1 column so the meta line
sits under both; the line is hidden on an empty playlist.
2026-09-02 18:30:06 +02:00
peter 95c33e0b0f refactor(playlists): compact the playlist header to two rows (E9 12A)
The header portals into the shell's FIXED toolbar band, so its height is
list height the user never gets back — and it was spending four stacked
blocks on it: title, meta line, an action row, then a separate view-control
row under the cover. Option B collapses that into two tight rows beside a
smaller cover: title + rename/YouTube-link on the first, with Play-all,
push/revert/delete and the unsynced badge pushed right; the meta line
(count · playtime · filtered) and the view controls (group-by, reset view,
save order, view status, undo/redo) sharing the second.

Every control and its gating is unchanged — the view-control group still
appears only for `items.length > 1 || viewActive`, so a one-item playlist
shows just the meta line.

Measured on a 32-item playlist: the band goes 170px -> 88px at rest
(122px while a view is active, where the status sentence wraps), which is
~80px more visible list.
2026-09-02 03:45:07 +02:00
peter 4e6bcbc078 fix(playlists): stop the table header shifting up on first scroll (E9 12B)
The playlist content wrapper carried `pt-3`, so its sticky column header sat
12px below the scroller top at rest and jumped up to `top:0` on the first
scroll. Drop the top padding so the header is flush from rest and never
shifts — the fixed toolbar band above supplies the breathing room, matching
the channel table's already-correct pattern.
2026-09-02 03:39:17 +02:00
peter e17fb5bbf5 feat(playlists): total playtime in header + sticky-header fade fix (E9 S1)
Surface the playlist's total playtime beside the item count in the detail
header (12D) — sums every item's duration (formatRuntime "h m", matching the
approved mockup), hidden when nothing sums. Data is already client-side; no
backend change.

Pass fadeTop:false for the playlists page (12B) so the shell's top edge-fade
no longer sits over the already-sticky column header (tableHeader.tsx:257),
matching the channels/audit pages.
2026-09-02 03:39:17 +02:00
peter a9044ece50 release: 0.78.8 (improvement/email-html-alternative)
gate / frontend (push) Failing after 1m40s
gate / backend (push) Successful in 2m40s
2026-09-02 03:24:59 +02:00
peter 473aafd1c8 release: bump VERSION to 0.78.8 + release notes 2026-09-02 03:24:56 +02:00
peter 7019e3afdf fix(email): give every mail an HTML alternative so its link is clickable
The bodies are text/plain only, so whether the URL in a verification or
password-reset mail is a link at all was the client's decision. Gmail, Outlook and
Apple Mail linkify plain text; lighter web clients render the body verbatim and leave
the link as dead text — in a message whose entire purpose is that one link.

_send now attaches an HTML part generated from the SAME plain body, rather than
per-template HTML that would have to be kept in sync with it: one converter, all
eleven templates, and text/plain stays the primary part and the source of truth.
Escaping runs before linkification — bodies interpolate untrusted values (a
requester's address, the operator contact) — so no caller can inject markup, and a
query string's `&` is already `&amp;` when it reaches the href, which is what an HTML
attribute wants. Sentence-final punctuation is trimmed out of the href while
url-safe token characters (`-`, `_`) are left alone. Inline styles only, no images,
no external CSS, no tracking.
2026-09-02 03:13:23 +02:00
peter c81b110637 release: 0.78.7 (bug/verify-link-and-approval-lockout)
gate / frontend (push) Failing after 1m44s
gate / backend (push) Successful in 2m42s
2026-09-02 02:31:19 +02:00
peter 2ecc70ca6d release: bump VERSION to 0.78.7 + release notes 2026-09-02 02:31:14 +02:00
peter f8c63e80c2 fix(auth): widen the 0078 backfill so revoke victims who verified are rescued too
Review found the backfill's fourth predicate `AND email_verified = false` strands
exactly the subset that needs it most. The revoke does not invalidate the account's
outstanding verification token, so clicking the emailed link is the natural next
thing such a user does — and it sets email_verified=True while password_hash stays
NULL. That row then has no route back at all: reset ineligible, resend refuses an
already-verified address, set-password needs a session the account cannot obtain,
and nothing re-stamps it because the revoke itself is guarded on being unverified.

The remaining three predicates already pin the victim set exactly: registration and
the setup wizard always set a hash, every Google row carries a google_sub (adoption
assigns it before revoking), and nothing in the codebase ever clears google_sub — so
a local, non-demo, passwordless row can only have come from the revoke. Widening is
therefore safe, and it is the argument the docstring was already making while the
SQL contradicted it.

Also corrects that docstring: set-password is unreachable because the account cannot
authenticate at all, not because the epoch bump killed a session — an unverified
account is refused at login before a session is ever minted, so it never had one.
2026-09-02 02:21:46 +02:00
peter 3ac3d382e7 fix(auth): stop the address-bar mirror eating emailed auth tokens, and unlock revoked accounts
Two independent bugs made an email+password registration unrecoverable in prod.

The verification link did nothing. urlWithParams/clearParams rebuilt the URL as
pathname+query and dropped the fragment — but the one-time auth tokens ride the
fragment (#verify=/#reset=, SB3), and the providers that mirror nav params
(useUrlMirror in FeedFilters/Playlists/Plex/Channels) sit ABOVE the lazily-loaded
<App> in main.tsx. Their mount effect replaceState'd the token away before the auth
card's chunk had loaded, so the card read an empty hash: the link opened a bare
sign-in page, no request, no banner. Both helpers now carry the fragment over.

The account was then locked out for good. SEC-10 nulls password_hash when an
unverified account is attested (here: admin approval landing before the user could
verify — which the bug above made impossible). That is indistinguishable from a
Google-only account, and reset eligibility keyed off password_hash alone, so no
reset link was ever issued: no password to sign in with, and no way to get one.
users.password_revoked_at records the revocation, reset eligibility honours it, and
setting a password again (reset-confirm or set-password) clears it. An account that
never had a password stays ineligible on purpose — its owner adds one from Settings.

/auth/verify now also answers password_reset_required so the card can say why the
registered password stopped working, instead of leaving the user at the uniform
"Invalid email or password". Telling the holder of a valid verify token leaks
nothing: they have already proved they own the mailbox.

The 0078 migration backfills the stamp for rows carrying the exact lockout signature
(local, non-demo, no password, unverified), so existing victims recover unaided.
2026-09-02 01:28:45 +02:00
peter 94c62ca699 release: 0.78.6 (feat/admin-invite-delete-email-health)
gate / frontend (push) Failing after 1m43s
gate / backend (push) Successful in 2m53s
2026-09-02 00:36:52 +02:00
peter 1fe24d374a docs(tests): correct the deny-vs-delete rationale in the invites test file
Round 4's only finding: the file header and one comment still said Deny stops a
refused person re-asking. It does not — upsert_pending_invite reopens a denied
row as pending — and the delete_invite docstring in the same branch already says
so. The real difference is that Deny records an auditable decision and mails the
requester; Delete forgets the address.
2026-09-02 00:15:21 +02:00
peter 2cba6b9fca docs: round-3 copy corrections, and pin the two behaviours nothing was guarding
Round 3's correctness/security pass came back clean; what was left said more than
the code does.

- emailOff promised the access-request list is where sign-ups show up. A sign-up
  on an address that is already approved or already pending adds nothing to it
  (roadmap UX-25), and "only" was falsified by this release's own inbox notice.
- deleteConfirm claimed the admin was dropping "an approval you already gave" —
  it is the confirmation for pending and denied rows too.
- sendFailed blamed the mail server for rejecting a message; the reason covers
  connect, DNS, TLS and login failures alike, so it now describes the symptom and
  leaves the diagnosis to the log.
- ADMIN_EMAILS re-promotion happens in the Google callback only, not on any
  sign-in.

Tests: the decided-list copy of the delete affordance was never rendered (both
new cases fail if that branch is dropped), and the throttle release — a
deliberate behaviour the notes advertise — was asserted by nothing; a drop with
no admin yet, or with a failing write, must not silence the following hour.
2026-09-02 00:06:01 +02:00
peter 8d7aaad9a8 fix: warn before the delete, not after — plus round-2 copy and a11y corrections
Round 2 on the feature diff (15 → 12):

- still_allowed_by_env now rides the invite LIST as well as the delete response,
  so the confirmation warns while the admin can still change their mind. It
  mattered most for a DENIED row on an env-allow-listed address: deleting it
  silently converted a refusal into a grant, and the only warning arrived after
  the irreversible click.
- Both inbox bodies claimed to name "the most recent" dropped message. The
  throttle records the FIRST of each hour and never updates it, so they now say
  "one of them", and the not-configured body finally names where to fix it.
- settings.invites.emailOff told the admin they "get no notice when someone
  signs up" — rendered directly above the queue that shows exactly that. Only
  the email is lost, and it now says so.
- The disabled delete buttons put their reason out of reach: `disabled` drops a
  control from the tab order and suppresses its title. Switched to
  aria-disabled + preventDefault, the pattern Playlists.tsx already established
  for the same reason.
- deleteInvite is quiet, so the 409 the UI handles with a translated toast no
  longer also raises the server's English detail as a modal.
- After a completed password reset, Back landed on the reset form again with a
  consumed token — the history change made that state reachable. The token is
  cleared on success so the reset view stops being the card's root.
- The hu deleteHasAccount named a tab that renders under a different label.

Tests: a dom test for the Access-requests panel covering both has_account
variants and all three delete outcomes, /api/me flag cases, an ADMIN_EMAILS
case, and the env-allowlist test now patches the raw setting so the parsing
stays under test.
2026-09-01 23:50:29 +02:00
peter 85043013f0 fix: address the feature review — accurate copy, reachable controls, real coverage
Round 1 on the feature diff found 15; the substantive ones:

- The delete control VANISHED on any invite whose address had registered, which
  is the ordinary sign-up shape (register writes the User and the pending invite
  together). The guard stays broad — that row is the account's approval gate, and
  orphaning it strands someone who can then neither sign in nor be approved — but
  the control is now a disabled button carrying the reason, on both lists, so an
  admin never sees a row silently missing a button its neighbours have.
- Deleting an entry is not always a revoke: is_allowed falls back to the env
  ALLOWED_EMAILS / ADMIN_EMAILS lists once no row exists, and an ADMIN_EMAILS
  address is force-promoted on its next Google sign-in. The route now reports
  still_allowed_by_env and the UI says so instead of claiming access was removed.
- "Deny keeps the row so they can't immediately re-ask" was simply false —
  upsert_pending_invite reopens a denied row. Corrected everywhere it was stated.
- The Configuration warning reads me.email_configured, which save() did not
  invalidate, so it kept warning about fields the admin had just filled in.
- The read-only variant was an aria-hidden span explained only by a Tooltip that
  the hints toggle can switch off; now a disabled button with title + aria-label.
- The delete error path threw away the 409's reason (the stale-list case) and
  left the stale row on screen.
- The throttle window was spent even when no notice was written, silencing the
  next hour on the path whose job is to stop silent drops.
- email_enabled() opened a second session and decrypted the SMTP password on
  every /api/me poll; it now takes the caller's session.
- welcomeMode.dom.test.ts never rendered the card, so reverting the history
  wiring to useState left it green. A render test now drives the real links and
  fails on that revert (verified).

Plus /api/me flag tests, the registration-shaped pending case, the env-allowlist
case, and a release note that matches what the code does.
2026-09-01 23:31:49 +02:00
peter 44f626f732 feat(admin): delete a whitelist entry, and stop hiding a broken mail setup
Three operator gaps, all found the same way — a person could not sign up and the
app showed nobody anything.

- DELETE /api/admin/invites/{id} with a delete control on both the pending
  requests and the previously-decided list. Deny keeps the row so the person
  cannot immediately re-ask; delete forgets the address. Refused with 409 while
  a User row exists for it, because is_allowed reads that row and removing it
  would revoke a live user's access with no other sign — the list ships
  has_account so the UI withholds the button instead of offering one that fails.
- Email is fail-soft by design, which had come to mean invisible. _send() now
  records an email_undelivered notification for every admin when a message is
  skipped or rejected, naming the message that was lost, throttled to one an
  hour. /api/me carries email_configured, driving a standing warning on
  Configuration → Email / SMTP and above the access-request list.
- The sign-in card's Create account / Forgot password / Didn't get the email
  views are history entries (useHistorySubview), so Back returns to the previous
  view instead of only moving the address bar.

Also: the per-test rate-limiter reset sweeps every loaded app.* module rather
than a hand-kept list of three — this change added a limiter outside it.
2026-09-01 23:09:25 +02:00
peter 24c93cbb83 docs: name the Configuration tab as it is rendered, drop an unused fixture
Round 6's two remaining nits. The operator note said "Configuration → Email"
but the group renders as "Email / SMTP" in both locales, and
test_an_admin_can_activate_a_pre_approved_registrant requested make_user without
using it — a leftover from the reverted rounds that made the test look like it
seeded an admin when it did not.
2026-09-01 22:15:56 +02:00
peter cb1f4f00b0 docs(auth): correct the last round of over-claiming comments and notes
Round 5's correctness/security pass came back clean; what was left was text that
said more than the code does.

- The register-path notify comment (and its test) claimed the admin "is still
  told". The call fires, but on a no-SMTP instance _send drops it — which is the
  open half of the problem, not something this branch fixed. Both now say so.
- The release notes tell admins what to actually do: an account whose address was
  whitelisted in advance has no pending request to approve, so the whitelist box
  is what activates it. That instruction only existed in the roadmap and in
  chores, which the dialog does not render.
- "That one line is the whole production change" now accounts for the test-only
  RateLimiter.reset() that also ships.
- The rate-limiter test claimed to pin that blocked attempts are not counted, on
  an assertion that held either way. Replaced with one that drives the clock
  across the window boundary and fails if hammering extends it.
2026-09-01 22:02:43 +02:00
peter f6e1d1e69f fix(auth): keep the admin notice unconditional on the register path
Round 4 showed the `inv is not None` gate added in b5a1002 removed a signal that
matters: for an address the admin had already approved, that email is their only
notice that someone has now created an account on it — the invite is not pending
so nothing lands in the Access-requests queue either. Suppressing it left the
registrant at "awaiting admin approval" with the admin unaware they existed. The
duplicate it avoided (Request access, then sign up) is mild noise by comparison.

The production change is now one db.commit(), which is all the reported bug ever
needed. Release notes corrected again: a previously-denied address was also
unaffected, the pending-request case never lost its queue entry (only the
account behind it), and the test inventory was missing a guard.
2026-09-01 21:48:40 +02:00
peter b64c3f514d fix(auth): drop the invite reopen — keep the fix to the lost account alone
Round 3 of the review killed the reopen from b5a1002: flipping an approved
invite back to pending let an unauthenticated POST /auth/register REVOKE a
standing approval for any whitelisted-but-unclaimed address, which then fails
is_allowed() and locks the real owner out of Google sign-in — the primary way
people get in here. It also re-armed the pre-emption window it was meant to
close, because with no SMTP email_verified is set True and the SEC-10 guard the
design leaned on never fires.

Two attempts at fixing the stranded-account UX in the same change both came back
worse than the wart. What remains is the narrow, real bug: the account was
silently rolled back and is now committed. An admin still binds the approval to
the account, exactly as before this branch. UX-25 in the roadmap records the
follow-up and both dead ends so they are not retried.

Also from round 3: the upsert docstring no longer claims the admin UI cannot
reach these accounts (re-entering the address in the same panel's whitelist box
does, and a test now pins it), the release notes are scoped to what the diff
actually changes with the operator note moved where the dialog renders it, and
RateLimiter has unit tests so reset() is not production code only a fixture
calls.
2026-09-01 21:34:38 +02:00
peter b5a1002412 fix(auth): reopen a pre-dated approval instead of auto-granting it
Replaces the activation added in ea931dc, which round 2 of the review showed was
a real weakening: on a no-SMTP instance it made a standing whitelist entry a
bearer grant — the first anonymous party to register an approved-but-unclaimed
address got an immediately usable account under that identity, with no
attestation, no admin action and no signal anywhere. The verify-time activation
had the same shape one step removed, and neither site ran the SEC-10 preset-
password guard or left an audit trail.

Keep a human in the loop instead. An approval granted before any account existed
was a decision about the ADDRESS, so registration reopens the invite
(upsert_pending_invite gains revive_approved, used only by the register worker).
The request lands in the Access-requests queue, approving it runs
_activate_user_by_email — and with it revoke_preset_password_if_unverified — and
the account activates. One mechanism, already audited, no new grant path, and
the "awaiting admin approval" copy is honest again.

The explicit commit from the previous commit still matters: with an invite
already pending, upsert returns without committing and _issue_token is gated on
SMTP, so a no-SMTP registration was still being rolled back.

Also from round 2: gate the admin notice on a fresh decision (matching the two
other upsert callers) so a retry no longer re-pings every admin, and reset the
module-level rate limiters between tests — the suite pooled every login into one
300s window and was two logins short of failing an unrelated test with a 429.
2026-09-01 21:18:14 +02:00
peter ea931dca30 fix(auth): honour an approval granted before sign-up instead of parking the account
Follow-up to the commit fix, from the deep-review. Persisting the account was
only half the story: when the admin had approved the address BEFORE the person
registered, the account landed is_active=False against an invite that is no
longer pending — absent from the Access-requests queue, and approve_invite 409s
on an already-decided invite. The sign-up worked and still could not sign in.

Derive activation from is_allowed() at the moment ownership is attested:
- verify_email_confirm activates a pre-approved account when the emailed link is
  confirmed (only the mailbox owner can follow it), so SEC-10 stays intact;
- _register_account activates only when there is no SMTP to verify with, where
  the admin whitelist is by design the sole gate — the same one add_invite
  already activates on.

Tests: the pre-approved no-SMTP case and the verify-activates case are real
regression tests (both fail without this change); guards cover the unknown-email
and denied->pending branches, verification of an unapproved account, and the
first end-to-end coverage of the /auth/register route and its
allow_registration gate. Release notes rewritten — the previous entry miscounted
its regression tests and advertised an email fix this release has no code for.
2026-09-01 20:56:43 +02:00
peter d954e71a77 fix(auth): commit the account on registration so a pre-approved sign-up isn't lost
_register_account() flushed the new User but relied on someone else to commit:
upsert_pending_invite() commits only when it actually writes an invite row, and
_issue_token() runs only when SMTP is enabled. For an email an admin had ALREADY
approved, on an instance where email is unavailable, neither fired — the session
closed and the account was rolled back. The would-be user got the uniform "ok"
and no account, could not sign in, and re-registering was a silent no-op.

Commit explicitly after the invite upsert (a no-op when upsert already committed
the same transaction). Three DB-lane regression tests cover the pre-approved
no-SMTP path (fails without the fix), the unknown-email path that always worked,
and the pre-approved path with SMTP on.
2026-09-01 16:25:05 +02:00
peter 602f62967a release: 0.78.4 (bug/plex-transcode-hls-startposition)
gate / frontend (push) Failing after 1m55s
gate / backend (push) Successful in 2m57s
2026-08-30 20:33:15 +02:00
peter ba00b50dea release: 0.78.4 — fix Plex transcode playback stall/rebuffer
hls.js startPosition:0 so the transcode/remux HLS session starts VOD-style
at the requested position instead of chasing the live edge of the growing
EVENT playlist.
2026-08-30 20:32:56 +02:00
peter 92adb593c8 fix(plex): start hls.js at position 0 for the transcode/remux HLS session
Our HLS is a growing EVENT playlist with no ENDLIST, which hls.js reads as
LIVE. Its default startPosition (-1) then starts at the live edge and hugs it
(no forward buffer → rebuffer spinners), and on a resume/seek session the
segments' absolute PTS (-copyts, offset by the server's -ss) don't line up
with the live-edge timeline → 0 fragments load → permanent stall. The
seek-restart model makes fragment 0 the requested content position, so start
VOD-style at 0 and buffer forward. Fixes the transcode playback stall/rebuffer
found in 0.78.2 UAT.
2026-08-30 20:09:00 +02:00
1427 changed files with 179427 additions and 31991 deletions
+1
View File
@@ -4,6 +4,7 @@
*.env.local
**/__pycache__/
**/*.pyc
**/.coverage
.venv/
venv/
# Docker patterns are anchored at the context root with NO implicit `**/` (unlike .gitignore, where
+18 -8
View File
@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
# Admin emails (subset of the above) get the admin role.
ADMIN_EMAILS=
# Logging. LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# the worker/download child read; the API's running level is the admin Configuration page's
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
# line — what a log collector ingests) or "text" (a readable line for a local terminal).
LOG_LEVEL=INFO
LOG_FORMAT=json
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
FRONTEND_ORIGIN=
@@ -60,18 +68,20 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Download center ---
# The Download Center adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# --- Files library & downloads ---
# Downloading adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# `bgutil-pot` sidecar (mints YouTube tokens) — both come up automatically with docker compose.
# Downloaded media defaults to a Docker-managed named volume. Set DOWNLOAD_HOST_PATH to a host
# The Files library defaults to a Docker-managed named volume. Set FILES_HOST_PATH to a host
# directory instead — e.g. one your Plex server can read — to keep the Plex-style tree there.
# The path must be writable by the container user (uid of `appuser`, 1000): chown 1000:1000 <dir>.
# DOWNLOAD_HOST_PATH=/mnt/media/youtube
# FILES_HOST_PATH=/mnt/media/youtube
# --- Plex integration (optional) ---
# The optional Plex module browses/plays your Plex library, playing the LOCAL physical file
@@ -81,6 +91,6 @@ SCHEDULER_ENABLED=true
# to /plex-media inside the container (the local side of the admin "path map", whose default is
# /data=/plex-media). Leave unset if you don't use Plex.
# PLEX_MEDIA_HOST_PATH=/mnt/media
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to DOWNLOAD_ROOT; point it at
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to FILES_ROOT; point it at
# fast local storage if your download volume is slow.
# PLEX_HLS_DIR=/var/tmp/plex-hls
+4 -1
View File
@@ -2,8 +2,11 @@
# whole-repo reformatting does not bury the real authorship of each line.
# Configure once per clone: git config blame.ignoreRevsFile .git-blame-ignore-revs
#
# Prettier one-time repo-wide format pass (R2 S1b, 2026-07-21)
# Prettier one-time repo-wide format pass (2026-07-21)
51c36c2071728d2cd7ae580b7bfdf21f6867a347
# ruff format one-time backend + scripts pass (line-length 100, 2026-08-23)
b1832afd180866b922521148603db447020c57a4
# Plan-code purge from comments, one-time (2026-10-04)
4aaaaa786017278460e0b4f94cc0d3374e0e37e2
-71
View File
@@ -1,71 +0,0 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two parallel jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip. node 22 matches the prod build image
# (Dockerfile `node:22-alpine`).
name: gate
on: [push, pull_request]
jobs:
frontend:
runs-on: ubuntu-latest
container: node:22-bookworm
defaults:
run:
working-directory: frontend
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run typecheck
- run: npm run lint
- run: npm run format:check
- run: npm run knip
- run: npm run test
backend:
runs-on: ubuntu-latest
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
container: nikolaik/python-nodejs:python3.12-nodejs20
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: siftlode
POSTGRES_PASSWORD: siftlode
POSTGRES_DB: siftlode_test
options: >-
--health-cmd "pg_isready -U siftlode -d siftlode_test"
--health-interval 2s
--health-timeout 3s
--health-retries 30
env:
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
SIFTLODE_TEST_REQUIRE_DB: "1"
steps:
- uses: actions/checkout@v4
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
- name: pip-audit (advisory)
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
continue-on-error: true
+8 -2
View File
@@ -9,6 +9,7 @@ __pycache__/
venv/
.mypy_cache/
.pytest_cache/
.coverage
.ruff_cache/
# Node / frontend
@@ -28,12 +29,17 @@ frontend/e2e/.auth/
*.sql.gz
backups/
# Download center: locally downloaded media (the DOWNLOAD_ROOT bind mount at the repo root).
# Download center: locally downloaded media (the FILES_ROOT bind mount at the repo root).
# Anchored with a leading slash so it does NOT also ignore the backend/app/downloads package.
/downloads/
/files/
# OS / editor
.DS_Store
Thumbs.db
.idea/
.vscode/
# TypeScript incremental build artifacts — machine-local, absolute-path-bearing, and no gate lane
# reads or regenerates them, so a committed one is stale from the commit that adds it.
*.tsbuildinfo
frontend/e2e/.speed/
+1
View File
@@ -0,0 +1 @@
24
+1
View File
@@ -0,0 +1 @@
3.12
+15
View File
@@ -0,0 +1,15 @@
# The CVE gate's allowlist. The publish gate scans the built image and the committed tree with
# Trivy and fails on any HIGH or CRITICAL finding that has a fix available. A finding that cannot be
# fixed yet is let through ONLY by an entry here, and every entry carries both:
# statement — why it is acceptable for now (what is exposed, what is not)
# expired_at — the date it stops being accepted; after it, the gate fails again on its own
# A contract test (frontend/src/lib/gates.contract.test.ts) refuses an entry without either, and
# docs/gates.md lists them.
#
# Example:
# vulnerabilities:
# - id: CVE-2026-00000
# statement: in libfoo's XML path, which nothing in the image calls; no fixed Debian build yet
# expired_at: 2026-12-31
vulnerabilities: []
secrets: []
+1767
View File
File diff suppressed because it is too large Load Diff
+80 -38
View File
@@ -5,19 +5,23 @@ be developed that way. This file is the rules of the road: read it before you to
loaded into every agent session, so keep it **short and true**. A rule here that has gone stale is
worse than no rule — if you change how the code works, change this file in the same commit.
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a download center). It is a
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a files library with downloads). It is a
**modular monolith**: one Postgres, one FastAPI process (uvicorn + APScheduler), one worker process
(yt-dlp/ffmpeg), one PO-token sidecar. React 18 + react-query frontend with a router-free navigation
provider.
(yt-dlp/ffmpeg), one PO-token sidecar. React 19 (with the React Compiler) + react-query frontend, with a
router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every scope and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` (no burn-down mandate, but none was ever triaged: a `reportOptional*` entry in code you touch may be a real bug). A run that fixed some rewrites it — commit that.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
## Workflow
@@ -26,8 +30,8 @@ The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc project
keep commits logical and Conventional-Commit-styled.
- **No AI attribution in commits.** No `Co-Authored-By` trailer, no "generated by" lines. (This file
being AI-written is the intended, visible exception; commits are not.)
- **Release: user-owned in timing, agent-run in execution.** The user decides *when* to ship; on their
go-ahead *you* run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
- **Release: user-owned in timing, agent-run in execution.** The user decides _when_ to ship; on their
go-ahead _you_ run `siftlode release` from the feature branch (merge to `main` → publish → deploy →
verify `/api/version`). Never self-ship — bring work to a green, committed state and stop for the
go-ahead first, then run it yourself.
- **Public repo.** Keep ops/topology/secrets out — those live in a separate private repo. Nothing here
@@ -41,49 +45,86 @@ how it gets built. Concretely:
- **Plan before you build.** For anything non-trivial, present a short plan (sprint-level for larger
work) and get a go-ahead before writing code.
- **Offer choices, mark the optimal — and explain the trade-off.** When several solutions are viable,
list them and mark the recommended one. This is the *one* place to spend extra words: give the *why*
list them and mark the recommended one. This is the _one_ place to spend extra words: give the _why_
behind each option and the weight of the decision. These decision points are where the product owner
engages with the build and learns from it, so make them understandable — a reasoned menu, not a bare one.
- **Mockups for design/UX decisions.** For any visual or interaction choice, show mockup options to
choose from before building.
choose from before building. How the UI looks and behaves is decided in
[`docs/DESIGN.md`](docs/DESIGN.md) (its variants render in the dev-only `/design-lab`): build to it,
and change it only through its decision log.
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change.
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
change runs its step 4 alone: every state the new text renders in, in both languages).
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → components; data
via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through `lib/api/*`
slices over `core.req<T>`.
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
only along a declared edge and writes them only through that unit's service; core reaches a unit's tables through its
`queries`. **Fence-held, with no exemptions** (`backend/tests/test_unit_fences.py`) — as far as an AST sees: what it
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
[`backend/app/README.md`](backend/app/README.md) with the conventions.
Frontend: `main.tsx` provider tree → `App.tsx` shell (a shared `/watch` link boots from its own
entry, `watch.html` → `watchMain.tsx`, held app-free by a contract test) → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
feature component or a domain API slice, and one layer down `lib/` may not import from `components/` at
all, not even a type (`no-restricted-imports` in `eslint.config.js`, with its own test). A new generic
component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
`lib/api/*` slices over `core.req<T>`.
**Not wanted** (reviewed and rejected as over-engineering for this size): microservices, event bus,
CQRS, a repository/DI framework, a router library, a generic CRUD-router factory or base serializer.
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
|---|---|---|
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| Area | Use | Not |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
@@ -96,10 +137,11 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
"no results". Reads degrade honestly; writes are conditional (`UPDATE … WHERE status='running'`).
- **i18n parity.** HU/EN, zero key drift. Every user-facing string is translated.
- **Comment the "why", keep it current.** This codebase uses comments as its architecture doc; a stale
"why" is a liability, so update it when the behavior changes.
"why" is a liability, so update it when the behavior changes. Say the why itself, never the plan
step that produced it: no epic, sprint or review-round ids in the code (`planCodes.contract.test.ts`).
---
*Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
_Run-it-locally and the full gate commands live in [`docs/development.md`](docs/development.md); backend
code conventions in [`backend/app/README.md`](backend/app/README.md). This file stays the source of
truth for the canonical-patterns table and the gate/workflow/collaboration rules above.*
truth for the canonical-patterns table and the gate/workflow/collaboration rules above._
+23 -9
View File
@@ -10,7 +10,9 @@ ARG GIT_SHA=unknown
ARG BUILD_DATE=
# Stage 1: build the frontend SPA
FROM node:22-alpine AS frontend
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image and the dev machine the gate runs on must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
COPY frontend/package.json frontend/package-lock.json ./
@@ -37,22 +39,31 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
# Create the app user BEFORE the copies so files land already-owned via COPY --chown (a trailing
# `chown -R /app` would rewrite every file into a duplicate layer). /downloads must be appuser-owned
# `chown -R /app` would rewrite every file into a duplicate layer). /files must be appuser-owned
# so a fresh named volume mounted there inherits that ownership (Docker copies the image dir's
# ownership into a new empty volume) — the worker/API can then write without a manual chown. A bind
# mount to a host path instead needs that path writable by this uid (see docs/self-hosting.md).
RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
&& mkdir -p /files \
&& chown appuser /app /files
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode (E6 S1).
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode.
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
# no driver, so without this `vainfo`/`h264_vaapi` fail and transcode silently falls back to libx264.
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
ARG APT_REFRESH=
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
@@ -74,11 +85,14 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock: nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls.
COPY backend/requirements.lock .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && pip install -r requirements.lock
pip install --upgrade pip && pip install -r requirements.lock && pip uninstall -y pip
COPY --chown=appuser backend/ .
# Read at runtime by config.py as the app_version (a committed single source of truth).
+661
View File
@@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
+23 -6
View File
@@ -22,9 +22,11 @@ blocker and SponsorBlock keep working.
tags to slice the feed by.
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
- **In-app player** with resume, plus keyboard/scroll controls.
- **Download Center** — save videos to the server with yt-dlp in a Plex-friendly layout (format
presets, per-user storage quota), trim / crop / split & join them in a built-in editor, then save
to your device, share with another user, or hand out a public watch link.
- **Files** — one library for everything you keep: paste a link anywhere in the app to download it
with yt-dlp (format presets, per-user storage quota, the running downloads in a tray), upload your
own files, sort them into folders, trim / crop / split & join videos in a built-in editor, play them
where you left off, then save to your device, share with another user, or hand out a public watch
link that plays on any phone.
- **Plex library** *(optional)* — point Siftlode at your own Plex server to browse and play its
movies and shows alongside your feed, with watch state kept in sync.
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
@@ -94,9 +96,18 @@ docker compose -f docker-compose.selfhost.yml pull # or: docker compose pull
docker compose -f docker-compose.selfhost.yml up -d
```
Database migrations run automatically on startup. Your data (accounts, subscriptions, playlists, the
video catalog) lives in a Postgres volume — back it up with `scripts/backup.sh` (or `backup.ps1` on
Windows) and restore with `scripts/restore.sh`.
That follows `latest`, which moves with every release. Set `IMAGE_TAG=0.80.0` in `.env` to pin a
version and upgrade on your own schedule; released versions are listed in [`CHANGELOG.md`](CHANGELOG.md)
and in the app's Release Notes. The registry keeps only the newest few of those, so a pin decides
*when* you take the next hop rather than letting you stay put forever — see
[docs/self-hosting.md](docs/self-hosting.md#upgrading).
**Back up before upgrading.** Migrations run automatically on startup and are not undone by pulling
the old image back. A backup is the database dump **plus your `.env`** — `TOKEN_ENCRYPTION_KEY` is
what decrypts the Google, SMTP and Plex secrets stored in that database, so a dump restored without it
is only half an instance. Use `scripts/backup.sh` (or `backup.ps1` on Windows) and `scripts/restore.sh`;
the equivalent one-liners, the pinning recipe and the rollback steps are in
[docs/self-hosting.md](docs/self-hosting.md#backups).
## How it works
@@ -111,6 +122,12 @@ Windows) and restore with `scripts/restore.sh`.
FastAPI + PostgreSQL (SQLAlchemy, Alembic) backend; React + Vite + Tailwind + TanStack Query
frontend; packaged as a single Docker image with Docker Compose.
## License
[GNU AGPL-3.0](LICENSE). You can run, study, modify and share Siftlode; if you modify it and let other
people use it over a network, they are entitled to your modified source. Copyright © 2026 Siftlode
contributors.
## Note
This project is developed with AI assistance.
+1 -1
View File
@@ -1 +1 @@
0.78.3
0.108.0
+918
View File
@@ -0,0 +1,918 @@
{
"files": {
"./app/audit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/debugsession.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/diagnostics.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 40,
"endColumn": 49,
"lineCount": 1
}
}
],
"./app/discover/pins.py": [
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
}
],
"./app/discover/queries.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
}
],
"./app/discover/rails.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 4,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 18,
"endColumn": 41,
"lineCount": 1
}
}
],
"./app/downloads/adapters.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 38,
"lineCount": 1
}
}
],
"./app/downloads/backfill.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportReturnType",
"range": {
"startColumn": 23,
"endColumn": 27,
"lineCount": 1
}
}
],
"./app/downloads/credentials.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 27,
"endColumn": 30,
"lineCount": 1
}
}
],
"./app/downloads/gc.py": [
{
"code": "reportOperatorIssue",
"range": {
"startColumn": 16,
"endColumn": 28,
"lineCount": 1
}
}
],
"./app/downloads/pipeline.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 22,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportGeneralTypeIssues",
"range": {
"startColumn": 10,
"endColumn": 18,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 31,
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 43,
"lineCount": 1
}
}
],
"./app/downloads/service.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/downloads/urlguard.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 11,
"endColumn": 41,
"lineCount": 1
}
}
],
"./app/feedquery.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 37,
"endColumn": 5,
"lineCount": 3
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 37,
"endColumn": 5,
"lineCount": 3
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 97,
"lineCount": 1
}
}
],
"./app/logsink.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/notifications.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 52,
"endColumn": 58,
"lineCount": 1
}
}
],
"./app/plex/lookups.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 48,
"endColumn": 58,
"lineCount": 1
}
}
],
"./app/plex/people.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 15,
"endColumn": 23,
"lineCount": 1
}
}
],
"./app/plex/sync.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 21,
"endColumn": 69,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 21,
"endColumn": 69,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 70,
"endColumn": 83,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 76,
"endColumn": 89,
"lineCount": 1
}
}
],
"./app/plex/tmdb.py": [
{
"code": "reportAssignmentType",
"range": {
"startColumn": 16,
"endColumn": 9,
"lineCount": 10
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 35,
"lineCount": 1
}
}
],
"./app/plex/watch_sync.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 11,
"endColumn": 5,
"lineCount": 11
}
}
],
"./app/ratelimit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 14,
"endColumn": 25,
"lineCount": 1
}
}
],
"./app/routes/admin.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 9,
"endColumn": 13,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 37,
"endColumn": 45,
"lineCount": 1
}
}
],
"./app/routes/discover.py": [
{
"code": "reportInvalidTypeForm",
"range": {
"startColumn": 19,
"endColumn": 33,
"lineCount": 1
}
}
],
"./app/routes/plex/collections.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 20,
"endColumn": 45,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 20,
"endColumn": 46,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 39,
"endColumn": 47,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 46,
"endColumn": 49,
"lineCount": 1
}
}
],
"./app/routes/scheduler.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 25,
"endColumn": 28,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 25,
"endColumn": 28,
"lineCount": 1
}
}
],
"./app/sync/explore.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
}
],
"./app/sync/maintenance.py": [
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 19,
"endColumn": 42,
"lineCount": 1
}
}
],
"./app/sync/playlists.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 55,
"endColumn": 72,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 51,
"endColumn": 68,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 39,
"endColumn": 56,
"lineCount": 1
}
}
],
"./app/sync/runner.py": [
{
"code": "reportAssignmentType",
"range": {
"startColumn": 19,
"endColumn": 88,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 16,
"endColumn": 24,
"lineCount": 1
}
},
{
"code": "reportOptionalIterable",
"range": {
"startColumn": 23,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 45,
"endColumn": 72,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 19,
"endColumn": 9,
"lineCount": 7
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 19,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportOptionalIterable",
"range": {
"startColumn": 23,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 47,
"endColumn": 55,
"lineCount": 1
}
}
],
"./app/sync/videos.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/worker.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 40,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 40,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 35,
"endColumn": 43,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 24,
"endColumn": 32,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 36,
"lineCount": 1
}
}
]
}
}
-19
View File
@@ -1,19 +0,0 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN adduser --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
EXPOSE 8000
CMD ["sh", "entrypoint.sh"]
+2
View File
@@ -1,6 +1,8 @@
[alembic]
script_location = alembic
prepend_sys_path = .
# Split the path list the way the OS does (alembic's legacy space/comma/colon split is deprecated).
path_separator = os
[loggers]
keys = root,sqlalchemy,alembic
+5 -5
View File
@@ -1,17 +1,17 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
import app.models # noqa: F401 (import so models register on Base.metadata)
from app.logs import configure_logging
import app.tables # noqa: F401 (import so every table registers on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
# stream — one format with everything else, not alembic.ini's own fileConfig shape.
configure_logging("alembic")
target_metadata = Base.metadata
@@ -6,7 +6,7 @@ Create Date: 2026-07-07
Some sources (a direct media URL like a reddit v.redd.it HLS manifest) carry no thumbnail, so the
download shows a blank image box. The worker now extracts a representative frame with ffmpeg and
saves it as a poster sidecar; `poster_path` (relative to download_root) records it so the card /
saves it as a poster sidecar; `poster_path` (relative to files_root) records it so the card /
watch page / link preview can fall back to it. Nullable — only set when there was no thumbnail.
"""
@@ -0,0 +1,58 @@
"""users.password_revoked_at — remember that SEC-10 dropped a preset password
Revision ID: 0078_password_revoked_at
Revises: 0077_plex_playback_class
Create Date: 2026-09-02
SEC-10 (`auth.revoke_preset_password_if_unverified`) nulls `password_hash` when an account whose email
was never verified is attested (Google adoption, or admin approval). That left NO trace, and
password-reset eligibility keyed off `password_hash IS NOT NULL` — so such an account could neither
sign in (no password) nor request a reset link (not "eligible"): a permanent lockout with a silent,
uniform response at every step.
This column is that trace. A row with `password_revoked_at` set is a local account that HAD a password
and had it revoked, so the reset flow treats it as eligible again. Setting a password (reset-confirm or
/auth/set-password) clears the stamp. A Google-only account that never had a password keeps it NULL and
stays reset-ineligible by design — its owner adds a password from Settings while signed in.
Backfill: the signature is exact and narrow — a local (google_sub IS NULL), non-demo account with NO
password could ONLY have got there through the revoke. Registration always sets a hash, the setup
wizard sets one too, every Google row carries a google_sub (the adoption path assigns it BEFORE
revoking), and nothing in the codebase ever clears google_sub. Stamp those with the current time so
they become recoverable without operator intervention.
Deliberately NOT also requiring `email_verified = false`, even though the revoke only fires on an
unverified account: verifying is exactly what such a victim does NEXT. The revoke does not invalidate
the outstanding verification token, so clicking the emailed link sets email_verified=True while
password_hash stays NULL — and that row has no route back at all (reset ineligible, resend refuses an
already-verified address, and set-password needs a signed-in session the account can no longer obtain:
no password to sign in with, and no Google identity). Those are the victims who most need the stamp,
so the predicate must not exclude them.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0078_password_revoked_at"
down_revision: Union[str, None] = "0077_plex_playback_class"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"users",
sa.Column("password_revoked_at", sa.DateTime(timezone=True), nullable=True),
)
op.execute(
"UPDATE users SET password_revoked_at = now() "
"WHERE password_hash IS NULL "
"AND google_sub IS NULL "
"AND is_demo = false"
)
def downgrade() -> None:
op.drop_column("users", "password_revoked_at")
@@ -0,0 +1,65 @@
"""email_canonical on users + invites, for matching one mailbox across Gmail spellings
Revision ID: 0079_email_canonical
Revises: 0078_password_revoked_at
Create Date: 2026-09-04
Gmail delivers `x.y@gmail.com`, `xy@gmail.com` and `xy+tag@gmail.com` to ONE mailbox; the invite gate
only lowercased. On 2026-09-03 that turned an approved user's Google sign-in into a silent second
access request (OAuth returns the account's stored primary, which need not be the spelling they typed)
and left prod with two user rows for one mailbox.
The backfill below reproduces `app.utils.canonical_email` in SQL. That duplication is deliberate — a
migration must not import application code that will keep changing — but it means the two have to
agree; `test_canonical_email_backfill_matches_the_helper` pins them together.
Deliberately NOT unique. Prod already carries canonical duplicates: two approved invites for one
mailbox dating back to 2026-06-16, and two active user rows from the 2026-09-03 walkthrough. Merging
accounts is a judgement call about people's data, not something a migration gets to do on its own, so
this migration only records the canonical form and leaves the duplicates in place for a human.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0079_email_canonical"
down_revision: str | None = "0078_password_revoked_at"
branch_labels: str | None = None
depends_on: str | None = None
# Mirrors app.utils.canonical_email: Gmail (and its googlemail alias) loses dots and any `+tag` from
# the local part and lands on gmail.com; every other domain is only lowercased, because the dot rule
# is not guaranteed elsewhere — and does not hold for Workspace custom domains unless their admin
# enabled it.
_CANON_SQL = """
CASE
WHEN split_part(lower({col}), '@', 2) IN ('gmail.com', 'googlemail.com')
-- an empty local part after stripping falls through to plain lowercase, as the helper does
AND replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') <> ''
THEN replace(split_part(split_part(lower({col}), '@', 1), '+', 1), '.', '') || '@gmail.com'
ELSE lower({col})
END
"""
def _backfill(table: str) -> None:
op.execute(
f"UPDATE {table} SET email_canonical = {_CANON_SQL.format(col='email')} "
f"WHERE email IS NOT NULL"
)
def upgrade() -> None:
for table in ("users", "invites"):
op.add_column(
table,
sa.Column("email_canonical", sa.String(length=320), nullable=False, server_default=""),
)
_backfill(table)
op.create_index(f"ix_{table}_email_canonical", table, ["email_canonical"])
def downgrade() -> None:
for table in ("users", "invites"):
op.drop_index(f"ix_{table}_email_canonical", table_name=table)
op.drop_column(table, "email_canonical")
@@ -0,0 +1,32 @@
"""users.google_email — the address Google asserts, kept beside the one the user registered with
Revision ID: 0080_google_email
Revises: 0079_email_canonical
Create Date: 2026-09-04
Since 0079's follow-up a Google sign-in no longer overwrites `users.email`, so a deliberate
`me+siftlode@gmail.com` survives being linked to a Google account whose primary is `me@gmail.com`.
That leaves two true addresses for one account and only one of them stored. This column holds the
other, so the account menu and the Settings sign-in row can show which address Google actually
answers with.
No backfill is possible: Google's answer is only known at sign-in. Accounts linked before this
column existed carry NULL until their owner next signs in with Google — and for the majority the two
addresses are identical anyway, where the UI shows nothing either way.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0080_google_email"
down_revision: str | None = "0079_email_canonical"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("google_email", sa.String(length=320), nullable=True))
def downgrade() -> None:
op.drop_column("users", "google_email")
@@ -0,0 +1,34 @@
"""invites.invited_at — when the admin actually told this person the door was open
Revision ID: 0081_invite_invited_at
Revises: 0080_google_email
Create Date: 2026-09-05
Whitelisting an address stays SILENT by design: the operator is opening a door as a possibility,
usually after agreeing it with the person, and a row can sit approved for weeks without anyone
registering. Telling them is therefore a separate, deliberate act — a send button on the row.
Which raises the one question the admin cannot otherwise answer: "did I already write to them?"
Without a record the only failure mode is writing to a real person twice, which is exactly the
nudging the silent-by-default design exists to avoid. The audit log holds the event, but not where
the decision is made.
NULL means "never invited from here", which is true of every existing row: no backfill is possible
or wanted, since the historic approval mails answered a REQUEST rather than inviting anyone.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0081_invite_invited_at"
down_revision: str | None = "0080_google_email"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("invites", sa.Column("invited_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.drop_column("invites", "invited_at")
@@ -0,0 +1,34 @@
"""invites.lang — the language this person asked us in
Revision ID: 0082_invite_lang
Revises: 0081_invite_invited_at
Create Date: 2026-09-05
The mails an access request produces — approved, denied, invited — go to an address that often has
no account yet, so there is no `preferences.language` to read. The signal does exist, though, and
only at the moment of the request: the language of the page the form was submitted from, or the
locale Google reported for a sign-in we refused. Recording it there is what lets the decision mail
answer in the requester's own language instead of an admin's guess about a stranger.
Captured rather than derived, for the third time in this epic: `preferences.language` (S6) and
`google_email` (S3) both taught that a signal which exists only in one moment cannot be backfilled
later. So no backfill here either — NULL means "we were never told", which is the honest state for
every historic row, and for every address an admin whitelists by hand. Those fall back to English
unless the admin picks otherwise on the invitation send.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0082_invite_lang"
down_revision: str | None = "0081_invite_invited_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("invites", sa.Column("lang", sa.String(length=8), nullable=True))
def downgrade() -> None:
op.drop_column("invites", "lang")
@@ -0,0 +1,78 @@
"""comfort_pins — the shelf you fall back to when you cannot decide
Revision ID: 0083_comfort_pins
Revises: 0082_invite_lang
Create Date: 2026-09-06
E24's one new table. The shelf holds five different things — a video, a YouTube channel, a Plex show,
a Plex movie, a playlist — because comfort media is usually "put on X", a series or a channel, and
almost never one particular video.
Three shapes were possible and two are wrong here:
- a boolean on `video_states` / `plex_states` cannot ORDER a shelf, and has nowhere to put a channel
or a show (neither has a per-user state row at all);
- a polymorphic `(kind, ref)` string pair orders and holds anything, but drops the foreign key — and a
Plex item removed from the library, or a channel the user unsubscribes and prunes, would then leave
a dead tile on the one surface whose entire job is to be reliable when nothing else is.
So: one nullable CASCADE-ing FK per kind, plus `num_nonnulls(...) = 1`. The union invariant lives in
the schema rather than in a comment at every insert site, which is the same rule that the
`cancelled`/`canceled` bug taught this codebase.
NULLs are DISTINCT in a Postgres unique index, so the five per-kind `(user_id, <ref>)` uniques each
stop a double-pin without colliding with the other four kinds' NULL rows.
No backfill: an empty shelf is the honest state for every existing account. The surface renders its
own empty state and invites the first pin.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0083_comfort_pins"
down_revision: str | None = "0082_invite_lang"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"comfort_pins",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("video_id", sa.String(length=16), sa.ForeignKey("videos.id", ondelete="CASCADE")),
sa.Column(
"channel_id", sa.String(length=32), sa.ForeignKey("channels.id", ondelete="CASCADE")
),
sa.Column("plex_item_id", sa.Integer(), sa.ForeignKey("plex_items.id", ondelete="CASCADE")),
sa.Column("plex_show_id", sa.Integer(), sa.ForeignKey("plex_shows.id", ondelete="CASCADE")),
sa.Column("playlist_id", sa.Integer(), sa.ForeignKey("playlists.id", ondelete="CASCADE")),
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.CheckConstraint(
"num_nonnulls(video_id, channel_id, plex_item_id, plex_show_id, playlist_id) = 1",
name="ck_comfort_pin_one_ref",
),
sa.UniqueConstraint("user_id", "video_id", name="uq_comfort_pin_video"),
sa.UniqueConstraint("user_id", "channel_id", name="uq_comfort_pin_channel"),
sa.UniqueConstraint("user_id", "plex_item_id", name="uq_comfort_pin_plex_item"),
sa.UniqueConstraint("user_id", "plex_show_id", name="uq_comfort_pin_plex_show"),
sa.UniqueConstraint("user_id", "playlist_id", name="uq_comfort_pin_playlist"),
)
op.create_index("ix_comfort_pins_user_id", "comfort_pins", ["user_id"])
def downgrade() -> None:
op.drop_index("ix_comfort_pins_user_id", table_name="comfort_pins")
op.drop_table("comfort_pins")
@@ -0,0 +1,41 @@
"""add download_jobs.error_detail — the domain-specific entry of the failure catalog (E11.5 S1)
Revision ID: 0084_dl_job_error_detail
Revises: 0083_comfort_pins
Create Date: 2026-09-13
`error_code` is the generic category (it decides whether a Retry is offered). `error_detail` names
the host-specific catalog entry `downloads.errors.classify` matched on top of it — `facebook.login_wall`
and its siblings — so the UI can say what went wrong ON THAT SITE and what to do about it, instead of
the category's generic sentence. Nullable: most failures have no host-specific entry.
Backfill: a job's code is written once, at failure time, so a new rule never reaches the rows that
failed before it. The one shape measured on prod (2026-09-13, jobs 9/66/75/145/156/157/182/256/257) is
yt-dlp's `[facebook] <id>: Cannot parse data`, stored as `unknown`. Without a cookie that is Facebook's
login wall. Whether a cookie was attached at the time is not recorded anywhere, so every old row gets
the no-cookie entry — on prod no account had a Facebook cookie, so that is what they all were.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0084_dl_job_error_detail"
down_revision: str | None = "0083_comfort_pins"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("error_detail", sa.String(length=48), nullable=True))
op.execute(
sa.text(
"UPDATE download_jobs SET error_code = 'login_required', "
"error_detail = 'facebook.login_wall' "
"WHERE (error_code IS NULL OR error_code = 'unknown') "
"AND error ILIKE '%[facebook]%cannot parse data%'"
)
)
def downgrade() -> None:
op.drop_column("download_jobs", "error_detail")
@@ -0,0 +1,26 @@
"""add users.last_login_at — when the account last signed in (E22 S1)
Revision ID: 0085_users_last_login_at
Revises: 0084_dl_job_error_detail
Create Date: 2026-09-16
Stamped by the one sign-in path (`routes/auth._establish_session`), which the Google callback and
password login share. No backfill is possible: the signal exists only at sign-in, so existing rows
read NULL until each account signs in again, and the admin UI says so rather than inventing a date.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0085_users_last_login_at"
down_revision: str | None = "0084_dl_job_error_detail"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.drop_column("users", "last_login_at")
@@ -0,0 +1,48 @@
"""add log_events — stored log events for diagnostics (E22 S3)
Revision ID: 0086_log_events
Revises: 0085_users_last_login_at
Create Date: 2026-09-17
Written by app/logsink.py's background writer (WARNING and above, request lines), pruned by the log_gc
job after `log_retention_days`. Sized against prod on 2026-09-16: ~2k rows on a normal day, ~14k on a
YouTube-RSS-outage day before flood control — a few MB over the 7-day window. `user_id` carries no
foreign key on purpose (see models.LogEvent).
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0086_log_events"
down_revision: str | None = "0085_users_last_login_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"log_events",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
sa.Column("level", sa.String(10), nullable=False),
sa.Column("sev", sa.SmallInteger(), nullable=False),
sa.Column("source", sa.String(16), nullable=False),
sa.Column("logger", sa.String(128), nullable=False),
sa.Column("event", sa.String(64), nullable=True),
sa.Column("msg", sa.Text(), nullable=False),
sa.Column("attrs", postgresql.JSONB(), nullable=True),
sa.Column("exc", sa.Text(), nullable=True),
sa.Column("request_id", sa.String(64), nullable=True),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.Column("debug_session_id", sa.String(64), nullable=True),
sa.Column("job_id", sa.Integer(), nullable=True),
sa.Column("version", sa.String(64), nullable=True),
)
op.create_index("ix_log_events_ts", "log_events", ["ts"])
op.create_index("ix_log_events_request_id", "log_events", ["request_id"])
op.create_index("ix_log_events_user_ts", "log_events", ["user_id", "ts"])
def downgrade() -> None:
op.drop_table("log_events")
@@ -0,0 +1,31 @@
"""add users.diagnostics_level + diagnostics_until — a user's consented diagnostics level (E22 S5)
Revision ID: 0087_users_diagnostics_level
Revises: 0086_log_events
Create Date: 2026-09-17
Both NULL = the base level (Alap). A raised level is ALWAYS paired with its expiry, computed by the
server at consent time (`app/diagnostics.py`), so no client can ask for a level that never ends.
Columns rather than `users.preferences`: that JSON is written wholesale by the client, and a consent
with a lifetime must not be something a client can write.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0087_users_diagnostics_level"
down_revision: str | None = "0086_log_events"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("diagnostics_level", sa.String(8), nullable=True))
op.add_column(
"users", sa.Column("diagnostics_until", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("users", "diagnostics_until")
op.drop_column("users", "diagnostics_level")
@@ -0,0 +1,30 @@
"""add channels.feed_fallback_at — when a channel's uploads were last read through the API because
its RSS feed would not answer
Revision ID: 0088_channels_feed_fallback_at
Revises: 0087_users_diagnostics_level
Create Date: 2026-09-19
YouTube's RSS feeds go dark for hours at a time, per channel (2026-09-18/19: 393 of 396 answered 404
for four hours; the same channel answers 404 to every retry). The RSS poll then reads the newest
uploads through the Data API instead, at a quota cost, and at most once per `rss_fallback_hours` per
channel: this column is that clock. NULL = never needed it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0088_channels_feed_fallback_at"
down_revision: str | None = "0087_users_diagnostics_level"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"channels", sa.Column("feed_fallback_at", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("channels", "feed_fallback_at")
@@ -0,0 +1,81 @@
"""add debug_sessions + debug_screenshots, and index log_events.debug_session_id (E16)
Revision ID: 0089_debug_sessions
Revises: 0088_channels_feed_fallback_at
Create Date: 2026-09-19
A debug session is one recording: Start → reproduce → Stop (app/debugsession.py). Every log line
of that user's requests while it runs carries its id in `log_events.debug_session_id` — the column
E22 created and left NULL — and the issue it ends in is built from exactly those rows, so the column
gets its index now. Partial: nearly every row has no session, and an index of NULLs would be most
of the table for nothing.
At most one OPEN session per user, held by a partial unique index rather than by the start handler
alone: two tabs pressing Start at once would otherwise both pass a "none open" check.
Screenshots are rows, not files: bounded (size and count per session, app/debugsession.py), gone
with their session by cascade, and a self-host has no file volume to configure for them.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0089_debug_sessions"
down_revision: str | None = "0088_channels_feed_fallback_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"debug_sessions",
sa.Column("id", sa.String(16), primary_key=True),
sa.Column(
"user_id",
sa.Integer,
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("level", sa.String(8), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("until", sa.DateTime(timezone=True), nullable=False),
sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("ended_reason", sa.String(16), nullable=True),
sa.Column("issue_url", sa.String(512), nullable=True),
)
op.create_index(
"uq_debug_sessions_open_per_user",
"debug_sessions",
["user_id"],
unique=True,
postgresql_where=sa.text("ended_at IS NULL"),
)
op.create_table(
"debug_screenshots",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"session_id",
sa.String(16),
sa.ForeignKey("debug_sessions.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("mime", sa.String(16), nullable=False),
sa.Column("size", sa.Integer, nullable=False),
sa.Column("data", sa.LargeBinary, nullable=False),
)
op.create_index(
"ix_log_events_debug_session",
"log_events",
["debug_session_id"],
postgresql_where=sa.text("debug_session_id IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_log_events_debug_session", table_name="log_events")
op.drop_table("debug_screenshots")
op.drop_index("uq_debug_sessions_open_per_user", table_name="debug_sessions")
op.drop_table("debug_sessions")
@@ -0,0 +1,27 @@
"""add debug_sessions.draft — the issue being written from a session (E16 S3)
Revision ID: 0090_debug_session_draft
Revises: 0089_debug_sessions
Create Date: 2026-09-19
The composer saves as it goes: the title, the rendered markdown body that an export or an issue
carries, the log rows the user took out, and the form's own fields so reopening it later (Settings →
Diagnosztika lists past sessions) shows what was written. NULL until the composer first saves.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0090_debug_session_draft"
down_revision: str | None = "0089_debug_sessions"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("debug_sessions", sa.Column("draft", JSONB, nullable=True))
def downgrade() -> None:
op.drop_column("debug_sessions", "draft")
@@ -0,0 +1,27 @@
"""add tagline_hu to plex_items + plex_shows — the HU overlay's tagline (E30 S1)
Revision ID: 0091_plex_tagline_hu
Revises: 0090_debug_session_draft
Create Date: 2026-09-29
The Hungarian overlay section lists a tagline beside the title and summary; the info page showed the
English one to a Hungarian reader. NULL until the next overlay sync fills it (serving falls back to EN).
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0091_plex_tagline_hu"
down_revision: str | None = "0090_debug_session_draft"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("plex_items", sa.Column("tagline_hu", sa.Text(), nullable=True))
op.add_column("plex_shows", sa.Column("tagline_hu", sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column("plex_shows", "tagline_hu")
op.drop_column("plex_items", "tagline_hu")
@@ -0,0 +1,31 @@
"""add biography_hu to plex_people — TMDB's Hungarian biography (E30 S4)
Revision ID: 0092_plex_person_biography_hu
Revises: 0091_plex_tagline_hu
Create Date: 2026-09-29
The person page showed a Hungarian reader the English biography although TMDB has a Hungarian one for
most people. The person fetch now appends `translations` and keeps the Hungarian biography beside the
English one. Already-enriched people are un-stamped, so their next page open refetches them once (the
same lazy path as a first open) and fills it; until then the English biography stands in.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0092_plex_person_biography_hu"
down_revision: str | None = "0091_plex_tagline_hu"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("plex_people", sa.Column("biography_hu", sa.Text(), nullable=True))
op.execute(
"UPDATE plex_people SET tmdb_enriched_at = NULL "
"WHERE tmdb_person_id IS NOT NULL AND tmdb_enriched_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_column("plex_people", "biography_hu")
@@ -0,0 +1,29 @@
"""re-fetch linked people — "Known for" ranked by votes, without guest visits (E30 S6)
Revision ID: 0093_plex_person_known_for_votes
Revises: 0092_plex_person_biography_hu
Create Date: 2026-09-30
A person's "Known for" pool was their credits ranked by TMDB popularity, so talk-show visits as
themselves crowded out the films (Tom Cruise: 14 of 15 were talk shows, none in the library). The pool is
now ranked by vote count, without guest visits, with directed works, and deeper. Already-enriched people
are un-stamped so the people_enrich job fetches them again; their biography stays on the row meanwhile.
"""
from alembic import op
revision: str = "0093_plex_person_known_for_votes"
down_revision: str | None = "0092_plex_person_biography_hu"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute(
"UPDATE plex_people SET tmdb_enriched_at = NULL "
"WHERE tmdb_person_id IS NOT NULL AND tmdb_enriched_at IS NOT NULL"
)
def downgrade() -> None:
pass # the watermark only says "fetched"; a refetch under the old ranking is not worth undoing to
@@ -0,0 +1,45 @@
"""Hungarian genres on every Plex row, not only on overlaid ones (E30 review)
Revision ID: 0094_plex_genres_hu_everywhere
Revises: 0093_plex_person_known_for_votes
Create Date: 2026-09-30
The HU genre facets and filter read `genres_hu`, which only the overlay pass wrote — so a library with
no HU twin, or a guid-less row, offered English genres a Hungarian filter could never match. The sync
now writes `genres_hu` for every row (plex/sync.py `_apply_facet_fields`); this fills the rows it has
not reached yet, from their own English list through the same map.
"""
import json
import sqlalchemy as sa
from alembic import op
from app.plex.genres import hu_genres
revision: str = "0094_plex_genres_hu_everywhere"
down_revision: str | None = "0093_plex_person_known_for_votes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
conn = op.get_bind()
for table in ("plex_items", "plex_shows"):
rows = conn.execute(
sa.text(
f"SELECT id, genres FROM {table} "
"WHERE genres_hu IS NULL AND jsonb_typeof(genres) = 'array'"
)
).fetchall()
updates = [
{"id": rid, "g": json.dumps(hu)} for rid, genres in rows if (hu := hu_genres(genres))
]
if updates:
conn.execute(
sa.text(f"UPDATE {table} SET genres_hu = CAST(:g AS jsonb) WHERE id = :id"), updates
)
def downgrade() -> None:
pass # derived from `genres`; the sync writes the same values again
@@ -0,0 +1,83 @@
"""videos: the per-channel index covers what the channel manager and the feed sidebar read (E31 S4d)
The channel manager (`GET /api/channels`) sums each subscribed channel's videos: count, last upload, total
duration. The feed sidebar (`/feed/summary`, `/feed/years`) counts the reader's videos per channel and per
year under the default filters: available, not via explore, not a short, not live. Every one of them found
its rows through `ix_videos_channel_pub_desc` (channel_id, published_at DESC NULLS LAST, id) and then read
each video's heap page for the rest: ~270k rows at ~2.5 KB each, 681 MB on prod. That stays in cache only
while nothing else needs the memory. Measured on prod (2026-10-02): `/api/channels` 51–61 ms warm, but
2.25 s and 1.96 s on a first open in the browser and 6.4 s under EXPLAIN with 30k blocks read from disk;
`/feed/summary` and `/feed/years` ~140 ms warm, ~800 ms cold.
The same key with those columns INCLUDEd makes them Index Only Scans. On a prod-copy localdev, after a
VACUUM, the manager's sums read 0 heap pages and the summary reads the 25 MB index instead of the heap.
The visibility map is what lets them skip the heap; prod's autovacuum kept it complete (relallvisible ==
relpages). The key columns are unchanged, so every reader of the old index (the per-channel feed, the FK
cascade, the sync upsert, channel detail) rides this one. The old index is dropped.
tests/test_channel_list_index_only.py holds the queries and this column list together.
CONCURRENTLY (autocommit_block) builds it without blocking a live prod. IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent. A re-run after a build that failed part-way rebuilds
the INVALID leftover, and the old index is dropped only once the new one is valid: a re-run that kept the
leftover and dropped the old one left `videos` with no usable channel_id index (E31 S5 review).
"""
from alembic import op
from sqlalchemy import text
revision: str = "0095_videos_channel_cover_index"
down_revision: str | None = "0094_plex_genres_hu_everywhere"
branch_labels: str | None = None
depends_on: str | None = None
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def _swap(build: str, ddl: str, drop: str) -> None:
_build(build, ddl)
if not _valid(build):
raise RuntimeError(f"{build} did not build valid; {drop} is kept")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {drop}")
def upgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_cover",
"ON videos (channel_id, published_at DESC NULLS LAST, id) "
"INCLUDE (duration_seconds, unavailable_since, via_explore, is_short, live_status)",
"ix_videos_channel_pub_desc",
)
def downgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_desc",
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)",
"ix_videos_channel_pub_cover",
)
@@ -0,0 +1,86 @@
"""videos: an index per feed sort, so a sorted page stops after its rows (E31 S4e)
The feed's views, duration and title sorts had no index. Every page (the first and each next cursor)
read the whole videos heap, filtered it and kept a top-N: on prod ~350 ms a page for views/duration,
550–780 ms for title (the ICU comparison on every row), 2.3 s for page 2 when the copies of one request
queued. With an index in the exact ORDER BY (`feedquery.order_by`: the key, NULLS LAST, then id) the
planner walks it and stops after the page. On a prod-copy localdev every plain sort fell from 180–310 ms
to ~1 ms, deep pages (20) to 3–5 ms, and narrow filters (a tag, a channel, a year, nothing found) stayed
at 0.3–25 ms.
One index per direction: a backward scan of `DESC NULLS LAST` gives `ASC NULLS FIRST`, not the
`ASC NULLS LAST` the asc sorts ask for. The title pair is per reader language (`app.collation.ICU`):
an index only serves an ORDER BY in its own collation.
Cost: ~130 MB (localdev: 10 + 10 + 8 + 8 + 4 × 23). Writes stay cheap: view_count and duration are
written by the enrich, which rewrites indexed columns already (the revalidation job only touches
last_checked_at), so no HOT update is lost. A search combined with one of these sorts can now walk the
index too: a term matching ~2 % of the videos took 130–200 ms instead of 7 (still under the plain sort it
replaces); rarer and broader terms stay fast. Fencing the index off under a search made broad terms
(`m`, `the`) take 2 s, so it is left to the planner.
The title indexes are the schema's first ones on an ICU collation: an ICU upgrade in the Postgres image
reorders them, and `app.collation.heal_drift` rebuilds them at startup when it sees one.
tests/test_feed_sort_indexes.py holds each sort and its index together.
CONCURRENTLY (autocommit_block) builds them without blocking a live prod; IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent.
"""
from alembic import op
from sqlalchemy import text
revision: str = "0096_videos_sort_indexes"
down_revision: str | None = "0095_videos_channel_cover_index"
branch_labels: str | None = None
depends_on: str | None = None
# name → the indexed ORDER BY (feedquery.sort_keys + order_by: key NULLS LAST, then id).
INDEXES = {
"ix_videos_views_desc": "view_count DESC NULLS LAST, id",
"ix_videos_views_asc": "view_count ASC NULLS LAST, id",
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
"ix_videos_title_en": '(lower(title) COLLATE "en-x-icu") ASC NULLS LAST, id',
"ix_videos_title_en_desc": '(lower(title) COLLATE "en-x-icu") DESC NULLS LAST, id',
"ix_videos_title_hu": '(lower(title) COLLATE "hu-x-icu") ASC NULLS LAST, id',
"ix_videos_title_hu_desc": '(lower(title) COLLATE "hu-x-icu") DESC NULLS LAST, id',
}
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def upgrade() -> None:
with op.get_context().autocommit_block():
for name, columns in INDEXES.items():
_build(name, f"ON videos ({columns})")
def downgrade() -> None:
with op.get_context().autocommit_block():
for name in INDEXES:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
@@ -0,0 +1,69 @@
"""messages: drop direct messaging (the owner's decision, 2026-10-04)
Chat — end-to-end encrypted direct messages plus the server's welcome note — was never used: prod held
two welcome notes, no message between people, and one key. Its tables go with its code. The rows are
not migrated anywhere; the nightly backup holds them.
The downgrade restores the schema as it stood (0026 + 0027), empty: a message can't be brought back
from a migration, and the E2EE key blobs were only ever readable by their owner's browser anyway.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0097_drop_messages"
down_revision: str | None = "0096_videos_sort_indexes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_table("message_keys")
op.drop_index("ix_messages_created_at", table_name="messages")
op.drop_index("ix_messages_recipient_id", table_name="messages")
op.drop_index("ix_messages_sender_id", table_name="messages")
op.drop_table("messages")
def downgrade() -> None:
op.create_table(
"messages",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"sender_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column(
"recipient_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("body", sa.Text(), nullable=True),
sa.Column("read_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("kind", sa.String(length=16), nullable=False, server_default="user"),
sa.Column("ciphertext", sa.Text(), nullable=True),
sa.Column("iv", sa.String(length=32), nullable=True),
)
op.create_index("ix_messages_sender_id", "messages", ["sender_id"])
op.create_index("ix_messages_recipient_id", "messages", ["recipient_id"])
op.create_index("ix_messages_created_at", "messages", ["created_at"])
op.create_table(
"message_keys",
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column("public_key", sa.Text(), nullable=False),
sa.Column("wrapped_private_key", sa.Text(), nullable=False),
sa.Column("salt", sa.String(length=64), nullable=False),
sa.Column("wrap_iv", sa.String(length=32), nullable=False),
sa.Column("key_check", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
@@ -0,0 +1,55 @@
"""demo: drop the shared demo account (the owner's decision, 2026-10-04)
The demo account — one shared user that whitelisted emails entered from the login page, reset on a
schedule — goes with its code. Prod held the demo user and 10 whitelist rows.
The upgrade deletes the demo user first (its playlists, states and subscriptions cascade; audit and
quota rows keep their history with the actor nulled, as for any deleted account), then the traces
that only the demo code could render: its audit rows (whitelist edits, manual resets, and every row
about the reset job — its runs and its interval changes), the reset job's interval override and its
completion notices. Nothing is migrated anywhere; the nightly backup holds the rows.
The downgrade restores the schema as it stood (0014), empty: the demo user is created lazily on its
first sign-in, so an empty `demo_whitelist` is the whole of its state.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0098_drop_demo_account"
down_revision: str | None = "0097_drop_messages"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute("DELETE FROM users WHERE is_demo")
op.execute(
"DELETE FROM audit_log WHERE action IN ('demo_add', 'demo_remove', 'demo_reset')"
" OR (target_type = 'job' AND target_id = 'demo_reset')"
)
op.execute("DELETE FROM scheduler_settings WHERE job_id = 'demo_reset'")
op.execute("DELETE FROM notifications WHERE data->>'job_id' = 'demo_reset'")
op.drop_index("ix_demo_whitelist_email", table_name="demo_whitelist")
op.drop_table("demo_whitelist")
op.drop_index("ix_users_is_demo", table_name="users")
op.drop_column("users", "is_demo")
def downgrade() -> None:
op.add_column(
"users",
sa.Column("is_demo", sa.Boolean(), nullable=False, server_default=sa.false()),
)
op.create_index("ix_users_is_demo", "users", ["is_demo"])
op.create_table(
"demo_whitelist",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("email", sa.String(length=320), nullable=False),
sa.Column("note", sa.Text(), nullable=True),
sa.Column("added_by", sa.String(length=320), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_demo_whitelist_email", "demo_whitelist", ["email"], unique=True)
@@ -0,0 +1,42 @@
"""library: a resume point per user and library item
The library player opened every item at 0:00. `library_states` keeps where a user stopped, per job
(their own download, or one shared with them) — the same shape as `video_states`' position, without a
status: the library has no "watched". It starts empty; the downgrade drops it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0099_library_states"
down_revision: str | None = "0098_drop_demo_account"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"library_states",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"job_id",
sa.Integer(),
sa.ForeignKey("download_jobs.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("position_seconds", sa.Integer(), nullable=False),
sa.Column("progress_updated_at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),
)
op.create_index("ix_library_states_job_id", "library_states", ["job_id"])
def downgrade() -> None:
op.drop_index("ix_library_states_job_id", table_name="library_states")
op.drop_table("library_states")
@@ -0,0 +1,22 @@
"""plex_items: drop `markers`, a column nothing writes or reads
Intro/credits markers are read live from Plex for the item the player opens (with Plex's `final` flag
on the block that ends it); this cached copy was never filled — every row is NULL. The downgrade
restores the empty column.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0100_drop_plex_item_markers"
down_revision: str | None = "0099_library_states"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_column("plex_items", "markers")
def downgrade() -> None:
op.add_column("plex_items", sa.Column("markers", sa.JSON(), nullable=True))
@@ -0,0 +1,46 @@
"""Plex summaries and taglines as plain text
Revision ID: 0101_plex_text_plain
Revises: 0100_drop_plex_item_markers
Create Date: 2026-10-08
Some Plex agents write a summary as HTML (`<p>…</p>`, `&aacute;`), and the app showed the tags as
text. The sync now stores every summary and tagline through `plain_text` (app/plex/plaintext.py);
this cleans the rows it wrote before, through the same function, so nothing waits for a full resync.
"""
import sqlalchemy as sa
from alembic import op
from app.plex.plaintext import plain_text
revision: str = "0101_plex_text_plain"
down_revision: str | None = "0100_drop_plex_item_markers"
branch_labels: str | None = None
depends_on: str | None = None
_COLUMNS = {
"plex_items": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_shows": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_collections": ("summary",),
}
def upgrade() -> None:
conn = op.get_bind()
for table, columns in _COLUMNS.items():
for col in columns:
rows = conn.execute(
sa.text(f"SELECT id, {col} FROM {table} WHERE {col} ~ '[<&]'")
).fetchall()
updates = [
{"id": rid, "v": clean}
for rid, value in rows
if (clean := plain_text(value)) != value
]
if updates:
conn.execute(sa.text(f"UPDATE {table} SET {col} = :v WHERE id = :id"), updates)
def downgrade() -> None:
pass # the markup is not wanted back; the next sync would clean it again
@@ -0,0 +1,193 @@
"""The Files library: folders, and a file's retention on the user's job
Revision ID: 0102_file_folders
Revises: 0101_plex_text_plain
Create Date: 2026-10-11
Every download, clip and upload of a user now sits in a folder of their Files library
(`app.downloads.folders`): a source folder made per origin (YouTube, Plex, uploads, one per other
site) or one of their own. How long a file stays moves from the shared asset to each user's job
(`download_jobs.expires_at`, stamped from the folder's `retention_days`), so one user keeping a file
no longer depends on another's copy; the asset's own `expires_at` / `gc_notified` go.
This places every existing job in its source folder and carries each done job's expiry over from
the asset it holds (an upload keeps none). The source-key rule is copied here, frozen, from
`folders.source_key_for` as it is today — a migration must not follow the code's later changes.
"""
from urllib.parse import urlsplit
import sqlalchemy as sa
from alembic import op
revision: str = "0102_file_folders"
down_revision: str | None = "0101_plex_text_plain"
branch_labels: str | None = None
depends_on: str | None = None
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
_DEFAULT_RETENTION_DAYS = 30 # config.download_retention_days on the day this was written
def _site_key(url: str) -> str:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return "youtube"
return f"site:{host or 'unknown'}"
def _asset_key(kind: str | None, ref: str | None) -> str:
if kind == "youtube":
return "youtube"
if kind == "url" and ref:
return _site_key(ref)
return "upload"
def _key(job, jobs_by_id, assets) -> str:
kind = job.source_kind
if kind == "youtube":
return "youtube"
if kind == "upload":
return "upload"
if kind == "clip":
spec = job.edit_spec or {}
return "plex" if spec.get("source") == "plex" else "youtube"
if kind == "url":
return _site_key(job.source_ref)
if kind == "edit":
parent = jobs_by_id.get(job.source_job_id)
if parent is not None and parent.id != job.id:
return _key(parent, jobs_by_id, assets)
asset = assets.get(job.source_asset_id)
if asset is not None:
return _asset_key(asset.source_kind, asset.source_ref)
return "upload"
def upgrade() -> None:
op.create_table(
"file_folders",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(16), nullable=False),
sa.Column("source_key", sa.String(260)), # "site:" + a host (<= 253)
sa.Column("name", sa.String(255)),
sa.Column("retention_days", sa.Integer()),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
sa.CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
op.create_index("ix_file_folders_user_id", "file_folders", ["user_id"])
op.create_index(
"uq_file_folder_own_name",
"file_folders",
["user_id", sa.text("lower(name)")],
unique=True,
postgresql_where=sa.text("name IS NOT NULL"),
)
op.add_column(
"download_jobs",
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="RESTRICT"),
),
)
op.add_column(
"download_jobs",
sa.Column("kept", sa.Boolean(), server_default="false", nullable=False),
)
op.add_column("download_jobs", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"download_jobs",
sa.Column("expiry_warned", sa.Boolean(), server_default="false", nullable=False),
)
op.create_index("ix_download_jobs_folder_id", "download_jobs", ["folder_id"])
op.create_index("ix_download_jobs_expires_at", "download_jobs", ["expires_at"])
conn = op.get_bind()
raw = conn.execute(
sa.text("SELECT value FROM system_config WHERE key = 'download_retention_days'")
).scalar()
retention = int(raw) if raw not in (None, "") else _DEFAULT_RETENTION_DAYS
jobs = conn.execute(
sa.text(
"SELECT j.id, j.user_id, j.source_kind, j.source_ref, j.edit_spec, j.source_job_id,"
" j.source_asset_id, j.status, j.asset_id, a.expires_at AS asset_expires_at"
" FROM download_jobs j LEFT JOIN media_assets a ON a.id = j.asset_id"
)
).fetchall()
jobs_by_id = {j.id: j for j in jobs}
assets = {
a.id: a
for a in conn.execute(sa.text("SELECT id, source_kind, source_ref FROM media_assets"))
}
folder_ids: dict[tuple[int, str], int] = {}
for job in jobs:
key = _key(job, jobs_by_id, assets)
slot = (job.user_id, key)
if slot not in folder_ids:
folder_ids[slot] = conn.execute(
sa.text(
"INSERT INTO file_folders (user_id, kind, source_key, retention_days)"
" VALUES (:u, 'source', :k, :r) RETURNING id"
),
{"u": job.user_id, "k": key, "r": None if key == "upload" else retention},
).scalar_one()
# A done file keeps the expiry it had (its asset's); an upload had none and keeps none.
expires = job.asset_expires_at if job.status == "done" and key != "upload" else None
conn.execute(
sa.text("UPDATE download_jobs SET folder_id = :f, expires_at = :e WHERE id = :id"),
{"f": folder_ids[slot], "e": expires, "id": job.id},
)
op.drop_column("media_assets", "expires_at")
op.drop_column("media_assets", "gc_notified")
def downgrade() -> None:
op.add_column("media_assets", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"media_assets",
sa.Column("gc_notified", sa.Boolean(), server_default="false", nullable=False),
)
# The asset goes when its last holder would: the latest of its holders' expiries (a holder that
# never expires keeps the asset unexpired).
op.execute(
"UPDATE media_assets a SET expires_at = sub.e FROM ("
" SELECT asset_id, CASE WHEN bool_or(expires_at IS NULL) THEN NULL ELSE max(expires_at) END"
" AS e FROM download_jobs WHERE asset_id IS NOT NULL AND status = 'done' GROUP BY asset_id"
") sub WHERE a.id = sub.asset_id AND a.source_kind <> 'upload'"
)
op.drop_index("ix_download_jobs_expires_at", table_name="download_jobs")
op.drop_index("ix_download_jobs_folder_id", table_name="download_jobs")
op.drop_column("download_jobs", "expiry_warned")
op.drop_column("download_jobs", "expires_at")
op.drop_column("download_jobs", "kept")
op.drop_column("download_jobs", "folder_id")
op.drop_index("uq_file_folder_own_name", table_name="file_folders")
op.drop_index("ix_file_folders_user_id", table_name="file_folders")
op.drop_table("file_folders")
@@ -0,0 +1,32 @@
"""The Files library: a file's SHA-256, and the expired rows retired
Revision ID: 0103_files_sha256
Revises: 0102_file_folders
Create Date: 2026-10-11
`media_assets.sha256` holds the file's hash once it is known (an upload is hashed while it streams
in, any other file on the first ask), so a machine fetching a file can check what it got.
An expired file now leaves its user's library (`downloads.gc`). Before, an expired download stayed
as a fileless "expired · download again" row (a done job with no asset). Those rows hold nothing on
disk and the library no longer shows them, so they are deleted here rather than left behind
unseen; their shares and public links go with them (ON DELETE CASCADE), and those already served
nothing. The downgrade cannot bring them back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0103_files_sha256"
down_revision: str | None = "0102_file_folders"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("sha256", sa.String(64)))
op.execute("DELETE FROM download_jobs WHERE status = 'done' AND asset_id IS NULL")
def downgrade() -> None:
op.drop_column("media_assets", "sha256")
@@ -0,0 +1,38 @@
"""A file's "Added" is when it became done
Revision ID: 0104_job_ready_at
Revises: 0103_files_sha256
Create Date: 2026-10-12
`download_jobs.ready_at` is when a job became done, and the Files library sorts and dates by it: a
download that failed for days and then went through used to sit days down the list, under its
request date. Every path to done stamps it, held by `ck_download_jobs_ready_at`.
A done job from before has no record of that moment (its "ready to watch" notice never named the
job, so it cannot be matched back), so it takes its request date, which is where it already sat.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0104_job_ready_at"
down_revision: str | None = "0103_files_sha256"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("ready_at", sa.DateTime(timezone=True)))
op.execute(
"""
UPDATE download_jobs SET ready_at = created_at WHERE status = 'done'
"""
)
op.create_check_constraint(
"ck_download_jobs_ready_at", "download_jobs", "status <> 'done' OR ready_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_constraint("ck_download_jobs_ready_at", "download_jobs", type_="check")
op.drop_column("download_jobs", "ready_at")
@@ -0,0 +1,37 @@
"""A file remembers where it came from
Revision ID: 0105_job_source_key
Revises: 0104_job_ready_at
Create Date: 2026-10-12
`download_jobs.source_key` is the origin a file came from — the source folder it may go back to —
recorded when it is placed. An edit used to work its origin out through its parent each time, and
once the parent was deleted or expired it fell back to "upload": moved into an own folder, it could
no longer go back to its folder, and deleting that folder sent it to Uploads to stay forever.
A file in a source folder takes that folder's key. One already in an own folder is left NULL and
keeps being worked out (`folders.source_key_for`), as before.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0105_job_source_key"
down_revision: str | None = "0104_job_ready_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("source_key", sa.String(260)))
op.execute(
"""
UPDATE download_jobs j SET source_key = f.source_key
FROM file_folders f
WHERE f.id = j.folder_id AND f.kind = 'source'
"""
)
def downgrade() -> None:
op.drop_column("download_jobs", "source_key")
+110
View File
@@ -0,0 +1,110 @@
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
Revision ID: 0106_api_tokens
Revises: 0105_job_source_key
Create Date: 2026-10-14
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
read, write, delete — so a token reaches nothing it was not given, folder by folder.
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
One per user.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0106_api_tokens"
down_revision: str | None = "0105_job_source_key"
branch_labels: str | None = None
depends_on: str | None = None
_KIND_OLD = (
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
)
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
def upgrade() -> None:
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
op.create_index(
"uq_file_folder_hub",
"file_folders",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'hub'"),
)
op.create_table(
"api_tokens",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("name", sa.String(80), nullable=False),
sa.Column("prefix", sa.String(16), nullable=False),
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
sa.Column("expires_at", sa.DateTime(timezone=True)),
sa.Column("last_used_at", sa.DateTime(timezone=True)),
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_table(
"api_token_grants",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
sa.CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
def downgrade() -> None:
op.drop_table("api_token_grants")
op.drop_table("api_tokens")
op.drop_index("uq_file_folder_hub", table_name="file_folders")
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
conn = op.get_bind()
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
for folder_id, user_id in hubs:
name, n = f"Hub {folder_id}", 1
while conn.execute(
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
{"u": user_id, "n": name},
).first():
n += 1
name = f"Hub {folder_id} ({n})"
conn.execute(
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
{"n": name, "i": folder_id},
)
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
@@ -0,0 +1,66 @@
"""Share-link tokens no longer stored in the clear
Revision ID: 0107_link_token_at_rest
Revises: 0106_api_tokens
Create Date: 2026-10-15
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
address just cannot be shown again (`app/downloads/links.py`).
Every existing link is converted here, so the addresses already sent keep working. The downgrade
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
changed) gets a fresh random token, which is the same as revoking it.
"""
import hashlib
import secrets
import sqlalchemy as sa
from alembic import op
from app import security
revision: str = "0107_link_token_at_rest"
down_revision: str | None = "0106_api_tokens"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
conn = op.get_bind()
sealing = security.can_encrypt()
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
conn.execute(
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
{
"id": link_id,
"h": hashlib.sha256(token.encode()).hexdigest(),
"e": security.encrypt(token) if sealing else None,
},
)
op.alter_column("download_links", "token_hash", nullable=False)
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
op.drop_column("download_links", "token")
def downgrade() -> None:
op.add_column("download_links", sa.Column("token", sa.String(64)))
conn = op.get_bind()
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
for link_id, enc in rows:
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
conn.execute(
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
{"id": link_id, "t": token},
)
op.alter_column("download_links", "token", nullable=False)
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
op.drop_column("download_links", "token_enc")
op.drop_column("download_links", "token_hash")
@@ -0,0 +1,40 @@
"""A share link may open a folder
Revision ID: 0108_folder_links
Revises: 0107_link_token_at_rest
Create Date: 2026-10-15
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
the two set; a folder's links go with the folder.
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0108_folder_links"
down_revision: str | None = "0107_link_token_at_rest"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"download_links",
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
)
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
op.alter_column("download_links", "job_id", nullable=True)
op.create_check_constraint(
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
)
def downgrade() -> None:
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
op.alter_column("download_links", "job_id", nullable=False)
op.drop_index("ix_download_links_folder_id", table_name="download_links")
op.drop_column("download_links", "folder_id")
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
@@ -0,0 +1,70 @@
"""One Watch later list per user, held by the database
Revision ID: 0112_one_watch_later
Revises: 0111_audit_request_id
Create Date: 2026-10-20
Watch later was made on a user's first save by a read then an insert, so two first saves at once
each made a list, and the reader then picked between them at random. A partial unique index holds
"one per user" now. An instance that already has a second list keeps the oldest: the others' videos
move into it (after its own, in their order; one it already holds is not added twice), a Home pin
of one of them moves to it (unless it is pinned itself; of several, the first on the shelf), then
they go — a pin left on them goes with them, as the kept list already holds one.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0112_one_watch_later"
down_revision: str | None = "0111_audit_request_id"
branch_labels: str | None = None
depends_on: str | None = None
_EXTRA = """
SELECT p.id, k.keep_id
FROM playlists p
JOIN (
SELECT user_id, min(id) AS keep_id FROM playlists WHERE kind = 'watch_later' GROUP BY user_id
) k ON k.user_id = p.user_id
WHERE p.kind = 'watch_later' AND p.id <> k.keep_id
"""
def upgrade() -> None:
op.execute(
f"""
INSERT INTO playlist_items (playlist_id, video_id, position, added_at)
SELECT e.keep_id, i.video_id,
(SELECT coalesce(max(position), -1) FROM playlist_items WHERE playlist_id = e.keep_id)
+ row_number() OVER (PARTITION BY e.keep_id ORDER BY i.playlist_id, i.position, i.id),
i.added_at
FROM playlist_items i
JOIN ({_EXTRA}) e ON e.id = i.playlist_id
ON CONFLICT ON CONSTRAINT uq_playlist_video DO NOTHING
"""
)
op.execute(
f"""
UPDATE comfort_pins c SET playlist_id = m.keep_id
FROM (
SELECT DISTINCT ON (e.keep_id) p.id AS pin_id, e.keep_id
FROM comfort_pins p
JOIN ({_EXTRA}) e ON e.id = p.playlist_id
WHERE NOT EXISTS (SELECT 1 FROM comfort_pins k WHERE k.playlist_id = e.keep_id)
ORDER BY e.keep_id, p.position, p.id
) m
WHERE c.id = m.pin_id
"""
)
op.execute(f"DELETE FROM playlists WHERE id IN (SELECT id FROM ({_EXTRA}) e)")
op.create_index(
"uq_playlist_watch_later",
"playlists",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'watch_later'"),
)
def downgrade() -> None:
op.drop_index("uq_playlist_watch_later", table_name="playlists")
+73 -25
View File
@@ -1,9 +1,8 @@
# Backend conventions
The rules below are already followed across `backend/app` — they lived only in a few module
docstrings (`demo.py`, `userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
docstrings (`userscope.py`, `routes/_common.py`) until this file collected them. When you add code,
match these so the next reader doesn't have to guess "which pattern do I copy".
## Module layering
@@ -13,35 +12,81 @@ Three layers, by *who* runs the code — not by feature:
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
2. **`plex/`, `downloads/`, `sync/` — domain logic shared by more than one caller.** Anything the
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
route, the worker, and the scheduler share one definition. Examples from this codebase:
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
2. **The units and the other domain packages — logic shared by more than one caller.** A unit's
database work (below), and anything the **scheduler or the worker** runs (not just a request), lives
in a domain package, so the route, the worker, and the scheduler share one definition. Examples from
this codebase: `downloads/service.py` owns enqueue + the job state transitions (routes call it);
`plex/sync.py` is run by both the admin route and the scheduler; `downloads/pipeline.py` is the
worker's child pipeline; `issues/bundle.py` is the one definition of a bug report's files, for the
zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
layers import. They depend on nothing above them (a core module must never import a `routes/…`
module).
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
module must never import a `routes/…` module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't
pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
## Units
The domain is four units, each owning its tables (`models.py` in its package):
| Unit | Package | Its routes |
| --- | --- | --- |
| Capture | `downloads/`, the worker | `downloads.py`, `files.py`, `public.py` |
| Reader | `reader/`, `discover/`, `feedquery.py`, the playlist sync and tag suggestions in `sync/` | `feed.py`, `playlists.py`, `saved_views.py`, `discover.py`, `tags.py` |
| Catalog | `catalog/`, the ingests in `sync/` | `channels.py`, `search.py`, `sync.py` |
| Plex | `plex/` | `plex/` |
- **A unit's route does no database work.** It validates, reads through the unit's `queries.py`,
writes through its `service.py`, serializes, and commits. It may name a unit's model as a type
(under `TYPE_CHECKING`), never use one as a value, and it passes a kind (`"movie"`), not a model, to
a query.
- **A unit's code keeps to its edges.** It reads another unit's tables only along a declared edge
(Capture and Reader read Catalog's and Plex's: the feed joins `Video` with `VideoState` in one
query) and writes them only through the owner's service. Core code reaches a unit's tables through
that unit's `queries`.
- **Whose a row is, `userscope` says** — in every module, core included: `owned_by(Model.user_id,
user)` for a filter, `owned_row(db, Model, id, user)` for a row by id (someone else's row is the
same answer as a missing one). A unit's `queries.own_x` returns None and its route answers
`not_found`; a core route calls `owned_or_404`. A rule `owned_row` cannot say — a job its owner
shared, a built-in profile nobody owns, a folder read under `FOR SHARE` — is written out in the
unit's own code, beside the reason.
All of it is held by `tests/test_unit_fences.py`, with no exemptions: the unit map, each unit's
packages and edges (`UNIT_PACKAGES`, `READS`) and what may commit (`UNIT_COMMITS`, `CORE_COMMITS`)
are listed there. It reads the code's AST, so it holds the forms written here: an owner column
compared inline (a model's, an import alias's or an `aliased()` one's, by `==`, `!=`, `.in_()`, or a
`filter_by(user_id=…)`), a model built or an `insert`/`update`/`delete` (under any import name, of a
model or its `__table__`) across units, and a `.commit()` call off the lists all fail it. What an AST
cannot tell is review's to catch: an ORM write on another unit's loaded row (an attribute, a
`db.delete`), SQL in `text()`, a column or model reached another way (bound to a variable, through
`__table__.c` or a module path), and a commit made by a `begin()` block or a `commit` handed in.
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
This follows CLAUDE.md's canonical rule — **the handler commits; a unit's `service` and every helper
flush** — so read it that way, not as an exception to it. The commit belongs to the **operation
owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- The **operation owner** is the route handler, or a job. A unit's `service` flushes even when it
makes a whole transition (`service.cancel`): the handler commits it, together with whatever else the
request changed. What owns its OWN session commits it: `quota` and `jobruns`' run rows, the
scheduler's jobs, a unit's jobs (the worker's `jobdb` writes, the syncs and ingests that commit per
page or batch so a later failure loses nothing already stored), and a few core steps that must land
whatever their caller does next (a token's last-used stamp, a refreshed YouTube grant) — the fence's
`UNIT_COMMITS` and `CORE_COMMITS` name each with its reason, `test_route_conventions` a route's own
committing helpers. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -63,13 +108,16 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
Everything below must be green before a change is done (run via `siftlode check`):
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
`ruff` + `basedpyright` (backend) · the lock check + `tsc` ×3 + `eslint` + `prettier` + `knip` +
`vite build` (frontend) · `vitest` · `pytest`. The pytest lane runs in `backend/docker-compose.test.yml`
— an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the `db`
fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks
for pure-logic only with `SIFTLODE_TEST_REQUIRE_DB=0`. See `docs/development.md`, and `docs/gates.md`
for what each gate deliberately leaves out.
+276
View File
@@ -0,0 +1,276 @@
"""Machine access to the Files library: API tokens, and what each one may do where.
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
manage the account, the tokens, or anything outside the library. And only in the folders its grants
name, with the rights they give: list, read, write, delete. Every route asks the one question,
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
token stops working at once. A token about to run out warns its user once, a week before
(`warn_expiring`), so a machine does not lose its key unannounced.
"""
import secrets
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.models import ApiToken, ApiTokenGrant, User
from app.notifications import create_notification
from app.ratelimit import RateLimiter, client_key
from app.refusal import refused
from app.security import hash_token
from app.utils import iso
log = logs.get_logger(__name__)
PREFIX = "sfl_"
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_BODY_BYTES = 32
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
# narrow a guess (8 of its 43 random characters).
SHOWN_LEN = len(PREFIX) + 8
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
# held back per client, counting only the refusals.
_calls = RateLimiter(600, 60)
_bad = RateLimiter(20, 300)
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
# minute, not one per request.
_TOUCH_EVERY = timedelta(minutes=1)
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
# "expires soon" for the same span.
EXPIRY_WARNING = timedelta(days=7)
class Right(StrEnum):
LIST = "list"
READ = "read"
WRITE = "write"
DELETE = "delete"
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
GRANT_COLUMNS = {
Right.LIST: "can_list",
Right.READ: "can_read",
Right.WRITE: "can_write",
Right.DELETE: "can_delete",
}
def _b62(value: int, width: int) -> str:
out = []
for _ in range(width):
value, digit = divmod(value, 62)
out.append(_ALPHABET[digit])
return "".join(reversed(out))
def _check(body: str) -> str:
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
def mint() -> str:
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
return PREFIX + body + _check(body)
def well_formed(raw: str) -> bool:
"""Whether `raw` has a token's shape and its checksum adds up."""
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
return False
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
if any(c not in _ALPHABET for c in body + check):
return False
return secrets.compare_digest(_check(body), check)
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
@dataclass(frozen=True)
class FilesCaller:
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
user (everything in their own folders); for a token, folder id → its rights there."""
user: User
token_id: int | None = None
token_prefix: str | None = None
grants: Mapping[int, frozenset[Right]] | None = None
def may(self, folder_id: int | None, right: Right) -> bool:
if self.grants is None:
return True
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
def folders_with(self, right: Right) -> set[int] | None:
"""The folders `right` is granted in; None = any of the user's own."""
if self.grants is None:
return None
return {f for f, rights in self.grants.items() if right in rights}
@property
def via(self) -> str | None:
"""How an audit row says a token did it, None when the user did."""
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
def bearer_token(request: Request) -> str | None:
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
scheme counts as none — it is not ours to judge)."""
scheme, _, value = request.headers.get("authorization", "").partition(" ")
if scheme.lower() != "bearer":
return None
return value.strip()
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
"""The signed-in user, or None when the request brings a token instead: a token wins over a
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
if bearer_token(request) is not None:
return None
return current_user(request, db)
def files_caller(
request: Request,
db: Session = Depends(get_db),
user: User | None = Depends(session_caller),
) -> FilesCaller:
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
if user is not None:
return FilesCaller(user=user)
return _token_caller(request, db, bearer_token(request) or "")
def _refuse_token(key: str, reason: str) -> Exception:
_bad.allow(key)
log.info(
"a request brought an API token that does not hold",
extra=logs.event("apitoken.refused", reason=reason),
)
return refused(401, "not_authenticated")
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
key = client_key(request)
if _bad.exhausted(key):
raise refused(429, "api_token_rate_limited")
if not well_formed(raw):
raise _refuse_token(key, "malformed")
token = db.execute(
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
).scalar_one_or_none()
if token is None:
raise _refuse_token(key, "unknown")
now = datetime.now(timezone.utc)
if token.expires_at is not None and token.expires_at <= now:
raise _refuse_token(key, "expired")
user = db.get(User, token.user_id)
if user is None or not user.is_active or user.is_suspended:
raise _refuse_token(key, "account")
if not _calls.allow(str(token.id)):
raise refused(429, "api_token_rate_limited")
grants = {
g.folder_id: grant_rights(g)
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
.scalars()
.all()
}
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
db.execute(
update(ApiToken)
.where(
ApiToken.id == token.id,
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
)
.values(last_used_at=now)
.execution_options(synchronize_session=False)
)
# Committed here, ahead of the route: a read-only route never commits, and a write route must
# not carry this along into a rollback of its own.
db.commit()
logs.set_user(user.id, *debugsession.level_for(user, None))
request.state.user_id = user.id
return caller
def revoke_all(db: Session, user_id: int) -> int:
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
credential like a session, so whatever ends every session of an account — a password reset or
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(
delete(ApiToken).where(owned_by(ApiToken.user_id, user_id)).returning(ApiToken.id)
)
.scalars()
.all()
)
for token_id in gone:
audit.record(
db,
user_id,
AuditAction.API_TOKEN_REVOKE,
target_type="api_token",
target_id=token_id,
summary="revoked with every session of the account",
)
return len(gone)
def expiring(token: ApiToken, now: datetime) -> bool:
"""Whether `token` runs out within the warning span (and has not yet)."""
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
def warn_expiring(db: Session) -> int:
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
now = datetime.now(timezone.utc)
hits = db.execute(
update(ApiToken)
.where(
ApiToken.expiry_warned.is_(False),
ApiToken.expires_at.is_not(None),
ApiToken.expires_at > now,
ApiToken.expires_at <= now + EXPIRY_WARNING,
)
.values(expiry_warned=True)
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
.execution_options(synchronize_session=False)
).all()
for token_id, user_id, name, expires_at in hits:
create_notification(
db,
user_id,
"api_token_expiring",
title=f"API token {name} expires soon", # English fallback; the client names it from data
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
commit=False,
)
db.commit()
return len(hits)
+41 -12
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -27,12 +28,18 @@ class AuditAction:
USER_SUSPEND = "user_suspend"
USER_UNSUSPEND = "user_unsuspend"
USER_DELETE = "user_delete"
# An admin let a stuck registrant in, or re-sent their confirmation link. Both are
# recorded because both are the admin acting ON someone's access, not merely reading a list.
USER_ACTIVATE = "user_activate"
USER_VERIFY_RESEND = "user_verify_resend"
USER_SETPW_LINK = "user_setpw_link"
INVITE_APPROVE = "invite_approve"
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
DEMO_ADD = "demo_add"
DEMO_REMOVE = "demo_remove"
DEMO_RESET = "demo_reset"
INVITE_DELETE = "invite_delete"
# The admin chose to tell someone their address is whitelisted. Adding the address is
# silent, so this is the act that actually reaches a person — and the one worth a record.
INVITE_SEND = "invite_send"
DISCOVERY_PURGE = "discovery_purge"
# Config
CONFIG_SET = "config_set"
@@ -42,13 +49,35 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
# A user sent a debug session's issue to the tracker — the one act that takes their log
# off this instance, so it is recorded (target = the session, summary = the issue's URL).
ISSUE_CREATE = "issue_create"
# The Files library (`routes.files`): what a user put in, took out, kept or reorganised. Ids
# only — a file's or folder's name is the user's own text, not the admin's to read here.
FILE_UPLOAD = "file_upload"
FILE_DELETE = "file_delete"
FILE_KEEP = "file_keep"
FILE_UNKEEP = "file_unkeep"
FILE_MOVE = "file_move"
FILE_FOLDER_CREATE = "file_folder_create"
FILE_FOLDER_RENAME = "file_folder_rename"
FILE_FOLDER_RETENTION = "file_folder_retention"
FILE_FOLDER_DELETE = "file_folder_delete"
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
# What a token then did is the file rows above, their summary naming it.
API_TOKEN_CREATE = "api_token_create"
API_TOKEN_REVOKE = "api_token_revoke"
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
# revoked.
# Never its token or password — only whether it has one.
DOWNLOAD_LINK_CREATE = "download_link_create"
DOWNLOAD_LINK_UPDATE = "download_link_update"
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
def record(
@@ -74,16 +103,16 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
def gc(db: Session, retention_days: int) -> int:
"""Delete audit rows older than `retention_days`. Returns the number removed. A
retention_days <= 0 disables pruning (keep forever)."""
retention_days <= 0 disables pruning (keep forever). Flushes; the job commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
result = db.execute(delete(AuditLog).where(AuditLog.created_at < cutoff))
db.commit()
return result.rowcount or 0
+230 -70
View File
@@ -1,17 +1,21 @@
import logging
import httpx
from fastapi import BackgroundTasks, Depends, HTTPException, Request
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app.userscope import owned_by
from app import debugsession
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.config import settings
from app.db import get_db
from app.models import Invite, User
from app.models import Invite, LogEvent, User
from app.security import decrypt_optional
from app.utils import canonical_email
from app.refusal import refused
PASSWORD_MIN_LEN = 10
@@ -21,8 +25,6 @@ def operator_contact() -> str | None:
return next(iter(sorted(settings.admin_email_set)), None)
DEMO_GOOGLE_SUB = "__demo__"
DEMO_EMAIL = "demo@siftlode.local"
MAX_SESSION_ACCOUNTS = 6
# OAuth scope vocabulary — the atoms + the request bundles, kept together as core. has_*_scope check
@@ -35,7 +37,7 @@ READ_SCOPES = f"{BASE_SCOPES} {READ_SCOPE}"
WRITE_SCOPES = f"{BASE_SCOPES} {READ_SCOPE} {WRITE_SCOPE}"
_YOUTUBE_SCOPES = frozenset({READ_SCOPE, WRITE_SCOPE})
log = logging.getLogger("siftlode.auth")
log = logs.get_logger(__name__)
def google_enabled(db: Session) -> bool:
@@ -63,34 +65,125 @@ def has_write_scope(user: User) -> bool:
return WRITE_SCOPE in tok.scopes.split()
def env_allowed_canonicals() -> set[str]:
"""The env bootstrap allow-list, in CANONICAL form — the only shape it may be compared in.
Canonicalising only the Invite branch of `is_allowed` left this one matching the literal string,
which is the 2026-09-03 lockout with a different source of truth: an operator who writes
`first.last@gmail.com` into ALLOWED_EMAILS is refused when Google answers with the undotted
primary, and a `+tag`ged entry can never match at all, because a Google account cannot contain a
`+`. The addresses themselves stay literal wherever we MAIL them (`admin_notify_groups`); only
the comparison is canonical.
"""
return env_admin_canonicals() | {canonical_email(e) for e in settings.allowed_email_set}
def env_admin_canonicals() -> set[str]:
"""The env ADMIN_EMAILS bootstrap list, canonical — for the role check only. Kept separate from
`env_allowed_canonicals` because being listed as an admin also grants access, but being allowed
does not make anyone an admin."""
return {canonical_email(e) for e in settings.admin_email_set}
def is_allowed(db: Session, email: str) -> bool:
"""May this email sign in? An approved Invite is the source of truth; the env
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB)."""
ALLOWED_EMAILS/ADMIN_EMAILS remain a bootstrap fallback (e.g. a fresh DB).
Matched on the CANONICAL form (utils.canonical_email), not the literal string. Gmail delivers
every dotted / `+tagged` spelling to one mailbox, so approving one spelling always meant approving
that mailbox — matching literally only produced the 2026-09-03 lockout, where an approved user's
Google sign-in became a silent second access request because OAuth returned the undotted primary.
One canonical form can match SEVERAL invite rows (prod carries approved pairs for one mailbox,
the oldest from 2026-06-16). An approved row among them grants access: the API cannot flip an
approved invite to denied (`_decide` only accepts a pending one), so approved-beside-denied means
the admin approved that mailbox under a different spelling, not that they revoked it. Revoking is
deleting the invite row or suspending the account — both still work, since a deleted row is no
longer here to match and suspension is enforced at sign-in.
"""
if not email:
return False
email = email.lower()
inv = db.execute(select(Invite).where(Invite.email == email)).scalar_one_or_none()
if inv is not None:
return inv.status == "approved"
return email in settings.allowed_email_set or email in settings.admin_email_set
canon = canonical_email(email)
statuses = (
db.execute(select(Invite.status).where(Invite.email_canonical == canon)).scalars().all()
)
if statuses:
return "approved" in statuses
return canon in env_allowed_canonicals()
def get_or_create_demo_user(db: Session) -> User:
"""The one shared demo user, created on first use. No OAuth token / YouTube scope, so
has_read_scope/has_write_scope are False and every YouTube-touching path is closed to it."""
user = db.query(User).filter(User.is_demo.is_(True)).one_or_none()
if user is None:
user = User(
google_sub=DEMO_GOOGLE_SUB,
email=DEMO_EMAIL,
display_name="Demo",
role="user",
is_demo=True,
preferences={"language": "en"},
def can_activate(db: Session, user: User) -> bool:
"""Is this account merely waiting for an admin to finish letting it in?
True when it is inactive but its address is ALREADY allowed — the shape roadmap UX-25 describes:
the admin approved the address, the person signed up afterwards, and nothing connects the two
(the Access-requests queue only offers Approve on a still-pending invite).
Gated on `is_allowed`, the same function the sign-in paths consult, because activating an address
that is NOT allowed manufactures an inconsistent account: password sign-in only checks
`is_active` and would let them in, while their Google sign-in would be turned away and silently
reopened as a fresh access request. Letting in someone with no approved invite is the whitelist
box's job — that action says so out loud.
Lives in core so the admin surface and the /api/me badge count answer from ONE definition; a
button that appears and a number that counts must never disagree about who is stuck.
"""
if user.is_suspended or user.is_active:
return False
return is_allowed(db, user.email)
def count_pending_activations(db: Session) -> int:
"""How many sign-ups are waiting on an admin click (see can_activate). Drives the admin badge.
Deliberately NOT "everyone who cannot sign in": an account that has simply not confirmed its
address yet is waiting on its owner, not on the admin, and badging that would nag about something
the admin did not cause. It is still listed on the admin screen — with a re-send — because the
mail may never have arrived.
"""
# Narrowed in SQL, then decided by can_activate itself — so the number the badge shows and the
# button the admin sees cannot drift apart by one of them re-deriving the rule.
rows = (
db.execute(
select(User).where(
User.is_suspended.is_(False),
User.is_active.is_(False),
)
)
db.add(user)
db.commit()
return user
.scalars()
.all()
)
return sum(1 for u in rows if can_activate(db, u))
def find_user_by_email(db: Session, email: str) -> User | None:
"""The account someone means when they type an address at sign-in, password-reset or
verify-resend. None when there isn't exactly one answer.
Exact spelling wins. Failing that, the address is resolved to its mailbox
(utils.canonical_email) — so a password chosen at registration still works when the person types
another spelling of their own Gmail, and so the three flows that look an account up by typed
address stop being dead ends. Before this, requesting a reset for `xy@gmail.com` on an account
registered as `x.y@gmail.com` issued nothing and said nothing: the endpoint answers uniformly by
design, so the user got silence and no way in.
SEVERAL canonical candidates return None rather than a guess. That case is real — prod carries two
accounts for one mailbox — and picking one of them for a password sign-in would hand someone the
wrong account. The callers all treat None as "no such account", which is the uniform,
non-enumerating answer they already gave.
"""
if not email:
return None
email = email.strip().lower()
exact = db.execute(select(User).where(User.email == email)).scalar_one_or_none()
if exact is not None:
return exact
candidates = (
db.execute(select(User).where(User.email_canonical == canonical_email(email)))
.scalars()
.all()
)
return candidates[0] if len(candidates) == 1 else None
def revoke_google_token(token: str | None) -> None:
@@ -108,7 +201,7 @@ def revoke_google_token(token: str | None) -> None:
timeout=10,
)
log.info("Google token revoke: status=%s", resp.status_code)
except Exception:
except Exception: # noqa: BLE001 — a revoke we cannot make must not fail the sign-out
log.warning("Google token revoke failed", exc_info=True)
@@ -119,6 +212,9 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
touch any browser session — each caller handles its own."""
email = user.email.lower()
user_id = user.id
# Read the language off the row while it still exists — the confirmation below is queued as a
# background task that runs after this session (and this user) are gone.
lang = mailtext.lang_for_user(user)
tok = user.token
# Both are stored encrypted — decrypt for the revoke call (the access-token fallback was passing
# ciphertext, so a token row with no refresh token silently failed to revoke). decrypt_optional so a
@@ -130,64 +226,114 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
)
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
db.execute(delete(User).where(User.id == user_id))
db.execute(delete(Invite).where(Invite.email == email))
# Stored log events name the account by id with no foreign key to cascade from (models.LogEvent),
# so they go explicitly — they are request history, i.e. personal data. Forgotten FIRST: this very
# request's access line, a queued row, a later background task would otherwise put rows back
# under the id after this DELETE (logs.forget_user).
logs.forget_user(user_id)
db.execute(delete(LogEvent).where(owned_by(LogEvent.user_id, user_id)))
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
# account was just erased — and prod really does carry approved pairs for one mailbox.
#
# Unless ANOTHER account still lives on that mailbox — prod carries that too (two user rows for
# one Gmail mailbox). Then NO invite goes, not even the literal one: is_allowed matches
# canonically, so every row here is what keeps the survivor allowed, and removing any of them
# would refuse their next Google sign-in and reopen it as a fresh access request — this epic's
# founding bug, caused by its own cleanup. `delete_invite` refuses for exactly this reason; the
# user row is already gone by now, so this query sees only the others.
canon = canonical_email(email)
if db.execute(select(User.id).where(User.email_canonical == canon)).first() is None:
db.execute(delete(Invite).where(Invite.email_canonical == canon))
db.commit()
if google_token:
background.add_task(revoke_google_token, google_token)
# Confirm the erasure to the (now former) user — GDPR good practice, and covers both the
# self-service and admin deletion paths since both funnel through here.
background.add_task(email_mod.send_account_deleted, email, operator_contact())
background.add_task(email_mod.send_account_deleted, email, lang, operator_contact())
ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
def resolved_user_id(conn: HTTPConnection) -> tuple[int | None, bool]:
def resolved_user_id(conn: HTTPConnection, *, strict: bool = True) -> tuple[int | None, bool]:
"""Which account this connection acts as, and whether that's the session's default account.
Takes any HTTPConnection — an HTTP Request OR a WebSocket — so the WS push channel shares this
exact per-tab resolution instead of re-implementing it.
Takes an HTTPConnection rather than a Request: it reads only the session and the headers.
The signed cookie holds the *wallet* (`account_ids`, every account signed into this browser)
plus a default `user_id`. A caller may override the default with the X-Siftlode-Account header,
but ONLY for an account already in the wallet — so a tab can't impersonate an account that never
authenticated here. Everything else (WebSocket, plain navigations) keeps using the cookie default.
plus a default `user_id`. A caller may name its account with the X-Siftlode-Account header (or
`?account=`), but ONLY an account in the wallet — so a tab can't impersonate an account that
never authenticated here. A connection that names no account acts as the cookie default.
A connection naming an account NOT in the wallet acts as nobody (`None`) — typically a tab still
pinned to an account that was signed out in another tab. It must not fall back to the default:
that tab would go on acting as an account it never showed, its player checkpoints and pref saves
landing there and its Sign out signing THAT account out. Only the bootstrap /api/me probe passes
`strict=False` and gets the default, which is how such a tab learns who it now is (the frontend
re-pins it and reloads). An account whose session ended while it stayed in the wallet (deleted,
suspended, revoked) leaves the wallet when it is refused (`_session_user`), so a tab pinned to it
takes the same way back.
"""
default_id = conn.session.get("user_id")
wallet = conn.session.get("account_ids") or []
# Header for normal XHR; ?account= for contexts that can't set headers (WebSocket, and a
# plain <a> file download). Both are wallet-gated below, so neither can impersonate an
# account that never signed into this browser.
# Header for normal XHR; ?account= for a context that can't set headers (a plain <a> file
# download). Both are wallet-gated below.
hdr = conn.headers.get(ACTIVE_ACCOUNT_HEADER) or conn.query_params.get("account")
if hdr:
try:
hid = int(hdr)
except ValueError:
# Under `strict` this becomes a 401 and the tab authenticates as nobody, which reads
# in the browser exactly like a session that expired. The value itself stays out —
# it came from a request, and `wallet` says which accounts were actually on offer.
log.debug(
"the active-account header could not be read as an id",
extra=logs.event("auth.account_header.invalid", accounts=len(wallet)),
)
hid = None
if hid is not None and hid in wallet:
return hid, hid == default_id
if strict:
return None, False
return default_id, True
def drop_from_wallet(session: dict, user_id: int) -> None:
"""Take one account out of this browser's wallet (`account_ids`); the session default is untouched."""
session["account_ids"] = [a for a in (session.get("account_ids") or []) if a != user_id]
def _end_account_session(request: Request, user_id: int, is_default: bool) -> None:
"""An account whose session here is no longer valid. Only nuke the whole session when it is the
SESSION DEFAULT; a per-tab account just leaves the wallet (other tabs' default may still be valid).
Leaving is what lets a tab pinned to it recover: its next /api/me names an account no longer signed
in here and gets the default. Left in, it answered "signed out" — the landing page, with the default
still signed in, and no way off it but closing the tab."""
if is_default:
request.session.clear()
else:
drop_from_wallet(request.session, user_id)
def current_user(request: Request, db: Session = Depends(get_db)) -> User:
user_id, is_default = resolved_user_id(request)
return _session_user(request, db, *resolved_user_id(request))
def _session_user(request: Request, db: Session, user_id: int | None, is_default: bool) -> User:
if not user_id:
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
user = db.get(User, user_id)
if user is None or not user.is_active or user.is_suspended:
# Suspended/deactivated mid-session → drop the session so the block takes effect at once.
# But only nuke the whole session when the SESSION DEFAULT identity is gone; a stale
# per-tab header account just 401s that one tab (other tabs' default may still be valid).
if is_default:
request.session.clear()
raise HTTPException(status_code=401, detail="Not authenticated")
# Suspended/deactivated mid-session → end it so the block takes effect at once.
_end_account_session(request, user_id, is_default)
raise refused(401, "not_authenticated")
# SA4: reject a cookie whose recorded epoch is behind the account's current one (revoked by a
# password reset/change or "log out everywhere"). A cookie with no recorded epoch (pre-SA4, or an
# account signed in before this shipped) is treated as epoch 0 — so the FIRST bump revokes it too.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(user_id), 0)) != (user.session_epoch or 0):
if is_default:
request.session.clear()
raise HTTPException(status_code=401, detail="Not authenticated")
_end_account_session(request, user_id, is_default)
raise refused(401, "not_authenticated")
# Always keep the session default in the switchable list — covers sessions created before
# multi-session existed (their account_ids was never seeded), so the switcher isn't empty.
default_id = request.session.get("user_id")
@@ -195,39 +341,53 @@ def current_user(request: Request, db: Session = Depends(get_db)) -> User:
if default_id and default_id not in accounts:
accounts.append(default_id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
# Attribute this request's log lines to the account. Here, not in the request middleware:
# this is the one place a session becomes an account, and the middleware runs outside both the
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
# diagnostics level rides along: their requests are logged — and stored — in that detail.
# So does a debug session the request names: only the caller's own open one counts, and a
# header naming anything else is ignored rather than refused — a stale tab must keep working.
session = debugsession.from_header(db, user, request.headers.get(debugsession.HEADER))
level, until = debugsession.level_for(user, session)
logs.set_user(user.id, level, until, session.id if session else None)
request.state.user_id = user.id
return user
def optional_current_user(request: Request, db: Session = Depends(get_db)) -> User | None:
"""Like `current_user` but returns None instead of raising 401 when there's no valid session.
Used by the bootstrap /api/me probe so the logged-out landing doesn't log a failed request to
the browser console (a non-2xx fetch is a console error no matter how the app handles it)."""
the browser console (a non-2xx fetch is a console error no matter how the app handles it).
The one lenient resolution (`strict=False`): a tab naming an account no longer signed in here gets
the default account, which is how it learns who it now is — see `resolved_user_id`. That includes an
account refused right here, which `_session_user` has just taken out of the wallet."""
user_id, is_default = resolved_user_id(request, strict=False)
try:
return current_user(request, db)
return _session_user(request, db, user_id, is_default)
# silent-ok: this is the anonymous case as much as the refused one — every landing page hit
# comes through here. The second attempt below is the handling, and it reports.
except HTTPException:
if is_default:
return None
try:
return _session_user(request, db, *resolved_user_id(request, strict=False))
except HTTPException:
# Both the pinned account AND the default one were refused, so the tab is signed out
# without being told why — and it got here from a session that named a real account.
log.info(
"a session was resolved to nobody: neither the named nor the default account held",
extra=logs.event("auth.session.probe_rejected", user_id=user_id),
)
return None
def require_human(user: User = Depends(current_user)) -> User:
"""Reject the shared demo account from actions that need a real Google/YouTube identity
or spend the shared quota. Most YouTube paths are already closed to it implicitly (it has
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
where there's no scope check to lean on."""
if user.is_demo:
raise HTTPException(status_code=403, detail="Not available in the demo account.")
return user
def require_write_scope(user: User = Depends(require_human)) -> User:
def require_write_scope(user: User = Depends(current_user)) -> User:
"""Dependency: require the YouTube write scope (subscribe/unsubscribe, playlist export/edit).
Single source of truth for the write gate — replaces the inline `if not has_write_scope(user)`
checks that had drifted into three different 403 messages across channels/playlists. Layers on
`require_human` because writing to YouTube needs a real, non-demo identity."""
checks that had drifted into three different 403 messages across channels/playlists."""
if not has_write_scope(user):
raise HTTPException(
status_code=403,
detail="Enable YouTube editing in Settings first.",
)
raise refused(403, "youtube_editing_off")
return user
@@ -235,7 +395,7 @@ def admin_user(user: User = Depends(current_user)) -> User:
"""Dependency: require the signed-in user to be an admin. Single source of truth for the
admin gate — every admin route/action depends on this rather than re-checking the role."""
if user.role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
raise refused(403, "admin_only")
return user
+231
View File
@@ -0,0 +1,231 @@
"""The global request-body cap (SEC-20).
**The hole.** FastAPI reads a declared body IN FULL before auth or any other dependency runs — the
body parameter has to be parsed to be passed in. So `POST /auth/password-login` drains a 32 MiB
upload from an anonymous caller and only then answers 401. 70 of the 72 body-declaring routes behave
the same way: one unauthenticated client can spend the process's memory and time at will.
**Why a middleware and not `capped_body` on every route.** `routes/_common.py:capped_body` is the
right tool where a route genuinely takes bulk bytes, and the two routes that use it use it
correctly. But putting it on all 70 others would (a) fight the canonical "Request body → a Pydantic
model" row, since capping means taking the body as raw bytes and parsing it by hand, and (b) leave
the NEXT route added uncovered, because nothing would remind its author. A default that has to be
opted into is not a default. This module inverts it: the cap applies to everything, and the few
routes that need more opt OUT, in one list, next to the code that reads it.
**Why raw ASGI and not `@app.middleware("http")`.** Starlette's BaseHTTPMiddleware cannot wrap
`receive`, so it can only read `Content-Length` — which a client may omit entirely (chunked) or
simply understate. The bytes have to be counted as they arrive, which is the same rule `capped_body`
already applies per-route, and counting needs the receive channel.
**Why the refusal is byte-identical to `capped_body`'s.** Two shapes for one fact is how the
`cancelled`/`canceled` bug happened. The 413 body here is built through the same `JSONResponse`
FastAPI renders `HTTPException(413, {"code": "too_large"})` with — not a hand-rolled `json.dumps`,
whose default separators differ by two bytes — and `tests/test_body_limit.py` compares the RAW BYTES
of the two against each other rather than against a literal, so they cannot drift apart. The same
reasoning decides where the middleware is registered (`main.py`): just inside CORS, so this 413
carries the same headers `capped_body`'s does.
**No admin knob, deliberately.** The ceiling is a constant, not a `sysconfig` value: an attacker-
facing limit that an admin can raise from the UI is a limit that can be raised by whoever gets the
admin session, and nobody has asked to tune it. If a self-hoster ever needs a different ceiling it
belongs in `config.py` (env, set before the process starts), not in the database.
"""
import re
from starlette.responses import JSONResponse
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from app import debugsession
from app.config import settings
# 1 MiB. The largest body any route on the default ceiling legitimately takes is 512 KiB (the client
# log sink, `routes/diagnostics.py:BODY_MAX_BYTES`), so this leaves 2x headroom — a ceiling that
# refuses a legitimate request would be a worse bug than the one this module fixes. That ratio is
# the load-bearing fact, so `test_body_limit.py` holds it rather than the figure above.
DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# The routes that need MORE than the default, as (method, path TEMPLATE) → their own ceiling. The
# template, not a concrete path, so the list reads like the route decorators it mirrors and
# `test_body_limit.py` can assert that each one still resolves to a real route: a renamed route
# fails the gate instead of silently losing its allowance in prod.
#
# These are RAISED CEILINGS, NOT EXEMPTIONS, and the difference is the whole point. An exemption
# would mean "this route is counted by something else" — which is only true when the route's own
# limit applies to the bytes as they ARRIVE. That, in turn, is only true of a route with NO DECLARED
# BODY PARAMETER: FastAPI parses and spools a declared body BEFORE the first dependency runs, so a
# route that declares one has already stored the bytes by the time its own cap — or its auth — gets
# a say.
#
# Both listed routes now satisfy that. The screenshot route always did (`capped_body` streams it,
# and the owner check resolves first). The upload route did NOT at first: it declared
# `file: UploadFile = File(...)`, so an anonymous caller could make the process write the whole
# per-file allowance to temp before the auth dependency refused them — this middleware bounded that
# number but could not stop the writing. It now reads its own body from a dependency declared after
# the auth one, so its in-handler cap measures bytes as they arrive, like everything else here.
#
# That rule is no longer only written here: `test_body_limit.py` parses the routes and fails the
# gate if any route in this dict declares a body parameter. A comment could not have stopped the
# defect it describes; the test can.
#
# A route belongs here ONLY IF ITS OWN CAP IS LARGER THAN THE DEFAULT. That is why
# `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
# default, so listing it could only widen what it accepts.
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file now — no multipart envelope to allow for —
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
#
# WHICH LAYER ANSWERS, measured rather than assumed:
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
# figure, after draining the rest of the body so the client can read it.
# * knob at the DEFAULT — the two numbers are EQUAL, and the edge always wins: a declared
# Content-Length over the cap is refused here before a byte is read, and an undeclared body
# trips this counter on the same chunk the route would have tripped on. The route's 422 is
# unreachable in that configuration, and no amount of headroom would change it — the
# declared-length check fires first for any body big enough to matter, and the route's drain
# pushes the count past the edge for the rest.
#
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
("POST", "/api/files/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
}
# The refusal, byte-for-byte what `capped_body`'s `HTTPException(413, {"code": "too_large"})` renders
# as — built through the same JSONResponse so the separators match too, and asserted against a real
# route's raw bytes in `test_body_limit.py` rather than against a literal here.
TOO_LARGE_BODY = JSONResponse({"detail": {"code": "too_large"}}, status_code=413).body
_PATH_PARAM = re.compile(r"\{[^/{}]+\}")
def template_to_regex(template: str) -> re.Pattern[str]:
"""`/api/debug-sessions/{sid}/screenshots` → a regex matching one concrete path.
A path parameter matches `[^/]+` — one segment, never across a `/`, so `{sid}` cannot swallow a
deeper path and hand an exemption to a route that was never granted one.
"""
literals = [re.escape(part) for part in _PATH_PARAM.split(template)]
return re.compile("^" + "[^/]+".join(literals) + "$")
def _declared_length(scope: Scope) -> int | None:
"""The `Content-Length` header as an int, or None when absent or unparseable (a chunked body has
none, and a malformed one is not a number we may trust — both fall through to counting)."""
for name, value in scope.get("headers", ()):
if name == b"content-length":
try:
return int(value)
except ValueError:
# silent-ok: a malformed Content-Length is the caller's doing, not a fault of ours,
# and it is about to be answered anyway — returning None drops the request into the
# counting path, which refuses on the real bytes. Logging it would hand an anonymous
# client a free line in our log for every request it cares to malform.
return None
return None
async def _send_too_large(send: Send) -> None:
await send(
{
"type": "http.response.start",
"status": 413,
"headers": [
(b"content-type", b"application/json"),
(b"content-length", str(len(TOO_LARGE_BODY)).encode()),
],
}
)
await send({"type": "http.response.body", "body": TOO_LARGE_BODY})
class BodyLimitMiddleware:
"""Refuses any request body over its route's ceiling, before the application sees it.
Registered just inside CORS in `main.py` — so the 413 is decorated like every other response
and a browser can actually read it — but outside the session, the setup gate and every auth
dependency, none of which run for a request that is already too large.
Every request gets a finite ceiling: `ROUTE_MAX_BODY_BYTES` raises it for the two routes that
genuinely take bulk bytes, and there is no path that skips counting altogether.
"""
def __init__(
self,
app: ASGIApp,
*,
limit: int = DEFAULT_MAX_BODY_BYTES,
routes: dict[tuple[str, str], int] | None = None,
) -> None:
self.app = app
self.limit = limit
self._routes = tuple(
(method, template_to_regex(path), cap)
for (method, path), cap in (ROUTE_MAX_BODY_BYTES if routes is None else routes).items()
)
def limit_for(self, method: str, path: str) -> int:
"""This request's ceiling: the route's own if it has one, else the global default. Every
request gets a finite number — there is no path through here that is simply not counted."""
for m, rx, cap in self._routes:
if m == method and rx.match(path):
return cap
return self.limit
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
limit = self.limit_for(scope.get("method", ""), scope.get("path", ""))
declared = _declared_length(scope)
if declared is not None and declared > limit:
# Refused on the declaration alone: nothing downstream is called, and not one byte of
# the body is read.
await _send_too_large(send)
return
seen = 0
refused = False
started = False
async def counting_receive() -> Message:
nonlocal seen, refused
message = await receive()
if message["type"] != "http.request":
return message
seen += len(message.get("body", b""))
if seen > limit:
# The app is blocked on this receive, so it has not answered yet. Telling it the
# client went away is the only in-band way to stop it reading: `request.stream()`
# turns an `http.disconnect` into ClientDisconnect, which unwinds the handler. Our
# own 413 goes out below, once the app has finished unwinding.
refused = True
return {"type": "http.disconnect"}
return message
async def guarded_send(message: Message) -> None:
nonlocal started
if refused and not started:
return # discard the app's answer to the disconnect; the 413 below is the response
started = True
await send(message)
try:
await self.app(scope, counting_receive, guarded_send)
except Exception: # only swallowed when WE caused it, see below
# ClientDisconnect (or whatever a handler raises on a truncated read) is the expected
# consequence of the disconnect we just faked, and the caller must get the 413 rather
# than a 500. Anything raised when we did NOT refuse is a real failure and is re-raised
# untouched.
if not refused:
raise
if refused and not started:
await _send_too_large(send)
View File
+265
View File
@@ -0,0 +1,265 @@
"""Catalog's tables: the shared YouTube world every user reads — channels and their videos — and
each user's footprint in it: what they subscribe to, block, explored or found by search. Only this
unit writes them; another unit reads them along a declared edge, or asks this unit's service."""
from datetime import datetime
from typing import TYPE_CHECKING
from sqlalchemy import (
BigInteger,
Boolean,
Computed,
DateTime,
ForeignKey,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db import Base
from app.models import TimestampMixin
if TYPE_CHECKING:
from app.models import User
class Channel(Base, TimestampMixin):
"""A YouTube channel. Shared across all users (one channel's videos are the same
for everyone), so its expensive metadata is fetched and stored only once."""
__tablename__ = "channels"
id: Mapped[str] = mapped_column(String(32), primary_key=True) # "UC..." channel id
title: Mapped[str | None] = mapped_column(String(255))
handle: Mapped[str | None] = mapped_column(String(255))
description: Mapped[str | None] = mapped_column(Text)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
uploads_playlist_id: Mapped[str | None] = mapped_column(String(64))
subscriber_count: Mapped[int | None] = mapped_column(BigInteger)
video_count: Mapped[int | None] = mapped_column(BigInteger)
# Channel "About" metadata (fetched with the same channels.list call, no extra quota):
# lifetime view count, the channel's join date, banner image, and external links.
total_view_count: Mapped[int | None] = mapped_column(BigInteger)
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
banner_url: Mapped[str | None] = mapped_column(String(1024))
external_links: Mapped[list | None] = mapped_column(JSON)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
country: Mapped[str | None] = mapped_column(String(8))
# Creator keyword tags (brandingSettings.channel.keywords) — a single space-separated string,
# multi-word tags quoted; the client parses it into chips for the About tab.
keywords: Mapped[str | None] = mapped_column(Text)
# Sync bookkeeping.
details_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
recent_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_rss_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# When the newest uploads were last read through the Data API because the RSS feed would not
# answer (`sync/runner.py` `run_feed_fallback`): the once-per-`rss_fallback_hours` clock.
feed_fallback_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
backfill_cursor: Mapped[str | None] = mapped_column(String(128))
backfill_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this channel only entered the catalog because a live YouTube search
# surfaced one of its videos (nobody subscribes to it / it has no organic upload sync).
# Lets the Library hide search-only "noise" by default.
from_search: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
# True when this channel exists only because a user EXPLORED it (opened its page without
# subscribing) — an ephemeral, per-explorer addition. The cleanup job hard-deletes such a
# channel (and its via_explore videos) once nobody keeps it (no subscription, no live
# ExploredChannel row, no per-user state) after a grace period. Cleared when someone
# subscribes (the "keep" signal promotes it to organic).
from_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
videos: Mapped[list["Video"]] = relationship(back_populates="channel")
subscriptions: Mapped[list["Subscription"]] = relationship(back_populates="channel")
class Subscription(Base, TimestampMixin):
"""Per-user link to a channel, with the user's own priority/visibility overrides."""
__tablename__ = "subscriptions"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
# YouTube's subscription resource id, needed to unsubscribe via the API.
yt_subscription_id: Mapped[str | None] = mapped_column(String(128))
priority: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
hidden: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Per-user opt-in to full-history (deep) backfill of this channel. Default off so a
# new user's unique channels don't trigger a big shared-quota burst; the deep
# scheduler only picks up channels at least one user has requested.
deep_requested: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
subscribed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
user: Mapped["User"] = relationship(back_populates="subscriptions")
channel: Mapped["Channel"] = relationship(back_populates="subscriptions")
# Video.live_status values that mean "currently a live/upcoming broadcast" — transient states
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
LIVE_OR_UPCOMING = ("live", "upcoming")
class Video(Base, TimestampMixin):
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
__tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_cover`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0095 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) and the per-channel feed
# sort. Its INCLUDEd columns make the channel manager's sums and the feed sidebar's counts Index Only
# Scans (no heap): a column those queries start reading belongs in it (test_channel_list_index_only).
# Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
description: Mapped[str | None] = mapped_column(Text)
# NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
# forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
published_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
# `feedquery.order_by` by test_feed_sort_indexes.
duration_seconds: Mapped[int | None] = mapped_column(Integer)
view_count: Mapped[int | None] = mapped_column(BigInteger)
like_count: Mapped[int | None] = mapped_column(BigInteger)
category_id: Mapped[int | None] = mapped_column(Integer)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
detected_language: Mapped[str | None] = mapped_column(String(16))
# Creator-supplied keyword tags (snippet.tags joined by spaces) — fetched for free with the
# snippet we already request. Indexed into search_vector so the feed search matches the
# uploader's own keywords, not just words in the title.
keywords: Mapped[str | None] = mapped_column(Text)
# Accumulated distinct live-search queries that surfaced this video (across all users) —
# folds YouTube's own relevance judgement into the local search: a video YouTube returned
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
search_terms: Mapped[str | None] = mapped_column(Text)
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
# boolean btrees and replaced the useful ones with partial indexes over the target subset
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
# search_vector GIN.
is_short: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Whether the youtube.com/shorts/<id> probe has run for this video.
shorts_probed: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video first entered the catalog via a live YouTube search (not an
# organic channel/playlist sync). Used to hide search-discovered "noise" from the
# Library by default; left False by the organic ingest paths.
via_search: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video entered the catalog because a user EXPLORED its (un-subscribed)
# channel. Like via_search it's hidden from the shared Library by default, but it's
# additionally PER-USER private: a via_explore video is only visible to users who
# explored or subscribe to its channel (a gate in the feed query), so one user's
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
via_explore: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# none | live | upcoming | premiere | was_live
live_status: Mapped[str] = mapped_column(
String(16), default="none", server_default="none", index=True
)
enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# videos.list?part=status fields (populated during enrichment); used by the
# maintenance/validation job to decide whether a video is playable anywhere.
embeddable: Mapped[bool | None] = mapped_column(Boolean)
privacy_status: Mapped[str | None] = mapped_column(String(16)) # public|unlisted|private
upload_status: Mapped[str | None] = mapped_column(
String(16)
) # processed|failed|rejected|deleted
# Maintenance/validation lifecycle. last_checked_at drives the rolling re-validation
# (least-recently-checked first); unavailable_since marks a video as currently
# unplayable (hidden from the feed immediately, hard-deleted after a grace period);
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
# weights so title matches outrank description matches. Backed by a GIN index (migration
# 0032). The accent-insensitive `unaccent_simple` config comes from migration 0031.
search_vector: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(keywords, '') || ' ' || "
"coalesce(search_terms, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(description, ''), 1000)), 'C')",
persisted=True,
),
)
channel: Mapped["Channel"] = relationship(back_populates="videos")
class SearchFind(Base, TimestampMixin):
"""Per-user record of a video the user surfaced via their own live YouTube search.
Lets "videos I searched for" appear in that user's own (Mine) feed and the Mine Source
filter — distinct from the GLOBAL `Video.via_search` provenance, which marks a video as
search-discovered by *anyone* and drives the shared Library's Source filter."""
__tablename__ = "search_finds"
__table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_user_search_video"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
class ExploredChannel(Base, TimestampMixin):
"""Per-user record of a channel the user opened the page of WITHOUT subscribing.
Exploring an un-subscribed channel ingests its recent uploads (flagged via_explore) so the
user can browse them, but keeps them out of everyone else's catalog. This row (a) gates the
explorer's access to those ephemeral videos in the feed query and (b) drives the grace-period
purge: a from_explore channel with no live ExploredChannel row (and no subscription / no
per-user state on its videos) is hard-deleted by the cleanup job. Subscribing is the "keep"
signal — it promotes the channel to organic and these rows become irrelevant. `created_at`
(from TimestampMixin) is the grace clock; a re-visit refreshes it."""
__tablename__ = "explored_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_explored_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
class BlockedChannel(Base, TimestampMixin):
"""Per-user channel blocklist. Videos from a blocked channel are excluded from this user's
live YouTube search (and not ingested on their behalf) and hidden from their feed / explore
views — a personal "never show me this channel again" that doesn't affect other users."""
__tablename__ = "blocked_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_blocked_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
+296
View File
@@ -0,0 +1,296 @@
"""Catalog's reads: channels and videos, and each user's footprint in them (subscriptions, blocks,
explorations), for the HTTP layer and for the core code that asks about them.
A user's row is read through `userscope`, so someone else's is the same answer as a missing one.
Catalog does not read another unit's tables: what it needs from Reader (a playlist's channels, a
user's tag links) it asks `reader.queries`.
"""
from datetime import datetime
from sqlalchemy import Select, and_, case, func, or_, select
from sqlalchemy.orm import Session
from app.catalog.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
ExploredChannel,
Subscription,
Video,
)
from app.collation import collated
from app.models import User
from app.sync.explore import followed_channel
from app.userscope import owned_by
# --- channels --------------------------------------------------------------------------------
def channel(db: Session, channel_id: str) -> Channel | None:
return db.get(Channel, channel_id)
def channel_title(db: Session, channel_id: str) -> str | None:
return db.scalar(select(Channel.title).where(Channel.id == channel_id))
def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
"""Each channel's title by id; a channel with no title yet is left out."""
if not ids:
return {}
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(ids)))
return {cid: title for cid, title in rows.tuples() if title is not None}
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
)
def aggregates(db: Session, channel_ids: list[str]) -> list[tuple[str, int, datetime | None, int]]:
"""`channel_aggregates`, run: (channel id, stored videos, last upload, total seconds)."""
if not channel_ids:
return []
return list(db.execute(channel_aggregates(channel_ids)).tuples())
def video_counts(db: Session, channel_id: str) -> tuple[int, int, int]:
"""A channel's stored videos, its Shorts and its live/upcoming ones, in one grouped pass."""
n_all, n_short, n_live = db.execute(
select(
func.count(Video.id),
func.count(Video.id).filter(Video.is_short.is_(True)),
func.count(Video.id).filter(Video.live_status.in_(LIVE_OR_UPCOMING)),
).where(Video.channel_id == channel_id)
).one()
return int(n_all or 0), int(n_short or 0), int(n_live or 0)
def deep_queued(db: Session, channel_ids: list[str]) -> set[str]:
"""Which of these channels at least one user (any user) has asked full history for."""
if not channel_ids:
return set()
return set(
db.execute(
select(Subscription.channel_id)
.where(
Subscription.channel_id.in_(channel_ids),
Subscription.deep_requested.is_(True),
)
.distinct()
).scalars()
)
def followed(db: Session, channel_id: str) -> bool:
"""Whether anyone subscribes to the channel."""
return (
db.execute(
select(Subscription.id).where(Subscription.channel_id == channel_id).limit(1)
).first()
is not None
)
# --- a user's subscriptions, blocks and explorations ----------------------------------------
def subscriptions(db: Session, user: User, lang: str) -> list[tuple[Subscription, Channel]]:
"""The user's subscriptions with their channels, highest priority first, then by title in the
reader's alphabet."""
return list(
db.execute(
select(Subscription, Channel)
.join(Channel, Channel.id == Subscription.channel_id)
.where(owned_by(Subscription.user_id, user))
.order_by(Subscription.priority.desc(), collated(Channel.title, lang))
).tuples()
)
def subscription(db: Session, user: User, channel_id: str) -> Subscription | None:
return db.execute(
select(Subscription).where(
owned_by(Subscription.user_id, user), Subscription.channel_id == channel_id
)
).scalar_one_or_none()
def subscribed_channels(user: User):
"""The channel ids the user subscribes to, as a subquery."""
return select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
def exploration(db: Session, user: User, channel_id: str) -> ExploredChannel | None:
return db.execute(
select(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
).scalar_one_or_none()
def is_blocked(db: Session, user: User, channel_id: str) -> bool:
return (
db.execute(
select(BlockedChannel.id).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
).first()
is not None
)
def blocked_ids(db: Session, user: User) -> set[str]:
return set(
db.scalars(select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user)))
)
def blocked_channels(db: Session, user: User, lang: str) -> list[Channel]:
return list(
db.execute(
select(Channel)
.join(BlockedChannel, BlockedChannel.channel_id == Channel.id)
.where(owned_by(BlockedChannel.user_id, user))
.order_by(collated(Channel.title, lang))
).scalars()
)
# --- sync progress ---------------------------------------------------------------------------
def catalog_totals(db: Session, user: User) -> dict:
"""The instance's catalog size and this user's subscription count (the sync status card)."""
return {
"subscriptions": db.scalar(
select(func.count())
.select_from(Subscription)
.where(owned_by(Subscription.user_id, user))
),
"channels_total": db.scalar(select(func.count()).select_from(Channel)),
"channels_backfilling": db.scalar(
select(func.count()).select_from(Channel).where(Channel.backfill_done.is_(False))
),
"videos_total": db.scalar(select(func.count()).select_from(Video)),
"pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
),
}
def my_progress(db: Session, user: User) -> tuple[int, int, int, int]:
"""Over the channels this user subscribes to: how many, how many have details, how many their
recent uploads, how many their whole history."""
total, details_synced, recent_synced, all_deep_done = db.execute(
select(
func.count(Channel.id),
func.count(Channel.details_synced_at),
func.count(Channel.recent_synced_at),
func.sum(case((Channel.backfill_done.is_(True), 1), else_=0)),
)
.select_from(Channel)
.join(
Subscription,
and_(Subscription.channel_id == Channel.id, owned_by(Subscription.user_id, user)),
)
).one()
return total or 0, details_synced or 0, recent_synced or 0, int(all_deep_done or 0)
def my_deep_requested(db: Session, user: User) -> list[Channel]:
"""The channels this user asked full history for."""
return list(
db.execute(
select(Channel).join(
Subscription,
and_(
Subscription.channel_id == Channel.id,
owned_by(Subscription.user_id, user),
Subscription.deep_requested.is_(True),
),
)
).scalars()
)
def my_video_count(db: Session, user: User) -> int:
"""The videos of the channels this user subscribes to and has not hidden."""
return (
db.scalar(
select(func.count(Video.id)).join(
Subscription,
and_(
Subscription.channel_id == Video.channel_id,
owned_by(Subscription.user_id, user),
Subscription.hidden.is_(False),
),
)
)
or 0
)
def video_total(db: Session) -> int:
return db.scalar(select(func.count(Video.id))) or 0
def deep_backlog(db: Session) -> list[Channel]:
"""Every channel someone asked full history for that does not have it yet."""
deep_requested = (
select(Subscription.id)
.where(Subscription.channel_id == Channel.id, Subscription.deep_requested.is_(True))
.exists()
)
return list(
db.execute(
select(Channel).where(Channel.backfill_done.is_(False), deep_requested)
).scalars()
)
def work_queue(db: Session) -> dict:
"""What the catalog's jobs still have queued, instance-wide (the scheduler card)."""
return {
# The same selection run_recent_backfill works through, or the card counts channels the
# job will never touch and never reaches zero.
"channels_recent_pending": db.scalar(
select(func.count())
.select_from(Channel)
.where(Channel.recent_synced_at.is_(None), followed_channel())
)
or 0,
"videos_pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
)
or 0,
"videos_pending_shorts": db.scalar(
select(func.count())
.select_from(Video)
.where(Video.shorts_probed.is_(False), Video.enriched_at.is_not(None))
)
or 0,
"videos_live_refresh": db.scalar(
select(func.count())
.select_from(Video)
.where(
or_(
Video.live_status.in_(("live", "upcoming")),
and_(Video.live_status == "was_live", Video.duration_seconds.is_(None)),
)
)
)
or 0,
}
+232
View File
@@ -0,0 +1,232 @@
"""Live YouTube search, the catalog's side: what a page of results leaves in the shared catalog.
A result becomes a catalog row so it renders with the normal feed cards and the in-app player and
gains per-user state. The ingest runs in steps with a YouTube call between each, and each step's
rows are committed before the next call: an enrich that fails rolls back to the stubs, which stay
(the FK targets for what follows, backfilled later). Results are flagged (`videos.via_search`,
`channels.from_search`) so the Library can hide this search-discovered noise; Shorts and
live/upcoming videos are never kept. The HTTP layer (`routes/search.py`) runs the paging, the caps
and the refusals.
"""
from collections.abc import Sequence
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from sqlalchemy import delete, func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import logs, quota, sysconfig
from app.catalog import queries
from app.catalog.models import LIVE_OR_UPCOMING, Channel, SearchFind, Subscription, Video
from app.models import User
from app.reader import queries as reader_queries
from app.sync.subscriptions import apply_channel_details
from app.sync.videos import apply_video_details, insert_stubs, parse_dt
from app.userscope import owned_by
from app.utils import like_escape
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short
log = logs.get_logger(__name__)
def classify_shorts(db: Session, videos: Sequence[Video]) -> None:
"""Confirm/deny Shorts for the freshly-ingested videos so none slip into the feed.
Mirrors the scheduler's run_shorts_classification, but scoped to just these rows and run
synchronously (the search is an explicit, awaited action). Non-candidates (too long, no
duration, or live) are finalised without a probe; the rest hit youtube.com/shorts (no
quota). Undetermined probes are left for the scheduler to retry."""
probe_max = sysconfig.get_int(db, "shorts_probe_max_seconds")
to_probe: list[Video] = []
for v in videos:
if v.shorts_probed:
continue
if v.enriched_at is None:
# Enrichment failed (proxy blip) or YouTube omitted the row — duration is unknown,
# so DON'T finalise shorts_probed off a null duration; leave it for the scheduler
# (which only classifies enriched rows) to retry. Otherwise a real Short would leak.
continue
if v.live_status != "none" or v.duration_seconds is None or v.duration_seconds > probe_max:
v.is_short = False
v.shorts_probed = True
else:
to_probe.append(v)
if to_probe:
with make_client() as client:
def work(v: Video):
return v, probe_is_short(client, v.id)
with ThreadPoolExecutor(max_workers=16) as pool:
for v, result in pool.map(work, to_probe):
if result is None:
continue
v.is_short = result
v.shorts_probed = True
db.commit()
def ingest_candidates(
db: Session, yt: YouTubeClient, user: User, candidates: list[dict]
) -> list[str]:
"""Materialise one page of search candidates into the catalog (channel stubs + enrich,
video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
video ids — non-Short, non-live — in the input relevance order."""
ids = [it["id"] for it in candidates]
if not ids:
return []
# 1) Channels are the FK target — create stubs for unknown ones (flagged from_search),
# then enrich the new ones (channels.list, 1 unit). Stubs survive an enrich failure.
channel_ids = list({it["channel_id"] for it in candidates})
existing = set(db.scalars(select(Channel.id).where(Channel.id.in_(channel_ids))).all())
new_channel_ids = [c for c in channel_ids if c not in existing]
if new_channel_ids:
titles = {it["channel_id"]: it["channel_title"] for it in candidates}
for cid in new_channel_ids:
db.add(Channel(id=cid, title=titles.get(cid), from_search=True))
db.commit()
try:
with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
details = yt.get_channels(new_channel_ids)
apply_channel_details(db, details, new_channel_ids)
db.commit()
except YouTubeError:
# The stubs remain and the details can be backfilled later — but until they are, those
# channels sit in the results with no thumbnail and no subscriber count, looking like
# channels that have neither.
log.warning(
"the channels found by a search could not be enriched",
extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
)
db.rollback() # details can be backfilled later; the stubs remain
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
# NOTHING), so an organically-synced video stays organic.
# published_at is NOT NULL (migration 0068); the scrape source exposes no publish date
# (only a relative "3 years ago"), so fall back to now() to satisfy the constraint — the
# enrich in step 3 overwrites it with the video's real snippet date.
now = datetime.now(timezone.utc)
insert_stubs(
db,
[
{
"id": it["id"],
"channel_id": it["channel_id"],
"title": it["title"],
"published_at": parse_dt(it["published_at"]) or now,
"thumbnail_url": it["thumbnail_url"],
"via_search": True,
}
for it in candidates
],
)
# 3) Enrich (videos.list, 1 unit/50) so cards have duration/stats and we can judge live.
videos = db.scalars(select(Video).where(Video.id.in_(ids))).all()
try:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# Every candidate then stays unenriched, and an unenriched video cannot be judged live or
# Short — so it is dropped. A working search returns nothing and reads as "no results".
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
detail_by_id = {}
enriched_at = datetime.now(
timezone.utc
) # when THIS enrich ran (distinct from the stub placeholder)
for v in videos:
item = detail_by_id.get(v.id)
if item is None:
continue
apply_video_details(v, item)
v.enriched_at = enriched_at
db.commit()
# 4) Confirm/deny Shorts (no quota) so none enter via search.
classify_shorts(db, videos)
keep = {
v.id
for v in videos
if v.enriched_at is not None and not v.is_short and v.live_status not in LIVE_OR_UPCOMING
}
return [vid for vid in ids if vid in keep]
def _fold_term(db: Session, video_ids: list[str], term: str) -> None:
"""Append `term` to each video's space-joined `search_terms`, unless it is already one of them.
"Already one of them" is a whole-term match, as a literal: a substring test skipped `cat` on a
video already found for `category`, and an unescaped one read the `_` in `my_channel` as a
wildcard — either way the term was never folded in, and local search lost that relevance."""
padded = func.concat(" ", func.coalesce(Video.search_terms, ""), " ")
db.execute(
update(Video)
.where(Video.id.in_(video_ids))
.where(padded.notilike(f"% {like_escape(term)} %"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
def new_first(db: Session, user: User, ids: list[str]) -> list[str]:
"""Reorder results so genuinely-new discoveries come first and things the user already has
(a subscribed channel, or a video they've watched / in-progress / saved / playlisted) sink
to the back — keeping the relevance order within each group."""
if not ids:
return ids
have = set(
db.scalars(
select(Video.id).where(
Video.id.in_(ids), Video.channel_id.in_(queries.subscribed_channels(user))
)
)
) | reader_queries.held_by(db, user, ids)
return [i for i in ids if i not in have] + [i for i in ids if i in have]
def record_finds(db: Session, user: User, video_ids: list[str], term: str) -> None:
"""These results as THIS user's search finds (the Mine feed, the Source filter), and the term
folded into each one's `search_terms` so local full-text search inherits YouTube's relevance.
Idempotent."""
db.execute(
pg_insert(SearchFind)
.values([{"user_id": user.id, "video_id": vid} for vid in video_ids])
.on_conflict_do_nothing(index_elements=["user_id", "video_id"])
)
_fold_term(db, video_ids, term)
def clear(db: Session, user: User, video_ids: list[str]) -> int:
"""Discard these results "as if never added": drop this user's finds of them, then delete the
ones nobody kept — a `via_search` video no reader holds (`reader.queries.kept_by_a_reader`),
whose channel nobody subscribes to, and that no other user still has as a find. Returns how
many videos went."""
db.execute(
delete(SearchFind).where(
owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids)
)
)
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
result = db.execute(
delete(Video).where(
Video.id.in_(video_ids),
Video.via_search.is_(True),
~reader_queries.kept_by_a_reader(Video.id),
~subscribed,
~other_find,
)
)
return result.rowcount if isinstance(result, CursorResult) else 0
+133
View File
@@ -0,0 +1,133 @@
"""Catalog's writes a request makes: a user's subscription settings, what they explore or block, a
subscription recorded or dropped, a channel's sync markers reset.
Every function here flushes and leaves the commit to its caller (the handler). The YouTube side of a
subscribe or an unsubscribe is the handler's (it needs the user's client and its refusals); what
lands here is the local record of it. The ingest jobs (`sync/`, `catalog/search`) write the
catalog's shared rows themselves.
"""
from datetime import datetime, timezone
from sqlalchemy import delete, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.catalog import queries
from app.catalog.models import BlockedChannel, Channel, ExploredChannel, Subscription, Video
from app.models import User
from app.userscope import owned_by
def remember_own_channel(user: User, channel_id: str) -> None:
"""The user's own YouTube channel, so discovery never offers it to them."""
user.yt_channel_id = channel_id
def update_subscription(
db: Session,
sub: Subscription,
*,
priority: int | None = None,
hidden: bool | None = None,
deep_requested: bool | None = None,
) -> bool:
"""Change the settings given (None leaves one as it is). Returns whether full history was just
turned ON — the caller then pulls the channel now rather than at the scheduler's next tick."""
turned_on = False
if priority is not None:
sub.priority = priority
if hidden is not None:
sub.hidden = hidden
if deep_requested is not None:
# Turning it off won't undo already-fetched videos; it stops further deep paging if nobody
# else still wants it.
turned_on = deep_requested and not sub.deep_requested
sub.deep_requested = deep_requested
db.flush()
return turned_on
def set_deep_all(db: Session, user: User, on: bool) -> int:
"""Every one of the user's channels in (or out of) full-history backfill; how many rows moved."""
result = db.execute(
update(Subscription).where(owned_by(Subscription.user_id, user)).values(deep_requested=on)
)
return result.rowcount if isinstance(result, CursorResult) else 0
def explore(db: Session, user: User, channel: Channel) -> None:
"""Record (or refresh the grace clock of) the user's exploration of a channel, and flag the
channel ephemeral unless someone follows it — the cleanup job's keep rule reads both."""
existing = queries.exploration(db, user, channel.id)
if existing is None:
db.add(ExploredChannel(user_id=user.id, channel_id=channel.id))
else:
existing.created_at = datetime.now(timezone.utc)
if not channel.from_explore and not queries.followed(db, channel.id):
channel.from_explore = True
db.flush()
def reset_backfill(db: Session, channel: Channel) -> None:
"""Fetch the channel again from scratch: its backfill markers cleared, and full history asked
for on EVERY subscription to it — the deep scheduler gates on an EXISTS over those, so the admin
who resets it need not subscribe themselves."""
channel.backfill_done = False
channel.backfill_cursor = None
channel.recent_synced_at = None
db.execute(
update(Subscription)
.where(Subscription.channel_id == channel.id)
.values(deep_requested=True)
)
db.flush()
def record_subscription(db: Session, user: User, channel: Channel, yt_subscription_id: str) -> None:
"""The subscription YouTube now holds, recorded here. Subscribing is the "keep" signal: an
explored channel becomes an organic one, so the cleanup job leaves it alone and its videos join
the shared Library like any subscribed channel's uploads."""
db.add(
Subscription(
user_id=user.id,
channel_id=channel.id,
yt_subscription_id=yt_subscription_id or None,
subscribed_at=datetime.now(timezone.utc),
)
)
if channel.from_explore:
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel.id, Video.via_explore.is_(True))
.values(via_explore=False)
)
db.flush()
def drop_subscription(db: Session, sub: Subscription) -> None:
db.delete(sub)
db.flush()
def block(db: Session, user: User, channel_id: str) -> None:
"""Block a channel for this user (idempotent), and end any exploration of it so the cleanup job
can reclaim it."""
if queries.is_blocked(db, user, channel_id):
return
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
db.execute(
delete(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
)
db.flush()
def unblock(db: Session, user: User, channel_id: str) -> None:
db.execute(
delete(BlockedChannel).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
)
+83
View File
@@ -0,0 +1,83 @@
"""Alphabetical order in the reader's language.
Why not the database default: the Postgres image is Alpine, and musl's `en_US.utf8` collation is BYTE
order — every accented initial sorted after "Z" ("Zorró < Életrajzi < Ötödik"), in every A→Z list the
server orders. Postgres ships the ICU collations on the same image (dev, test, prod alike), so a text
sort names one: Hungarian puts "Ö" after "O" and "Cs" after "Cu", English folds the accents in.
`collated` wraps an ORDER BY expression — and because a keyset cursor compares with the SAME expression
(`feedquery.sort_keys`), the cursor pages in the same order it sorted in. `sorted_words` orders a Python
list the same way through the database, so a list built in Python matches one built in SQL exactly.
The feed's title sorts ride indexes built in these collations (migration 0096), and an index keeps the
order of the ICU that built it. `postgres:16-alpine` is a floating tag: an image update can bring a newer
ICU, after which those indexes disagree with the comparisons a query makes (a keyset page skips or
repeats a title). `heal_drift` rebuilds them at startup once Postgres reports the version moved."""
from typing import Any
from sqlalchemy import Engine, SQLColumnExpression, column, func, select, text, values
from sqlalchemy.orm import Session
from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.types import String
from app import logs
from app.mailtext import normalize
log = logs.get_logger(__name__)
ICU: dict[str, str] = {"en": "en-x-icu", "hu": "hu-x-icu"}
def collation(lang: str | None) -> str:
"""The ICU collation for a UI language; anything unknown sorts as English."""
return ICU[normalize(lang)]
def collated(expr: SQLColumnExpression[Any], lang: str | None) -> ColumnElement[Any]:
"""`expr` compared in the reader's alphabet. Case-folded, so "apple" and "Apple" sit together."""
return func.lower(expr).collate(collation(lang))
def sorted_words(db: Session, words: list[str], lang: str | None) -> list[str]:
"""`words` in the reader's alphabetical order, by the same ICU collation as `collated`."""
if not words:
return []
w = values(column("w", String), name="words").data([(x,) for x in words])
return list(db.scalars(select(w.c.w).order_by(collated(w.c.w, lang), w.c.w)))
def heal_drift(engine: Engine) -> list[str]:
"""Rebuild every index on an ICU collation whose library version moved, then record the new version.
Returns the rebuilt indexes. CONCURRENTLY: the feed keeps reading while it runs."""
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
drifted = conn.execute(
text(
"SELECT oid, collname FROM pg_collation WHERE collname = ANY(:names) "
"AND collversion IS DISTINCT FROM pg_collation_actual_version(oid)"
),
{"names": list(ICU.values())},
).all()
if not drifted:
return []
indexes = list(
conn.execute(
text(
"SELECT DISTINCT d.objid::regclass::text FROM pg_depend d "
"JOIN pg_class c ON c.oid = d.objid AND c.relkind = 'i' "
"WHERE d.classid = 'pg_class'::regclass "
"AND d.refclassid = 'pg_collation'::regclass AND d.refobjid = ANY(:oids)"
),
{"oids": [row.oid for row in drifted]},
).scalars()
)
log.warning(
"ICU collation version changed (%s): rebuilding %d indexes",
", ".join(row.collname for row in drifted),
len(indexes),
)
for name in indexes:
conn.execute(text(f"REINDEX INDEX CONCURRENTLY {name}"))
for row in drifted:
conn.execute(text(f'ALTER COLLATION "{row.collname}" REFRESH VERSION'))
return indexes
+125 -26
View File
@@ -1,6 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import model_validator
from pydantic import AliasChoices, Field, ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -15,11 +16,40 @@ def _version_from_file() -> str:
v = p.read_text(encoding="utf-8").strip()
if v:
return v
# silent-ok: this runs at import, and `app.logs` imports THIS module — a log call here
# would be a circular import. The loop tries the other location, then falls back to "dev".
except OSError:
continue
return "dev"
# Closed value sets. A setting typed with one of these is a CHOICE: pydantic rejects anything else at
# boot, and sysconfig derives the admin picker's options from the same Literal — so the list lives in
# exactly one place and a consumer never has to guess what a typo should fall back to.
SearchSource = Literal["scrape", "api"]
DownloadLayout = Literal["plex", "flat"]
LogLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
LogFormat = Literal["json", "text"]
TranscodeHw = Literal["auto", "on", "off"]
# libx264's presets, fastest first (`placebo` left out on purpose: no measurable gain for its cost).
TranscodePreset = Literal[
"ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow"
]
def choices_of(annotation: object) -> tuple[str, ...]:
"""The allowed values of a choice-typed setting, or () for any other type."""
return tuple(get_args(annotation)) if get_origin(annotation) is Literal else ()
def canonical_choice(options: tuple[str, ...], raw: object) -> str | None:
"""The option `raw` names, in its canonical spelling (any letter case, surrounding space
ignored), or None. The ONE matching rule for a choice value from the env and from the admin
API, so a spelling one accepts the other cannot refuse."""
folded = str(raw).strip().casefold()
return next((option for option in options if option.casefold() == folded), None)
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
@@ -57,7 +87,7 @@ class Settings(BaseSettings):
# VPS Caddy front door. A request from any other peer (someone hitting the app port directly)
# is NOT trusted and its raw socket IP is used, so it can't spoof its rate-limit identity via
# XFF. Empty = trust no proxy (use the direct peer) — correct for a directly-exposed / local-dev
# deploy. Comma-separated. See routes.auth._client_ip + the deploy docs.
# deploy. Comma-separated. See ratelimit.client_key + the deploy docs.
trusted_proxy_ips: str = ""
# --- Outbound email (onboarding: access-request + approval notices) ---
@@ -85,10 +115,13 @@ class Settings(BaseSettings):
search_daily_limit_per_user: int = 70
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
search_source: str = "scrape"
search_source: SearchSource = "scrape"
# Recent-first backfill: how far back to fetch on the first pass per channel.
backfill_recent_max_videos: int = 100
backfill_recent_max_days: int = 365
# A channel whose RSS feed has not answered for this many hours gets its newest uploads read
# through the Data API instead (1 unit), at most once per this many hours. 0 = never.
rss_fallback_hours: int = 2
# Shorts are confirmed by probing youtube.com/shorts/<id>. Only videos at or below
# this duration (and not livestreams) are probed; longer videos are never Shorts.
@@ -113,8 +146,6 @@ class Settings(BaseSettings):
autotag_interval_minutes: int = 30
subscriptions_resync_minutes: int = 360
playlist_sync_minutes: int = 360
# How often the shared demo account is auto-reset to a clean baseline (communal sandbox).
demo_reset_minutes: int = 720
# --- Maintenance / validation job ---
# Runs once a day by default (admin-tunable via the Scheduler dashboard). Grace periods
# are in days because that's the granularity that matters; the rolling re-validation
@@ -140,9 +171,13 @@ class Settings(BaseSettings):
feed_default_hidden_live: str = "live,upcoming"
# --- Download center (yt-dlp) ---
# Root directory (inside the container) for downloaded media — a Docker volume/bind mount.
# Root directory (inside the container) for the files library — a Docker volume/bind mount.
# Env-only (a mount path can't live in the DB). The worker lays a Plex-style tree under it.
download_root: str = "/downloads"
# DOWNLOAD_ROOT is its old name: an install still on the old compose file (mounted at
# /downloads, DOWNLOAD_ROOT=/downloads) keeps finding its files after it pulls a new image.
files_root: str = Field(
default="/files", validation_alias=AliasChoices("FILES_ROOT", "DOWNLOAD_ROOT", "files_root")
)
# Whether THIS process runs the download worker loop. On for the dedicated worker
# container, off for the API (which keeps the scheduler). Mirrors scheduler_enabled.
worker_enabled: bool = False
@@ -150,12 +185,18 @@ class Settings(BaseSettings):
# for YouTube Proof-of-Origin tokens (bypasses bot-detection + unlocks high-quality
# formats). Empty disables it (yt-dlp still works, but may hit "confirm you're not a bot").
download_pot_base_url: str = "http://bgutil-pot:4416"
# yt-dlp player clients (comma-separated). PRIMARY = android_vr: high-quality DASH, no PO
# token / JS runtime needed, reliable for normal use. On a bot/DRM/format failure (YouTube's
# per-client responses are flaky, and android_vr can get bot-flagged under heavy load) the
# worker retries with the FALLBACK set — the POT-capable web clients (web needs the token +
# Deno) plus android for audio. Both admin-tunable when YouTube shifts what works.
download_player_clients: str = "android_vr"
# yt-dlp player clients (comma-separated). PRIMARY = tv_simply: measured 2026-10-10 as the one
# client that is offered 1080p+ AND finishes the fetch, its stream token minted by the bgutil
# sidecar. android_vr (the primary until then) was still offered 2160p, but every fetch died
# with HTTP 403 — it now needs a stream PO token the sidecar cannot mint — and the fallback
# below served 360p only, so every download came out at 360p without failing. On a
# bot/DRM/format failure the worker retries with the FALLBACK set — the POT-capable web clients
# (web needs the token + Deno) plus android, whose 360p is the last resort. Both admin-tunable
# (env) when YouTube shifts what works.
# A SIGNED-IN download (the user added YouTube cookies) keeps only the cookie-capable clients
# of these two sets — yt-dlp drops the rest, so android_vr / android / ios never run there.
# See adapters.YouTubeAdapter._COOKIE_CAPABLE.
download_player_clients: str = "tv_simply"
download_player_clients_fallback: str = "web_safari,web,android"
@staticmethod
@@ -208,10 +249,14 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# Per-file cap for local uploads. Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# ge=1: this number is also the EDGE ceiling (`bodylimit.ROUTE_MAX_BODY_BYTES`), where 0 is a
# real limit of zero rather than the route's "no per-file cap" sentinel — an instance set to 0
# would refuse every upload, including the admin page's own way back. To lift the cap in
# practice, set a large number.
download_upload_max_bytes: int = Field(default=524_288_000, ge=1) # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -220,10 +265,32 @@ class Settings(BaseSettings):
# Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs.
audit_retention_days: int = 180
audit_gc_minutes: int = 1440
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: str = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
download_layout: DownloadLayout = "plex"
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
# startup and on change without a restart. This is what a process logs at before it can reach
# the database — and the only knob the worker/runner children read, since they are short-lived.
# TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
log_level: LogLevel = "INFO"
# "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
# line for a local terminal). Redaction applies to both.
log_format: LogFormat = "json"
# Also write events to the `log_events` table (app/logsink.py): WARNING and above, the request
# lines, and — once they exist — client batches. Off only where no table is wanted (the tests).
log_db_sink: bool = True
# Days a `log_events` row is kept (the privacy policy promises this), and how often the log_gc job
# enforces it — hourly, so a row never outlives the promise by more than an hour.
log_retention_days: int = 7
log_gc_minutes: int = 60
# --- Plex integration (optional) ---
# The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
@@ -241,7 +308,7 @@ class Settings(BaseSettings):
plex_path_map: str = ""
# Which Plex library section keys are exposed (comma-separated). Empty = all enabled sections.
plex_libraries: str = ""
# HU-overlay mapping (E5 S5): comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# HU-overlay mapping: comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# section is synced as a non-browsable overlay whose Hungarian metadata is written onto the EN
# twin section's rows (matched by guid). Empty = no overlays. E.g. "3:4,5:1".
plex_overlay_map: str = ""
@@ -251,7 +318,7 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# 15 min: with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
@@ -262,25 +329,34 @@ class Settings(BaseSettings):
# re-encode is CPU-heavy (the VAAPI hardware path is far cheaper and could sustain more); direct-serve
# and remux (stream-copy) are not capped by this.
plex_max_transcodes: int = 4
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# Max concurrent playback sessions of ANY kind — remux (stream-copy) included. Cheap on the CPU but
# each one is a whole segmented copy of the film on the scratch volume, written faster than
# realtime. One account holds at most two regardless (plex/stream.py `_MAX_SESSIONS_PER_USER`).
plex_max_sessions: int = 8
# --- Transcode (playable="transcode") re-encode tuning ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
plex_transcode_hw: str = "auto"
plex_transcode_hw: TranscodeHw = "auto"
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
# encodes at source resolution.
plex_transcode_preset: str = "veryfast"
# Loudness normalisation of the downmixed stereo audio on the HLS path (EBU R128 `loudnorm`).
# OFF by default: it makes a film ~11 dB louder and narrows its dynamic range, which is a taste
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
plex_audio_loudnorm: bool = False
plex_transcode_preset: TranscodePreset = "veryfast"
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
# dir under files_root. On localdev the files_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's files_root is fast
# local disk, so the default is fine there.
plex_hls_dir: str = ""
# --- TMDB enrichment (E5 S2; optional — enriches the Plex catalog from TMDB) ---
# --- TMDB enrichment (optional — enriches the Plex catalog from TMDB) ---
# TMDB API key (v3 api_key, 32 hex chars). Server-side only; stored encrypted when set via the
# admin Config page. Empty → the TMDB enrichment job no-ops. Feeds keywords / crew / budget /
# revenue / TMDb score / recommendations / backdrops onto the mirrored Plex rows, matched by the
@@ -289,10 +365,22 @@ class Settings(BaseSettings):
tmdb_api_base: str = "https://api.themoviedb.org/3"
# Metadata language for TMDB fetches (keyword/overview locale). Stored EN by default.
tmdb_language: str = "en-US"
# --- Bug reports: where a debug session's issue goes. All three empty = no tracker; the
# composer still exports a zip. The repository must be PRIVATE — an issue carries the reporter's
# log — and app/issues/gitea.py refuses a public one. Runtime knobs (sysconfig, admin UI). ---
issues_gitea_url: str = ""
issues_gitea_repo: str = ""
issues_gitea_token: str = ""
# How often the TMDB enrichment job fetches a batch of not-yet-enriched Plex titles, in minutes.
tmdb_enrich_interval_min: int = 120
# Titles fetched per TMDB enrichment run (TMDB is fast + generously rate-limited).
tmdb_enrich_batch: int = 100
# How often the people job fetches a batch of person bios from TMDB and portraits through Plex,
# in minutes, and how many of each per run: ~18k linked people at 300 per 10 min is about ten hours
# once, then only the newcomers.
people_enrich_interval_min: int = 10
people_enrich_batch: int = 300
@property
def allowed_email_set(self) -> set[str]:
@@ -320,6 +408,17 @@ class Settings(BaseSettings):
https OAUTH_REDIRECT_URL as the signal for 'real deployment' vs local dev."""
return self.oauth_redirect_url.lower().startswith("https://")
@field_validator("*", mode="before")
@classmethod
def _canonical_choice(cls, value: object, info: ValidationInfo) -> object:
"""A choice setting accepts its value in any letter case (`LOG_LEVEL=debug`) and stores the
canonical spelling; an unknown value still fails the boot with pydantic's own message."""
field = cls.model_fields.get(info.field_name or "")
options = choices_of(field.annotation) if field else ()
if options and isinstance(value, str):
return canonical_choice(options, value) or value
return value
@model_validator(mode="after")
def _enforce_production_secrets(self) -> "Settings":
"""Fail fast rather than boot a public instance with the shipped placeholder
+55 -2
View File
@@ -1,6 +1,6 @@
from collections.abc import Iterator
from sqlalchemy import create_engine
from sqlalchemy import create_engine, text
from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
from app.config import settings
@@ -12,7 +12,9 @@ class Base(DeclarativeBase):
# Pool sized above the sync-endpoint threadpool (~40) so a burst of short DB sessions can't starve on
# checkout. The image proxies deliberately DON'T hold a connection during their slow external fetch
# (see routes/plex/), so this headroom is only for the many brief sessions, not long ones.
# (see routes/plex/), so this headroom is only for the many brief sessions, not long ones. The one
# long hold — a playlist sync's advisory claim, for a whole push — opens its own unpooled connection
# (sync/playlists.py `_claim_engine`) instead of drawing a third one from here.
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
@@ -20,6 +22,10 @@ engine = create_engine(
max_overflow=30,
pool_timeout=30,
future=True,
# A database error renders its bound parameters into its message — a title, an uploader, a
# file path, a URL — and that message is logged and stored as a job's error text. The SQL
# stays; the values are data, and the ids an operator needs are on the line already.
hide_parameters=True,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
@@ -30,3 +36,50 @@ def get_db() -> Iterator[Session]:
yield db
finally:
db.close()
def release_read_connection(db: Session) -> None:
"""End a request's READ transaction so its pooled connection goes back BEFORE a long await.
SQLAlchemy 2.0 autobegins on the first SELECT, so a request that has merely authenticated is
already holding one of the connections above — `idle in transaction` to Postgres, and nothing
times it out. A route that then awaits something slow (a request body still arriving, an
external fetch) pins it for the whole wait, which is precisely the invariant the pool is sized
on in the comment above. Measured: the next checkout after this call reuses the very backend
that was released, and a 1.8 s upload goes from one sustained `idle in transaction` to none.
Every route that reads its own body from a dependency needs this, and `test_body_limit.py`
holds that — the three that do are the ones with a raised (or self-imposed) body ceiling, and
they got here one at a time, which is how the upload route ended up the only one fixed.
The caller must have NOTHING PENDING, and that is checked rather than described. The session is
shared with every dependency that ran before, so a rollback here would silently discard a write
one of them staged — `service.enqueue` already learned that lesson the expensive way and uses a
SAVEPOINT for it. CLAUDE.md: a rule stated in a comment must also be enforced in code.
TWO checks, because the obvious one is blind to the shapes this codebase actually writes in.
`db.new`/`deleted`/`dirty` hold only UNFLUSHED unit-of-work: the moment a helper calls
`db.flush()` — CLAUDE.md's canonical split is "the handler commits; helpers flush" — the object
is persistent with no unflushed history and all three sets are empty again, and Core DML
(`db.execute(update(...))`) never enters them at all. Measured: after `add`+`flush` the sets are
empty, the old check passed, and the rollback lost the row. So the transaction is asked
directly. Postgres assigns a transaction id on the first WRITE and never for a read, so
`pg_current_xact_id_if_assigned()` is NULL for exactly the state this function is safe in — and
it sees a raw `text("UPDATE …")` too, which no session-level bookkeeping would.
"""
if db.new or db.deleted or db.dirty:
raise RuntimeError(
"release_read_connection() with pending work: something ahead of this dependency "
"staged a write, which ending the transaction would discard. Commit it first, or do "
"it after the await."
)
if (
db.in_transaction()
and db.execute(text("SELECT pg_current_xact_id_if_assigned()")).scalar() is not None
):
raise RuntimeError(
"release_read_connection() on a transaction that has already WRITTEN: something ahead "
"of this dependency flushed a row or ran an UPDATE/INSERT/DELETE, which ending the "
"transaction would discard. Commit it first, or do it after the await."
)
db.rollback()
+484
View File
@@ -0,0 +1,484 @@
"""Debug sessions: Start → reproduce → Stop, and the evidence a bug report is built from.
While a session is open its user's browser names it in `X-Debug-Session` on every request. The auth
dependency accepts the header only for the caller's OWN open session (anything else is ignored, never
an error — a stale tab must not break the app) and then logs that request at the session's level with
every line tagged by its id; the browser tags its own events the same way. The issue is later built
from exactly the rows carrying the tag, so what the review list shows is what gets sent.
Starting a session IS the consent for its level, for its duration: the start dialog carries the
consent text. It needs no change to the user's own diagnostics level, and none is made — when the
session ends the level in force is simply the user's own again.
A session is capped at `MAX_MINUTES`, set by the server at Start. An open one past its cap is over
whether or not anything has closed it yet: it reads as over at once, a new Start closes it, and the
hourly log job sweeps the rest. Sessions go with the logs they point at, after `log_retention_days`.
Core layer: imports `models`, `diagnostics`.
"""
import secrets
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from typing import Literal, get_args
from uuid import uuid4
from sqlalchemy import and_, delete, func, not_, select, update
from sqlalchemy.orm import Session, defer
from app.userscope import owned_by, owned_row
from app import diagnostics
from app.models import DebugScreenshot, DebugSession, LogEvent, User
HEADER = "x-debug-session"
MAX_MINUTES = diagnostics.SESSION_MINUTES
SCREENSHOT_MAX_BYTES = 10 * 1024 * 1024
SCREENSHOTS_MAX = 8
# What one user may keep in screenshots across ALL their sessions (until retention takes them): the
# per-session cap alone would let Start → 8 images → Stop repeat without end.
SCREENSHOT_USER_MAX_BYTES = 100 * 1024 * 1024
# The screenshot types kept, and the extension each is exported and sent under — one table, so an
# accepted type always has a name.
IMAGE_TYPES = {"image/png": "png", "image/jpeg": "jpg", "image/webp": "webp"}
# What a screenshot is called in the zip and on the issue. `issues/bundle.py` builds the names from
# this and `IMAGE_TYPES`; this module and `issues/gitea.py` read them back to tell a screenshot's
# link from a log's. One definition, because three copies of a name is how the next rename breaks
# `images_pending` into silence.
SCREENSHOT_PREFIX = "screenshot-"
# Rows an issue can carry. A 60-minute session at Teljes is a few thousand rows; past this the
# bundle says how many it left out rather than growing without bound.
BUNDLE_MAX = 5_000
SessionLevel = Literal["DEBUG", "TRACE"]
SESSION_LEVELS: tuple[str, ...] = get_args(SessionLevel)
# Detail order of the diagnostics levels: a session raises the level in force, never lowers it.
_DETAIL = {"INFO": 0, "DEBUG": 1, "TRACE": 2}
_ID_LEN = 16
class EndReason(StrEnum):
STOPPED = "stopped"
EXPIRED = "expired"
def is_live(session: DebugSession, now: datetime | None = None) -> bool:
"""Open and within its cap — THE liveness rule: an open session past its cap is over."""
return session.ended_at is None and session.until > (now or datetime.now(timezone.utc))
def open_session(db: Session, user: User) -> DebugSession | None:
"""The user's live session, or None. Read-only: one past its cap is over and reads as None;
`start` and the hourly sweep are what close it."""
session = _unclosed(db, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
def _unclosed(db: Session, user: User) -> DebugSession | None:
return db.scalar(
select(DebugSession).where(
owned_by(DebugSession.user_id, user), DebugSession.ended_at.is_(None)
)
)
def start(db: Session, user: User, level: SessionLevel) -> DebugSession:
"""Open a session, or return the live one — a second tab pressing Start joins the recording
rather than ending it. One past its cap is closed first: the one-open-per-user index would
refuse the new row beside it. Flushes; the caller commits. Two concurrent Starts can still both
get here: the index refuses the second, and the caller joins the one that won."""
now = datetime.now(timezone.utc)
current = _unclosed(db, user)
if current is not None:
if is_live(current, now):
return current
end(current, EndReason.EXPIRED)
db.flush()
session = DebugSession(
id=secrets.token_hex(_ID_LEN // 2),
user_id=user.id,
level=level,
started_at=now,
until=now + timedelta(minutes=MAX_MINUTES),
)
db.add(session)
db.flush()
return session
def stop(session: DebugSession) -> None:
"""The user's Stop. Past its cap the session had already ended on its own, and a late Stop must
not rename that."""
end(session, EndReason.STOPPED if is_live(session) else EndReason.EXPIRED)
def end(session: DebugSession, reason: EndReason) -> None:
"""Close a session. An expired one is dated at its cap, not at whenever it was noticed."""
if session.ended_at is not None:
return
session.ended_at = (
session.until
if reason is EndReason.EXPIRED
else min(datetime.now(timezone.utc), session.until)
)
session.ended_reason = reason
def from_header(db: Session, user: User, value: str | None) -> DebugSession | None:
"""The session a request names, if it is the caller's own and still live; else None."""
if not value or len(value) != _ID_LEN or not all(c in "0123456789abcdef" for c in value):
return None
session = owned_row(db, DebugSession, value, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
def level_for(user: User, session: DebugSession | None) -> tuple[str, datetime | None]:
"""The diagnostics level (and its expiry) a request is logged at: the user's own, raised to the
session's while one is named — whichever records more.
A named session also caps the expiry at its own: this pair is what the process tracks under the
session (app/logs.py), and an expiry outliving the recording would hold the root logger down
after it ended."""
own = diagnostics.effective_level(user)
until = user.diagnostics_until if own != diagnostics.BASE else None
if session is None:
return own, until
if _DETAIL[session.level] <= _DETAIL[own]:
return own, min(until, session.until) if until else session.until
return session.level, session.until
def lock(db: Session, session: DebugSession) -> None:
"""Re-read the session under a row lock, held until the caller commits. Everything that changes
what an issue is made of — the draft, the screenshots, the send itself — takes it first: a send
in flight sees none of them change under it, and each of them sees a send that committed."""
db.refresh(session, with_for_update=True)
def save_draft(session: DebugSession, draft: dict) -> None:
"""Keep the composer's draft (the caller validated, locked and commits). Shape: `title`, `body`
(the rendered markdown an export or an issue carries), `excluded` (log row ids taken out of the
evidence) and `form` (the composer's own fields, to reopen it as it was). A delivery's progress
is not the composer's to replace, and is kept."""
progress = (session.draft or {}).get("delivery")
session.draft = {**draft, "delivery": progress} if progress else draft
# --- a delivery's progress -----------------------------------------------------------------
# Written by app/issues/gitea.py into the draft; read HERE, so the send's decision and what the
# composer shows are one reading of it.
def delivery(session: DebugSession) -> dict:
return dict((session.draft or {}).get("delivery") or {})
def not_uploaded(session: DebugSession) -> list[str]:
"""The files a send that stopped half-way has still to upload — by name, as the issue carries
them. What the composer names to the user."""
return list(delivery(session).get("pending") or [])
def images_pending(session: DebugSession) -> bool:
"""Whether the issue's body still has to be edited to show the screenshots it already carries
(every upload went, the edit that shows them did not)."""
state = delivery(session)
shown = state.get("images_shown")
return not shown and any(n.startswith(SCREENSHOT_PREFIX) for n in state.get("links") or {})
def unfinished(session: DebugSession) -> bool:
"""Whether a send owes the issue anything — the retry's question."""
return bool(not_uploaded(session)) or images_pending(session)
# A send claims the session while it talks to the tracker, instead of holding the row's lock across
# those calls: a Stop, an autosave or a screenshot would queue behind a lock for the whole send.
# They see the claim and are refused at once instead. A claim not renewed for this long was left by
# a request that died mid-send (the process restarted), and the next send takes it over — which is
# why a live send RENEWS it every time it reports progress (`set_delivery`). A claim is a lease, and
# a send slower than the lease must not lose it while it is still running.
SEND_CLAIM_SECONDS = 600
def _claim(session: DebugSession) -> dict:
claim = delivery(session).get("sending")
return claim if isinstance(claim, dict) else {}
def _stamp(token: str) -> dict:
return {"at": datetime.now(timezone.utc).isoformat(), "by": token}
def sending(session: DebugSession) -> bool:
"""Whether a send is under way right now (a live claim)."""
at = _claim(session).get("at")
if not at:
return False
return (datetime.now(timezone.utc) - datetime.fromisoformat(at)).total_seconds() < (
SEND_CLAIM_SECONDS
)
def claim_send(session: DebugSession) -> str:
"""Mark the session as being sent, and return the token that OWNS this claim. The caller holds
the lock and commits, which is what makes the claim visible to everyone else."""
token = uuid4().hex
_write_delivery(session, {**delivery(session), "sending": _stamp(token)})
return token
def release_send(session: DebugSession, token: str) -> None:
"""Give back our OWN claim, and only ours: one that aged out and was taken over belongs to the
send holding it now, and handing that one back would let a third send in beside it."""
if _claim(session).get("by") != token:
return
_write_delivery(session, {k: v for k, v in delivery(session).items() if k != "sending"})
class SendClaimLost(Exception):
"""This send's claim is no longer on the row: it aged out and another send took it over. The
send must stop — what it uploads from here it can no longer record, so a retry would send it
again — and it must not write, because the row now holds the other send's progress."""
def claim_owner(db: Session, session: DebugSession) -> str | None:
"""Who holds the row's claim RIGHT NOW, read from the ROW and not from the object.
`expire_on_commit` is off (app/db.py), so a checkpoint does not re-read: a send that stalled
past its lease still carries its own dead token in memory long after another send took over.
A targeted scalar, not `lock`/`db.refresh` — a refresh here would discard the `issue_url` that
`gitea.deliver` assigns one line before its first progress report, and that link is the one
thing the send must not lose."""
return db.scalar(
select(DebugSession.draft["delivery"]["sending"]["by"].as_string()).where(
DebugSession.id == session.id
)
)
def set_delivery(db: Session, session: DebugSession, state: dict, token: str) -> None:
"""Write a delivery's progress (app/issues/gitea.py keeps it; this module reads it), and renew
the claim while doing it — a send still reporting progress is still alive, so one slower than
the lease cannot expire while it is still running.
The claim is not the caller's to carry: `gitea.deliver` threads ONE snapshot of this dict
through a whole send, so a progress report that carried the claim wrote a stale one back at
every step. It is stamped from `token` here instead, and only after the ROW confirms the claim
is still ours. Without that check the write is a full-column overwrite with no WHERE: a send
that lost its lease came back and stomped the take-over send's committed links, then put its
own dead token back on an idle session. `release_send` has always made
the same check; this is the other half of it."""
if claim_owner(db, session) != token:
raise SendClaimLost(session.id)
state = {k: v for k, v in state.items() if k != "sending"}
state["sending"] = _stamp(token)
_write_delivery(session, state)
def _write_delivery(session: DebugSession, state: dict) -> None:
# A new dict, so the JSONB column sees the change.
session.draft = {**(session.draft or {}), "delivery": state}
def serialize(session: DebugSession | None, screenshots: int = 0) -> dict | None:
if session is None:
return None
return {
"id": session.id,
"level": session.level,
"started_at": session.started_at.isoformat(),
"until": session.until.isoformat(),
"ended_at": session.ended_at.isoformat() if session.ended_at else None,
"ended_reason": session.ended_reason,
"issue_url": session.issue_url,
# A send that stopped half-way: the files it has still to upload, and whether the issue's
# body has still to be edited to show the screenshots it already carries. Both are true of a
# send that is RUNNING too — every step commits its progress — so `sending` goes with them:
# without it a reload or a second tab renders a live send as a failed one, and offers a
# retry that is certain to be refused.
"pending_files": not_uploaded(session),
"images_pending": images_pending(session),
"sending": sending(session),
"screenshots": screenshots,
"draft": session.draft,
}
def view(db: Session, session: DebugSession) -> dict:
"""The session as the API serves it — one shape wherever it is served."""
return serialize(session, screenshot_count(db, session)) or {}
def screenshot_count(db: Session, session: DebugSession) -> int:
return (
db.scalar(
select(func.count())
.select_from(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
)
or 0
)
# --- screenshots ---------------------------------------------------------------------------
# By content, not by the declared type: what is stored is served back to a browser as an image.
_SIGNATURES = (
(b"\x89PNG\r\n\x1a\n", "image/png"),
(b"\xff\xd8\xff", "image/jpeg"),
)
def sniff(data: bytes) -> str | None:
"""The image type `data` really is — one of IMAGE_TYPES — or None."""
for magic, mime in _SIGNATURES:
if data.startswith(magic):
return mime
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return "image/webp"
return None
def lock_user(db: Session, user_id: int) -> None:
"""Hold this user's own lock for the rest of the transaction. A session's row lock bounds what
THAT session may keep; the byte cap below is the user's across all of them, and a lock on one
session cannot serialize a check that reads them all — two uploads to two sessions both read the
total before either commits, and both pass."""
db.execute(select(func.pg_advisory_xact_lock(user_id)))
def stored_screenshot_bytes(db: Session, user_id: int) -> int:
return (
db.scalar(
select(func.coalesce(func.sum(DebugScreenshot.size), 0))
.join(DebugSession, DebugSession.id == DebugScreenshot.session_id)
.where(owned_by(DebugSession.user_id, user_id))
)
or 0
)
class ScreenshotRefused(Exception):
"""A screenshot that cannot be kept; `code` is the refusal's machine word."""
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
def add_screenshot(db: Session, session: DebugSession, data: bytes) -> DebugScreenshot:
"""Keep one screenshot. Locks the user and the session, and flushes; the caller commits.
Refused while a send is under way or once the issue was sent (it can no longer carry it), past
the per-session count or the user's storage, or when it is not an image we can show."""
if not data:
raise ScreenshotRefused("empty")
mime = sniff(data)
if mime is None:
raise ScreenshotRefused("not_an_image")
# Under both locks: two uploads at once cannot both pass the count (the session's), nor both
# pass the user's byte cap from two different sessions (the user's). Always in this order.
lock_user(db, session.user_id)
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if session.issue_url is not None:
raise ScreenshotRefused("already_sent")
if screenshot_count(db, session) >= SCREENSHOTS_MAX:
raise ScreenshotRefused("too_many")
if stored_screenshot_bytes(db, session.user_id) + len(data) > SCREENSHOT_USER_MAX_BYTES:
raise ScreenshotRefused("storage_full")
shot = DebugScreenshot(
session_id=session.id,
created_at=datetime.now(timezone.utc),
mime=mime,
size=len(data),
data=data,
)
db.add(shot)
db.flush()
return shot
def remove_screenshot(db: Session, session: DebugSession, shot: DebugScreenshot) -> None:
"""Delete one screenshot, under the lock. Refused while a send is under way or unfinished: a
delivery names the screenshots by their order, and removing one would renumber what it still
has to send. Once the issue has everything, this only drops OUR copy — the issue keeps its
own. The caller commits."""
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if unfinished(session):
raise ScreenshotRefused("already_sent")
db.delete(shot)
db.flush()
def screenshots(db: Session, session: DebugSession) -> list[DebugScreenshot]:
"""The session's screenshots, oldest first, WITHOUT their bytes loaded."""
return list(
db.scalars(
select(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
.options(defer(DebugScreenshot.data))
.order_by(DebugScreenshot.id)
)
)
# --- the evidence --------------------------------------------------------------------------
# The browser's own log upload (`POST` here, every few seconds while recording) is how the evidence
# arrives, not part of it: its server lines would fill the bundle with the recording's plumbing. The
# browser leaves its copies out for the same reason (frontend lib/diagnostics `isOwnRequest`).
INGEST_PATH = "/api/diagnostics/events"
def _evidence(session: DebugSession):
upload_line = and_(
LogEvent.event == "http.request", LogEvent.attrs["path"].astext == INGEST_PATH
)
# `IS TRUE`, not the bare condition: for a row with no event or no attrs it is NULL, and
# `NOT NULL` would drop exactly the rows that are evidence.
return (LogEvent.debug_session_id == session.id, not_(upload_line.is_(True)))
def bundle(
db: Session, session: DebugSession, *, drop_excluded: bool = False
) -> tuple[list[LogEvent], int]:
"""The session's log rows, browser and server interleaved in time, capped at BUNDLE_MAX — and
how many rows it has in all, so a cut bundle can say so. `drop_excluded` leaves out the rows
the user took out in the composer: what an export or an issue carries."""
where = list(_evidence(session))
excluded = (session.draft or {}).get("excluded") or [] if drop_excluded else []
if excluded:
where.append(LogEvent.id.not_in(excluded))
rows = list(
db.scalars(
select(LogEvent).where(*where).order_by(LogEvent.ts, LogEvent.id).limit(BUNDLE_MAX)
)
)
total = len(rows)
if total == BUNDLE_MAX:
total = db.scalar(select(func.count()).select_from(LogEvent).where(*where)) or total
return rows, total
# --- housekeeping (the hourly log job) -----------------------------------------------------
def sweep(db: Session) -> int:
"""Stage closing every open session past its cap; returns how many."""
now = datetime.now(timezone.utc)
result = db.execute(
update(DebugSession)
.where(DebugSession.ended_at.is_(None), DebugSession.until <= now)
.values(ended_at=DebugSession.until, ended_reason=EndReason.EXPIRED.value)
.execution_options(synchronize_session=False)
)
return result.rowcount or 0
def gc(db: Session, retention_days: int) -> int:
"""Stage deleting sessions (and by cascade their screenshots) started before the log retention
window — the rows they would bundle are gone by then too. Returns how many."""
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
result = db.execute(delete(DebugSession).where(DebugSession.started_at < cutoff))
return result.rowcount or 0
-82
View File
@@ -1,82 +0,0 @@
"""Demo-account service: reset the shared demo sandbox to a clean, seeded baseline.
Shared by the admin's manual reset endpoint (routes/admin.py) and the scheduled demo-reset job
(scheduler.py). Lives in the service layer, not routes, so the scheduler can call it without an
`app.routes -> app.scheduler` import cycle (the reason its import used to be lazy).
"""
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from app.models import Playlist, PlaylistItem, Subscription, User, Video, VideoState
def _seed_demo_playlists(db: Session, demo: User) -> int:
"""Seed a few sample playlists from the shared catalog so a freshly-reset demo isn't
empty — gives visitors something to play with. Best-effort: skips silently if the
catalog is too small."""
recent = (
db.execute(
select(Video.id)
.where(Video.is_short.is_(False), Video.live_status == "none")
.order_by(Video.published_at.desc().nullslast())
.limit(60)
)
.scalars()
.all()
)
if not recent:
return 0
samples = [
("Fresh picks", recent[0:10]),
("Quick watch", recent[10:18]),
("Saved for later", recent[18:26]),
]
created = 0
for pos, (name, vids) in enumerate(samples):
if not vids:
continue
pl = Playlist(user_id=demo.id, name=name, position=pos)
db.add(pl)
db.flush()
for ipos, vid in enumerate(vids):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=ipos))
created += 1
return created
def _seed_demo_subscriptions(db: Session, demo: User, n: int = 14) -> int:
"""Seed the demo with a handful of catalog channels (the ones with the most videos) so its
Channel manager isn't empty — gives visitors something to explore and matches the landing-page
screenshot. Idempotent: clears the demo's subscriptions first, then re-adds the top N."""
db.execute(delete(Subscription).where(Subscription.user_id == demo.id))
top = (
db.execute(
select(Video.channel_id)
.where(Video.channel_id.is_not(None))
.group_by(Video.channel_id)
.order_by(func.count().desc())
.limit(n)
)
.scalars()
.all()
)
for channel_id in top:
db.add(Subscription(user_id=demo.id, channel_id=channel_id))
return len(top)
def reset_demo_state(db: Session, demo: User) -> int:
"""Wipe the demo account's per-user state (watch/save/hide states, playlists, preferences)
back to a clean baseline and re-seed a few sample playlists + channel subscriptions. Returns
the seeded playlist count. Shared by the admin's manual reset button and the scheduled demo
reset (see scheduler)."""
db.execute(delete(VideoState).where(VideoState.user_id == demo.id))
# Playlist items cascade on playlist delete (ondelete="CASCADE").
db.execute(delete(Playlist).where(Playlist.user_id == demo.id))
demo.preferences = {"language": "en"}
db.commit()
seeded = _seed_demo_playlists(db, demo)
_seed_demo_subscriptions(db, demo)
db.commit()
return seeded
+64
View File
@@ -0,0 +1,64 @@
"""A user's diagnostics level: what they consented to, and for how long.
Three levels, named in the UI Alap / Részletes / Teljes:
* INFO — the base, for everyone, no consent. The browser keeps its log in memory; nothing extra
is sent or stored.
* DEBUG — consented. The server logs this user's requests in detail and stores every line of
them, and the browser streams its own log to `POST /api/diagnostics/events`.
* TRACE — consented. As DEBUG, plus what the user typed into a search.
A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
never taken from the client, and an expired level reads as INFO without anything having to clear it.
Core layer: imports `models` and `sysconfig`.
"""
from datetime import datetime, timedelta, timezone
from typing import Literal, get_args
from sqlalchemy.orm import Session
from app import sysconfig
from app.models import User
DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
BASE: DiagnosticsLevel = "INFO"
ELEVATION_HOURS = 24
# How long a debug session records at most (app/debugsession.py). Here, beside the other
# consent window, because the start dialog's consent text states it and reads it from `state`.
SESSION_MINUTES = 60
def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
"""The level in force for `user` right now: the consented one until it expires, else BASE."""
level, until = user.diagnostics_level, user.diagnostics_until
if level not in LEVELS or level == BASE or until is None:
return BASE
if until <= (now or datetime.now(timezone.utc)):
return BASE
return level # type: ignore[return-value] — narrowed by the LEVELS check above
def state(db: Session, user: User) -> dict:
"""The `/api/me` view of it: the level in force, when a raised one lapses, and how long stored
log lines are kept — the consent text states that number, so it must be the instance's own."""
level = effective_level(user)
return {
"level": level,
"until": user.diagnostics_until.isoformat() if level != BASE else None,
"retention_days": sysconfig.get_int(db, "log_retention_days"),
"session_minutes": SESSION_MINUTES,
}
def set_level(user: User, level: DiagnosticsLevel) -> None:
"""Record a consent (or its withdrawal). The caller commits. Raising the level again restarts
the clock — each consent is for the next ELEVATION_HOURS, not an extension of an older one."""
if level == BASE:
user.diagnostics_level = None
user.diagnostics_until = None
return
user.diagnostics_level = level
user.diagnostics_until = datetime.now(timezone.utc) + timedelta(hours=ELEVATION_HOURS)
+10
View File
@@ -0,0 +1,10 @@
"""The read-model behind the Home surface ("Mit nézzek most?").
The anti-paralysis answer layer: one hero suggestion, a cross-source resume rail, and the user's own
comfort shelf, all served from our own Postgres with no upstream call at serve time (the same rule
the Plex Discover surface follows).
Layering, mirroring `app/plex/`: `budget` (the time bands) and `cards` (the unified row shape) are
pure; `queries` holds the SQL; `rails`, `pins` and `hero` compose. Nothing here imports from
`app.routes` — `routes/discover.py` is a thin HTTP shell over these.
"""
+62
View File
@@ -0,0 +1,62 @@
"""The time budget — the replacement for "what mood are you in?".
`random-gondolatok` 15/D asked for a mood questionnaire before the recommendations. The same file's
later BA-interview note (#23) says the mood half is not what makes this work: the friction is
deciding at all. A mood→genre mapping would also be invented data — we store no mood signal and never
will — whereas "how much time do you have" is a question the library can actually answer, from a
column we already have.
So the surface asks about TIME, and the answer is deterministic: four bands, no model, no latency.
The band also picks the source for free — ten minutes is a YouTube video, an evening is a movie —
which is exactly the cross-source behaviour the Home surface exists to provide.
The bands overlap on purpose. "I have 30 minutes" should not be answered with a two-minute Short just
because it fits; a lower bound is as much a part of the promise as the upper one.
"""
from typing import Literal
Budget = Literal["short", "medium", "long", "any"]
DEFAULT_BUDGET: Budget = "any"
# budget -> (min_seconds, max_seconds); None = unbounded on that side.
BANDS: dict[str, tuple[int | None, int | None]] = {
"short": (None, 12 * 60), # "10 perc" — a bit of slack over the round number the chip shows
"medium": (8 * 60, 40 * 60), # "30 perc"
"long": (40 * 60, None), # "Egy este"
"any": (None, None), # "Mindegy"
}
def band(budget: str) -> tuple[int | None, int | None]:
"""The (min, max) seconds for a budget. Unknown budgets fall back to the unbounded band rather
than raising: the route validates the vocabulary with a Literal, so anything reaching here with a
bad value is our own bug, and an unfiltered shelf is a better failure than a 500 on the landing
page."""
return BANDS.get(budget, BANDS["any"])
def remaining_seconds(duration_seconds: int | None, position_seconds: int) -> int | None:
"""What is actually LEFT to watch. The question is "how much time do you have", so a 90-minute
film with 8 minutes to go belongs in the ten-minute band, not the evening one. Unknown duration
stays unknown — see `fits`."""
if duration_seconds is None:
return None
return max(0, duration_seconds - max(0, position_seconds))
def fits(seconds: int | None, budget: str) -> bool:
"""Does a runtime fit the budget?
An unknown duration fits only the unbounded band. We cannot promise an item finishes inside ten
minutes when we do not know how long it is, and quietly including it would make the one control
on this page dishonest — the failure mode the house rules call an un-honest state."""
lo, hi = band(budget)
if seconds is None:
return lo is None and hi is None
if lo is not None and seconds < lo:
return False
if hi is not None and seconds > hi:
return False
return True
+142
View File
@@ -0,0 +1,142 @@
"""The unified discover card: one row shape the Home surface can render without knowing which
module it came from.
This is deliberately a THIN mapping, not a new serializer layer. The per-source projections already
exist and are already the shapes the frontend's `VideoCard` and `PlexPosterCard` consume —
`feedquery.serialize` for YouTube, `plex.cards.movie_card`/`show_card`/`leaf_card` for Plex — so each
unified card carries the native payload under `video` / `plex` alongside the handful of fields the
rail chrome itself needs (title, thumbnail, progress, runtime). The Home page therefore renders with
the existing card components rather than a third one, and a field added to a feed row or a Plex card
still gets added in exactly one place. CLAUDE.md's "not wanted" list names a base serializer for
precisely this reason.
`aspect` is the one genuinely new field. A rail has to know whether a tile is a 16:9 still or a 2:3
poster, and that is a property of the ARTWORK, not of the module: a Plex episode ships a landscape
still exactly like a YouTube video does, while a movie and a show ship posters. Emitting it lets the
rail layer stay domain-free instead of branching on `source`.
`subtitle` only ever carries DATA — a channel name, a show title, a year. Nothing here composes an
English phrase: this repo is HU+EN with zero key drift, and a sentence assembled in the backend is a
string no translation file can reach. Anything that needs wording (season counts, "N videó") is left
to the client, which has both the native payload and the translations.
"""
from app.discover.budget import remaining_seconds
from app.catalog.models import Channel
from app.reader.models import Playlist
# Plex artwork is a poster for the things you browse and a landscape still for the things you play.
_POSTER_KINDS = ("movie", "show")
def _base(
*,
source: str, # "youtube" | "plex"
kind: str, # "video" | "movie" | "show" | "episode" | "channel" | "playlist"
id: str,
title: str | None,
aspect: str, # "wide" (16:9) | "poster" (2:3)
subtitle: str | None = None,
thumb: str | None = None,
duration_seconds: int | None = None,
position_seconds: int = 0,
status: str = "new",
) -> dict:
return {
"source": source,
"kind": kind,
"id": id,
"title": title or "",
"subtitle": subtitle,
"thumb": thumb,
"aspect": aspect,
"duration_seconds": duration_seconds,
"position_seconds": position_seconds,
"remaining_seconds": remaining_seconds(duration_seconds, position_seconds),
"status": status,
# Native payloads — at most one is ever set. The client opens the player from these.
"video": None,
"plex": None,
}
def key(card: dict) -> str:
"""The identity of a card ACROSS sources — `<source>:<id>`. A Plex rating_key and a YouTube video
id could in principle collide as bare strings, and the reroll's exclusion list has to be able to
say "not that one" about either."""
return f"{card['source']}:{card['id']}"
def from_video(row: dict) -> dict:
"""`row` is a `feedquery.serialize` dict — the same object the feed grid renders."""
card = _base(
source="youtube",
kind="video",
id=row["id"],
title=row["title"],
aspect="wide",
subtitle=row.get("channel_title"),
thumb=row.get("thumbnail_url"),
duration_seconds=row.get("duration_seconds"),
position_seconds=row.get("position_seconds") or 0,
status=row.get("status") or "new",
)
card["video"] = row
return card
def from_plex(row: dict) -> dict:
"""`row` is a `plex.cards` dict (movie / show / leaf). Its own `type` is the kind."""
kind = row.get("type") or "movie"
if kind == "episode":
subtitle = row.get("show_title")
elif kind == "show":
subtitle = None # season count lives in the native payload; the client words it
else:
subtitle = str(row["year"]) if row.get("year") else None
card = _base(
source="plex",
kind=kind,
id=row["id"],
title=row.get("title"),
aspect="poster" if kind in _POSTER_KINDS else "wide",
subtitle=subtitle,
thumb=row.get("thumb"),
duration_seconds=row.get("duration_seconds"),
position_seconds=row.get("position_seconds") or 0,
status=row.get("status") or "new",
)
card["plex"] = row
return card
def from_channel(ch: Channel, unwatched: int) -> dict:
"""A pinned channel. It is not itself playable — the shelf resolves it to its newest unwatched
video on click (see `pins._shelf_card`) — so the tile advertises what is waiting there instead of a
runtime it does not have."""
card = _base(
source="youtube",
kind="channel",
id=ch.id,
title=ch.title,
aspect="wide",
thumb=ch.thumbnail_url,
)
card["unwatched_count"] = unwatched
return card
def from_playlist(pl: Playlist, item_count: int, thumb: str | None) -> dict:
"""A pinned playlist. `thumb` is its first item's thumbnail — a shelf of blank tiles would defeat
the point of a surface you are supposed to be able to scan at a glance."""
card = _base(
source="youtube",
kind="playlist",
id=str(pl.id),
title=pl.name,
aspect="wide",
thumb=thumb,
)
card["item_count"] = item_count
return card
+109
View File
@@ -0,0 +1,109 @@
"""The hero: the single suggestion the Home surface leads with.
One rule, in one function, because the page's entire promise is that it answers rather than offers.
Three tiers, tried in order:
1. **resume** — something already started whose REMAINING time fits the budget. Almost always the
right answer: it is the thing the user themselves chose most recently, and finishing it costs no
decision at all.
2. **comfort** — a pinned shelf entry that fits. The user's own declared fallback, which beats
anything we could infer.
3. **surprise** — a seeded draw from what is unseen and fits.
That order is the PAGE's answer. "Mást kérek" rotates it (`start`), so each press offers a different
KIND of answer rather than the next row of the same list — the user's call, 2026-09-06, after the
button was measured cycling nine resume cards and never once reaching the shelf.
The tier travels with the card so the client can label it honestly ("Ott hagytad abba" vs "A
polcodról" vs "Meglepetés") instead of guessing from the shape of the payload — a kicker that says
"where you left off" over a film the user has never opened is exactly the kind of quiet lie the house
rules call a dishonest state.
Budgets are matched against REMAINING time throughout (`budget.remaining_seconds`, which equals the
runtime for anything not yet started): with eight minutes left, a 90-minute film is a ten-minute
answer.
"""
from sqlalchemy.orm import Session
from app.discover import cards, pins, rails
from app.discover.budget import fits
from app.models import User
HERO_TIERS = ("resume", "comfort", "surprise")
# How deep the surprise tier draws before giving up. Deliberately deeper than one, so an excluded
# pick moves the tier along instead of emptying it (the shuffle is seeded — the same seed always
# hands back the same order, so without the headroom "already shown" would mean "nothing left").
SURPRISE_DRAW = 12
def pick(
db: Session,
user: User,
budget: str,
seed: int,
*,
resume: list[dict] | None = None,
shelf: list[dict] | None = None,
exclude: set[str] | None = None,
start: int = 0,
lang: str,
) -> dict | None:
"""`{"tier": ..., "card": ...}`, or None when nothing at all fits.
`resume` and `shelf` may be passed pre-computed so the consolidated `/api/discover` response does
not build the same two rails twice; the reroll endpoint passes neither and pays only for what it
reaches.
`exclude` is what makes "Mást kérek" mean something. A new seed alone only re-draws the SURPRISE
tier: the resume tier deterministically returns the most recently progressed thing that fits, so
pressing reroll on a user who has something half-finished handed back the identical card and the
button looked dead (measured in the browser, 2026-09-06). Skipping the cards already shown makes
every press produce a different answer, at every tier.
`start` ROTATES which tier is tried first — 0 keeps the priority order (the page's own first
answer), 1 leads with comfort, 2 with surprise. Exclusion alone was not enough: the resume rail
holds up to 20 rows, so the button walked down it and never reached the shelf or a surprise at
all (measured, 2026-09-06: twelve presses returned nine cards, every one of them `resume`). The
fallthrough is kept, so a press does not come back empty just because the tier it led with has
nothing to offer."""
skip = exclude or set()
def _resume() -> dict | None:
for card in rails.continue_watching(db, user, lang=lang) if resume is None else resume:
playable = rails.playable(db, user.id, card, lang=lang)
# Excluded on the card the USER saw (the show), not only on what it resolves to, so a
# rerolled-past show does not come straight back as its own next episode.
if cards.key(card) in skip or (playable is not None and cards.key(playable) in skip):
continue
if playable is not None and fits(playable["remaining_seconds"], budget):
return playable
return None
def _comfort() -> dict | None:
if shelf is None:
own = pins.shelf(db, user, include_plex=rails.plex_available(db), lang=lang)
else:
own = shelf
for card in pins.playables(own):
if cards.key(card) not in skip and fits(card["remaining_seconds"], budget):
return card
return None
def _surprise() -> dict | None:
# A wider draw than the one card we need: the shuffle is seeded, so without the headroom an
# excluded pick would leave the tier empty rather than move past it.
for card in rails.surprise(db, user, budget, seed, limit=SURPRISE_DRAW, lang=lang):
if cards.key(card) not in skip:
return card
return None
tiers = {"resume": _resume, "comfort": _comfort, "surprise": _surprise}
offset = start % len(HERO_TIERS)
for tier in HERO_TIERS[offset:] + HERO_TIERS[:offset]:
card = tiers[tier]()
if card is not None:
return {"tier": tier, "card": card}
return None
+298
View File
@@ -0,0 +1,298 @@
"""The comfort shelf: pin a thing you can always fall back to, and have it be one click from playing.
Five kinds are pinnable — a video, a YouTube channel, a Plex show, a Plex movie, a playlist — because
comfort media is almost always "put on X", a series or a channel, and only sometimes one particular
video. `models.ComfortPin` carries that union as five nullable FKs under a `num_nonnulls(...) = 1`
check; this module is where the union is spoken about in one vocabulary, so no call site invents a
second one.
Every shelf card carries a resolved `next`: the thing that actually plays when you click the tile —
a show's in-progress-or-next episode, a channel's newest unwatched video, a playlist's first
unwatched item. That buys the property the whole surface exists for: one click, no intermediate page,
no second decision. Channels and playlists are resolved for the whole shelf at once (`_Batch`), so the
cost does not grow a query or two per pin.
Domain module, so nothing here raises HTTP. `resolve_ref` returns None for "no such thing, or not
yours" and the route turns that into its 404 — core never imports routes.
"""
from dataclasses import dataclass, field
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.discover import cards, queries
from app.models import User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.reader.models import ComfortPin, Playlist
from app.plex import rollup
from app.plex.cards import SHOW_NEW, leaf_card, show_card
from app.plex.lookups import by_rating_key
from app.userscope import owned_by, owned_row
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
PIN_COLUMNS: dict[str, str] = {
"video": "video_id",
"channel": "channel_id",
"plex_item": "plex_item_id",
"plex_show": "plex_show_id",
"playlist": "playlist_id",
}
PIN_KINDS = tuple(PIN_COLUMNS)
def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, object] | None:
"""Translate the client's (kind, ref) into the (column, value) the row stores, or None.
The client speaks the ids the rest of the API speaks: a video id, a channel id, a Plex
`rating_key`, a playlist id. Plex's internal integer keys never leave the backend, so the
translation belongs here rather than in five call sites.
The COLUMN half of every answer comes from `PIN_COLUMNS`; only the LOOKUP differs per kind
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before a
review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
all — a constant that called itself the one vocabulary while nothing spoke it."""
col = PIN_COLUMNS.get(kind)
if col is None:
return None
if kind == "video":
return (col, ref) if db.get(Video, ref) is not None else None
if kind == "channel":
return (col, ref) if db.get(Channel, ref) is not None else None
if kind == "plex_item":
it = by_rating_key(db, PlexItem, ref)
return (col, it.id) if it is not None else None
if kind == "plex_show":
sh = by_rating_key(db, PlexShow, ref)
return (col, sh.id) if sh is not None else None
try:
pid = int(ref)
# silent-ok: validating a reference the client supplied. "Not an integer" is answered as "no
# such pin target", which is the same answer this function gives for one that does not exist.
except ValueError:
return None
# A playlist is per-user, unlike the other four (which are shared catalog rows whose PRIVACY
# lives in the state tables). Someone else's list must read as "no such thing", not as a 403 —
# the same reason `owned_or_404` 404s rather than 403s.
pl = owned_row(db, Playlist, pid, user)
return (col, pl.id) if pl is not None else None
def reorder(db: Session, user: User, ids: list[int]) -> None:
"""Put the listed pins in the listed order, WITHIN the slots they already hold.
The client sends the order it can SEE, which is not always the whole shelf: with Plex off the Home
rail hides the Plex pins, and a stale tab may not know a pin added elsewhere. Numbering only the
listed ids 0..n-1 collided with every pin left out — two tiles on one position, and the hidden
ones jumped about when they came back. So the pins left out keep their slots, and the listed ones
are dealt into the slots the listed set occupies. Ids that are not this user's, or repeated, are
ignored rather than failing a stale replay wholesale. Flushes; the route commits."""
# Locked: the rows it leaves alone are part of the answer too. Two overlapping reorders each
# skipped the rows that already matched in their own snapshot and interleaved into two pins on
# one slot; the second now waits and deals from the first one's result. Locked in id order and
# sorted HERE: with `ORDER BY position … FOR UPDATE` Postgres sorts before it locks, so the one
# that waited got the fresh rows in its stale order and dealt the pins it left out back to their
# old slots — and one global lock order means two reorders cannot deadlock.
locked = db.execute(
select(ComfortPin)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
current = sorted(locked, key=lambda p: (p.position, p.id))
owned = {p.id for p in current}
listed: list[int] = []
for i in ids:
if i in owned and i not in listed:
listed.append(i)
chosen = set(listed)
queue = iter(listed)
order = [next(queue) if p.id in chosen else p.id for p in current]
by_id = {p.id: p for p in current}
for position, pin_id in enumerate(order):
if by_id[pin_id].position != position:
by_id[pin_id].position = position
db.flush()
def list_pins(db: Session, user: User) -> list[ComfortPin]:
return list(
db.execute(
select(ComfortPin)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.position.asc(), ComfortPin.id.asc())
)
.scalars()
.all()
)
def next_position(db: Session, user: User) -> int:
"""New pins land at the end of the shelf.
`max(position) + 1`, not the row count. Positions are never renumbered on removal — `remove_pin`
deletes the row, and every FK cascade silently removes a pin when the thing it points at goes
away — so past the first hole a count under-counts and would drop the new pin into the MIDDLE of
the shelf, or onto a position another pin already holds (nothing in the schema forbids that).
Same idiom as a new playlist's place in the rail (`reader.service`)."""
return (
db.scalar(
select(func.coalesce(func.max(ComfortPin.position), -1)).where(
owned_by(ComfortPin.user_id, user.id)
)
)
+ 1
)
def own_pin(db: Session, user: User, pin_id: int) -> ComfortPin | None:
return owned_row(db, ComfortPin, pin_id, user)
def pin_of(db: Session, user: User, column: str, value: object) -> ComfortPin | None:
"""The user's pin of this thing (`column`/`value` from `resolve_ref`), if it is on the shelf."""
return db.execute(
select(ComfortPin).where(
owned_by(ComfortPin.user_id, user), getattr(ComfortPin, column) == value
)
).scalar_one_or_none()
def add(db: Session, user: User, column: str, value: object) -> ComfortPin:
"""A new pin at the end of the shelf. A second pin of the same thing (another tab, at the same
moment) raises `IntegrityError` at the flush: the unique constraint is the guard."""
pin = ComfortPin(user_id=user.id, position=next_position(db, user), **{column: value})
db.add(pin)
db.flush()
return pin
def remove(db: Session, pin: ComfortPin) -> None:
db.delete(pin)
db.flush()
@dataclass
class _Batch:
"""What the shelf resolves for every pinned channel and playlist in one go."""
channels: dict[str, Channel] = field(default_factory=dict)
channel_summary: dict[str, tuple[int, dict | None]] = field(default_factory=dict)
playlists: dict[int, Playlist] = field(default_factory=dict)
playlist_summary: dict[int, tuple[int, dict | None]] = field(default_factory=dict)
@classmethod
def load(cls, db: Session, user: User, pins: list[ComfortPin]) -> "_Batch":
ch_ids = [p.channel_id for p in pins if p.channel_id is not None]
pl_ids = [p.playlist_id for p in pins if p.playlist_id is not None]
b = cls()
if ch_ids:
b.channels = {
c.id: c for c in db.scalars(select(Channel).where(Channel.id.in_(ch_ids)))
}
b.channel_summary = queries.channels_summary(db, user, ch_ids)
if pl_ids:
b.playlists = {
p.id: p for p in db.scalars(select(Playlist).where(Playlist.id.in_(pl_ids)))
}
b.playlist_summary = queries.playlists_summary(db, user, pl_ids)
return b
def _shelf_card(db: Session, user: User, pin: ComfortPin, batch: _Batch, lang: str) -> dict | None:
"""One pin as a unified card with its resolved `next`. None when the pinned row has gone (the FK
cascade removes the pin itself, so this is only the window between a delete and a refetch)."""
card: dict | None = None
nxt: dict | None = None
if pin.video_id is not None:
row = queries.yt_video_row(db, user, pin.video_id)
if row is None:
return None
card = cards.from_video(row)
nxt = dict(card) # a video plays itself
elif pin.channel_id is not None:
ch = batch.channels.get(pin.channel_id)
if ch is None:
return None
count, row = batch.channel_summary.get(ch.id, (0, None))
card = cards.from_channel(ch, count)
nxt = cards.from_video(row) if row else None
elif pin.plex_item_id is not None:
it = db.get(PlexItem, pin.plex_item_id)
if it is None:
return None
st = rollup.item_state_map(db, user.id, [it.id]).get(it.id)
card = cards.from_plex(leaf_card(it, st, lang=lang))
nxt = dict(card) # a movie or a single episode plays itself
elif pin.plex_show_id is not None:
sh = db.get(PlexShow, pin.plex_show_id)
if sh is None:
return None
state = rollup.state_map(db, user.id, [sh.id]).get(sh.id, SHOW_NEW)
card = cards.from_plex(show_card(sh, state, lang=lang))
ep = queries.next_episode(db, user.id, sh.id, lang=lang)
nxt = cards.from_plex(ep) if ep else None
elif pin.playlist_id is not None:
pl = batch.playlists.get(pin.playlist_id)
if pl is None:
return None
count, row = batch.playlist_summary.get(pl.id, (0, None))
card = cards.from_playlist(pl, count, row.get("thumbnail_url") if row else None)
nxt = cards.from_video(row) if row else None
if card is None:
return None
# A COPY, never the card itself: a self-playing pin (a video, a movie) would otherwise make
# `card["next"] is card`, which serializes as a circular reference and 500s the whole shelf. The
# copy is taken before `pin_id`/`next` are attached, so `next` stays a plain playable card.
card["pin_id"] = pin.id
card["next"] = nxt
return card
_PLEX_COLUMNS = ("plex_item_id", "plex_show_id")
def shelf(db: Session, user: User, *, include_plex: bool = True, lang: str) -> list[dict]:
"""The whole shelf, in the user's own order.
`include_plex=False` is the Home surface with the deployment's Plex toggle off (the owner's call,
2026-09-19): the rails beside it go YouTube-only, and a Plex tile there would offer a Play that
lands on a gated module. The pins themselves are kept — the pin menus still read the full shelf
(`GET /pins`), and switching Plex back on brings the tiles back in their old slots
(`reorder` keeps a hidden pin's place)."""
out = []
pins = list_pins(db, user)
if not include_plex:
pins = [p for p in pins if all(getattr(p, c) is None for c in _PLEX_COLUMNS)]
batch = _Batch.load(db, user, pins)
for pin in pins:
card = _shelf_card(db, user, pin, batch, lang)
if card is not None:
out.append(card)
return out
def playables(shelf_cards: list[dict]) -> list[dict]:
"""The `next` of every shelf entry that has one — what the hero may draw from. A pinned channel
with nothing unwatched left, or an empty playlist, simply is not an answer right now.
Each carries `pinned_from`: the pin it was resolved from. A show's next EPISODE is not itself
pinned, so without it the hero's menu looked the episode up, missed, and offered "Pin" under a
kicker that says "From your shelf" — and taking it inserted a second pin. With it, the hero's
pin row acts on the shelf entry the card came from (the owner's call, 2026-09-19: "Unpin")."""
return [
{
**c["next"],
"pinned_from": {k: c[k] for k in ("source", "kind", "id", "title")},
}
for c in shelf_cards
if c.get("next")
]
+372
View File
@@ -0,0 +1,372 @@
"""The row fetchers behind the Home surface — one place where "what is startable right now" is
expressed as SQL, for both sources.
Two rules shape this module:
**Reuse the query layer, not the route.** The YouTube half calls `feedquery.filtered_query` /
`sort_keys`, so the Home surface inherits the feed's own definitions of subscribed, blocked,
explore-visible and in-progress for free, and cannot drift from them. The Plex half calls
`plex.rollup` and the `plex.cards` serializers rather than `plex/discover.py`: that read-model
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
sidebar and wrong for a landing page that does not.
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
here takes the feed's seeded-md5 idiom instead of `random()`.
"""
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session, aliased
from app.discover.budget import band
from app.feedquery import (
feed_columns,
filtered_query,
not_blocked_cond,
order_by,
serialize,
sort_keys,
)
from app.models import User
from app.plex.models import PlexItem, PlexShow, PlexState
from app.catalog.models import Channel, Video
from app.reader.models import PlaylistItem, VideoState
from app.plex import rollup
from app.plex.cards import display_title, leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
from app.userscope import owned_by
# How many rows a single rail may carry — Home's rails and the Plex module's Discover rails alike
# (`plex/discover.py` imports it): the two surfaces should feel like the same app, and a rail
# is a rail. One definition, because a comment was all that held two copies in step.
RAIL_LIMIT = 20
# The feed parameters a Home rail runs with. Spelled out rather than borrowed from the route's
# `_feed_params` defaults, because those are an HTTP contract that may change for the feed's own
# reasons; the landing page's scope should not move with them silently.
_FEED_BASE: dict = {
"tags": [],
"tag_modes": {},
"untagged": False,
"channel_ids": [],
"q": None,
"min_duration": None,
"max_duration": None,
"max_age_days": None,
"published_after": None,
"published_before": None,
"show_normal": True,
"include_shorts": False,
"include_live": False,
"show": "unwatched",
"scope": "my",
"library_source": "organic",
}
def _seeded(col, seed: int):
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
return func.md5(func.concat(col, str(seed)))
# ── YouTube ───────────────────────────────────────────────────────────────────────────────────────
def yt_in_progress(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
"""Started-but-not-finished videos, most recently progressed first, as (progressed_at, row).
The predicate is `feedquery`'s `show="in_progress"` verbatim (a stored resume position, status not
watched/hidden) — but the query is built here rather than through `filtered_query` because this
rail wants an ordering by `progress_updated_at`, and that column lives on the state alias the feed
query keeps private. Driving from `video_states` is also the faster plan: it is a per-user table of
a few hundred rows, against a catalog of many thousands.
Building it here means the feed's OTHER predicates have to be chosen deliberately rather than
inherited. `not_blocked_cond` is shared, because "never show me this channel" outranks "you
started it" — blocking leaves the state row untouched, so without this the video the user just
banned heads the resume rail. The feed's SCOPE clauses (subscribed / via_explore accessibility)
are deliberately NOT applied, for the reason `yt_video_row` gives: something you have already
started must stay resumable even after you unsubscribe or an exploration expires."""
st = aliased(VideoState)
q = (
select(*feed_columns(user, st), st.progress_updated_at.label("progressed_at"))
.join(Channel, Channel.id == Video.channel_id)
.join(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(
st.position_seconds > 0,
st.status.notin_(("watched", "hidden")),
Video.unavailable_since.is_(None),
not_blocked_cond(user),
)
.order_by(st.progress_updated_at.desc().nullslast())
.limit(limit)
)
return [(r.progressed_at, serialize(r)) for r in db.execute(q).all()]
def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) -> list[dict]:
"""Unwatched videos from the user's own feed scope, in seeded-shuffle order, inside the budget.
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
actually a good response to, and the feed's own default keeps them out otherwise."""
lo, hi = band(budget)
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("shuffle", seed, rank_expr)
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
# go: the resume tier has already had its chance at them, and it may have passed precisely because
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
out = [serialize(r) for r in rows]
return [r for r in out if not r["position_seconds"]][:limit]
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
"""One video as a feed row, with this user's watch state.
Built directly rather than through `filtered_query` on purpose: a PINNED video must keep working
even when it falls outside the feed's scope — the user unsubscribed from the channel, or blocked
it, or the video only ever arrived through a search. The shelf is the surface you fall back to;
it silently losing an entry because a filter moved is exactly the failure it must not have."""
st = aliased(VideoState)
row = db.execute(
select(*feed_columns(user, st))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(Video.id == video_id)
).first()
return serialize(row) if row is not None else None
def channels_summary(
db: Session, user: User, channel_ids: list[str]
) -> dict[str, tuple[int, dict | None]]:
"""For each pinned channel: (unwatched count, its newest unwatched video) — the tile's only honest
promise, since a channel has no runtime of its own, and what "put on that channel" resolves to.
ONE query for the whole shelf, not two per channel: the count and the first row per channel are
window functions over the same filtered set. `library_source="all"`: a pinned channel means the
whole channel, including videos that only ever arrived through a search — the feed's "organic"
default is a feed-scoping choice, not a statement about what belongs to a channel."""
if not channel_ids:
return {}
params = dict(_FEED_BASE, channel_ids=list(channel_ids), scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
newest = order_by(sort_keys("newest", 0, rank_expr))
per_channel = {"partition_by": Video.channel_id}
sub = query.add_columns(
func.row_number().over(**per_channel, order_by=newest).label("_rn"),
func.count().over(**per_channel).label("_total"),
).subquery()
rows = db.execute(select(sub).where(sub.c._rn == 1)).all()
return {r.channel_id: (int(r._total), serialize(r)) for r in rows}
def playlists_summary(
db: Session, user: User, playlist_ids: list[int]
) -> dict[int, tuple[int, dict | None]]:
"""For each pinned playlist: (item count, the row to play next) — its first not-yet-watched item,
or its first item once the whole list is watched. The same row doubles as the tile's artwork, so
a shelf of playlists is scannable instead of a row of blank boxes. One query for the whole shelf."""
if not playlist_ids:
return {}
st = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, st), PlaylistItem.playlist_id.label("_playlist"))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(PlaylistItem.playlist_id.in_(playlist_ids))
.order_by(PlaylistItem.playlist_id, PlaylistItem.position.asc())
).all()
grouped: dict[int, list] = {}
for r in rows:
grouped.setdefault(r._playlist, []).append(r)
out: dict[int, tuple[int, dict | None]] = {}
for pid, items in grouped.items():
nxt = next((r for r in items if (r.status or "new") not in ("watched", "hidden")), items[0])
out[pid] = (len(items), serialize(nxt))
return out
# ── Plex ──────────────────────────────────────────────────────────────────────────────────────────
def _unseen_movie_conds(uid: int):
"""A movie the user has neither finished nor hidden. A LEFT-JOIN-IS-NULL would also drop rows that
merely carry a resume position, which are exactly the ones Continue watching wants back."""
seen = (
select(1)
.select_from(PlexState)
.where(
PlexState.item_id == PlexItem.id,
owned_by(PlexState.user_id, uid),
or_(
PlexState.status.in_(("watched", "hidden")),
# Started counts as "not a surprise" too — see the note in `yt_surprise`.
PlexState.position_seconds > 0,
),
)
.exists()
)
return ~seen
def plex_in_progress(
db: Session, uid: int, limit: int = RAIL_LIMIT, *, lang: str
) -> list[tuple[object, dict]]:
"""In-progress movies + partially-watched shows as (progressed_at, card).
Mirrors `plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
NULL status still counts as started, so `notin_` alone would drop it), and the same shared
`rollup.agg_subquery` + `apply_show_state_filter` for shows, so the two rails cannot disagree on
what "in progress" means. What it does NOT inherit is that route's `LibraryFilters` narrowing:
this surface has no Plex sidebar to read a filter from."""
movie_libs, show_libs = content_libs(db)
out: list[tuple[object, dict]] = []
if movie_libs:
mq = (
select(PlexItem, PlexState)
.join(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
PlexState.position_seconds > 0,
or_(
PlexState.status.is_(None),
PlexState.status.notin_(("watched", "hidden")),
),
)
.order_by(PlexState.progress_updated_at.desc().nullslast())
.limit(limit)
)
for it, st in db.execute(mq).all():
out.append((st.progress_updated_at, movie_card(it, st, lang=lang)))
if show_libs:
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at, prog.c.total, prog.c.watched, prog.c.inprog).join(
prog, prog.c.show_id == PlexShow.id
),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_libs))
for sh, resumed_at, total, watched, inprog in db.execute(
sq.order_by(prog.c.resumed_at.desc().nullslast()).limit(limit)
).all():
out.append(
(resumed_at, show_card(sh, rollup.show_state(total, watched, inprog), lang=lang))
)
return out
def plex_surprise(
db: Session, uid: int, budget: str, seed: int, limit: int, *, lang: str
) -> list[dict]:
"""Unseen playable movies inside the budget, in seeded-shuffle order.
Movies only. A show is not a thing you can start playing, and resolving every candidate show to
its next episode just to draw three cards would be a query per row for a rail whose whole value is
that it answers instantly — shows reach this surface through Continue watching and the comfort
shelf, which is where "put the next episode on" actually belongs.
No `playable` filter, matching the Plex surprise rail: `PlexItem.playable` is a playback CLASS
("direct" / "remux" / "transcode"), not a yes/no, and NULL there means "not probed yet" rather than
"will not play". Excluding un-probed items would quietly hide a fresh library from the one surface
that is supposed to have an answer. The class travels on the card, so the client still shows it."""
movie_libs, _ = content_libs(db)
if not movie_libs:
return []
lo, hi = band(budget)
q = (
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
_unseen_movie_conds(uid),
)
.order_by(_seeded(PlexItem.rating_key, seed))
.limit(limit)
)
if lo is not None:
q = q.where(PlexItem.duration_s >= lo)
if hi is not None:
q = q.where(PlexItem.duration_s <= hi)
items = db.execute(q).scalars().all()
states = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, states.get(it.id), lang=lang) for it in items]
# Watch states that take an episode out of the running for "what would play next".
_DONE = ("watched", "hidden")
def next_episode(db: Session, uid: int, show_id: int, *, lang: str) -> dict | None:
"""What "put that show on" means: the episode in progress, else the first unwatched one, else the
first — over the same `(season, episode)` broadcast ordering `detail.py::_episode_nav` uses.
Three ordered `LIMIT 1` probes rather than one scan picked apart in Python. The Python version
loaded EVERY episode of the show to choose one of them — 236 rows for Friends, 227 for JAG — which
the comfort shelf could just about afford (a handful of hand-picked pins) and the resume rail
could not, since it resolves a show per rail row on the landing page.
Returned as a `leaf_card` so it carries the show title the episode row itself does not."""
asc = (
PlexItem.season_number.asc().nullsfirst(),
PlexItem.episode_number.asc().nullsfirst(),
)
base = (
select(PlexItem, PlexState)
.outerjoin(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
.limit(1)
)
# In progress — the LAST one in broadcast order, which is `plex.rollup.rollup`'s `inprog[-1]`,
# i.e. what the show page's own Resume button plays. Two episodes of one show are in progress at
# once more often than it sounds (`watch_sync._reconcile_inprogress_movies` documents that a
# scrub to the end leaves a stale `status='new', position>0` row until the daily reconcile), and
# taking opposite ends of that set is how one show came to have two different "continue" answers:
# the Home tile ranked it by its LATEST touch and then offered the EARLIEST started episode.
# `position_seconds > 0` already requires the state row to exist, so no null guard.
pick = db.execute(
base.where(PlexState.position_seconds > 0, PlexState.status.notin_(_DONE)).order_by(
PlexItem.season_number.desc().nullslast(),
PlexItem.episode_number.desc().nullslast(),
)
).first()
# Unwatched — INCLUDING an episode with no state row at all, which is the common case and the one
# a bare `status NOT IN (...)` silently drops (NULL is not false, it is unknown, so the row fails
# the filter).
pick = (
pick
or db.execute(
base.where(or_(PlexState.id.is_(None), PlexState.status.notin_(_DONE))).order_by(*asc)
).first()
)
# Fully watched show: start it again from the top.
pick = pick or db.execute(base.order_by(*asc)).first()
if pick is None:
return None
ep, st = pick
show = db.get(PlexShow, show_id)
return leaf_card(
ep,
st,
show_title=display_title(show, lang) if show else None,
show_rk=show.rating_key if show else None,
lang=lang,
)
+105
View File
@@ -0,0 +1,105 @@
"""The Home surface's rails: cross-source lists, assembled from `queries.py` and emitted as unified
cards.
The whole point of this module is that a rail is ONE list, not a YouTube list beside a Plex list. Two
side-by-side rails would just move the choice the user could not make ("which half do I look at?") one
level up, which is the opposite of what this epic is for.
"""
from sqlalchemy.orm import Session
from app import sysconfig
from app.discover import cards, queries
from app.discover.queries import RAIL_LIMIT
from app.models import User
from app.plex.models import PlexShow
from app.plex.lookups import by_rating_key
from app.plex.rollup import NO_TS
def plex_available(db: Session) -> bool:
"""Plex is a deployment-level toggle, not a per-user one (see `routes/me.py`). With it off, the
Home rails are YouTube-only and say nothing about Plex."""
return sysconfig.get_bool(db, "plex_enabled")
def playable(db: Session, uid: int, card: dict, *, lang: str) -> dict | None:
"""The thing that would actually START from this card — itself for anything with a runtime, and
for a Plex SHOW the episode that is next (in progress, else the first unwatched, else the first).
ONE function, because three surfaces need the same answer and a second implementation is how the
shelf and the rail came to disagree in the first place: the hero calls it to decide what it is
offering, `continue_watching` calls it to fill each show tile's `next`, and the comfort shelf
resolves the same way in `pins`. None when the show has no episodes mirrored yet.
A card that already carries a resolved `next` (a rail row) is taken at its word rather than
re-queried — the hero reads the rail it was handed."""
if not (card.get("source") == "plex" and card.get("kind") == "show"):
return card
if "next" in card:
return card["next"]
sh = by_rating_key(db, PlexShow, card["id"])
if sh is None:
return None
ep = queries.next_episode(db, uid, sh.id, lang=lang)
return cards.from_plex(ep) if ep else None
def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT, *, lang: str) -> list[dict]:
"""One resume rail across both sources, most recently progressed first.
A Plex SHOW enters as a show CARD — that is what the Plex rail shows, and it is the artwork the
user recognises — but it carries `next`: the episode that would actually start. Without it a
click on a half-watched series landed on the show page with six seasons to choose between, which
is the decision this whole surface exists to remove (the comfort shelf and the hero already
resolved; only this rail did not). The client plays `next ?? card`, so the tile stays a poster and
the click stays one click."""
rows: list[tuple[object, dict]] = [
(ts, cards.from_video(r)) for ts, r in queries.yt_in_progress(db, user, limit)
]
if plex_available(db):
rows += [
(ts, cards.from_plex(c))
for ts, c in queries.plex_in_progress(db, user.id, limit, lang=lang)
]
rows.sort(key=lambda r: r[0] or NO_TS, reverse=True)
out = [card for _, card in rows[:limit]]
# Resolve AFTER the slice: only the rows that survive it are worth a query.
for card in out:
if card["kind"] == "show":
card["next"] = playable(db, user.id, card, lang=lang)
return out
def surprise(
db: Session, user: User, budget: str, seed: int, limit: int = 3, *, lang: str
) -> list[dict]:
"""The hero's draw pool: unseen things that fit the budget, from both sources under ONE seed.
Interleaved rather than concatenated — the same alternation `plex/discover.py` uses for its
movie/show split — so a short pool never comes back all-YouTube or all-Plex just because one side
happened to be queried first.
WHICH source leads is decided by the seed, not by a hardcoded order. The hero draws a pool of ONE,
so a fixed order would mean the Plex library could never answer "egy este" while a single YouTube
video also fit — the reroll would look broken and the cross-source promise would quietly be a
YouTube promise. Parity on the same seed that orders the rows keeps it deterministic: the same
seed always gives the same answer, and pressing "Mást kérek" moves both halves together."""
halves = [
(cards.from_video, queries.yt_surprise(db, user, budget, seed, limit)),
(
cards.from_plex,
queries.plex_surprise(db, user.id, budget, seed, limit, lang=lang)
if plex_available(db)
else [],
),
]
if seed % 2:
halves.reverse()
out: list[dict] = []
for i in range(max(len(rows) for _, rows in halves)):
for to_card, rows in halves:
if i < len(rows):
out.append(to_card(rows[i]))
return out[:limit]
+161 -7
View File
@@ -11,9 +11,22 @@ Resolution works both before extraction (from source_kind / URL host) and after
`extractor_key` in the info dict), so a non-YouTube "url" job still lands on the right adapter.
"""
import re
from urllib.parse import urlparse
import yt_dlp
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
from app.downloads.stated import StatedFailure
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
# below and the failure catalog (errors.py) decide on it, so the wording exists once.
FACEBOOK_PARSE_WALL = "cannot parse data"
# A source's own verdicts on a link, raised from `resolve_url`. errors.py classifies them; it imports
# them from here because this module must not import errors (errors → credentials → adapters).
POST_WITHOUT_VIDEO_MESSAGE = "No single video was found in this post."
POST_NOT_SHOWN_MESSAGE = "Facebook did not show this post to the signed-in session."
class SourceAdapter:
@@ -22,9 +35,21 @@ class SourceAdapter:
key = "generic"
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
def client_sets(self) -> list:
"""Player-client sets to attempt, in order. A generic source needs no client override, so a
single default attempt (`[None]`) — the per-client fallback is a YouTube-only concept."""
def resolve_url(self, url: str, ydl_opts: dict) -> str:
"""The link to hand yt-dlp for `url`, resolved BEFORE the first attempt. `ydl_opts` are the
download's own options (cookie, timeouts, address family) for any page this needs to read.
Raises a RuntimeError carrying one of this module's verdicts when the page read shows the
download cannot go ahead: the content is there but holds no single video
(`POST_WITHOUT_VIDEO_MESSAGE`), or the page is not the content at all — the login page a
refused cookie gets (`POST_NOT_SHOWN_MESSAGE`), never the no-video verdict. A failed page read
is raised as itself, so it is classified like any other download error. Generic: `url`."""
return url
def client_sets(self, authed: bool = False) -> list:
"""Attempts to make, in order — each entry is handed back to `ydl_opts_extra`, and the next one
is tried when `retriable` says so. A generic source makes a single default attempt (`[None]`).
`authed` is whether the acting user's cookie file goes with the download."""
return [None]
def ydl_opts_extra(self, clients) -> dict:
@@ -37,6 +62,10 @@ class SourceAdapter:
return False
# --- metadata -----------------------------------------------------------------------------
def title(self, info: dict) -> str | None:
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
return info.get("title")
def uploader_url(self, info: dict) -> str | None:
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
@@ -65,12 +94,47 @@ class YouTubeAdapter(SourceAdapter):
"Requested format is not available",
"Only images are available",
"Sign in to confirm",
# Measured 2026-09-16: a gated video whose EXTRACTION succeeds and whose media fetch is then
# refused by the CDN ("unable to download video data: HTTP Error 403"). The wall moves
# between the player response and the stream, so this is the same per-client flakiness one
# step later — and it used to end the job on attempt 1, because the text classifies as
# `network` and nothing here matched it. The format URLs belong to the client that minted
# them, so the next client set is a real second chance.
"unable to download video data",
)
# Measured in the worker against a REAL account cookie (2026-09-16, yt-dlp 2026.07.04): with
# `player_client=android_vr` yt-dlp logs `Skipping client "android_vr" since it does not support
# cookies`, drops it, and the request that is left with no client fails with "Requested format is
# not available" — which is already a retriable marker above, so the cost is a wasted attempt
# rather than a hard failure. Our primary set is tv_simply ALONE, which cannot carry cookies either
# (measured on yt-dlp 2026.08.19: `SUPPORTS_COOKIES` False), so without this filter every
# signed-in download would burn its first attempt that way before reaching the POT-backed web
# fallback the cookie exists for. The list is yt-dlp's own INNERTUBE_CLIENTS[...]
# ["SUPPORTS_COOKIES"], pinned by test_adapters against the installed yt-dlp, so a version bump
# that moves the table turns the gate red instead of silently un-authing every YouTube download.
_COOKIE_CAPABLE = frozenset(
{
"mweb",
"tv",
"tv_downgraded",
"web",
"web_creator",
"web_embedded",
"web_music",
"web_safari",
}
)
# Where a signed-in download lands when the admin's knobs hold no cookie-capable client at all.
# Both clients in ONE set: yt-dlp queries each and merges their formats, which is the cover for
# the per-client flakiness that the two-tier ladder gives a signed-out download.
_AUTHED_DEFAULT = ("web_safari", "web")
def client_sets(self) -> list:
return [
c for c in (settings.player_client_list, settings.player_client_fallback_list) if c
] or [None]
def client_sets(self, authed: bool = False) -> list:
sets = [c for c in (settings.player_client_list, settings.player_client_fallback_list) if c]
if authed:
kept = [[c for c in s if c in self._COOKIE_CAPABLE] for s in sets]
return [s for s in kept if s] or [list(self._AUTHED_DEFAULT)]
return sets or [None]
def ydl_opts_extra(self, clients) -> dict:
# == --extractor-args "youtube:player_client=…;youtubepot-bgutilhttp:base_url=…" (list values).
@@ -93,6 +157,63 @@ class YouTubeAdapter(SourceAdapter):
class FacebookAdapter(SourceAdapter):
key = "facebook"
# Measured (yt-dlp 2026.07.04): with the user's cookie, some reels answer a plain request
# with "Cannot parse data" and parse only when the request looks like Chrome. Impersonation is a
# SECOND attempt behind that one error, not the default: the links that already download with the
# cookie keep their proven path, and only the ones stuck at the wall go through curl_cffi. Without
# a cookie the same error is the login wall, which no disguise gets past.
_IMPERSONATE = "impersonate"
# A post page is ~4–5 MB of inline JSON; anything far past that is not a post page.
_POST_PAGE_MAX_BYTES = 16 * 1024 * 1024
def resolve_url(self, url: str, ydl_opts: dict) -> str:
# Measured: given a group post's link, yt-dlp either finds no formats or — worse —
# downloads ANOTHER video from the page's feed data. The post's own video is the one Video node
# in the page scripts that carry the post id; its watch link downloads (plain first, then the
# Chrome retry above). Needs the cookie: signed out, the page is the login wall.
post_id = facebook_group_post_id(url)
if post_id is None or not ydl_opts.get("cookiefile"):
return url
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
page = raw.decode("utf-8", "replace")
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise StatedFailure(POST_NOT_SHOWN_MESSAGE)
video_id = facebook_post_video_id(page, post_id)
if video_id is None:
# Never fall back to the post link: that is exactly where the wrong video comes from.
raise StatedFailure(POST_WITHOUT_VIDEO_MESSAGE)
return f"https://www.facebook.com/watch/?v={video_id}"
def client_sets(self, authed: bool = False) -> list:
return [None, self._IMPERSONATE] if authed else [None]
def ydl_opts_extra(self, clients) -> dict:
if clients == self._IMPERSONATE:
return {"impersonate": ImpersonateTarget.from_str("chrome")}
return {}
def retriable(self, error_text: str) -> bool:
return FACEBOOK_PARSE_WALL in error_text.lower()
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
# signed-out read titles the page with that same text behind the engagement counts, cut short.
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
def title(self, info: dict) -> str | None:
raw = info.get("title")
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
return raw
line = next(
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
)
# The whole first line, never cut here with an ellipsis: normalize_title's NFKC turns "…" into
# "..." and then collapses it to ".", which read as a finished sentence ("… live at."). The
# stored title is unbounded; display_name and the file name clip it themselves.
return line or raw # no description either (the 18+ reel): nothing better on the page
def uploader_url(self, info: dict) -> str | None:
# Facebook gives neither channel_url nor uploader_url, but exposes a numeric `uploader_id`
@@ -107,6 +228,39 @@ class FacebookAdapter(SourceAdapter):
return True
_FB_GROUP_POST_PATH = re.compile(r"^/groups/[^/]+/(?:permalink|posts)/(\d+)/?$")
_SCRIPT_BLOCK = re.compile(r"<script[^>]*>(.*?)</script>", re.S)
_VIDEO_NODE = re.compile(r'"__typename":"Video","id":"(\d+)"')
def _quoted(post_id: str) -> str:
# How the page JSON carries the id: quoted, so a longer id that merely starts with it never counts.
return f'"{post_id}"'
def facebook_group_post_id(url: str) -> str | None:
"""The post id of a Facebook group post link (`/groups/<group>/permalink|posts/<id>/`), else None."""
parsed = urlparse(url)
if not _host_is(parsed.netloc.lower(), "facebook.com"):
return None
match = _FB_GROUP_POST_PATH.match(parsed.path)
return match.group(1) if match else None
def facebook_post_video_id(page: str, post_id: str) -> str | None:
"""The id of the one video attached to `post_id`, read from its page: the Video nodes of the script
blocks that carry the post id (the page's other ~10 videos — feed, reels — sit in other blocks).
None for no video AND for more than one: guessing between two would be the wrong-video bug again."""
quoted = _quoted(post_id)
ids = {
vid
for block in _SCRIPT_BLOCK.findall(page)
if quoted in block
for vid in _VIDEO_NODE.findall(block)
}
return ids.pop() if len(ids) == 1 else None
class RedditAdapter(SourceAdapter):
key = "reddit"
+29 -9
View File
@@ -11,24 +11,30 @@ Run once per environment:
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
"""
import logging
from urllib.parse import urlsplit
import yt_dlp
from sqlalchemy import select
from app import logs
from app.config import settings
from app.db import SessionLocal
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, service
from app.models import MediaAsset
from app.downloads.models import MediaAsset
log = logging.getLogger("siftlode.backfill")
log = logs.get_logger(__name__)
def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
needs the POT/player-client strategy even just to extract)."""
with formats.hiding(url):
return _extract_meta_scoped(url, adapter, asset_id)
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
for clients in adapter.client_sets():
opts = formats.build_ydl_opts(
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
@@ -40,7 +46,19 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
if info:
return info
except Exception as exc: # noqa: BLE001 — best-effort backfill, keep going
log.info("meta extract failed for %s (%s): %s", url, clients, str(exc)[:100])
# The asset id, not the URL. A source URL is a row of this user's download
# history, and `redact_text` only takes credential params out of it — the rest
# of it goes in whole. The id names the same asset to anyone holding the
# database and nothing at all to anyone who is not; the host is kept because it
# is what decides which client set was tried. yt-dlp's message goes through
# `log_safe` for the same reason: it opens with the URL or the video id.
log.info(
"meta extract failed for asset %s on %s (%s): %s",
asset_id,
urlsplit(url).hostname or "?", # log-pii-ok: the host, never the path or query
clients,
formats.log_safe(exc)[:100],
)
return None
@@ -60,20 +78,22 @@ def run() -> dict:
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
if a.uploader_url is None and adapter.wants_metadata_backfill():
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
info = _extract_meta(url, adapter)
info = _extract_meta(url, adapter, a.id)
uu = adapter.uploader_url(info) if info else None
if uu:
a.uploader_url = uu
filled_url += 1
log.info("asset %s uploader_url <- %s", a.id, uu)
log.info(
"asset %s uploader_url filled", a.id
) # the URL names what they downloaded
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
rel = editmod.ensure_poster(a, settings.download_root)
rel = editmod.ensure_poster(a, settings.files_root)
if rel:
a.poster_path = rel
filled_poster += 1
log.info("asset %s poster <- %s", a.id, rel)
log.info("asset %s poster filled", a.id) # the path carries the title
db.commit()
result = {
"assets": len(assets),
+289
View File
@@ -0,0 +1,289 @@
"""Ad-hoc clips: a marked range of a YouTube video or a Plex item, cut into a new per-user asset.
A clip never needs the whole source first. A YouTube clip fetches only its range (yt-dlp's
`download_ranges`; measured: 3-7 s whether the video runs four minutes or an hour, and the fetched
range starts on the marked frame), and a Plex clip reads the library file in place. Either way the
range then goes through ONE re-encode built here, so every clip has the same shape whatever its
source: frame-exact at both ends, H.264 at most 1080p (an HDR source tonemapped to SDR — kept as
10-bit PQ it played washed out), stereo AAC, in an .mp4 that plays on any phone.
A clip may also crop the picture to a frame the user drew over the playing video. The frame is
given in fractions of the box it was drawn in, with that box's aspect: the YouTube iframe never
tells its picture's size, so the frame there is drawn over the player's 16:9 box (a short's 9:16),
and the cut maps it onto the real picture (`crop_filter`) — what lies over a letterbox bar is dropped.
The spec is stored on the job (`DownloadJob.edit_spec`) and is the clip's whole identity: its
signature is the asset's cache key, so marking the same range again is an instant cache hit, and a
different range or audio track is a different clip.
"""
import hashlib
import json
import math
import re
from pathlib import Path
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
from app.plex.sync import HDR_TRANSFERS
SOURCES = ("youtube", "plex")
# Owner's cap: a clip is a moment, not a way to export a whole film to a public link.
MAX_CLIP_S = 600
MIN_CLIP_S = 1
MAX_HEIGHT = 1080
# What yt-dlp fetches the range in (formats.normalize fills the rest): the clip's own ceiling, so the
# re-encode never upscales and never throws away a better source than it keeps.
YOUTUBE_FETCH_SPEC = {"mode": "av", "max_height": MAX_HEIGHT, "container": "mp4"}
# Bump when the spec → output-bytes mapping changes, so a cached clip re-derives.
_SIG_VERSION = 1
_YOUTUBE_ID = re.compile(r"[A-Za-z0-9_-]{11}")
_RATING_KEY = re.compile(r"[0-9]{1,32}")
_MAX_AUDIO_TRACK = 63
# HDR → SDR as measured on the owner's 4K HDR files (2026-10-10): linearise, tonemap (hable),
# convert the primaries and transfer to BT.709 and back to 8-bit 4:2:0.
_TONEMAP = (
"zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable:desat=0,"
"zscale=t=bt709:m=bt709:r=tv,format=yuv420p"
)
_SCALE = f"scale=-2:'min({MAX_HEIGHT},ih)'"
# A crop frame is at least this share of its box on each side (the player's frame holds the same
# floor), and what is left of it on the real picture at least _MIN_VISIBLE of it.
MIN_CROP = 0.1
_MIN_VISIBLE = 0.05
# The box aspects a frame may be drawn in: a 9:16 short to a 21:9 film, with room either side.
_CROP_ASPECTS = (0.25, 4.0)
# Float slack on the frame's far edges (x + w, y + h), which the client rounds to 4 decimals.
_EDGE_SLACK = 1e-3
class ClipError(Exception):
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
message names (the cap itself, so the client never says a stale one)."""
def __init__(self, reason: str, **params: float):
self.reason = reason
self.params = params
super().__init__(reason)
def normalize(
source: str,
ref: str,
start_s: float,
end_s: float,
audio_track: int | None = None,
crop: dict | None = None,
) -> dict:
"""The validated, canonical clip spec. Times are rounded to the millisecond and a crop to four
decimals, so the same marks always give the same cache key. Raises ClipError."""
if source not in SOURCES:
raise ClipError("bad_source")
pattern = _YOUTUBE_ID if source == "youtube" else _RATING_KEY
if not isinstance(ref, str) or not pattern.fullmatch(ref):
raise ClipError("bad_source")
if not (math.isfinite(start_s) and math.isfinite(end_s)) or start_s < 0:
raise ClipError("bad_range")
start, end = round(start_s, 3), round(end_s, 3)
if end - start < MIN_CLIP_S:
raise ClipError("too_short")
if end - start > MAX_CLIP_S:
raise ClipError("too_long", limit_s=MAX_CLIP_S)
spec: dict = {"source": source, "ref": ref, "start_s": start, "end_s": end}
if source == "plex" and audio_track is not None:
if not 0 <= audio_track <= _MAX_AUDIO_TRACK:
raise ClipError("bad_audio")
spec["audio_track"] = audio_track
frame = _crop(crop)
if frame is not None:
spec["crop"] = frame
return spec
def _crop(raw: dict | None) -> dict | None:
"""A crop frame, canonical — or None for no crop, which a frame over the whole box also is (the
same clip as one cut without a frame, so the same cache key)."""
if raw is None:
return None
try:
x, y, w, h, aspect = (float(raw[k]) for k in ("x", "y", "w", "h", "aspect"))
except (KeyError, TypeError, ValueError) as e:
raise ClipError("bad_crop") from e
if not all(math.isfinite(v) for v in (x, y, w, h, aspect)):
raise ClipError("bad_crop")
lo, hi = _CROP_ASPECTS
if not lo <= aspect <= hi or x < 0 or y < 0:
raise ClipError("bad_crop")
if w < MIN_CROP - _EDGE_SLACK or h < MIN_CROP - _EDGE_SLACK:
raise ClipError("bad_crop")
if x + w > 1 + _EDGE_SLACK or y + h > 1 + _EDGE_SLACK:
raise ClipError("bad_crop")
frame = {"x": x, "y": y, "w": min(w, 1 - x), "h": min(h, 1 - y), "aspect": aspect}
frame = {k: round(v, 4) for k, v in frame.items()}
if (frame["x"], frame["y"], frame["w"], frame["h"]) == (0, 0, 1, 1):
return None
return frame
def duration(spec: dict) -> float:
return spec["end_s"] - spec["start_s"]
def sig(spec: dict) -> str:
"""The clip's cache signature: the spec itself, versioned."""
body = json.dumps({**spec, "_v": _SIG_VERSION}, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(body.encode()).hexdigest()[:24]
def asset_ref(user_id: int, spec: dict) -> str:
"""The asset's `source_ref`. The user id makes a clip private to whoever cut it, like an edit:
never dedup-shared, never cache-served to someone else."""
return f"{user_id}:{spec['source']}:{spec['ref']}"
def reference_url(spec: dict) -> str | None:
"""Where a YouTube clip came from: its video at the clip's start. A Plex clip has no public page."""
if spec.get("source") != "youtube" or not spec.get("ref"):
return None
return f"https://www.youtube.com/watch?v={spec['ref']}&t={int(spec.get('start_s') or 0)}s"
def is_hdr(probe: dict) -> bool:
"""Whether the first video stream of an ffprobe `-show_streams` JSON carries an HDR transfer."""
for s in probe.get("streams") or []:
if s.get("codec_type") == "video":
# PQ / HLG: an 8-bit SDR encode must tonemap them (the one set the Plex stream uses).
return s.get("color_transfer") in HDR_TRANSFERS
return False
def display_aspect(probe: dict) -> float | None:
"""The picture's shape as a player shows it, from an ffprobe `-show_streams` JSON: the first video
stream's width × its sample aspect ratio over its height, turned a quarter when the stream says
it is rotated (ffmpeg turns such a stream upright before any filter of ours, so a crop is cut
from the upright picture). None when the stream has no size."""
for s in probe.get("streams") or []:
if s.get("codec_type") != "video":
continue
w, h = s.get("width"), s.get("height")
if not isinstance(w, int) or not isinstance(h, int) or w <= 0 or h <= 0:
return None
dar = w * (_ratio(s.get("sample_aspect_ratio")) or 1.0) / h
return 1 / dar if _quarter_turned(s) else dar
return None
def _ratio(v: object) -> float | None:
"""'32:27' → 1.185…; '0:1', 'N/A', a missing or malformed value → None."""
if not isinstance(v, str) or ":" not in v:
return None
num, _, den = v.partition(":")
try:
n, d = float(num), float(den)
except ValueError: # silent-ok: a malformed ratio reads as none given; the caller takes 1:1
return None
return n / d if n > 0 and d > 0 else None
def _quarter_turned(stream: dict) -> bool:
rotation: object = (stream.get("tags") or {}).get("rotate")
for sd in stream.get("side_data_list") or []:
if "rotation" in sd:
rotation = sd["rotation"]
try:
return round(float(str(rotation))) % 180 == 90
except ValueError: # silent-ok: no (None) or a malformed rotation: the stream is upright
return False
def crop_filter(crop: dict, dar: float) -> str | None:
"""The ffmpeg `crop` for a frame drawn in a box of `crop["aspect"]`, over a picture of display
aspect `dar` fitted into that box as a player fits it (whole, centred, bars on the short sides).
The frame's part over a bar is dropped: "" when what is left is the whole picture, None when too
little of the frame lies on the picture at all.
In fractions of the input's own width and height, so a sample aspect ratio other than 1:1 (an
anamorphic DVD) needs nothing of its own, and floored to even pixels: an even 4:2:0 frame, and a
window that never runs past the picture's far edge."""
box = crop["aspect"]
pic_w, pic_h = (1.0, box / dar) if dar > box else (dar / box, 1.0)
def span(start: float, size: float, extent: float) -> tuple[float, float]:
origin = (1 - extent) / 2
a = min(1.0, max(0.0, (start - origin) / extent))
b = min(1.0, max(0.0, (start + size - origin) / extent))
return a, b - a
x, w = span(crop["x"], crop["w"], pic_w)
y, h = span(crop["y"], crop["h"], pic_h)
if w < _MIN_VISIBLE or h < _MIN_VISIBLE:
return None
if w > 1 - 1e-4 and h > 1 - 1e-4:
return ""
def px(axis: str, f: float) -> str:
return f"trunc({axis}*{f:.6f}/2)*2"
return f"crop={px('iw', w)}:{px('ih', h)}:{px('iw', x)}:{px('ih', y)}"
def audio_track_count(probe: dict) -> int:
return sum(1 for s in probe.get("streams") or [] if s.get("codec_type") == "audio")
def build_cmd(
src: Path,
dest: Path,
start_s: float,
dur_s: float,
*,
audio_track: int | None,
hdr: bool,
threads: int,
crop: str = "",
) -> list[str]:
"""ffmpeg for one clip: an input seek (decodes from the keyframe before `start_s` and drops the
frames up to it, so the clip starts on the marked frame), the first video stream, the chosen audio
track (the first when none was chosen; none at all is fine), no subtitles, data or chapters.
A `crop` (crop_filter) comes first: the tonemap and the scale then work on the frame it keeps."""
vf = ",".join([*([crop] if crop else []), *([_TONEMAP] if hdr else []), _SCALE, EVEN_FRAME])
audio = f"0:a:{audio_track}" if audio_track is not None else "0:a:0?"
return [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-ss",
f"{start_s:.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-t",
f"{dur_s:.3f}",
"-map",
"0:v:0",
"-map",
audio,
"-sn",
"-dn",
# A source's chapters would ride into the .mp4 as a data track of their own (an mkv's
# chapter list came out as `bin_data`); a clip of a moment has no chapters to keep.
"-map_chapters",
"-1",
"-vf",
vf,
*X264_ARGS,
"-threads",
str(threads),
*AAC_ARGS,
"-ac",
"2",
"-movflags",
"+faststart",
str(dest),
]
+103 -25
View File
@@ -1,15 +1,17 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
Two sides depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
authed download becomes a PRIVATE asset that is never cache-served to another user. Retry
(`service.resume`) and Re-download (`service.redownload`) ask the same question again, so a
cookie the user added after a failure goes with the next attempt.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Both key the private identity on `cookie_salt` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
@@ -21,19 +23,21 @@ Security controls enforced here:
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs
from app.downloads import clip as clipmod
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.downloads.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
from app.userscope import owned_by
log = logging.getLogger("siftlode.downloads.credentials")
log = logs.get_logger(__name__)
class HostSpec:
@@ -53,6 +57,11 @@ class HostSpec:
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273: every player client fails on that one video while its neighbours download fine, and
# the newest yt-dlp fails the same way — a signed-in session is the only way through). Without a
# row here the `login_required` copy sent the user to a dropdown that could not help them.
HostSpec("youtube", "YouTube", ("youtube.com", "youtu.be", "youtube-nocookie.com")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
@@ -99,11 +108,38 @@ def cookie_salt(user_id: int, host_key: str) -> str:
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
"""The salted `format_sig` an authed download for (user, host) resolves to under the current
signature version — the private asset enqueue creates, and one of the identities `is_authed_asset`
recognises. Neither calls it; the tests use it to hold the two to each other."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def rebuild_salt(asset_id: int, cookie: str | None) -> str:
"""The salt of a Re-download's fresh asset: its OWN row id (so every rebuild is a new identity no
enqueue can ever compute), prefixed with the user's `cookie_salt` when a cookie applies. The id is
what makes it recognisable — the worker has the asset row, so `is_authed_asset` can recompute this
one, where a salt built from the job's previous asset id could not be."""
return f"{cookie}:rebuild:{asset_id}" if cookie else f"rebuild:{asset_id}"
def is_authed_asset(asset: MediaAsset, spec: dict, user_id: int, url: str | None) -> bool:
"""Whether an asset is `user_id`'s private, cookie-salted identity for `url`'s host — the one
kind of asset a cookie is ever attached to: the one enqueue creates (`authed_format_sig`), or a
Re-download's rebuild of it (`rebuild_salt`). The worker's cookie gate asks it before a download,
and the failure diagnosis asks it after one (a failure on this asset happened WITH the cookie).
Either signature version counts (`formats.format_sigs`): a job queued before a deploy that bumped
the version still downloads into the asset it was given."""
host_key = host_key_for_url(url)
if host_key is None:
return False
salt = cookie_salt(user_id, host_key)
if asset.format_sig in formats.format_sigs(spec, salt=salt):
return True
return asset.id is not None and asset.format_sig in formats.format_sigs(
spec, salt=rebuild_salt(asset.id, salt)
)
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
@@ -122,7 +158,7 @@ def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
owned_by(DownloadCredential.user_id, user_id), DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
@@ -179,7 +215,7 @@ def status_list(db: Session, user_id: int) -> list[dict]:
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
select(DownloadCredential).where(owned_by(DownloadCredential.user_id, user_id))
).scalars()
}
out = []
@@ -199,43 +235,85 @@ def status_list(db: Session, user_id: int) -> list[dict]:
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
def usable_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, spec: dict
) -> str | None:
"""The user's cookie text a download of `url` into `asset` goes with, or None — the ONE decision
whether a download is signed in. The worker child writes it out for yt-dlp (`resolve_cookiefile`);
the worker asks it again for a job that inherits a failed asset. That second ask sees the
credential as it is when the job is claimed: a cookie the user removed or added since the failed
download changes this job's explanation to fit the cookie its Retry would go with.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
(`is_authed_asset`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
InvalidToken) also yields None; the download then fails honestly on the login wall (the
`login_required` category)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
if not is_authed_asset(asset, spec, user_id, url):
return None # shared asset — never attach a cookie to a non-private identity
return _readable(cred, user_id, host_key)
def usable_clip_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict
) -> str | None:
"""`usable_cookie` for an ad-hoc clip's range fetch. The second gate is the clip's own identity:
a clip asset is private by construction (`clip.asset_ref` names its user, see downloads/clip),
so a cookie goes with it exactly when that asset is THIS user's clip of `clip_spec`. Without it a
video YouTube gates signed out could never be clipped, while its download goes through."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset.source_kind != "clip" or asset.source_ref != clipmod.asset_ref(user_id, clip_spec):
return None
return _readable(cred, user_id, host_key)
def _readable(cred: DownloadCredential, user_id: int, host_key: str) -> str | None:
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
"treated as no cookie",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
return raw
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset: MediaAsset,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download (`usable_cookie`), or None."""
raw = usable_cookie(db, user_id, url, asset, spec)
return _write_cookiefile(raw, staging) if raw else None
def resolve_clip_cookiefile(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict, staging: Path
) -> Path | None:
"""`resolve_cookiefile` for an ad-hoc clip (`usable_clip_cookie`)."""
raw = usable_clip_cookie(db, user_id, url, asset, clip_spec)
return _write_cookiefile(raw, staging) if raw else None
def _write_cookiefile(raw: str, staging: Path) -> Path:
+163 -132
View File
@@ -23,7 +23,6 @@ for a player hover-preview later.
import hashlib
import json
import logging
import math
import os
import queue
@@ -33,9 +32,11 @@ import threading
import time
from pathlib import Path
from app import logs
from app.config import settings
from app.downloads.stated import StatedFailure
log = logging.getLogger("siftlode.edit")
log = logs.get_logger(__name__)
def encode_thread_cap() -> int:
@@ -54,6 +55,10 @@ class EditAborted(Exception):
# --- edit spec ------------------------------------------------------------------------------
# SEC-13: ceiling on a join's cut-list. An accurate join emits two filters per segment into a single
# ffmpeg `filter_complex`, so "however many the caller sent" is an unbounded command line.
_MAX_SEGMENTS = 100
def normalize_edit_spec(spec: dict | None) -> dict:
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
@@ -67,7 +72,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if crop:
try:
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
except (KeyError, TypeError, ValueError):
except (KeyError, TypeError, ValueError) as exc:
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
# and nothing anywhere called it a failure; this line is the only trace there is.
log.warning(
"the crop in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.crop_dropped", reason=type(exc).__name__),
)
c = None
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
out["crop"] = c
@@ -76,6 +87,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
segs = spec.get("segments")
if isinstance(segs, list) and segs:
norm: list[dict] = []
unreadable = 0
for s in segs:
try:
st = max(0.0, float((s or {}).get("start_s") or 0.0))
@@ -84,10 +96,25 @@ def normalize_edit_spec(spec: dict | None) -> dict:
continue
en = float(end_raw)
except (TypeError, ValueError):
# silent-ok: counted, not logged here — one cut-list can drop several, and the
# join that comes out is simply shorter than the one the user built. Reported
# once after the loop.
unreadable += 1
continue
if en > st:
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
if unreadable:
log.warning(
"some segments of a cut list could not be read and were dropped",
extra=logs.event("edit.spec.segments_dropped", dropped=unreadable, kept=len(norm)),
)
norm.sort(key=lambda x: x["start_s"])
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
# The cap is far above any real cut-list (the UI's own splitter produces a handful) and the
# excess is DROPPED rather than raising: the surplus segments are the attack, the first
# _MAX_SEGMENTS are a legitimate edit.
del norm[_MAX_SEGMENTS:]
if norm:
out["segments"] = norm
# A join always chooses a codec path: accurate=filter-concat re-encode,
@@ -104,7 +131,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
start = max(0.0, float(trim.get("start_s") or 0.0))
end_raw = trim.get("end_s")
end = float(end_raw) if end_raw is not None else None
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# Either a 400 the user cannot act on, or — beside a crop — a full-length clip where
# they asked for a cut. Both are worth one line saying the spec was the problem.
log.warning(
"the trim in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.trim_dropped", reason=type(exc).__name__),
)
start, end = 0.0, None
t: dict = {"start_s": round(start, 3)}
if end is not None and end > start:
@@ -155,6 +188,51 @@ def clip_duration(spec: dict, source_duration: int | None) -> int | None:
# --- ffmpeg: trim / crop --------------------------------------------------------------------
# Every libx264 / aac re-encode in this package uses these, so a re-encoded file is always the
# phone-safe shape (`playable`): `-pix_fmt yuv420p` pins 8-bit 4:2:0 — without it libx264 keeps a
# 10-bit or 4:4:4 source's pixel format and writes H.264 High 10 / High 4:4:4, which browsers decode
# only patchily (the one safe H.264 is 8-bit 4:2:0).
X264_ARGS = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p"]
AAC_ARGS = ["-c:a", "aac", "-b:a", "192k"]
# The last step of every video filter chain a libx264 re-encode runs: 4:2:0 halves the chroma, so
# libx264 refuses an odd width or height ("width not divisible by 2") — measured on an odd source
# that is not 4:2:0 already (a 4:4:4 or RGB screen capture, 1365x767). A crop of a 4:2:0 source is
# rounded to its chroma grid by ffmpeg itself; this holds every other path to the encoder to the same
# even frame. Rounding down to even costs at most one pixel row/column.
EVEN_FRAME = "scale=trunc(iw/2)*2:trunc(ih/2)*2"
# The demuxers ffmpeg/ffprobe may use to open a library file — single-file containers only. Without
# it ffmpeg picks the demuxer from the file's CONTENT and NAME, and an upload's name is the user's:
# a playlist-type demuxer (HLS, DASH, concat) opens whatever local paths its entries name, so a
# crafted upload could copy another user's clip or a Plex file into its own converted asset. A file
# no demuxer here can open fails the probe, and an unprobeable upload is a download-only 'file'.
MEDIA_DEMUXERS = (
"mov", # mp4, m4a, mov, 3gp
"matroska", # mkv, webm
"mpegts",
"mpeg",
"avi",
"asf",
"flv",
"ogg",
"mp3",
"aac",
"ac3",
"eac3",
"flac",
"wav",
"w64",
"aiff",
"caf",
"amr",
)
INPUT_GUARD = ["-format_whitelist", ",".join(MEDIA_DEMUXERS)]
# A downloaded thumbnail is the remote site's bytes under the suffix yt-dlp gave it (`.png`, `.webp`,
# `.jpg` — `pipeline._THUMB_SUFFIXES`): only the single-image demuxers those open, so a "thumbnail"
# that is really a playlist cannot pull a local file into the poster. Anything else fails, and the
# poster falls back to a cut frame (`ensure_poster`).
THUMB_GUARD = ["-format_whitelist", "png_pipe,jpeg_pipe,webp_pipe"]
def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]:
"""ffmpeg command for a trim/crop edit. `-progress pipe:1` streams machine-readable progress.
@@ -181,7 +259,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
]
if start:
cmd += ["-ss", f"{start:.3f}"]
cmd += ["-i", str(src)]
cmd += [*INPUT_GUARD, "-i", str(src)]
if end is not None:
dur = end - start
if dur > 0:
@@ -189,21 +267,9 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
if reencode:
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
# the exact cut). A crop adds the filter; an accurate trim re-encodes with no filter.
if crop:
cmd += ["-vf", f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"]
cmd += [
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
]
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
else:
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -251,11 +317,14 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
parts, labels = [], []
for i, s in enumerate(segs):
st, en = s["start_s"], s["end_s"]
parts.append(f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf}[v{i}]")
parts.append(
f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf},{EVEN_FRAME}[v{i}]"
)
parts.append(f"[0:a]atrim=start={st}:end={en},asetpts=PTS-STARTPTS[a{i}]")
labels.append(f"[v{i}][a{i}]")
parts.append(f"{''.join(labels)}concat=n={len(segs)}:v=1:a=1[v][a]")
cmd = head + [
*INPUT_GUARD,
"-i",
str(src),
"-filter_complex",
@@ -264,26 +333,23 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
"[v]",
"-map",
"[a]",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
*X264_ARGS,
*AAC_ARGS,
]
else:
listp = staging / "concat.txt"
lines = []
lines = ["ffconcat version 1.0"]
for s in segs:
lines.append(f"file '{_concat_quote(src)}'")
# ffmpeg hands the list's whitelist (which must name `concat` to read the list) down to
# the file it names, so a library file whose bytes are a concat list would be read as one
# and open the paths inside it. Each entry's own whitelist replaces the inherited one.
lines.append(f"option format_whitelist {INPUT_GUARD[1]}")
lines.append(f"inpoint {s['start_s']}")
lines.append(f"outpoint {s['end_s']}")
prep[listp] = "\n".join(lines) + "\n"
cmd = head + ["-f", "concat", "-safe", "0", "-i", str(listp), "-c", "copy"]
guard = ["-format_whitelist", "concat," + INPUT_GUARD[1]]
cmd = head + ["-f", "concat", "-safe", "0", *guard, "-i", str(listp), "-c", "copy"]
if out_ext in ("mp4", "m4a", "mov"):
cmd += ["-movflags", "+faststart"]
@@ -296,11 +362,14 @@ def _parse_out_time(line: str) -> float | None:
if line.startswith("out_time_us="):
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: one line of ffmpeg's `-progress` stream, several per second. None means "this
# line carried no usable time", which the caller already treats as "no progress yet".
except ValueError:
return None
if line.startswith("out_time_ms="): # (some builds mislabel this as microseconds)
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: as above, the other spelling of the same line.
except ValueError:
return None
return None
@@ -319,7 +388,9 @@ def _pump(stream, sink: "queue.Queue[str | None]") -> None:
try:
for line in stream:
sink.put(line)
except (ValueError, OSError): # pipe closed under us by the kill path
# silent-ok: the pipe was closed under us by our OWN kill path — the expected end of a
# cancelled or timed-out run, and the code that did the killing is the code that reports it.
except (ValueError, OSError):
pass
finally:
sink.put(None)
@@ -337,6 +408,8 @@ def _drain_tail(stream, sink: list[str]) -> None:
sink.append(line)
if len(sink) > 200:
del sink[:100]
# silent-ok: same closed pipe as `_pump` above, and the tail collected so far is still handed
# to whoever reports the failure.
except (ValueError, OSError):
pass
@@ -352,13 +425,17 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
try:
proc.wait(timeout=grace_s)
return
# silent-ok: the escalation below IS the handling of this timeout, and surviving THAT is what
# the line under the second wait reports.
except subprocess.TimeoutExpired:
pass
kill()
try:
proc.wait(timeout=grace_s)
except subprocess.TimeoutExpired:
log.error("%s pid %s survived SIGKILL", label, proc.pid)
# TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# `wait` call and nothing else — it would add three frames and no fact the line lacks.
log.error("%s pid %s survived SIGKILL", label, proc.pid) # noqa: TRY400
def _stop(proc: subprocess.Popen) -> None:
@@ -394,6 +471,8 @@ def run_ffmpeg(
while True:
try:
line = lines.get(timeout=1.0)
# silent-ok: not a failure — the one-second timeout is how this loop gets a turn to
# poll for cancellation while ffmpeg is quiet.
except queue.Empty:
line = ""
if line is None: # stdout hit EOF — ffmpeg is done (or dying)
@@ -406,11 +485,11 @@ def run_ffmpeg(
if should_cancel():
raise EditAborted
if time.monotonic() - last_output > stall_timeout_s:
raise RuntimeError(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
raise StatedFailure(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
try:
ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
except subprocess.TimeoutExpired: # pipes closed but the process lingers
raise RuntimeError("ffmpeg did not exit after closing its output")
raise StatedFailure("ffmpeg did not exit after closing its output") from None
except BaseException:
# ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
# function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
@@ -425,17 +504,10 @@ def run_ffmpeg(
if pipe:
pipe.close()
if ret != 0:
raise RuntimeError(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
raise StatedFailure(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
# --- ffmpeg: universal browser playability --------------------------------------------------
# A downloaded file must play in EVERY browser, whatever the source shipped. Universal = H.264
# video + AAC/MP3 audio in an mp4 container: H.264 is the only video codec all browsers decode
# (VP9/AV1 fail on Safari/iOS; HEVC fails on Chrome/Firefox), and mp4/AAC is the safe pairing.
# Anything else is fixed post-download: re-encode the offending stream(s), or just remux when the
# codecs are already fine but the container isn't.
_BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
# --- ffprobe ---------------------------------------------------------------------------------
def run_ffprobe_json(
@@ -447,10 +519,10 @@ def run_ffprobe_json(
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the shared media probe (`probe.probe_media`,
used by both the download worker's finalize and the upload ingest) both go through it, so the
package: the shared media probe (`probe.probe_json`: the stored facts and the phone-safe rule) goes
through it, so the
subprocess flags/timeout/error handling live in one place instead of drifting copies."""
args = ["ffprobe", "-v", "error"]
args = ["ffprobe", "-v", "error", *INPUT_GUARD]
if show_streams:
args.append("-show_streams")
if show_format:
@@ -461,39 +533,16 @@ def run_ffprobe_json(
try:
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# None makes every field it would have filled read as absent, and a file with no streams
# and no duration is indistinguishable here from one that is not media at all.
log.warning(
"ffprobe could not read a file",
extra=logs.event("media.ffprobe_failed", reason=type(exc).__name__),
)
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
ctype = (s.get("codec_type") or "").lower()
if ctype == "video" and vcodec is None:
vcodec = name
elif ctype == "audio" and acodec is None:
acodec = name
return vcodec, acodec
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
"""Media duration in whole seconds via ffprobe, or 0 if unknown — a progress denominator for a
re-encode when the yt-dlp info dict carried no `duration` (some URL/live extractions)."""
@@ -503,6 +552,7 @@ def probe_duration(path: Path) -> int:
"ffprobe",
"-v",
"error",
*INPUT_GUARD,
"-show_entries",
"format=duration",
"-of",
@@ -515,53 +565,16 @@ def probe_duration(path: Path) -> int:
check=True,
).stdout.strip()
return int(float(out)) if out else 0
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# Zero is the denominator of the re-encode progress bar, so a failed probe leaves the user
# watching a bar that never moves rather than a job that says it went wrong.
log.info(
"the duration probe failed; progress for this job has no total",
extra=logs.event("media.duration_probe_failed", reason=type(exc).__name__),
)
return 0
def browser_transcode_flags(
vcodec: str | None, acodec: str | None, is_mp4: bool
) -> tuple[bool, bool, bool]:
"""Decide what (if anything) makes a file universally browser-playable. Returns
(needs_change, reencode_video, reencode_audio). needs_change=False => already H.264 + AAC/MP3
in mp4. A None codec (no such stream, or a failed probe) is never re-encoded."""
rv = bool(vcodec) and vcodec not in _BROWSER_SAFE_VCODEC
ra = bool(acodec) and acodec not in _BROWSER_SAFE_ACODEC
return (rv or ra or not is_mp4), rv, ra
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-i",
str(src),
]
cmd += (
["-c:v", "libx264", "-preset", "veryfast", "-crf", "20"]
if reencode_video
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
# --- filmstrip (editor scrub track) ---------------------------------------------------------
_SB_COLS = 12
@@ -597,6 +610,7 @@ def build_storyboard_cmd(src: Path, dest: Path, duration_s: int | None, geom: di
"-loglevel",
"error",
"-nostdin",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -620,6 +634,7 @@ def build_poster_cmd(src: Path, dest: Path, at_s: float) -> list[str]:
"-nostdin",
"-ss",
f"{max(0.0, at_s):.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -641,6 +656,7 @@ def build_thumbnail_jpg_cmd(src: Path, dest: Path) -> list[str]:
"-loglevel",
"error",
"-nostdin",
*THUMB_GUARD,
"-i",
str(src),
"-frames:v",
@@ -660,15 +676,28 @@ def _run_ffmpeg_best_effort(cmd: list[str], timeout_s: int) -> Exception | None:
try:
subprocess.run(cmd, capture_output=True, timeout=timeout_s, check=True)
return None
# silent-ok: the exception is the RETURN VALUE. This helper exists so the three callers share
# one run/timeout policy, and each logs the failure with the context only it has.
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError) as exc:
return exc
def _step_failure(err: BaseException) -> str:
"""What a best-effort image step's failure line says: the kind and, for ffmpeg, its exit
status. Never `str(err)`: a CalledProcessError or TimeoutExpired quotes ffmpeg's argv, and an
OSError its file name — the source path either way, which is built from the channel and the
title."""
code = getattr(err, "returncode", None)
return f"{type(err).__name__} {code}" if code is not None else type(err).__name__
def _is_jpeg(path: Path) -> bool:
"""True if the file starts with the JPEG SOI marker (FF D8 FF)."""
"""True if the file starts with the JPEG SOI marker (0xFF 0xD8 0xFF)."""
try:
with path.open("rb") as f:
return f.read(3) == b"\xff\xd8\xff"
# silent-ok: False only means "do not copy this straight through" — the caller then goes the
# ffmpeg route, which opens the same file and reports it properly if it really is unreadable.
except OSError:
return False
@@ -689,19 +718,21 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
shutil.copyfile(src, dest)
return dest
except OSError as exc:
log.info("thumbnail copy failed (%s); poster falls back to a cut frame", str(exc)[:120])
log.info(
"thumbnail copy failed (%s); poster falls back to a cut frame", _step_failure(exc)
)
return None
err = _run_ffmpeg_best_effort(build_thumbnail_jpg_cmd(src, dest), timeout_s)
if err is not None:
log.info(
"thumbnail->jpg conversion failed (%s); poster falls back to a cut frame",
str(err)[:120],
_step_failure(err),
)
return None
return dest if dest.exists() else None
def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
def ensure_poster(asset, files_root: str, timeout_s: int = 30) -> str | None:
"""Generate a poster frame for a ready asset that has no thumbnail (e.g. a direct media URL
like a reddit HLS manifest, which yt-dlp can't get a thumbnail for). Written as the media's
`<base>.jpg` sidecar — so Plex uses it too and `delete_asset_files` cleans it up — and returned
@@ -709,7 +740,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
short clip) dodges black intro frames."""
if not asset.rel_path:
return None
root = Path(download_root)
root = Path(files_root)
src = root / asset.rel_path
if not src.exists():
return None
@@ -720,12 +751,12 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
at = min(3.0, (asset.duration_s or 10) * 0.1)
err = _run_ffmpeg_best_effort(build_poster_cmd(src, dest, at), timeout_s)
if err is not None:
log.info("poster gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("poster gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
return rel if dest.exists() else None
def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict | None:
def ensure_storyboard(asset, files_root: str, timeout_s: int = 90) -> dict | None:
"""Return the filmstrip meta for a ready source asset, generating the sprite on first use.
Best-effort + time-bounded: for a very long/high-res source the one-pass decode can be slow,
@@ -737,7 +768,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
if not asset.rel_path:
return None
geom = storyboard_geometry(asset.duration_s)
root = Path(download_root)
root = Path(files_root)
rel = f".storyboards/asset-{asset.id}.jpg"
dest = root / rel
@@ -751,7 +782,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
cmd = build_storyboard_cmd(src, dest, asset.duration_s, geom)
err = _run_ffmpeg_best_effort(cmd, timeout_s)
if err is not None:
log.info("storyboard gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("storyboard gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
if not dest.exists():
return None
+243 -1
View File
@@ -1,4 +1,5 @@
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code — and, where a site's
failure has been measured, into that site's entry of the failure catalog.
The worker stores the raw exception text (truncated) for technical detail, but a raw
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
@@ -6,11 +7,82 @@ to one of a small set of category codes; the frontend turns the code into a loca
explanation (and decides whether a retry makes sense). The raw text stays available behind a
"details" disclosure. This is the same string-matching approach the worker already trusts for
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
**The failure catalog** (`_CATALOG`). A category is generic: "Sign-in required" cannot tell a
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
its own sentence for — why it failed on that site, and what the user can do. `diagnose()` tries the
catalog first and falls back to the category. How an entry is added, every time a failure is reported:
1. reproduce it on dev (with and without the user's cookie) and keep the verbatim yt-dlp text;
2. add the entry below, scoped as narrowly as the measurement allows (a marker that is yt-dlp's
generic wording for every site must never match without its host; a marker another module also
decides on — the Facebook Chrome retry's FACEBOOK_PARSE_WALL — is imported, never retyped);
3. pin the real text in `tests/test_download_errors.py`;
4. add `downloads.errors.fail.catalog.<detail>.{short,body}` in EN and HU (a contract test checks it;
the group is not called `detail`, which `actionErrors.contract` reads as a raw `HttpError.detail`).
"""
from collections.abc import Callable
from dataclasses import dataclass
from typing import NamedTuple
from app.downloads.adapters import (
FACEBOOK_PARSE_WALL,
POST_NOT_SHOWN_MESSAGE,
POST_WITHOUT_VIDEO_MESSAGE,
facebook_group_post_id,
)
from app.downloads.credentials import host_key_for_url
# Worker-authored strings that this module owns because each is coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE and the three *_TOO_BIG_MESSAGEs are raised by the worker and must classify
# back to `quota_exceeded`; their rules below match a fragment of each, and test_download_errors pins
# each one, because a message and its rule drifting apart is exactly how a
# category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
# text (`worker._process_job`), so it has a rule of its own that gives the category back.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
#
# A source adapter's verdicts on a link (`SourceAdapter.resolve_url`) are adapters.py's
# constants, imported above; each has a rule below that gives its category back, and a catalog entry
# that gives the site's sentence.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
# A download, a cut clip, or an upload's converted file, held to the space left, which is not zero:
# "full" would be untrue. All three classify by the one phrase they share.
DOWNLOAD_TOO_BIG_MESSAGE = (
"The video is larger than the download storage you have left — free up space, then try again."
)
CLIP_TOO_BIG_MESSAGE = (
"The clip is larger than the download storage you have left — "
"free up space or cut a shorter clip, then try again."
)
CONVERTED_TOO_BIG_MESSAGE = (
"The converted file is larger than the download storage you have left — "
"free up space, then try again."
)
UPLOAD_MISSING_MESSAGE = "The uploaded file is missing."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
# case-insensitive on a substring, against the raw yt-dlp/ffmpeg message.
_RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
("larger than the download storage you have left", "quota_exceeded"),
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
# or the site answered the signed-in read with something that is not the post at all.
(POST_WITHOUT_VIDEO_MESSAGE.lower(), "format_unavailable"),
(POST_NOT_SHOWN_MESSAGE.lower(), "login_required"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -42,6 +114,8 @@ _RULES: tuple[tuple[str, str], ...] = (
# An edit's source download went away — our own wording, and MORE specific than the generic
# "no longer available" below, so it must be matched first.
("source download is no longer available", "source_missing"),
# An upload's original went away before its conversion ran — our wording (UPLOAD_MISSING_MESSAGE).
(UPLOAD_MISSING_MESSAGE.lower().rstrip("."), "upload_missing"),
("source file is missing", "source_missing"),
# Private / deleted / removed / terminated.
("private video", "unavailable"),
@@ -70,6 +144,13 @@ _RULES: tuple[tuple[str, str], ...] = (
("postprocessing", "postprocess"),
("produced no output file", "postprocess"),
("no output file", "postprocess"),
# yt-dlp's own verdict that its extractor did not understand the site's answer: it appends this
# request for a bug report to an UNEXPECTED extraction failure, never to an expected one (a private,
# removed or geo-blocked video says so in words matched above). Measured 2026-10-12: a TikTok link
# failed with it on every attempt under yt-dlp 2026.07.04 and downloads under 2026.08.19 — the
# site changed and a newer yt-dlp is the cure. After the media-format and post-processing rules,
# whose texts can carry the same request; before the network ones, which are about the transport.
("please report this issue", "site_changed"),
# Network / transport.
("unable to download", "network"),
("http error", "network"),
@@ -82,6 +163,120 @@ _RULES: tuple[tuple[str, str], ...] = (
)
class Diagnosis(NamedTuple):
"""What a failure was: the generic `code` (category) and, when a catalog entry matched, its
host-specific `detail`."""
code: str
detail: str | None = None
@dataclass(frozen=True)
class _Entry:
"""One measured failure of one site. It matches when the download URL belongs to `host`
(`credentials.SUPPORTED_HOSTS`), every one of `markers` is in the message, the cookie state is
`cookie` (None: either), and — when `link` is given — it recognises the URL (a truthy result). A
link predicate is the adapter's own, so the catalog and the download agree on what a link is."""
host: str
markers: tuple[str, ...]
code: str
detail: str
cookie: bool | None = None
link: Callable[[str], object] | None = None
_CATALOG: tuple[_Entry, ...] = (
# Facebook answers a signed-out client with a page yt-dlp cannot parse — an 18+ reel, a closed
# group's post (measured 2026-09-13: prod jobs 145, 156, 256 download once the owner's cookie is
# attached). "Cannot parse data" is yt-dlp's generic wording for a broken extractor on EVERY site,
# which is why the entry is scoped to Facebook and never becomes a general rule.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.login_wall",
cookie=False,
),
# The same text WITH a cookie: an expired / signed-out cookie, or a page that also needs
# impersonation (prod job 257). Telling this user to add cookies would be wrong — they have.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post read with a cookie that Facebook answered with its login page (measured with a
# refused cookie: no post id anywhere on the page) — the same refused cookie, found by the adapter.
_Entry(
host="facebook",
markers=(POST_NOT_SHOWN_MESSAGE.lower(),),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post with no single video of its own. With a cookie the adapter reads the post's page
# and says so itself (POST_WITHOUT_VIDEO_MESSAGE); without one yt-dlp is given the post link and
# answers "No video formats found" (measured on prod jobs 66 and 145, whose videos are reshared
# clips). With a cookie yt-dlp only ever gets the post's VIDEO link, so the same text there is about
# that video, not the post — the category says it. Only for group post links (the adapter's own
# test of one): a reel with no formats is a different failure.
_Entry(
host="facebook",
markers=(POST_WITHOUT_VIDEO_MESSAGE.lower(),),
code="format_unavailable",
detail="facebook.group_post_video",
link=facebook_group_post_id,
),
_Entry(
host="facebook",
markers=("[facebook]", "no video formats found"),
code="format_unavailable",
detail="facebook.group_post_video",
cookie=False,
link=facebook_group_post_id,
),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273, a festival set: all 12 of yt-dlp's player clients fail on that one video while its
# neighbours download fine, the POT sidecar is live and answering, and the newest yt-dlp fails
# identically — so this is the video's own signed-out gate, not a broken stack or a stale yt-dlp).
# The marker is quote-agnostic on purpose: YouTube writes the apostrophe as U+2019, so
# "you're not a bot" with a straight quote never matches the real text.
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.bot_wall",
cookie=False,
),
# The same wall WITH a cookie. Not yet reproduced with a real YouTube cookie (this user's first
# one is being added now) — but "add cookies" is provably the wrong advice for someone who has,
# and yt-dlp names the likeliest cause itself: "The provided YouTube account cookies are no
# longer valid. They have likely been rotated in the browser as a security measure."
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.cookie_rejected",
cookie=True,
),
# The same gate one step later: the player response comes back, and then the CDN refuses the
# stream itself ("unable to download video data: HTTP Error 403"). Measured 2026-09-16 on the
# very same video, ~90 minutes after it was answering with the bot message — signed out it
# 403'd on every attempt; with the owner's cookie attached it downloaded in full (519 MB). So
# the category this text carries generically, `network`, is a lie here: nothing was wrong with
# the network. Scoped to a cookie-less YouTube download, where that is what we measured.
_Entry(
host="youtube",
markers=("unable to download video data", "403"),
code="login_required",
detail="youtube.media_forbidden",
cookie=False,
),
)
def classify(msg: str | None) -> str:
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
if not msg:
@@ -91,3 +286,50 @@ def classify(msg: str | None) -> str:
if marker in low:
return category
return "unknown"
def classify_conversion(msg: str | None) -> str:
"""The category of an upload's failed conversion. The generic ffmpeg categories speak of a
download ("the video downloaded but…"), which an upload never was: a failed encode or an
unrecognised failure of one is `convert_failed`; a category of its own (a full quota, a missing
original) stays as it is."""
code = classify(msg)
return "convert_failed" if code in ("postprocess", "unknown") else code
def diagnose(
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
) -> Diagnosis:
"""The catalog entry for this failure of this link, else just its category (see the module doc).
`url` is the address yt-dlp was given; `cookie_attached` whether the acting user's cookie went
with it. Without a URL only the category can be known."""
host = host_key_for_url(url) if msg and url else None
if host is not None:
low = msg.lower() # type: ignore[union-attr] — `host` is only set when `msg` is truthy
for e in _CATALOG:
if (
e.host == host
and all(m in low for m in e.markers)
and (e.cookie is None or e.cookie == cookie_attached)
and (e.link is None or e.link(url)) # type: ignore[arg-type] — as `msg` above
):
return Diagnosis(e.code, e.detail)
return Diagnosis(classify(msg))
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
SEC-12: for a user-supplied URL, a transport-level failure text distinguishes "connection
refused" from "HTTP Error 401" from "timed out" per host:port — which is a port scanner with
extra steps, since the raw text is shown to the user behind the details disclosure. `urlguard`
stops the fetch aiming at our own network; this stops the remaining readout.
Only the network class, and only for ``url`` jobs: a YouTube job has no user-chosen host, and
every other class (login_required, geo_blocked, drm, …) is real, useful diagnosis. A separate
function rather than an inline branch in the worker so it can be pinned by a test — the raw
text leaking back is a security regression that would otherwise pass a green suite."""
if source_kind == "url" and code == "network":
return NETWORK_REDACTED_MESSAGE
return msg
+345
View File
@@ -0,0 +1,345 @@
"""The folders of the Files library — where each of a user's files sits.
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
uploads, one per other site). Its `source_key` names the origin; the client renders its name
from the key, in the reader's language.
- an OWN folder is the user's, with the name they gave it.
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
may be moved in like into an own folder, and it keeps a file until it is deleted.
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
"""
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from urllib.parse import urlsplit
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
from app.userscope import owned_by
class FolderKind(StrEnum):
SOURCE = "source"
OWN = "own"
HUB = "hub"
class SourceKey(StrEnum):
"""The fixed origins. Any other site is `site:<host>` (see `site_key`)."""
YOUTUBE = "youtube"
PLEX = "plex"
UPLOAD = "upload"
SITE_PREFIX = "site:"
# A host's leading label that only says "the same site, another front door".
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
def site_key(url: str) -> str:
"""The source key of a page on some site: `site:<host>` without its front door (www., m.),
or the YouTube key for a YouTube address that arrived as a plain URL."""
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return SourceKey.YOUTUBE
return f"{SITE_PREFIX}{host or 'unknown'}"
def source_key_for(db: Session, job: DownloadJob) -> str:
"""The origin a job's file came from — the source folder it belongs in.
The one recorded when it was placed, else: youtube → YouTube; a clip → the clip's source
(YouTube or Plex); an upload → Uploads; a URL → its site; an edit → wherever its parent's file
came from (its parent job, or, once that is deleted, the parent's asset)."""
if job.source_key:
return job.source_key
kind = job.source_kind
if kind == "youtube":
return SourceKey.YOUTUBE
if kind == "upload":
return SourceKey.UPLOAD
if kind == "clip":
return (
SourceKey.PLEX if (job.edit_spec or {}).get("source") == "plex" else SourceKey.YOUTUBE
)
if kind == "url":
return site_key(job.source_ref)
if kind == "edit":
parent = db.get(DownloadJob, job.source_job_id) if job.source_job_id else None
if parent is not None and parent.id != job.id:
return source_key_for(db, parent)
asset = db.get(MediaAsset, job.source_asset_id) if job.source_asset_id else None
if asset is not None:
if asset.source_kind == "youtube":
return SourceKey.YOUTUBE
if asset.source_kind == "upload":
return SourceKey.UPLOAD
if asset.source_kind == "url":
return site_key(asset.source_ref)
return SourceKey.UPLOAD
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.source_key == key
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
retention = (
None if key == SourceKey.UPLOAD else sysconfig.get_int(db, "download_retention_days")
)
folder = FileFolder(
user_id=user_id, kind=FolderKind.SOURCE, source_key=key, retention_days=retention
)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: the expected outcome of the race the savepoint exists for — another request made
# the same folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def place(db: Session, job: DownloadJob) -> None:
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
key = source_key_for(db, job)
job.source_key = key
job.folder_id = source_folder(db, job.user_id, key).id
def done_expiry(db: Session, folder_id: int | None, kept: bool) -> datetime | None:
"""When a file that finishes now goes: its folder's retention from now, or never (kept, or a
folder that keeps files)."""
if kept or folder_id is None:
return None
# Read under a share lock, held to the caller's commit: a retention change (`set_retention` locks
# the folder) waits for this file and restamps it too, or this read waits for the change and
# takes the new value — never the old value stamped on a file the change did not see.
days = db.execute(
select(FileFolder.retention_days)
.where(FileFolder.id == folder_id)
.with_for_update(read=True)
).scalar()
return None if days is None else datetime.now(timezone.utc) + timedelta(days=days)
def stamp_done(db: Session, job: DownloadJob) -> None:
"""Stamp a job that just became done (in this session): when it did, and its expiry."""
job.ready_at = datetime.now(timezone.utc)
job.expires_at = done_expiry(db, job.folder_id, job.kept)
job.expiry_warned = False
class FolderError(Exception):
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
def __init__(self, reason: str) -> None:
super().__init__(reason)
self.reason = reason
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The user's folder `folder_id` that a file is about to be put in — held (FOR SHARE) until the
caller's transaction ends, so a delete of the folder waits for the file and then takes it home
with the rest (`delete_own` locks it FOR UPDATE), never failing on its foreign key. `hold=False`
only looks (a check ahead of a long wait, which must not hold a lock through it). Another user's
folder answers like a missing one."""
stmt = select(FileFolder).where(FileFolder.id == folder_id)
if hold:
stmt = stmt.with_for_update(read=True)
folder = db.execute(stmt.execution_options(populate_existing=True)).scalar_one_or_none()
if folder is None or folder.user_id != user_id:
raise FolderError("file_folder_gone")
return folder
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
have."""
folder = own_folder(db, user_id, folder_id, hold=hold)
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
raise FolderError("file_folder_not_its_source")
return folder
def restamp(db: Session, job: DownloadJob) -> None:
"""Start a file's clock again from now: it moved to another folder, or its user stopped keeping
it. A file not done yet gets its expiry when it finishes (`stamp_done`), so it has none now."""
done = job.status == JobStatus.DONE
job.expires_at = done_expiry(db, job.folder_id, job.kept) if done else None
job.expiry_warned = False
def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
"""Keep a file until its user deletes it, or hand it back to its folder's retention."""
job.kept = kept
restamp(db, job)
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
it came from.
The caller has checked that `folder` is the job's user's."""
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
raise FolderError("file_folder_not_its_source")
if folder.id == job.folder_id:
return
job.folder_id = folder.id
restamp(db, job)
def _hold_for_change(db: Session, folder: FileFolder) -> None:
"""Lock `folder` FOR UPDATE and read it afresh into the same object: a change waits out a file
being put in it or stamped from it right now (`own_folder`, `done_expiry`), and works from what
another change that landed first left, not from what the caller read before the wait. The row is
consumed on purpose: an ORM result refreshes the object only as its rows are fetched."""
fresh = db.execute(
select(FileFolder)
.where(FileFolder.id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if fresh is None:
raise FolderError("file_folder_gone")
def set_retention(db: Session, folder: FileFolder, days: int | None) -> int | None:
"""Change how long `folder` keeps a file (None = until deleted). The done files already in it
move by the difference — each keeps the age it has — but none ends up closer than the GC's
warning window, so a shortened retention still warns before it takes anything. Returns the
retention it replaced (equal to `days` when nothing changed)."""
_hold_for_change(db, folder) # so the file arriving now is among the ones restamped below
old = folder.retention_days
if old == days:
return old
folder.retention_days = days
now = datetime.now(timezone.utc)
floor = now + timedelta(days=sysconfig.get_int(db, "download_gc_grace_days"))
# Locked as they are read: one moved out or deleted meanwhile is no longer this folder's to time.
jobs = db.execute(
select(DownloadJob)
.where(
DownloadJob.folder_id == folder.id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
for job in jobs:
if days is None:
job.expires_at = None
elif old is None or job.expires_at is None:
job.expires_at = max(now + timedelta(days=days), floor)
else:
job.expires_at = max(job.expires_at + timedelta(days=days - old), floor)
job.expiry_warned = False
return old
def delete_own(db: Session, folder: FileFolder) -> int:
"""Delete one of a user's own folders. Its files are not deleted: each goes back to the source
folder it came from, its clock starting there now. Returns how many moved."""
if folder.kind != FolderKind.OWN:
raise FolderError("file_folder_not_own")
_hold_for_change(db, folder) # so the file arriving now is taken home too
# Locked as they are read: a file moved out meanwhile stays where it went, and one deleted
# meanwhile is not written back (its gone row would fail the delete).
jobs = (
db.execute(
select(DownloadJob)
.where(DownloadJob.folder_id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
for job in jobs:
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
restamp(db, job)
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
def create_own(db: Session, user_id: int, name: str) -> FileFolder:
"""A new folder of the user's own; FolderError("file_folder_name_taken") when they already have
one by that name (the unique (user, lower(name)))."""
folder = FileFolder(user_id=user_id, kind=FolderKind.OWN, name=name)
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
return folder
def renamable(folder: FileFolder) -> bool:
"""Only a folder of the user's own making takes a name: a source folder's is its origin's."""
return folder.kind == FolderKind.OWN
def rename(db: Session, folder: FileFolder, name: str) -> None:
"""Rename one of the user's own folders (a source folder's name is its origin's)."""
if not renamable(folder):
raise FolderError("file_folder_not_own")
folder.name = name
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
+222 -17
View File
@@ -13,12 +13,162 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser; the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import contextlib
import contextvars
import hashlib
import json
import logging
import os
import re
import urllib.parse
from collections.abc import Iterator
from app import logs
log = logs.get_logger(__name__)
# yt-dlp names what it is working on in these shapes, and each one is a row of someone's download
# history: the source URL, the `[extractor] <id>:` prefix on nearly every warning and error, the
# file path it writes to (`storage.rel_path` builds it from the channel and the title), a
# playlist's title, and an id quoted mid-sentence. The job id is already on every line
# (`logs.set_job` in the download child), so none of them adds anything an operator needs. What
# stays: the host (it says which extractor is failing) and the extractor tag. Every pattern below
# is a shape measured in yt-dlp's own source (2026.07.04), pinned in test_ytdlp_logging.py.
_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
# `Destination:` is yt-dlp's own word in `[download] Destination: <path>`, not an id — the path
# after it is what `_PATH_RE` takes out.
_ID_PREFIX_RE = re.compile(r"(\[[\w:.-]+\] )(?!Destination:)[^\s:\[\]]+:")
# The generic extractor's "id" is the URL's basename, percent-DECODED: `My%20Holiday.mp4` is
# `My Holiday` — spaces, brackets, colons and all, which no pattern can bound (`Our Trip: Day 1`
# ends at which `: `?). The job knows its own URL, so the exact name is taken out as a literal
# (`hiding`). This pattern is the fallback for a line logged outside that scope: up to the first
# `: `. Only this extractor: every other one names a site id, and letting the prefix run to `: `
# everywhere would take the words out of lines like `[VideoConvertor] Converting …; Destination:`.
_GENERIC_ID_RE = re.compile(r"(\[generic\] )(?!Destination:)[^\[\]\n]+?:(?= |$)", re.M)
# `[youtube:tab] Playlist <title>: Downloading 5 items` — the title is not an id, and has spaces.
_PLAYLIST_TITLE_RE = re.compile(r"(\] Playlist ).+?(?=: Downloading )")
# `[download] Finished downloading playlist: <title>`, `[download] Downloading playlist: <title>`,
# `… Skipping already downloaded playlist: <title>` — the title runs to the end of the line.
_TITLE_TAIL_RE = re.compile(r"(playlist: |\[download\] Downloading \w+: ).*", re.I)
# An id quoted mid-sentence, not as a prefix: `… not available for <id>`, and `--no-playlist`'s
# `Downloading just the video <id>` (`_yes_playlist`) / `Downloading just video <id>` (multifeed).
_BARE_ID_RE = re.compile(
r"((?:subtitles not available for|Downloading just (?:the )?\w+) )[^\s,;]+"
)
# Two segments or more, and not glued to a word: `www.youtube.com/…` is what the URL pattern
# leaves behind, and it must survive this one.
_PATH_RE = re.compile(r"(?<![\w.…])/(?:[^/\s\"']+/)+[^/\s\"']*")
_HIDDEN: contextvars.ContextVar[tuple[str, ...]] = contextvars.ContextVar(
"formats_hidden", default=()
)
def generic_id(url: str) -> str:
"""The name yt-dlp's generic extractor gives a URL: its basename, extension off, percent-
decoded — `InfoExtractor._generic_id` (2026.07.04), pinned against it in the tests."""
return urllib.parse.unquote(os.path.splitext(url.rstrip("/").split("/")[-1])[0])
@contextlib.contextmanager
def hiding(url: str | None) -> Iterator[None]:
"""For the work on one URL: `log_safe` also takes out the name yt-dlp's generic extractor
gives it, exactly, where yt-dlp puts it (`[generic] <name>:`). A context variable, so it
holds for yt-dlp's logger calls too — they run on the same thread as the extraction."""
name = generic_id(url) if url else ""
token = _HIDDEN.set((*_HIDDEN.get(), name) if name else _HIDDEN.get())
try:
yield
finally:
_HIDDEN.reset(token)
def log_safe(text: object) -> str:
"""yt-dlp's text (its own lines, or a DownloadError's message) with the parts that name a
person's download taken out. Every log line that carries yt-dlp's words goes through here,
not just the adapter's: yt-dlp raises the same text it has just logged, so a line that skipped
it would put back one line later exactly what the adapter took out."""
text = str(text)
for name in _HIDDEN.get():
text = text.replace(f"[generic] {name}:", "[generic] …:")
text = _URL_RE.sub(lambda m: f"{m.group(1)}/…", text)
text = _ID_PREFIX_RE.sub(r"\1…:", text)
text = _GENERIC_ID_RE.sub(r"\1…:", text)
text = _PLAYLIST_TITLE_RE.sub(r"\1…", text)
text = _TITLE_TAIL_RE.sub(r"\1…", text)
text = _BARE_ID_RE.sub(r"\1…", text)
return _PATH_RE.sub("…", text)
# yt-dlp's HTTP downloader, refusing a file announced larger than `max_filesize`, logs this line and
# RETURNS — no exception and no file (measured; test_formats holds it against the pinned yt-dlp).
_OVER_CAP_LINE = "File is larger than max-filesize"
_OVER_CAP: contextvars.ContextVar[list[bool] | None] = contextvars.ContextVar(
"formats_over_cap", default=None
)
@contextlib.contextmanager
def watching_size_cap() -> Iterator[list[bool]]:
"""While yt-dlp works inside it: a one-slot flag, set when it refused a file over the size cap
(`build_ydl_opts(max_bytes=…)`). It then returns as if it had downloaded, and all the caller
would find is no file — a processing fault, to anyone reading it, for what is the owner's
storage. A context variable, like `hiding`: yt-dlp's logger calls run on the extraction's thread."""
flag = [False]
token = _OVER_CAP.set(flag)
try:
yield flag
finally:
_OVER_CAP.reset(token)
class _YdlLog:
"""yt-dlp's logger protocol, pointed at ours.
Without it yt-dlp explains itself to nobody: `quiet` sends its output to a stdout we do not
read, so "Requested format is not available", a signature-extraction failure or a POT refusal
reached us only as whatever text ended up in the raised exception. Measured against yt-dlp
2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
`warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
`debug` goes to TRACE: it is several lines per second during a download, so it shows only
when the whole process runs at TRACE. (Not under a user's own Teljes consent — that lives in
the API process's request scope, and the download child that runs yt-dlp has none.)
`error` goes to WARNING, not ERROR. yt-dlp calls it from `trouble()` just before it raises,
on EVERY failed attempt — including the ones the pipeline then recovers from with its fallback
client set. Whether the job failed is the pipeline's verdict (`download.job.failed`), and that
is the one line whose level should say so."""
def _emit(self, level: int, msg: str) -> None:
if _OVER_CAP_LINE in msg and (over := _OVER_CAP.get()) is not None:
over[0] = True
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
# a record that has args, this way the question never arises — and it is the shape ruff's
# `G` rules ask for. (The sink's flood limiter groups a bare `%s` by the text itself, so
# distinct yt-dlp warnings do not share one bucket — `logsink._template`.)
log.log(level, "%s", log_safe(msg), extra=logs.event("download.ytdlp"))
def debug(self, msg):
self._emit(logs.TRACE, msg)
def info(self, msg):
self._emit(logging.DEBUG, msg)
def warning(self, msg):
self._emit(logging.WARNING, msg)
def error(self, msg):
self._emit(logging.WARNING, msg)
_YDL_LOG = _YdlLog()
# Fields that affect the produced file — and therefore the cache identity.
_SIG_KEYS = (
@@ -56,7 +206,13 @@ _DEFAULTS = {
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
# v5: the phone-safe rule (`playable`) also judges the pixel format, so a cached 10-bit / 4:4:4 H.264
# asset (which v4 let through as "H.264") re-derives as 8-bit 4:2:0 instead of being reused.
_SIG_VERSION = 5
# Audio-only downloads version on their own, so a change to the audio bytes doesn't re-download every
# cached video. v5: an extracted .mp3/.m4a is stored as itself instead of remuxed into an .mp4 that
# iPad's WebKit refuses to play, so a cached remuxed audio asset must re-derive.
_AUDIO_SIG_VERSION = 5
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
@@ -74,21 +230,28 @@ def normalize(spec: dict) -> dict:
# rather than raising an unhandled 500 at enqueue.
try:
s["max_height"] = int(s["max_height"])
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# "best" is the opposite of what a height cap asks for: the user set a ceiling to save
# bandwidth or disk, and the download quietly takes the largest rendition there is.
log.warning(
"the height cap on a download could not be read and was dropped",
extra=logs.event("download.format.max_height_dropped", reason=type(exc).__name__),
)
s["max_height"] = None
for b in ("embed_subs", "embed_chapters", "embed_thumbnail", "sponsorblock"):
s[b] = bool(s[b])
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# Browser-playability guarantee: the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
# universally-safe combo is mp4 + H.264 + AAC (an .mp3/.m4a for audio), so we CLAMP the spec to
# it HERE — the single point every path (picker, stored profile, raw API) flows through — instead
# of only hiding options in the UI. Non-mp4 containers (mkv plays nowhere on iOS; AV1 fails there,
# and webm/VP9 plays only on recent devices) and any explicit vcodec are coerced to the compat
# profile; Opus audio falls back to AAC/m4a. A source that exposes no H.264 (or a 10-bit one) is
# still caught by the worker's phone-safe step (`downloads.playable`), so playback is guaranteed
# end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
@@ -96,26 +259,41 @@ def normalize(spec: dict) -> dict:
return s
def format_sig(spec: dict, salt: str | None = None) -> str:
def _sig_version(s: dict) -> int:
return _AUDIO_SIG_VERSION if s["mode"] == "a" else _SIG_VERSION
def format_sig(spec: dict, salt: str | None = None, *, version: int | None = None) -> str:
"""Stable short hash of the output-affecting spec — the MediaAsset cache key component.
`salt` forks a distinct, non-dedup identity for the same spec (used by force-redownload so it
always re-fetches into its own asset instead of sharing a cached file with other jobs)."""
always re-fetches into its own asset instead of sharing a cached file with other jobs).
`version` hashes under a signature version other than the current one (see `format_sigs`)."""
s = normalize(spec)
data = {k: s[k] for k in _SIG_KEYS}
data["_v"] = _SIG_VERSION
data["_v"] = _sig_version(s) if version is None else version
if salt is not None:
data["_salt"] = salt
payload = json.dumps(data, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode()).hexdigest()[:24]
def format_sigs(spec: dict, salt: str | None = None) -> tuple[str, str]:
"""`format_sig` under the current signature version and under the one before it. An asset keeps
the signature it was created with, so a job queued before a deploy that bumped the version and
run after it downloads into an asset carrying the previous one — which must still be recognised
as that user's private asset (`credentials.is_authed_asset`), or it downloads without its cookie."""
previous = _sig_version(normalize(spec)) - 1
return format_sig(spec, salt), format_sig(spec, salt, version=previous)
def build_ydl_opts(
spec: dict,
outtmpl: str,
progress_hook,
extra: dict | None = None,
cookiefile: str | None = None,
max_bytes: int | None = None,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
@@ -123,14 +301,25 @@ def build_ydl_opts(
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
`cookiefile` is the acting user's per-host cookies.txt path, attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it."""
merge so a source adapter can never clobber it.
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp's HTTP downloader refuses
a file announced larger than it rather than filling the disk and only then being caught by the
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
pipeline also holds every produced file to what is left by its real size."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
"outtmpl": outtmpl,
"noplaylist": True,
# SEC-13: `noplaylist` only demotes a video-inside-a-playlist URL to the single video. A
# URL that is PURELY a playlist or channel has no such video to fall back to, so it still
# expands into every entry — one enqueue, an unbounded number of downloads. One job means
# one item; a user who wants the whole list enqueues it item by item, under their quota.
"playlist_items": "1",
"quiet": True,
"no_warnings": True,
"noprogress": True,
@@ -154,7 +343,23 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# With a logger attached, `quiet` and `no_warnings` stop deciding whether yt-dlp explains itself
# and only decide where — so both stay on above and the explanation comes here instead of to a
# stdout nothing reads. AFTER the adapter merge, like `cookiefile` and the size cap: an adapter
# must not be able to silence the one account of what went wrong that a failed job leaves.
opts["logger"] = _YDL_LOG
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
#
# A non-positive budget is left unset because 0 is not an expressible yt-dlp filter, NOT because
# nothing needs to happen: the worker refuses that job outright before extraction (`_OwnerRefused`
# → `quota_exceeded`). Enqueue cannot make that call — it ran while the user was still under the
# cap. Delete the worker's refusal and this branch silently uncaps the one user already over it.
if max_bytes and max_bytes > 0:
opts["max_filesize"] = max_bytes
# The acting user's per-host cookie. AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
@@ -176,7 +381,7 @@ def build_ydl_opts(
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
# the worker's phone-safe step (`downloads.playable`).
if s["mode"] == "v":
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
+246 -101
View File
@@ -1,84 +1,167 @@
"""Download retention garbage collector (a scheduler job).
Five passes each run:
1. Pre-expiry warning — for ready assets expiring within the grace window, notify each holder
once (gc_notified flag) so a download isn't silently deleted.
2. Expiry deletion — delete assets past their TTL (files + row). The FK SET NULL nulls the
holders' jobs' asset_id, so those jobs surface as "expired" (re-downloadable) in the UI.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed
ready assets until back under it.
Six passes each run:
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
released, so the shared file goes with its last holder.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed ready
assets (never an upload, never one a user kept) until back under it; each holder's file leaves
their library, with a notice.
4. Orphan reap — delete errored, fileless, unreferenced asset rows (they carry no TTL, so
passes 1-3 skip them; benign row-count hygiene).
5. Orphan FILE sweep — delete on-disk files/dirs no surviving asset owns (cruft left when a
re-download renames an asset, e.g. old accented channel dirs), skipping system trees and
anything newer than an hour so an in-flight download is never touched.
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
so an in-flight download is never touched.
6. Token expiry warning — a machine's token for the library about to run out warns its user once
(`apitokens.warn_expiring`): the library's other clock.
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
"""
import logging
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import sysconfig
from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import storage
from app.models import DownloadJob, MediaAsset
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logging.getLogger("siftlode.downloads.gc")
# Dot-prefixed top-level trees that are NOT part of the Plex/flat asset layout — the orphan sweep
# never touches them (user clips, Plex HLS/img caches, in-progress staging, editor storyboards).
_SYSTEM_TREE = {".edits", ".plex-hls", ".plex-img-cache", ".staging", ".storyboards"}
log = logs.get_logger(__name__)
# The one dot-prefixed top-level tree the orphan sweep enters. The others are not part of the asset
# layout (user clips, Plex HLS/img caches, in-progress staging, editor storyboards) and are skipped.
_UPLOAD_TREE = ".uploads"
_ORPHAN_MIN_AGE_S = 3600 # only sweep files older than this, so an in-flight download is never hit
def _holders(db: Session, asset_id: int) -> list[int]:
return list(
db.execute(
select(DownloadJob.user_id).where(DownloadJob.asset_id == asset_id).distinct()
).scalars()
def _notify(db: Session, job: DownloadJob, asset: MediaAsset | None, ntype: str) -> None:
title = job.display_name or (asset.title if asset else None) or "download"
create_notification(
db,
user_id=job.user_id,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={
"kind": ntype,
"title": title,
"video_id": asset.source_ref if asset else None,
"job_id": job.id,
},
commit=False,
)
def _notify_holders(db: Session, asset: MediaAsset, ntype: str) -> None:
title = asset.title or "download"
for uid in _holders(db, asset.id):
create_notification(
db,
user_id=uid,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={"kind": ntype, "title": asset.title, "video_id": asset.source_ref},
commit=False,
def _relock_expired(
db: Session, job_id: int, asset_id: int | None, now: datetime
) -> DownloadJob | None:
"""Re-read an expired job under the lock order the user routes take (`own_job_for_update`: the
asset, then the job) and say whether it is still to go. Between the pass's SELECT and here a
user may have deleted or re-downloaded it (dropping it again would release its asset a second
time, freeing a file another job still holds) or kept or moved it (it is no longer expired)."""
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
still = (
job is not None
and job.asset_id == asset_id
and job.status == JobStatus.DONE
and not job.kept
and job.expires_at is not None
and job.expires_at <= now
)
return job if still else None
def _mark_warned(db: Session, job_id: int, expires_at: datetime) -> DownloadJob | None:
"""Mark one file warned if it still goes when the pass saw it go — not deleted, kept, moved or
re-timed since (each of those sets a fresh expiry, and its own warning to come) — and return it.
The UPDATE's row lock holds it so until the commit."""
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at == expires_at,
)
.values(expiry_warned=True)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
return db.get(DownloadJob, job_id) if hit is not None else None
def _delete_asset(db: Session, asset: MediaAsset, ntype: str) -> None:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
_notify_holders(db, asset, ntype)
db.delete(asset) # jobs.asset_id -> NULL via FK ON DELETE SET NULL
def _on_disk(db: Session) -> int:
"""The bytes the cache holds now (every asset whose file is on disk)."""
return (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).scalar()
or 0
)
def _sweep_orphan_files(db: Session, download_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under DOWNLOAD_ROOT that no `ready` asset
def _relock_evictable(db: Session, asset_id: int) -> tuple[MediaAsset | None, list[DownloadJob]]:
"""Re-read an eviction candidate and its holders under their locks (the asset, then its jobs —
the order the user routes take): between the pass's SELECT and here a holder may have kept it,
or every holder deleted it."""
asset = db.execute(
select(MediaAsset)
.where(MediaAsset.id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
holders = (
db.execute(
select(DownloadJob)
.where(DownloadJob.asset_id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
return asset, list(holders)
def _drop_job(db: Session, job: DownloadJob, ntype: str) -> None:
"""Take a file out of its user's library: tell them, release the job's hold on the asset (the
file goes with the last holder), delete the row (its shares and links cascade)."""
_notify(db, job, db.get(MediaAsset, job.asset_id) if job.asset_id else None, ntype)
service.release_asset(db, job)
db.delete(job)
def _sweep_orphan_files(db: Session, files_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under FILES_ROOT that no `ready` asset
owns — cruft left when a re-download changes an asset's on-disk name (e.g. an old accented
channel dir, or a poster from a superseded naming scheme). Skips the dot-prefixed system trees
and only removes files older than `_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download
is never deleted, and only files NOT owned by a surviving asset."""
root = Path(download_root)
channel dir, or a poster from a superseded naming scheme) — or, under `.uploads`, that no upload
row names at all. Skips the other dot-prefixed system trees and only removes files older than
`_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download is never deleted, and only files
NOT owned by a surviving asset."""
root = Path(files_root)
if not root.is_dir():
return 0
keep: set[Path] = set()
# An upload's file is on disk from ingest on, whatever its state (a queued conversion's original,
# a failed one kept for Retry), so every upload row keeps its file — not only a ready one.
for rel in db.execute(
select(MediaAsset.rel_path).where(
MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None)
or_(MediaAsset.status == "ready", MediaAsset.source_kind == "upload"),
MediaAsset.rel_path.is_not(None),
)
).scalars():
media = (root / rel).resolve()
@@ -99,8 +182,12 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
if pause_s and touched % batch == 0:
time.sleep(pause_s)
unreclaimed = 0
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
# .edits / .plex-* / .staging / .storyboards + any dotfile — but `.uploads` is swept: a
# conversion stopped between placing the new file and the commit that points at it (or the
# unlink of the original after it) leaves a file no row names, and nothing else finds it.
if entry.name.startswith(".") and entry.name != _UPLOAD_TREE:
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
@@ -111,7 +198,9 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
f.unlink()
removed += 1
except OSError:
pass
# silent-ok: counted, not logged per file — a broken mount would put one line
# under every orphan. The total is reported once at the end of the sweep.
unreclaimed += 1
if entry.is_dir(): # prune emptied dirs (deepest first), then the channel dir itself
for d in sorted(
(p for p in entry.rglob("*") if p.is_dir()),
@@ -120,12 +209,20 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
):
try:
d.rmdir()
# silent-ok: `rmdir` refusing means the directory still holds files, which is
# the ordinary answer here — this prune only takes the ones that emptied.
except OSError:
pass
try:
entry.rmdir()
# silent-ok: the same prune one level up, at the channel directory.
except OSError:
pass
if unreclaimed:
log.warning(
"some orphaned download files could not be reclaimed",
extra=logs.event("download.gc.unreclaimed", files=unreclaimed, removed=removed),
)
return removed
@@ -134,74 +231,95 @@ def run_download_gc(db: Session) -> dict:
grace_days = sysconfig.get_int(db, "download_gc_grace_days")
warned = expired = evicted = reaped = 0
# 1. Pre-expiry warnings.
# 1. Pre-expiry warnings — one per file, the first run inside the grace window. Each mark is
# conditional on the expiry the SELECT saw and committed on its own: a file deleted since is
# passed over (a write to its gone row would fail the whole run), and one kept, moved or re-timed
# since keeps its fresh expiry unwarned.
warn_before = now + timedelta(days=grace_days)
soon = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
)
soon = db.execute(
select(DownloadJob.id, DownloadJob.expires_at).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at > now,
DownloadJob.expires_at <= warn_before,
)
.scalars()
.all()
)
for asset in soon:
_notify_holders(db, asset, "download_expiring")
asset.gc_notified = True
warned += 1
if soon:
).all()
db.rollback() # the pass-1 snapshot holds no locks; each mark takes its own below
for job_id, expires_at in soon:
job = _mark_warned(db, job_id, expires_at)
if job is not None:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
_notify(db, job, asset, "download_expiring")
warned += 1
db.commit()
# 2. Expiry deletion (ready assets past TTL, and any errored leftovers with a TTL).
dead = (
db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
)
# 2. Expiry — a file past its folder's retention leaves the library (warned in pass 1).
# Each file is re-checked under its locks and committed on its own: the release unlinks the
# file at once, so it must never act on what the SELECT saw a while ago.
dead = db.execute(
select(DownloadJob.id, DownloadJob.asset_id).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at <= now,
)
.scalars()
.all()
)
for asset in dead:
_delete_asset(db, asset, "download_expired")
expired += 1
if dead:
).all()
db.rollback() # the pass-2 snapshot holds no locks; each file takes its own below
for job_id, asset_id in dead:
job = _relock_expired(db, job_id, asset_id, now)
if job is not None:
_drop_job(db, job, "download_expired")
expired += 1
db.commit()
# 3. LRU eviction over the total cache cap.
# 3. LRU eviction over the total cache cap: the disk's safety net. Never an upload (no source to
# fetch again) and never a file anyone kept; an evicted file leaves every holder's library.
cap = sysconfig.get_int(db, "download_total_max_bytes")
if cap > 0:
total = (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.status == "ready"
)
).scalar()
or 0
)
if total > cap:
candidates = (
if _on_disk(db) > cap:
kept_assets = select(DownloadJob.asset_id).where(
DownloadJob.kept.is_(True), DownloadJob.asset_id.is_not(None)
)
candidate_ids = (
db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready")
select(MediaAsset.id)
.where(
MediaAsset.status == "ready",
MediaAsset.source_kind != "upload",
MediaAsset.id.not_in(kept_assets),
)
.order_by(MediaAsset.last_access_at.asc().nulls_first())
)
.scalars()
.all()
)
for asset in candidates:
if total <= cap:
db.rollback()
for asset_id in candidate_ids:
# Counted afresh before each: a file deleted since (this one, an upload, a candidate
# further down) has freed its space, and no other user's file goes for that space.
if _on_disk(db) <= cap:
break
total -= asset.size_bytes or 0
_delete_asset(db, asset, "download_evicted")
# Locked and re-read like an expiry: since the SELECT a holder may have kept it,
# every holder deleted it, or someone asked for it again (a fresh cache hit).
asset, holders = _relock_evictable(db, asset_id)
if (
asset is None
or asset.status != "ready"
or any(j.kept for j in holders)
or (asset.last_access_at is not None and asset.last_access_at > now)
):
db.commit()
continue
for job in holders:
_notify(db, job, asset, "download_evicted")
db.delete(job)
if asset.rel_path:
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
evicted += 1
db.commit()
db.commit()
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
# referenced errored row to pending on re-enqueue, so these only accumulate once every
@@ -223,9 +341,35 @@ def run_download_gc(db: Session) -> dict:
if orphans:
db.commit()
# 4b. Reap unheld uploads. An upload carries no expires_at, is never evicted and never re-enqueued,
# so the passes above never free one that lost its last holder outside `release_asset` — a
# conversion that finished after its job was cancelled, or one cancelled mid-run (the asset
# was `downloading` then). Its file, ready or still the unconverted original, is at `rel_path`.
unheld = (
db.execute(
select(MediaAsset).where(
MediaAsset.source_kind == "upload",
MediaAsset.ref_count == 0,
MediaAsset.status != "downloading",
)
)
.scalars()
.all()
)
for asset in unheld:
if asset.rel_path: # no holder is left to tell, so no notice — just the file and the row
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
reaped += 1
if unheld:
db.commit()
# 5. Orphan FILE sweep — remove on-disk cruft no surviving asset owns (old accented dirs, stale
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
swept = _sweep_orphan_files(db, settings.download_root)
swept = _sweep_orphan_files(db, settings.files_root)
# 6. A machine's token about to run out.
tokens_warned = apitokens.warn_expiring(db)
result = {
"warned": warned,
@@ -233,6 +377,7 @@ def run_download_gc(db: Session) -> dict:
"evicted": evicted,
"reaped": reaped,
"swept": swept,
"tokens_warned": tokens_warned,
}
log.info("download_gc %s", result)
return result
+3 -3
View File
@@ -6,7 +6,7 @@ from sqlalchemy import text
from app.config import settings
from app.db import SessionLocal
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
@@ -68,8 +68,8 @@ def _reset_asset_pending(asset_id: int) -> None:
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
return Path(settings.files_root) / ".staging" / f"asset-{asset_id}"
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
return Path(settings.files_root) / ".staging" / f"edit-{asset_id}"
+196 -12
View File
@@ -1,27 +1,91 @@
"""Public "share by link" helpers for the download center.
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
login-free `/watch/{token}` page. So the token is never stored as written:
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
opens; only its address cannot be shown again, and the owner makes a new link to share.
Password-protected links additionally require a short-lived signed **grant** on the file request —
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
"""
import hashlib
import hmac
import secrets
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy import update as sql_update
from sqlalchemy.orm import Session
from app import audit, security
from app.config import settings
from app.models import DownloadLink
from app.downloads import storage
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.models import User
from app.utils import iso
from app.userscope import owned_by
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows
# (`queries.MAX_LIST`; a test holds the two equal — `queries` imports this module).
FOLDER_MAX = 500
def new_token() -> str:
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
def seal(link: DownloadLink, token: str) -> None:
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
link.token_hash = security.hash_token(token)
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
def find(db: Session, token: str) -> DownloadLink | None:
"""The link a `/watch` token names, by its hash."""
return db.execute(
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
).scalar_one_or_none()
def recover(link: DownloadLink) -> str | None:
"""The link's token from its encrypted copy, or None when there is none this key can read."""
return security.decrypt_optional(link.token_enc)
def in_folder(folder: FileFolder) -> tuple:
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
owned_by(DownloadJob.user_id, folder.user_id),
DownloadJob.status == JobStatus.DONE,
)
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
a dead row), newest first, at most `FOLDER_MAX`."""
rows = db.execute(
select(DownloadJob, MediaAsset)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(*in_folder(folder), MediaAsset.status == "ready")
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
.limit(FOLDER_MAX)
).all()
return [
(job, asset)
for job, asset in rows
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
]
def is_expired(link: DownloadLink) -> bool:
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
@@ -49,22 +113,28 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
exp_s, sig = grant.split(".", 1)
try:
exp = int(exp_s)
# silent-ok: an untrusted grant that does not parse is simply not a valid grant, and the
# honest answer to it — a password prompt — is the same as for one that expired.
except ValueError:
return False
if exp < datetime.now(timezone.utc).timestamp():
return False
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
# `sig` is whatever the URL carried.
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": link.token,
"url": f"{base_url}/watch/{link.token}",
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
@@ -104,3 +174,117 @@ def public_meta(
"file_url": file_url,
"poster_url": poster_url,
}
# --- the owner's writes --------------------------------------------------------------------
# Each flushes and records its audit row; the handler commits.
def expiry(days: int | None) -> datetime | None:
"""None/0/negative → no expiry; otherwise N days out, capped at ~10 years."""
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def audit_view(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
def create(
db: Session,
user: User,
*,
password: str,
allow_download: bool,
expires_days: int | None,
**target: int,
) -> tuple[DownloadLink, str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and the
token it is opened by — which is in hand only now."""
token = new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=allow_download,
password_hash=security.hash_password(password) if password else None,
expires_at=expiry(expires_days),
)
seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **audit_view(link)},
)
return link, token
def update(
db: Session,
user: User,
link: DownloadLink,
*,
allow_download: bool | None = None,
expires_days: int | None = None,
password: str | None = None,
sent: frozenset[str] = frozenset(),
) -> None:
"""Change what the caller sent (`sent` names the fields, so `expires_days=None` can mean "never
expires"). Setting, clearing or changing the password bumps its version, so any grant issued
under the previous one stops working now rather than living out its TTL."""
before = audit_view(link)
if "allow_download" in sent:
link.allow_download = bool(allow_download)
if "expires_days" in sent:
link.expires_at = expiry(expires_days)
if "password" in sent:
pw = (password or "").strip()
link.password_hash = security.hash_password(pw) if pw else None
link.password_version += 1
after = audit_view(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link.id,
before=before,
after=after,
)
db.flush()
def revoke(db: Session, user: User, link: DownloadLink) -> None:
link_id = link.id
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.flush()
def count_view(db: Session, link: DownloadLink) -> None:
"""One more opening of the link. In SQL, not `+= 1` on the loaded row: two opens at once would
both write the same count."""
db.execute(
sql_update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
+385
View File
@@ -0,0 +1,385 @@
"""Capture's tables: the downloaded files (`MediaAsset`), each user's jobs over them and the folders
they sit in, the shares and public links that hand them on, a user's quota, cookies and format
profiles, and where they stopped playing one. Only this unit writes them; another unit reads them
along a declared edge, or asks this unit's service."""
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
Boolean,
CheckConstraint,
Date,
DateTime,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, validates
from app.db import Base
from app.models import DISPLAY_TEXT_MAX, TimestampMixin, UpdatedAtMixin
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
Mirrors the app's "shared catalog, private per-user state" model: one row per unique
``(source_kind, source_ref, format_sig)`` is the global cache. When a second user requests
the same video with a format that hashes to the same `format_sig`, their `download_job`
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512)) # video id, or full URL for ad-hoc
format_sig: Mapped[str] = mapped_column(String(64)) # hash of the output-affecting spec
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
width: Mapped[int | None] = mapped_column(Integer)
height: Mapped[int | None] = mapped_column(Integer)
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
# yt-dlp's channel/uploader page URL (channel_url|uploader_url) — lets an auto-filled channel
# render as a clickable link. Null for sources without one (e.g. reddit) or older rows.
uploader_url: Mapped[str | None] = mapped_column(Text)
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
"""A named download preset — a reusable format profile.
`user_id` NULL + `is_builtin` marks the shipped presets (Best MP4, 1080p, audio-only, …);
a user's saved custom profiles carry their `user_id`. `spec` is the format definition
(mode av/v/a, max_height, container, audio_format, codec prefs, embed subs/chapters/
thumbnail, sponsorblock) that `downloads.formats` turns into a yt-dlp selector + a cache
`format_sig`."""
__tablename__ = "download_profiles"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
) # NULL for a builtin preset
name: Mapped[str] = mapped_column(String(80))
spec: Mapped[dict] = mapped_column(JSON)
is_builtin: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
(SET NULL if the asset is GC'd — the job then shows as expired).
Phase 2 (editor): `job_kind='edit'` jobs derive a new per-user clip from an already-downloaded
ready job. They still hang off a `MediaAsset` (with `source_kind='edit'`, a per-user cache key)
so all the file-serve/ref-count/GC/quota machinery is reused; the worker branches on the
asset's `source_kind` and runs ffmpeg instead of yt-dlp. `source_job_id`/`source_asset_id`
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL"), index=True
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="SET NULL")
)
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
String(16), default="queued", server_default="queued", index=True
) # queued | running | paused | done | error | canceled
progress: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
speed_bps: Mapped[int | None] = mapped_column(BigInteger)
eta_s: Mapped[int | None] = mapped_column(Integer)
phase: Mapped[str | None] = mapped_column(String(32))
error: Mapped[str | None] = mapped_column(Text) # raw yt-dlp/ffmpeg text (technical detail)
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
"""A user→user grant: `to_user` may access `from_user`'s downloaded file (private by
default). Since assets are already deduped/shared on disk, a share is just an ACL row."""
__tablename__ = "download_shares"
__table_args__ = (UniqueConstraint("job_id", "to_user_id", name="uq_download_share"),)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(ForeignKey("download_jobs.id", ondelete="CASCADE"))
from_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
to_user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
class DownloadQuota(Base, UpdatedAtMixin):
"""Per-user download limits (admin-managed). A row exists only when an admin customizes a
user; otherwise the sysconfig `download_default_*` values apply. `unlimited` bypasses the
byte cap entirely (e.g. for the admin's own account)."""
__tablename__ = "download_quotas"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
max_bytes: Mapped[int] = mapped_column(BigInteger)
max_concurrent: Mapped[int] = mapped_column(Integer)
max_jobs: Mapped[int] = mapped_column(Integer)
unlimited: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
# Bumped whenever the password changes; folded into the signed watch grant so rotating the
# password immediately invalidates any grants issued under the old one (instead of them living
# out the 6h grant TTL).
password_version: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
+39 -10
View File
@@ -11,13 +11,12 @@ import html
import re
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
@@ -66,28 +65,33 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
width = (data[3] << 8) + data[4]
return width, height
f.seek(length - 2, 1) # skip this segment's payload
# silent-ok: None omits width/height from the unfurl card, which is what the card does for
# every poster whose dimensions were never asked for. Nothing is claimed that is not true.
except OSError:
return None
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected video"),
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
)
if folder:
return _folder_tags(db, link, url)
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
@@ -133,7 +137,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
poster_abs = storage.safe_abs_path(settings.download_root, asset.poster_path)
poster_abs = storage.safe_abs_path(settings.files_root, asset.poster_path)
image_dims = _jpeg_size(poster_abs) if poster_abs else None
if image_dims:
tags.append(_tag("og:image:width", str(image_dims[0])))
@@ -145,13 +149,38 @@ def _watch_tags(token: str, db: Session) -> str | None:
return "\n ".join(tags)
def render_watch_html(token: str, index_html: Path, db: Session) -> str | None:
"""Return index.html with the OG block replaced for this watch token, or None to serve the
generic page unchanged (invalid/expired/not-ready link, or a template without the markers)."""
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
"""A folder link's card: the folder's name and how many files it holds — never their names,
which the page shows only to whoever opens it."""
folder = db.get(FileFolder, link.folder_id)
if folder is None or not folder.name:
return None
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
title = folder.name.strip()
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
return "\n ".join(
[
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", about),
_tag("og:type", "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
_tag("twitter:description", about, attr="name"),
_tag("twitter:card", "summary", attr="name"),
]
)
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
markers)."""
tags = _watch_tags(token, db)
if tags is None:
return None
text = index_html.read_text(encoding="utf-8")
text = page_html.read_text(encoding="utf-8")
if "<!--OG:START-->" not in text:
return None
# A function replacement avoids re.sub interpreting backslashes/group refs in `tags`.
File diff suppressed because it is too large Load Diff
+103
View File
@@ -0,0 +1,103 @@
"""The phone-safe rule: what a library file must be so it plays wherever a share link lands.
A shared `/watch` link is opened by whoever got it, on whatever they hold: an iPhone's WebKit, an
Android Chrome, a desktop Firefox. The one shape every one of them plays:
* video → H.264, 8-bit 4:2:0 (`yuv420p`, or its full-range twin `yuvj420p`), AAC audio, in .mp4
* audio only → MP3 in .mp3, or AAC in .m4a
Measured on the owner's iPhone + iPad (2026-10-05): AV1, Matroska, HEVC tagged `hev1` and MP3
inside an .mp4 all fail there. VP9/WebM, HEVC `hvc1` and 10-bit H.264 played on those devices, but
Chrome and Firefox do not decode HEVC and browsers decode 10-bit / 4:4:4 H.264 only patchily, so the
rule is the shape every browser plays, not the shape one device plays.
Every file that becomes a ready library asset passes through here — a download, an upload (the
worker converts it, see `upload`), and an editor clip — so a link can only ever serve this shape.
`plan` says what (if anything) must change; `build_cmd` is the one ffmpeg command that changes it:
re-encode only the stream that fails, copy the rest, or just remux when only the container is wrong.
"""
from dataclasses import dataclass
from pathlib import Path
from app.downloads import edit as editmod
from app.downloads import probe
_SAFE_VCODEC = "h264"
_SAFE_PIX_FMTS = frozenset({"yuv420p", "yuvj420p"})
_VIDEO_EXT = "mp4"
# An audio-only file: the codec, and the extension it plays in. Anything else becomes AAC in .m4a.
_AUDIO_HOME = {"mp3": "mp3", "aac": "m4a"}
_AUDIO_FALLBACK_EXT = "m4a"
@dataclass(frozen=True)
class Plan:
"""What makes one file phone-safe. `ext` is the extension the safe file has (no dot)."""
ext: str
has_video: bool
reencode_video: bool
reencode_audio: bool
needs_change: bool
def plan(data: dict | None, ext: str) -> Plan | None:
"""The phone-safe plan for a file whose `probe.probe_json` is `data` and whose extension is
`ext`; None when it is not media at all (an unreadable probe, a still image, no audio or video
stream) — such a file is a download-only 'file' and never plays inline, so there is nothing to
make safe."""
if data is None:
return None
video, audio = probe.primary_streams(data)
ext = ext.lower().lstrip(".")
if video is not None:
rv = (video.get("codec_name") or "").lower() != _SAFE_VCODEC or (
video.get("pix_fmt") or ""
).lower() not in _SAFE_PIX_FMTS
ra = audio is not None and (audio.get("codec_name") or "").lower() != "aac"
return Plan(_VIDEO_EXT, True, rv, ra, rv or ra or ext != _VIDEO_EXT)
if audio is not None:
home = _AUDIO_HOME.get((audio.get("codec_name") or "").lower())
target = home or _AUDIO_FALLBACK_EXT
return Plan(target, False, False, home is None, home is None or ext != target)
return None
def build_cmd(src: Path, dest: Path, p: Plan, threads: int | None = None) -> list[str]:
"""The ffmpeg command that writes `src` to `dest` (whose suffix is `p.ext`) in the phone-safe
shape: the failing stream(s) re-encoded with the package's shared encoder settings, the rest
stream-copied. An audio-only result drops any cover-art stream. `-progress pipe:1`
streams machine-readable progress; `threads` caps the encoder so a re-encode cannot claim every
core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
*editmod.INPUT_GUARD,
"-i",
str(src),
]
# Exactly the two streams `plan` judged: `V` is a video stream that is not cover art. Nothing
# else rides along — a subtitle or data track the .mp4 muxer cannot take would fail the run.
if p.has_video:
cmd += ["-map", "0:V:0", "-map", "0:a:0?"]
if p.reencode_video:
cmd += ["-vf", editmod.EVEN_FRAME, *editmod.X264_ARGS]
else:
cmd += ["-c:v", "copy"]
else:
cmd += ["-map", "0:a:0"]
cmd += editmod.AAC_ARGS if p.reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
if p.ext in ("mp4", "m4a"):
cmd += ["-movflags", "+faststart"]
cmd += [str(dest)]
return cmd
+107
View File
@@ -0,0 +1,107 @@
"""What a library file previews as — the single home of the preview allow-list.
A generic file (`media_kind == "file"`; video and audio have their players) previews by its stored
`container`, the upload's cleaned extension, and ONLY through this table: an image is shown as an
image, a PDF in the browser's own viewer, and a text format as plain text. Anything else has no
preview. The served Content-Type comes from here too, never from a guess on the name, and HTML / SVG
sit in the TEXT list on purpose: rendered on this origin, an uploaded page would run with the
reader's session. The route adds `CSP: sandbox` on top, so even a mislabelled file (a "png" that is
really a page) gets no script and no origin.
The serializer hands the kind to the client (`asset.preview`), so the library and the server can
never disagree on which files open a preview.
"""
import codecs
from typing import Literal
from app.downloads import mediakind
PreviewKind = Literal["image", "pdf", "text"]
IMAGE_TYPES: dict[str, str] = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"avif": "image/avif",
"bmp": "image/bmp",
}
PDF_TYPE = "application/pdf"
# Read as text: notes, data, subtitles, config and source. HTML and SVG belong here (see above).
TEXT_EXTS: frozenset[str] = frozenset(
{
"txt", "text", "md", "markdown", "log", "csv", "tsv", "json", "jsonl", "ndjson", "xml",
"yaml", "yml", "toml", "ini", "conf", "cfg", "properties", "env", "srt", "vtt", "ass", "ssa",
"sub", "nfo", "cue", "m3u", "m3u8", "html", "htm", "xhtml", "svg", "css", "scss", "less",
"js", "mjs", "cjs", "ts", "tsx", "jsx", "py", "rb", "php", "pl", "lua", "go", "rs", "java",
"kt", "kts", "swift", "c", "h", "cc", "cpp", "hpp", "cs", "sh", "bash", "zsh", "fish", "ps1",
"bat", "cmd", "sql", "r", "dart", "diff", "patch", "tex", "rst", "adoc", "gradle", "dockerfile",
}
) # fmt: skip
TEXT_TYPE = "text/plain; charset=utf-8"
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
TEXT_PREVIEW_BYTES = 1 << 20
# A NUL byte in the first this-many bytes means the "text" is not text.
BINARY_SNIFF_BYTES = 8192
def kind(media_kind: str | None, container: str | None) -> PreviewKind | None:
"""The preview a library file gets, or None (a media file plays instead; others download)."""
if media_kind != mediakind.FILE:
return None
ext = (container or "").lower()
if ext in IMAGE_TYPES:
return "image"
if ext == "pdf":
return "pdf"
if ext in TEXT_EXTS:
return "text"
return None
def media_type(preview: PreviewKind, container: str | None) -> str:
"""The Content-Type a preview is served with: from this table, never guessed from the name."""
if preview == "image":
return IMAGE_TYPES[(container or "").lower()]
if preview == "pdf":
return PDF_TYPE
return TEXT_TYPE
# A text that is not UTF-8 is read as Central European Windows: what an old Hungarian subtitle or
# note is written in (its ő and ű are not in Latin-1).
FALLBACK_ENCODING = "cp1250"
_BOMS = (
(codecs.BOM_UTF8, "utf-8-sig"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
)
def looks_binary(head: bytes) -> bool:
"""Whether the start of a "text" file is binary after all (a NUL in it)."""
return b"\x00" in head[:BINARY_SNIFF_BYTES]
def text_utf8(head: bytes) -> bytes | None:
"""A text file's head as UTF-8, the one charset the preview is served in; None when it is not text.
A byte-order mark names its encoding (UTF-16 is full of NULs, so it is read before the binary
check); else UTF-8 is kept as it is — a character cut at the size cap included, the reader shows
that one as a replacement mark — and anything else is read as `FALLBACK_ENCODING`."""
for bom, encoding in _BOMS:
if head.startswith(bom):
return head.decode(encoding, errors="replace").encode()
if looks_binary(head):
return None
try:
head.decode("utf-8")
except UnicodeDecodeError as e: # silent-ok: not UTF-8 is an answer — read it as the fallback
if not (e.reason == "unexpected end of data" and e.end == len(head)):
return head.decode(FALLBACK_ENCODING, errors="replace").encode()
return head
+70 -32
View File
@@ -34,6 +34,28 @@ _IMAGE_FORMATS = {
}
def primary_streams(data: dict) -> tuple[dict | None, dict | None]:
"""(video stream, audio stream) — the first of each kind — from a NON-None ffprobe JSON dict, by
the same reading the stored facts use: a still image / image-sequence container has neither (it
is a generic 'file'), and a video stream flagged `attached_pic` (an mp3/m4a's cover art) is not a
video track. The phone-safe rule (`playable`) judges exactly these two streams."""
fmt_names = {
n.strip() for n in ((data.get("format") or {}).get("format_name") or "").split(",")
}
if fmt_names & _IMAGE_FORMATS:
return None, None
video = audio = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and video is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
video = stream
elif kind == "audio" and audio is None:
audio = stream
return video, audio
def _facts_from_probe(
data: dict,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
@@ -45,63 +67,79 @@ def _facts_from_probe(
fmt_names = {n.strip() for n in (fmt.get("format_name") or "").split(",")}
if fmt_names & _IMAGE_FORMATS:
return None, None, None, None, None
vcodec = acodec = None
width = height = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and vcodec is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
vcodec = stream.get("codec_name")
width = stream.get("width")
height = stream.get("height")
elif kind == "audio" and acodec is None:
acodec = stream.get("codec_name")
video, audio = primary_streams(data)
vcodec = video.get("codec_name") if video else None
width = video.get("width") if video else None
height = video.get("height") if video else None
acodec = audio.get("codec_name") if audio else None
duration = None
try:
raw = fmt.get("duration")
if raw is not None:
duration = int(float(raw))
# silent-ok: an unusable duration is stored as no duration, which the UI renders as unknown
# rather than as zero — the honest state, and the one a re-probe can still fill in.
except (TypeError, ValueError):
duration = None
return vcodec, acodec, width, height, duration
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (an unprobeable file — or a still image — is treated as a generic
non-media file, so it becomes a download-only 'file' rather than a broken 'video')."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def probe_json(path: Path) -> dict | None:
"""The full ffprobe JSON (streams + format) of `path`, or None when ffprobe could not read it —
the one call both the stored facts (`facts`) and the phone-safe rule read from."""
return editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def facts(
data: dict | None,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) of a `probe_json` result; all None when it is None
(an unprobeable file is treated as a generic non-media file, so it becomes a download-only 'file'
rather than a broken 'video')."""
if data is None:
return None, None, None, None, None
return _facts_from_probe(data)
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (a still image is a generic 'file', see `primary_streams`)."""
return facts(probe_json(path))
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None]:
"""(vcodec, acodec, width, height) for the stored download asset: PROBE the finished file on disk
(the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources (Facebook among
them), so deriving media_kind from it there labelled a real H.264 video a download-only 'file'.
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) for the stored download asset: PROBE the finished
file on disk (the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources
(Facebook among them), so deriving media_kind from it there labelled a real H.264 video a
download-only 'file'. It omits the duration too (a Facebook reel, an Instagram reel), and a NULL
duration hides the Library row's trim action although the file plays for minutes.
Precedence is by whether the probe RAN, not per-field: if ffprobe read the file, its codec verdict
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
`probe or info` fallback would let `info`'s stale vcodec resurrect a video track the file doesn't
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions still gap-fill from `info` in either case."""
mislabelling an audio download as a broken in-app 'video'. When ffprobe could NOT read the file
at all, no codec is reported: the probe runs with the media demuxer whitelist, so a file it
cannot open is one the player and the editor cannot open either — and taking `info`'s codecs
for it would make it an editable 'video' whose bytes are whatever the remote site served. The
cosmetic dimensions and the duration still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
info_d = int(info["duration"]) if info.get("duration") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h
pv, pa, pw, ph, _dur = _facts_from_probe(data)
# Unreadable to the guarded probe: a download-only 'file' (no codecs → media_kind FILE).
return None, None, info_w, info_h, info_d
pv, pa, pw, ph, pd = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the cosmetic width/height the probe couldn't read.
return pv, pa, pw if pw is not None else info_w, ph if ph is not None else info_h
# gap-fills the width/height/duration the probe couldn't read.
return (
pv,
pa,
pw if pw is not None else info_w,
ph if ph is not None else info_h,
pd if pd is not None else info_d,
)
+404
View File
@@ -0,0 +1,404 @@
"""Capture's reads: what the library, the download tray, the share dialogs and the public watch pages
show. Each returns rows or plain values and writes nothing; whose a row is goes through
`userscope.owned_by` / `owned_row`, so a row of someone else's reads as a missing one.
The HTTP layer (`routes.downloads`, `routes.files`, `routes.public`) asks here and serializes; the
writes are the service's (`service`, `folders`, `links`, `quota`)."""
from collections.abc import Sequence
from datetime import datetime, timezone
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.apitokens import FilesCaller, Right
from app.collation import collated
from app.downloads import folders, links, quota
from app.downloads.models import (
DownloadJob,
DownloadLink,
DownloadProfile,
DownloadQuota,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
)
from app.downloads.status import FINISHED, JobStatus
from app.models import User
from app.userscope import owned_by, owned_row
from app.utils import like_escape
# Cap a listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# --- jobs and their files ------------------------------------------------------------------
def own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
return owned_row(db, DownloadJob, job_id, user)
def job(db: Session, job_id: int) -> DownloadJob | None:
"""A job by id, whoever's — for a public link, which already names its job."""
return db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""A file the caller may read: their own, or one shared with them."""
row = db.get(DownloadJob, job_id)
if row is None:
return None
if row.user_id == user.id or shared_with(db, job_id, user) is not None:
return row
return None
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob | None:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
row = own_job(db, caller.user, job_id)
if row is None or not caller.may(row.folder_id, Right.READ):
return None
return row
def asset_of(db: Session, job: DownloadJob | None) -> MediaAsset | None:
"""The file under a job, if it has one yet."""
return db.get(MediaAsset, job.asset_id) if job is not None and job.asset_id else None
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its
folder's key; one moved into an own folder (or someone else's, shared) is traced back by
`folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def queue(db: Session, user: User, recent: datetime) -> list[DownloadJob]:
"""The download tray: what is waiting, running, paused or failed, and what finished (done or
canceled) since `recent`. A done job is timed from when it became done, never from its last
write: a rename, a move or the GC's expiry warning touches an old file's row, and the tray would
announce it as just finished."""
return list(
db.execute(
select(DownloadJob)
.where(
owned_by(DownloadJob.user_id, user),
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
def statuses_by_source(db: Session, user: User, statuses: Sequence[str]) -> list[tuple[str, str]]:
"""(source_ref, status) of each of `user`'s jobs in one of `statuses`."""
return list(
db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
owned_by(DownloadJob.user_id, user), DownloadJob.status.in_(tuple(statuses))
)
).tuples()
)
def library(
db: Session,
user: User,
*,
folder_id: int | None,
listable: set[int] | None,
shared: bool,
term: str,
sort: str,
order: str,
lang: str,
) -> list[DownloadJob]:
"""The caller's done files (optionally one folder's, or only in the folders a token may list),
or the ones shared with them; searched by name and channel, sorted by when they were added, by
name (in the reader's alphabet) or by size. A file is added to its owner's list when it is done,
not when it was asked for (a download that failed for days and then went through is new); one
shared with the caller, when it was shared."""
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
owned_by(DownloadShare.to_user_id, user)
)
else:
stmt = stmt.where(owned_by(DownloadJob.user_id, user))
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
return list(db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars())
def recent_jobs(db: Session) -> list[DownloadJob]:
"""Everyone's newest jobs, for the admin's list."""
return list(
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
# --- shares --------------------------------------------------------------------------------
def shared_with(db: Session, job_id: int, user: User | int) -> DownloadShare | None:
"""The grant that hands this job to `user`, if there is one."""
return db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, user)
)
).scalar_one_or_none()
def shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
owned_by(DownloadShare.to_user_id, user),
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
# --- folders -------------------------------------------------------------------------------
def own_folder(db: Session, user: User, folder_id: int) -> FileFolder | None:
return owned_row(db, FileFolder, folder_id, user)
def folders_by_id(db: Session, ids: set[int]) -> dict[int, FileFolder]:
"""These folders, by id — for a reader that already knows which ones (a token's grants)."""
if not ids:
return {}
return {f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
def folders_with_totals(
db: Session, user: User, listable: set[int] | None
) -> list[tuple[FileFolder, int, int]]:
"""The user's folders (only `listable` ones, when given) with how many done files each holds and
their size. A source folder its files have all left stays (count 0)."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(owned_by(FileFolder.user_id, user))
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).tuples()
return [(f, count, int(size or 0)) for f, count, size in rows]
def folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
def linked_folders(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
# --- links ---------------------------------------------------------------------------------
def own_link(db: Session, user: User, link_id: int) -> DownloadLink | None:
"""A link to one of the user's files or folders."""
link = db.get(DownloadLink, link_id)
if link is None:
return None
target = (
owned_row(db, DownloadJob, link.job_id, user)
if link.job_id is not None
else owned_row(db, FileFolder, link.folder_id, user)
)
return link if target is not None else None
def links_to(
db: Session, *, job_id: int | None = None, folder_id: int | None = None
) -> list[DownloadLink]:
"""The links to one file or one folder, newest first."""
target = (
DownloadLink.job_id == job_id if job_id is not None else DownloadLink.folder_id == folder_id
)
return list(
db.execute(select(DownloadLink).where(target).order_by(DownloadLink.id.desc())).scalars()
)
def link_folder(db: Session, link: DownloadLink) -> FileFolder | None:
return db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def folder_job(db: Session, folder: FileFolder, job_id: int) -> DownloadJob | None:
"""One of the files a link to `folder` reaches (`links.in_folder`), while it is in it."""
return db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *links.in_folder(folder))
).scalar_one_or_none()
def owner_suspended(db: Session, job: DownloadJob) -> bool:
"""The job's owner is suspended (or gone): the worker leaves their queued jobs where they are."""
owner = db.get(User, job.user_id)
return owner is None or owner.is_suspended
# --- profiles ------------------------------------------------------------------------------
def profiles_for(db: Session, user: User) -> list[DownloadProfile]:
"""The built-in format profiles, then the user's own."""
return list(
db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True) | owned_by(DownloadProfile.user_id, user))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
).scalars()
)
def own_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""One of the user's own profiles (a built-in is no one's to change)."""
p = owned_row(db, DownloadProfile, profile_id, user)
return None if p is None or p.is_builtin else p
def usable_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""A profile the user may enqueue with: a built-in, or their own."""
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
return None
return p
def default_profile(db: Session) -> DownloadProfile | None:
"""The first built-in, for an enqueue that named no format."""
return db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
# --- storage and quota (admin) -------------------------------------------------------------
def storage_totals(db: Session) -> tuple[int, int, int]:
"""(files on disk, their bytes, every asset row). On disk is `quota.ON_DISK`: ready files, and
uploads waiting for or failed in their conversion — the same disk the footprint and the GC cap
count."""
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar_one()
return on_disk[0], int(on_disk[1]), total_assets
def done_files_per_user(db: Session) -> dict[int, int]:
"""How many finished files each user's Files lists."""
return dict(
db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
)
.tuples()
.all()
)
def custom_quota(db: Session, user_id: int) -> DownloadQuota | None:
"""The admin's own limits for this user, if any were set."""
return db.get(DownloadQuota, user_id)
+107 -12
View File
@@ -1,26 +1,45 @@
"""Per-user download limits + footprint accounting.
A user's *footprint* is the total size of the ready assets their library references (cache hits
count — it's "how much disk you're responsible for", not "how much you personally downloaded").
count — it's "how much disk you're responsible for", not "how much you personally downloaded"). An
upload counts from the moment it lands, ready or not: its original is on disk while its conversion
waits, and stays there when the conversion fails (for Retry) — bytes a ready-only sum never saw.
Limits come from a per-user DownloadQuota row if the admin set one, else the sysconfig defaults.
`unlimited` (e.g. the admin's own account) bypasses the byte cap.
Enforcement split: max_jobs + max_bytes are checked at enqueue (they bound holdings); per-user
max_concurrent is enforced by the worker at claim time (it bounds simultaneous downloads).
DOWNLOAD jobs additionally consult `remaining_bytes` in the worker, because enqueue can only compare
the footprint as it stood when the job was created and earlier jobs may have filled the cap since.
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
"""
from dataclasses import dataclass
from sqlalchemy import func, select
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app import sysconfig
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.downloads.status import JobKind, JobStatus
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.userscope import owned_by
# Jobs a user is considered to be "holding" (occupy a job slot / footprint). Terminal failures
# (error, canceled) don't count.
_HOLDING = ("queued", "running", "paused", "done")
_RUNNING = ("queued", "running")
_HOLDING = (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE)
_RUNNING = (JobStatus.QUEUED, JobStatus.RUNNING)
# An asset whose file is on disk: a ready one, or an upload in any state (its original lands at
# ingest, see the module docstring). Every byte sum — a user's footprint, the GC's total-cap pass,
# the admin's storage view — reads it, so they all count the same disk.
ON_DISK = or_(
MediaAsset.status == "ready",
and_(MediaAsset.source_kind == "upload", MediaAsset.rel_path.is_not(None)),
)
@dataclass
@@ -49,7 +68,8 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
# "at limit" checks — the worker's claim-window exclusion vs at_concurrency_limit — since a
# 0-limit user is "capped" with zero running jobs, which the SQL exclusion can't see. Keep
# them consistent by never letting the effective limit drop below 1. (To BLOCK a user's
# downloads, suspend them or set max_jobs=0, not max_concurrent=0.)
# downloads, suspend them — the worker leaves a suspended user's queued jobs unclaimed — or
# set max_jobs=0, not max_concurrent=0.)
return QuotaLimits(row.max_bytes, max(1, row.max_concurrent), row.max_jobs, row.unlimited)
return QuotaLimits(
max_bytes=sysconfig.get_int(db, "download_default_max_bytes"),
@@ -60,12 +80,17 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
def footprint(db: Session, user_id: int) -> int:
"""Total bytes of the distinct ready assets this user's active jobs reference."""
"""Total bytes of the distinct assets this user's jobs hold on disk: the ready ones their active
jobs reference, and every upload of theirs — including one whose conversion failed, which keeps
its original (see the module docstring)."""
held_assets = (
select(DownloadJob.asset_id)
.where(
DownloadJob.user_id == user_id,
DownloadJob.status.in_(_HOLDING),
owned_by(DownloadJob.user_id, user_id),
or_(
DownloadJob.status.in_(_HOLDING),
and_(DownloadJob.job_kind == JobKind.UPLOAD, DownloadJob.status == JobStatus.ERROR),
),
DownloadJob.asset_id.is_not(None),
)
.distinct()
@@ -73,7 +98,7 @@ def footprint(db: Session, user_id: int) -> int:
return int(
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.id.in_(held_assets), MediaAsset.status == "ready"
MediaAsset.id.in_(held_assets), ON_DISK
)
).scalar()
or 0
@@ -85,7 +110,7 @@ def _count(db: Session, user_id: int, states) -> int:
db.execute(
select(func.count())
.select_from(DownloadJob)
.where(DownloadJob.user_id == user_id, DownloadJob.status.in_(states))
.where(owned_by(DownloadJob.user_id, user_id), DownloadJob.status.in_(states))
).scalar()
or 0
)
@@ -104,7 +129,7 @@ def at_concurrency_limit(db: Session, user_id: int) -> bool:
lim = resolve(db, user_id)
if lim.unlimited:
return False
running = _count(db, user_id, ("running",))
running = _count(db, user_id, (JobStatus.RUNNING,))
return running >= lim.max_concurrent
@@ -122,6 +147,21 @@ def check_enqueue(db: Session, user_id: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, used)
def remaining_bytes(db: Session, user_id: int) -> int | None:
"""Storage this user may still spend on ONE download: ``None`` for an unlimited user, else the
cap minus the current footprint, floored at 0.
Zero is a real answer, not "no limit" — it means the cap is already met and the caller must
REFUSE rather than proceed unbounded. `check_enqueue` cannot make that refusal on the caller's
behalf: it ran when the job was queued, while the user was still under the cap, and the jobs
that filled it have landed since. Keeping the arithmetic here (rather than inline in the worker)
is what keeps every max_bytes decision in this module."""
lim = resolve(db, user_id)
if lim.unlimited:
return None
return max(0, lim.max_bytes - footprint(db, user_id))
def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
"""Like check_enqueue, but for an upload whose size is KNOWN at ingest: also reject when adding
this file would push the footprint over max_bytes. A download can't pre-check its size (unknown
@@ -138,6 +178,18 @@ def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def fits_after_conversion(db: Session, user_id: int, old_bytes: int, new_bytes: int) -> bool:
"""Whether an upload converted from `old_bytes` (what it is charged at now) to `new_bytes` keeps
its owner within the cap. A re-encode can come out larger than the original, and nothing else
re-checks it once the upload was let in."""
if new_bytes <= old_bytes: # a smaller file never breaks a cap the larger one was let in under
return True
lim = resolve(db, user_id)
if lim.unlimited:
return True
return footprint(db, user_id) - old_bytes + new_bytes <= lim.max_bytes
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
@@ -150,3 +202,46 @@ def usage(db: Session, user_id: int) -> dict:
"max_jobs": lim.max_jobs,
"unlimited": lim.unlimited,
}
def set_custom(
db: Session,
user_id: int,
*,
max_bytes: int | None = None,
max_concurrent: int | None = None,
max_jobs: int | None = None,
unlimited: bool | None = None,
) -> None:
"""The admin's own limits for a user: starts from what applies now, then each value given (a
size floored at 0, a count at 1). The handler commits."""
row = db.get(DownloadQuota, user_id)
if row is None:
cur = resolve(db, user_id)
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if max_bytes is not None:
row.max_bytes = max(0, max_bytes)
if max_concurrent is not None:
row.max_concurrent = max(1, max_concurrent)
if max_jobs is not None:
row.max_jobs = max(1, max_jobs)
if unlimited is not None:
row.unlimited = unlimited
db.flush()
def reset_custom(db: Session, user_id: int) -> bool:
"""Back to the defaults; False when the user had no limits of their own."""
row = db.get(DownloadQuota, user_id)
if row is None:
return False
db.delete(row)
db.flush()
return True
+53 -10
View File
@@ -11,18 +11,22 @@ writes the terminal job/asset state itself (done / error, or a cooperative pause
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
"""
import logging
import os
import shutil
import subprocess
import sys
import app.tables # noqa: F401 — a fresh interpreter: every table, before the first query
from app import logs
from app import logsink
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.models import DownloadJob
from app.downloads import formats
from app.downloads.status import JobKind
from app.downloads.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
log = logs.get_logger(__name__)
def _deprioritize() -> None:
@@ -52,12 +56,21 @@ def _deprioritize() -> None:
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
# T201: a usage line is for the person who typed the command. Logging is already
# configured here, and in a deployment it writes JSON records — the one reader of this
# line wants the bare sentence, not a record with a request id they do not have.
print( # noqa: T201
"usage: python -m app.downloads.runner <job_id>", file=sys.stderr
)
raise SystemExit(2)
job_id = int(sys.argv[1])
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
# output joins up with the worker's and the API's view of the same download.
logs.set_job(job_id)
logsink.start() # flushed at exit (atexit) — this process has no shutdown hook of its own
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
@@ -71,14 +84,44 @@ def main() -> None:
if asset_id is None:
return
_run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id)
def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) -> None:
"""Run the job, and be the last stop before Python's own exception hook.
What gets past the pipeline is a failure it could not even record (the database refused the
write that fails the job). Python's hook would print it to stderr raw — and its context is the
download's own failure, whose text is the URL or ffmpeg's stderr naming the file. Here it is
one line with the whole chain scrubbed; the non-zero exit then has the worker fail the job
(`cause=child_exit`)."""
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
from app.downloads.pipeline import _download, _process_edit
from app.downloads.pipeline import _convert_upload, _download, _process_clip, _process_edit
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
with formats.hiding(source_ref):
try:
if job_kind == JobKind.EDIT:
_process_edit(job_id, asset_id)
elif job_kind == JobKind.UPLOAD:
_convert_upload(job_id, asset_id)
elif job_kind == JobKind.CLIP:
_process_clip(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
# TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
log.error( # noqa: TRY400
"the download child for job %s failed",
job_id,
extra={
**logs.event(
"download.child.failed", job_id=job_id, error_type=type(exc).__name__
),
**logs.scrubbed_stack(exc, formats.log_safe),
},
)
raise SystemExit(1) from None
if __name__ == "__main__":
+460 -44
View File
@@ -1,25 +1,42 @@
"""Enqueue + cache resolution for the download center (the per-user layer over MediaAsset).
"""Capture's writes: enqueue + cache resolution, the job transitions, and what the library's
dialogs change (profiles, resume points, shares, a file's display fields).
`enqueue` is the single entry point used by the API: it resolves (or creates) the shared
MediaAsset for the requested source+format, links a per-user DownloadJob to it, and — on a
cache hit (asset already ready) — marks the job done instantly with no re-download. The worker
handles everything else. State transitions used by the routes (pause/resume/cancel/delete) and
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
handles everything else. Every function here flushes and leaves the commit to its caller (the
handler); the folder, link and quota writes are `folders`, `links` and `quota`.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete as delete_rows
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import clip as clipmod
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import DISPLAY_TEXT_MAX, User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.downloads.models import (
DownloadJob,
DownloadProfile,
DownloadShare,
LibraryState,
MediaAsset,
)
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.userscope import owned_by, owned_row
class EditError(Exception):
@@ -37,12 +54,15 @@ def source_url(source_kind: str, source_ref: str) -> str:
def reference_url(job, asset) -> str | None:
"""The clean "downloaded from" URL for a job — shared by the private Downloads page and the public
"""The clean "downloaded from" URL for a job — shared by the Files library and the public
watch page so both surfaces always agree. None for edit derivatives (a clip's source is the user's
own earlier download, not a web page); else the canonical page URL yt-dlp recorded; else one derived
from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == "edit":
own earlier download, not a web page) and for a Plex clip (the library is not a public page); a
YouTube clip links its video at the clip's start; else the canonical page URL yt-dlp recorded; else
one derived from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == JobKind.EDIT:
return None
if job.job_kind == JobKind.CLIP:
return clipmod.reference_url(job.edit_spec or {})
if asset is not None and asset.source_webpage_url:
return asset.source_webpage_url
if job.source_kind in ("youtube", "url"):
@@ -50,6 +70,23 @@ def reference_url(job, asset) -> str | None:
return None
def library_left_behind(db: Session, files_root: str) -> bool:
"""True when the library names files and none of the newest of them is under `files_root` — a
fresh, empty volume mounted in the library's place (a compose file changed under an install, its
old volume left behind). The mount check cannot see that: the new volume is a real mount."""
rels = (
db.execute(
select(MediaAsset.rel_path)
.where(MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None))
.order_by(MediaAsset.id.desc())
.limit(20)
)
.scalars()
.all()
)
return bool(rels) and not any(storage.abs_path(files_root, r).exists() for r in rels if r)
def get_or_create_asset(
db: Session, source_kind: str, source_ref: str, format_sig: str
) -> MediaAsset:
@@ -62,7 +99,12 @@ def get_or_create_asset(
MediaAsset.source_ref == source_ref,
MediaAsset.format_sig == format_sig,
)
asset = db.execute(stmt).scalar_one_or_none()
# Locked to the caller's commit: a release or the GC freeing this asset meanwhile (they lock it
# too) either waits for the new hold, or has deleted the row by the time this read returns — and
# the caller then gets a fresh one, never a job pointing at a row that is gone.
asset = db.execute(
stmt.with_for_update().execution_options(populate_existing=True)
).scalar_one_or_none()
if asset is not None:
# A previously-FAILED shared asset must be re-attempted for the new job about to hold it —
# reset it to pending (+clear the error) so the worker actually re-downloads instead of
@@ -80,11 +122,15 @@ def get_or_create_asset(
status="pending",
ref_count=0,
)
db.add(asset)
try:
db.flush()
# A SAVEPOINT: losing the race undoes only this insert, never the caller's own pending writes
# (Retry clears the job's error before it gets here; a whole-session rollback threw that away).
with db.begin_nested():
db.add(asset)
db.flush()
# silent-ok: the expected outcome of the race this savepoint exists for — two jobs wanting the
# same asset. Losing it means the other insert won, and re-selecting is the whole handling.
except IntegrityError:
db.rollback()
asset = db.execute(stmt).scalar_one()
return asset
@@ -174,29 +220,33 @@ def enqueue(
profile_id=profile_id,
profile_snapshot=spec,
display_name=display_name
or (asset.title[:255] if asset.status == "ready" and asset.title else None),
status="queued",
or (asset.title[:DISPLAY_TEXT_MAX] if asset.status == "ready" and asset.title else None),
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
# Cache hit: the file already exists — no worker round-trip needed.
if asset.status == "ready":
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
def _clip_title(source_title: str | None, display_name: str | None) -> str:
def clip_title(source_title: str | None, display_name: str | None) -> str:
# Clipped to DISPLAY_TEXT_MAX, not to a literal: this feeds `DownloadJob.display_name` through
# `enqueue_edit`, the same column `display_text` guards. A machine-made title, so it is clipped
# rather than run through `display_text` — blank here means "no title", not None.
if display_name:
return display_name[:255]
return display_name[:DISPLAY_TEXT_MAX]
base = source_title or "Clip"
return f"{base} (clip)"[:255]
return f"{base} (clip)"[:DISPLAY_TEXT_MAX]
def enqueue_edit(
@@ -231,7 +281,7 @@ def enqueue_edit(
ref = f"{user_id}:{src_asset.id}"[:512]
asset = get_or_create_asset(db, "edit", ref, sig)
if not asset.title:
asset.title = _clip_title(src_asset.title, display_name)
asset.title = clip_title(src_asset.title, display_name)
asset.uploader = src_asset.uploader
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
@@ -243,27 +293,116 @@ def enqueue_edit(
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="edit",
job_kind=JobKind.EDIT,
source_kind="edit",
source_ref=ref,
source_job_id=source_job.id,
source_asset_id=src_asset.id,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title,
status="queued",
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # identical clip already produced for this user → instant
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
def _clip_source_facts(db: Session, spec: dict) -> dict:
"""Title, uploader, thumbnail and length of what a clip is cut from, read from our own catalog.
Raises ClipError when a Plex item is unknown or its file cannot be read here — said at enqueue,
rather than as a job that fails in the queue a minute later."""
if spec["source"] == "youtube":
video = db.get(Video, spec["ref"])
if video is None: # not in the catalog: titled "Clip" until yt-dlp names it (worker)
return {"title": None, "uploader": None, "thumbnail_url": None, "length_s": None}
channel = db.get(Channel, video.channel_id)
return {
"title": video.title,
"uploader": channel.title if channel else None,
"thumbnail_url": video.thumbnail_url,
"length_s": video.duration_seconds,
}
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
if item is None:
raise clipmod.ClipError("not_found")
if plex_paths.local_media_path(db, item.file_path) is None:
raise clipmod.ClipError("media_missing")
title = item.title
show = db.get(PlexShow, item.show_id) if item.show_id else None
if show is not None:
episode = (
f"S{item.season_number}·E{item.episode_number}"
if item.season_number is not None and item.episode_number is not None
else None
)
title = " — ".join(p for p in (show.title, episode, item.title) if p)
return {
"title": title,
"uploader": None,
# Plex art is served through our own authenticated routes, never a public URL; the worker
# cuts the clip's poster from the clip itself.
"thumbnail_url": None,
"length_s": item.duration_s,
}
def enqueue_clip(
db: Session, user_id: int, spec: dict, display_name: str | None = None
) -> DownloadJob:
"""Queue an ad-hoc clip (`clipmod.normalize`d spec) for `user_id`: a NEW per-user asset cut
straight from the source — a YouTube range or a Plex library file (see downloads/clip).
Raises ClipError (unknown or unreachable source, a range past its end) or quota.QuotaExceeded."""
facts = _clip_source_facts(db, spec)
# One second of grace: a mark set on the last frame may round past the catalog's whole seconds.
if facts["length_s"] and spec["end_s"] > facts["length_s"] + 1:
raise clipmod.ClipError("past_end")
quota.check_enqueue(db, user_id)
ref = clipmod.asset_ref(user_id, spec)[:512]
asset = get_or_create_asset(db, "clip", ref, clipmod.sig(spec))
if not asset.title:
asset.title = clip_title(facts["title"], display_name)
asset.uploader = facts["uploader"]
asset.thumbnail_url = facts["thumbnail_url"]
asset.duration_s = round(clipmod.duration(spec))
asset.media_kind = mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind=JobKind.CLIP,
source_kind="clip",
source_ref=ref,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # the same range was already cut for this user → instant
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.flush()
return job
@@ -283,6 +422,8 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
if fields.get("status") == JobStatus.DONE:
return _finalize_done(db, job_id, {**fields, "ready_at": datetime.now(timezone.utc)})
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
@@ -296,6 +437,41 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
return result.rowcount > 0
def _finalize_done(db: Session, job_id: int, fields: dict) -> bool:
"""`finalize_if_running` for a job that finished: done is when a file's retention starts, its
folder's unless the user kept it. The expiry is written only while the job is still where it was
read from — a move, a keep, or its folder's delete (which sends its files home) landing while the
folder's retention was read sends the read round again, never stamping a folder's retention on a
file now in another one."""
while True:
held = db.execute(
select(DownloadJob.folder_id, DownloadJob.kept, DownloadJob.status).where(
DownloadJob.id == job_id
)
).one_or_none()
if held is None or held.status != JobStatus.RUNNING:
return False
stamped = {
**fields,
"expires_at": folders.done_expiry(db, held.folder_id, held.kept),
"expiry_warned": False,
}
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.RUNNING.value,
DownloadJob.folder_id.is_not_distinct_from(held.folder_id),
DownloadJob.kept == held.kept,
)
.values(**stamped)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
if hit is not None:
return True
def release_asset(db: Session, job: DownloadJob) -> None:
"""Drop this job's hold on its asset when it leaves a holding state. Once NO job holds the
asset anymore, delete the file + row immediately — a deleted download should free its disk,
@@ -310,24 +486,32 @@ def release_asset(db: Session, job: DownloadJob) -> None:
re-enqueuing the same (source,format) — get_or_create_asset would reset that row to `pending` and
+1 it, and deleting it under that would FK-null the new job's asset (a lost update on ref_count).
A ready asset is still freed at ref==0 (its file is the point); an orphaned errored row is cheap
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass."""
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass.
An UPLOAD is the exception: it is never shared or re-enqueued (its ref is unique), and while it
waits for or failed its phone-safe conversion the original file already sits at `rel_path` — so
it is freed at ref==0 whatever its state, except mid-conversion (`downloading`: the worker child
still reads it; the GC upload pass takes it once the child has stopped)."""
if not job.asset_id or job.status not in HOLDING:
return
asset = db.get(MediaAsset, job.asset_id)
if asset is None:
return
new_count = decr_ref_count(db, asset.id)
if new_count == 0 and asset.status == "ready":
freeable = asset.status == "ready" or (
asset.source_kind == "upload" and asset.status != "downloading"
)
if new_count == 0 and freeable:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
# --- Job state transitions ---------------------------------------------------------------------
# The single home for the pause/resume/cancel/delete/redownload transitions + their asset
# bookkeeping (release, failed-asset reset, fresh-asset rebuild), so the routes are thin callers and
# the worker/GC could share the same definitions (B-sub-8). Each commits its own transition; the
# route keeps only auth + HTTP mapping + serialization.
# the worker/GC could share the same definitions (B-sub-8). Each flushes its transition and the caller
# commits it; the route keeps only auth + HTTP mapping + serialization.
def pause(db: Session, job: DownloadJob) -> None:
@@ -335,24 +519,57 @@ def pause(db: Session, job: DownloadJob) -> None:
in any other state."""
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING):
job.status = JobStatus.PAUSED
db.commit()
db.flush()
def resume(db: Session, job: DownloadJob) -> None:
"""Requeue a paused/errored job for a fresh attempt, clearing its error. If the SHARED asset itself
failed, reset it to pending so the worker actually re-downloads — otherwise the requeued job would
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset)."""
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset).
A cookie the user added AFTER the job was queued moves the job onto their private asset first
(`_follow_the_cookie`): the worker only ever attaches a cookie to that identity, so without the move
the Retry a login-wall message asks for would fail on the same wall."""
if job.status in (JobStatus.PAUSED, JobStatus.ERROR):
job.progress = 0
job.error = None
job.error_code = None
job.error_detail = None
if job.asset_id:
asset = db.get(MediaAsset, job.asset_id)
if asset is not None:
asset = _follow_the_cookie(db, job, asset)
if asset and asset.status == "error": # MediaAsset.status (no enum), not a JobStatus
asset.status = "pending"
asset.error = None
job.status = JobStatus.QUEUED
db.commit()
db.flush()
def _follow_the_cookie(db: Session, job: DownloadJob, asset: MediaAsset) -> MediaAsset:
"""Re-point a requeued job at the asset enqueue would pick NOW, when the user has a cookie for the
link's host but the job still holds an asset the worker won't attach it to (queued before the
cookie existed). Returns the asset the job holds afterwards.
Only a `pending` / `error` asset moves — neither has a file nor a download in flight. A `ready` one
already has the file, and a `downloading` one may be this job's own child still being stopped by
the pause; the next Retry after it settles moves the job then. Removing a cookie never moves a job
back: the worker attaches nothing without a credential, so the private asset is as good as any."""
if asset.status not in ("pending", "error"):
return asset
url = source_url(job.source_kind, job.source_ref)
salt = credentials.salt_for_enqueue(db, job.user_id, url)
spec = formats.normalize(job.profile_snapshot)
if salt is None or credentials.is_authed_asset(asset, spec, job.user_id, url):
return asset
target = get_or_create_asset(
db, job.source_kind, job.source_ref, formats.format_sig(spec, salt=salt)
)
populate_from_catalog(db, target)
release_asset(db, job) # drop the old hold BEFORE re-pointing — it keys off job.asset_id
incr_ref_count(db, target.id)
job.asset_id = target.id
return target
def cancel(db: Session, job: DownloadJob) -> None:
@@ -361,25 +578,43 @@ def cancel(db: Session, job: DownloadJob) -> None:
if job.status not in (JobStatus.DONE, JobStatus.CANCELED):
release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
db.flush()
def delete(db: Session, job: DownloadJob) -> None:
"""Delete a job, releasing its asset hold first (which frees the file iff this was the last holder)."""
release_asset(db, job)
db.delete(job)
db.commit()
db.flush()
def redownload(db: Session, job: DownloadJob) -> None:
"""Rebuild a finished/failed job's file into a FRESH, private asset (the cache sig is salted with
the job id + current asset id, so it never reuses the dedup pool) and requeue the SAME job row —
the new asset's own id, so it never reuses the dedup pool) and requeue the SAME job row —
so any public share link (DownloadLink -> job_id) keeps resolving to the newly-downloaded file.
The old hold is dropped first (freeing the old file iff this job was its last holder). The caller
has already rejected edit clips + still-in-progress jobs (HTTP 409)."""
has already rejected edit clips + still-in-progress jobs (HTTP 409).
When the user has a cookie for the link's host, the salt carries it (`credentials.rebuild_salt`),
so the worker recognises the rebuild as their private identity and attaches the cookie — a cookie
download, or one the user added after a failure, rebuilds with the cookie."""
spec = formats.normalize(job.profile_snapshot)
sig = formats.format_sig(spec, salt=f"{job.id}:{job.asset_id}")
asset = get_or_create_asset(db, job.source_kind, job.source_ref, sig)
cookie = credentials.salt_for_enqueue(
db, job.user_id, source_url(job.source_kind, job.source_ref)
)
# The salt names the row's own id, which exists only once the row does: insert it under a unique
# placeholder, then write the real signature in the same transaction (nobody sees the placeholder).
asset = MediaAsset(
source_kind=job.source_kind,
source_ref=job.source_ref,
format_sig=f"rebuild:{uuid.uuid4().hex}",
status="pending",
ref_count=0,
)
db.add(asset)
db.flush()
asset.format_sig = formats.format_sig(spec, salt=credentials.rebuild_salt(asset.id, cookie))
db.flush()
populate_from_catalog(db, asset)
# Drop the old hold BEFORE re-pointing the job — release_asset keys off the job's current asset.
release_asset(db, job)
@@ -389,10 +624,191 @@ def redownload(db: Session, job: DownloadJob) -> None:
job.progress = 0
job.phase = None
job.error = None
job.error_code = None # a rebuild that fails again gets its own diagnosis, never the old one
job.error_detail = None
job.speed_bps = None
job.eta_s = None
# Re-derive the display name from the freshly cleaned title on completion (else an older, un-cleaned
# name would survive the rebuild).
job.display_name = None
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
db.commit()
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.flush()
# --- the library's writes ------------------------------------------------------------------
# What the HTTP layer changes for a user: each flushes, the handler commits.
def lock_own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's jobs, re-read under a row lock, for a transition that moves its asset's ref
count (resume, cancel, delete, redownload) or writes the status those decide on (pause). Two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds. None: not
theirs, or gone.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`finalize_if_running`), so the two can never deadlock."""
while True:
owned = owned_row(db, DownloadJob, job_id, user)
if owned is None:
return None
asset_id = owned.asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
return None
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def hold_for_share(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves None."""
if owned_row(db, DownloadJob, job_id, user) is None:
return None
return db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def create_profile(db: Session, user: User, name: str, spec: dict) -> DownloadProfile:
"""A format profile of the user's own, after their others."""
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
owned_by(DownloadProfile.user_id, user)
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
db.flush()
return p
def update_profile(profile: DownloadProfile, *, name: str | None, spec: dict | None) -> None:
"""Rename a profile and/or replace its spec; None leaves that one as it is."""
if name is not None:
profile.name = name[:80]
if spec is not None:
profile.spec = spec
def delete_profile(db: Session, profile: DownloadProfile) -> None:
db.delete(profile)
db.flush()
def set_position(db: Session, user: User, job_id: int, position: int) -> None:
"""The library player's resume point in a file the user can play; 0 clears it.
A clear is a statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon
of one stop may both clear it, and the second then deletes nothing rather than failing. A set is
an upsert, one row per (user, job) even when two checkpoints race, not a read-then-insert that
the second would fail on the unique key."""
if not position:
db.execute(
delete_rows(LibraryState).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id == job_id
)
)
return
now = datetime.now(timezone.utc)
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
def share(db: Session, job: DownloadJob, sender: User, recipient: User) -> bool:
"""Hand a file to another user, with a notice for them; False when it was already theirs."""
from app.notifications import create_notification
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job.id, owned_by(DownloadShare.to_user_id, recipient)
)
).first()
if exists is not None:
return False
db.add(DownloadShare(job_id=job.id, from_user_id=sender.id, to_user_id=recipient.id))
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": sender.email,
"job_id": job.id,
"title": job.display_name,
},
commit=False,
)
db.flush()
return True
def unshare(db: Session, job_id: int, recipient: User) -> bool:
"""Take a file back from one user (its owner's act, or the recipient's own dismissal); False
when it was not theirs."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, recipient)
)
).scalar_one_or_none()
if row is None:
return False
db.delete(row)
db.flush()
return True
# The display fields a user may set on their file; the file on disk keeps its own name.
DISPLAY_FIELDS = frozenset(
{"display_name", "display_uploader", "display_uploader_url", "extra_links"}
)
def set_display(job: DownloadJob, **fields: object) -> None:
"""Set a file's display fields (each already cleaned by the caller); a field not given is left."""
unknown = set(fields) - DISPLAY_FIELDS
if unknown:
raise ValueError(f"not a display field: {sorted(unknown)}")
for name, value in fields.items():
setattr(job, name, value)
def touch(asset: MediaAsset) -> None:
"""The file was just served: the cache's idle clock starts again."""
asset.last_access_at = datetime.now(timezone.utc)
def store_sha256(db: Session, asset_id: int, digest: str) -> None:
"""Keep a file's hash once it is known — only where none is, so a concurrent hash of the same
file never overwrites the first."""
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
+13
View File
@@ -0,0 +1,13 @@
"""The failures the download pipeline raises itself, with the whole story in the message.
`ffmpeg failed (1): …`, `yt-dlp produced no output file`, a post that shows no video: the message
says what happened, and the stack is only the `except` that caught it. A failure line leaves that
stack off (`pipeline._failure_stack`) — but ONLY for these. A plain `RuntimeError` is also what
Python raises for a bug of ours (a dict changed size during iteration, a generator's
StopIteration, `RecursionError`, `NotImplementedError`), and that one's stack is the only lead.
"""
class StatedFailure(RuntimeError):
"""A failure we raised on purpose, whose message is its full explanation. A `RuntimeError` so
every `except RuntimeError` that caught these before still does."""
+21
View File
@@ -33,6 +33,27 @@ HOLDING = frozenset(
{JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE, JobStatus.ERROR}
)
class JobKind(StrEnum):
"""What a job produces, which decides the worker path that runs it and what the library may do
with the result (only a DOWNLOAD has a web source to fetch again)."""
DOWNLOAD = "download" # yt-dlp fetches a source into a shared, deduplicated asset
EDIT = "edit" # a per-user trim/crop of a file already in the library
UPLOAD = "upload" # a file the user sent; the worker only converts it
CLIP = (
"clip" # a per-user range of a YouTube video or a Plex item, cut straight from the source
)
# A job kind with no web source to fetch again: Re-download refuses these.
NOT_REFETCHABLE = frozenset({JobKind.EDIT, JobKind.UPLOAD, JobKind.CLIP})
# States a route sets to STOP a running job. The worker's supervisor must step in (kill the child, free
# the asset, leave the route's status intact) when it observes one of these on a job it is running.
STOPPED_BY_ROUTE = frozenset({JobStatus.PAUSED, JobStatus.CANCELED})
# A job that will not move again on its own: the queue lists it only briefly after it got here.
FINISHED = frozenset({JobStatus.DONE, JobStatus.CANCELED})

Some files were not shown because too many files have changed in this diff Show More