Compare commits

...
809 Commits
Author SHA1 Message Date
peter 41ad538edc release: 0.114.0 (feature/e13-notifications-order) 2026-10-29 02:18:56 +01:00
peter 7489cab3bc chore(release): 0.114.0 2026-10-29 02:06:52 +01:00
peter 91256b10b0 test(nav): pin the restore's last look and the clearing of the name it looks for 2026-10-29 01:31:41 +01:00
peter b22e903174 fix(notifications): Earlier opens on Back only for a row that can be nowhere else, and Mark read reads read at once
A namesake in Earlier while the row itself was still in Needs you or Recent opened the fold, which took the place match away and left the name twice on the page: the focus fell to <body>. Row names are trimmed as Back saves them, so a title ending in a space still matches. Mark read handed the focus to a row that still said it was unread until the write landed; the row now reads read in the press's own render, and unread again if the write is refused.
2026-10-29 01:31:40 +01:00
peter d6e72d7576 fix(notifications): Back to a row that folded away since opens Earlier on it 2026-10-29 00:47:52 +01:00
peter e3a4593acf fix(nav): Back looks for a moved item only in the page's lists, once its own list has had its chance, and its reveal stops the scroll restore
The cross-list match took a same-named row of the app-wide download tray, and took it at the first look while a virtualized list had yet to render the item. The reveal of a moved item ran before the scroll restore's first frame, which put the old offset back over it. The pending restore's name is readable, for a page that folds rows away.
2026-10-29 00:47:52 +01:00
peter fa271a31b9 fix(notifications): Mark all read waits for the fresh list, is not offered where it cannot act, and a row says it is unread
The mask over a stale read lifted when the write settled, before the refetch landed, and the band took the rows back. A band holding only what asks by its kind (an access-requests nudge) left the button enabled with nothing to do. The unread dot was only drawn: the spoken label now says it, apart from the name Back finds the row by, and follows the row while it holds the focus.
2026-10-29 00:38:34 +01:00
peter 947efaba6e docs(design): the notification order's decision row carries what it was chosen over and why 2026-10-29 00:38:34 +01:00
peter 22d873ed49 fix(nav): an item Back finds in another list is brought into view 2026-10-28 22:04:07 +01:00
peter d825aff30d fix(notifications): a row is named by what it is about, not its kind alone 2026-10-28 22:04:06 +01:00
peter 9dd56e94d4 docs(design): the notification center's order and its Earlier fold 2026-10-28 21:54:00 +01:00
peter 25ad2ee1ba test(ui): a self-closing dialog leaves the focus in a menu that stays open 2026-10-28 21:54:00 +01:00
peter c86ec7f7a5 fix(nav): Back finds an item that moved to another list by its name 2026-10-28 21:54:00 +01:00
peter 0469812499 fix(notifications): Mark all read empties a band of read rows, and a read landing mid-write cannot refill it 2026-10-28 21:54:00 +01:00
peter ee658b8bfc test(plex): pin the late-checkpoint race, its window, and the bulk mark's push supersede 2026-10-28 21:53:59 +01:00
peter 52b93d0014 fix(plex): a restart that fails to start still deletes the session it moved aside 2026-10-28 21:53:59 +01:00
peter c3ad7684ea test(ui): pin the notifications Earlier fold as a plain Pressable
Its text is the disclosure's own label, not a title to select.
2026-10-28 20:44:01 +01:00
peter a5666bfde9 style(notifications): the Earlier fold reads as a section heading, like its siblings
It was a ghost button in body type under two uppercase section labels. It is now
the heading itself, its text the disclosure's button (24px tall, past the
WCAG 2.5.8 floor).
2026-10-28 20:42:18 +01:00
peter f49dc6532f feat(notifications): what asks for you comes first, then one stream; old read rows fold
The center showed two fixed sections, the server's above this browser's, so an
admin's "access requests waiting" sat under every server row, week-old read
"Download ready" notices among them. One order now, for both sources
(lib/inboxOrder): "Needs you" (unread, an access-requests nudge, a notice
waiting on an interaction), then everything else newest first, and read rows
older than a week fold under "Earlier (n)", closed by default.

A row that asked in this visit keeps its place once read: marking it read no
longer moves it to another list under the reader. Mark all read empties the
band, the client notices of the visit included (the server rows read at once,
not when the write lands). Marking a row read from its button hands the focus
to the row instead of dropping it on <body> as the button goes.
2026-10-28 20:01:21 +01:00
peter 0d410592e2 fix(plex): a stale finish refusal no longer sticks, and a refusal keeps the player's detail
A refusal that landed after a replay raised the refusal flag for the finish that watch made later,
so leaving skipped the watched patch. Each finish now resets the flag and a refusal acts only for
the finish it answers.

A refusal while the player still sat on the item (the up-next card) removed the detail query it
observes, dropping the card and restarting its countdown, or losing it if the refetch failed. It is
now refetched under the player and dropped only when the player leaves.
2026-10-28 19:05:24 +01:00
peter 7e5d4a7d93 fix(ui): a closing dialog's focus restore reaches the dialog left standing above it, and survives a menu
A dialog closing on its own under a standing modal skipped its restore, and the standing modal's
own target was a control inside the closed one: closing it left the focus on <body>. The skipped
target is now handed to the standing dialog, which falls back to it.

Focus on a row of one of the dialog's own menus (the popover engine portals them out of the dialog
root) read as the user walking away, so Back with a menu open left the focus on <body>. A focus in
a popover portal now counts as held, and one still standing is not taken from.
2026-10-28 19:05:24 +01:00
peter 24190e4c16 fix(plex): a checkpoint sent just before the finish no longer lands on the watched row
The 10 s tick in flight when the player finishes (or a refusal rollback whose state write in fact
committed) wrote its mid-item position onto the freshly watched row and scheduled a resume push
after the scrobble, so Plex showed the item in progress again. The checkpoint is now a conditional
write that skips a row marked watched in the last 30 s and answers like a finish; a rewatch past
that is checkpointed as before. Marking an item (or a season/show) watched also supersedes resume
pushes still in flight, as the near-end finish already did.
2026-10-28 18:56:12 +01:00
peter e96a562f14 fix(plex): a segment name ffmpeg never spells is refused; teardown no longer holds the registry lock
The artifact guard parsed the sequence with int(), so a zero-padded or non-ASCII-digit name read
as a segment the session had reached and parked the sync route for the full 20 s wait. It now
also requires the canonical spelling segment_name() produces.

Stopping, replacing, culling or reaping a session waited up to 3 s for ffmpeg and deleted a whole
segmented film under _lock, which every segment request and checkpoint takes. Teardown now signals
the process and moves its directory aside under the lock, and waits/deletes after it. A new session
on the same path still waits for the dying ffmpeg first (it writes by path), and the orphan sweep
recognises a moved-aside directory left by a crash.
2026-10-28 18:48:21 +01:00
peter 4084dd2f92 release: 0.113.0 (feature/e46-url-is-the-view) 2026-10-28 02:19:29 +01:00
peter 6aa6a6efdf chore(release): 0.113.0 2026-10-28 02:07:48 +01:00
peter 5686759bca fix(nav): a page settling keeps a reload's overlay entry; an old entry's sub-view goes to its own (review round 4)
- useUrlMirror: a change before the user has done anything since the load is the page settling (the
  rail stepping off a playlist deleted since, Files off a folder gone), not another view: the reload
  mark stays, and the player that reopens takes its entry over — one Back closes it, as before.
- useHistorySubview: an entry an older build wrote is a module's only when its _sub is the view the
  address names under that module's key (the card's push kept the season's `v` beside its `m`).
- Playlists: the rail's default selection and the page's "the rail moved there" read one helper.
- moduleUrl contract: writer literals in a choice's branches are scanned too; the sub-view census
  walks type arguments of any depth and refuses an aliased import.
- A stale comment on the nav press over a live search.
2026-10-28 01:39:26 +01:00
peter 99a33e7fd1 fix(history): a sub-view reads only its own module's from the entry it mounts on
The entry's _sub now names its owner (_subOf, the module's address key). Signed out on a Plex season,
the sign-in card took the season for one of its modes and rendered a card with an empty button; the
other way round, Plex took the card's register view for one of its own. An entry an older build wrote
is read as the module's only when its address names one of that module's views.
2026-10-28 00:29:40 +01:00
peter 8804bdde63 Merge branch 'main' into feature/e46-url-is-the-view 2026-10-28 00:11:30 +01:00
peter 35266fd7e8 release: 0.112.1 (bug/authlib-cve) 2026-10-28 00:08:40 +01:00
peter 6211e15611 chore(release): 0.112.1 2026-10-27 23:59:09 +01:00
peter 02906d0acb fix(nav): the address keeps what the app rewrites around it (review round 3)
- The Feed press over a live search ends only the search: the feed's own view stays named.
- Leaving a search steps back only off an entry the app pushed (_ytOwn); one a link or a reload
  opened is cleared in place, also where the Navigation API is missing.
- useUrlMirror rewrites a changed view through replaceView, so a reload's overlay marks go; its
  first write and its landing writes keep them for an overlay that reopens.
- Files and Playlists: a box emptied and filled again settles at once, never naming the ended search.
- Playlists: a linked player is scoped to the playlist the store started with, and only the rail's
  own move to the first playlist is no pick.
- Contract: writer objects are scanned to their matching brace, a computed or shorthand key is
  refused, and every useHistorySubview call hands a named definition.
- Lint: playwright/no-skipped-test also refuses fixme (gate snapshot regenerated).
- FEED_DEFAULT_SORT at the three remaining sites; linkedNotListed says to search by name.
2026-10-27 23:33:22 +01:00
peter 66e427dfa4 fix(nav): a search the address opened marks its entry, so a channel over it keeps it
A search reached by a reload or a link had no _yt mark on its entry, so a channel opened from its
results carried none, and a landing back on that channel (an overlay over it closing) ended the
search under it.
2026-10-27 22:43:17 +01:00
peter 011dea8eb7 fix(deps): Authlib 1.7.2 -> 1.9.1 for CVE-2026-96760 and CVE-2026-114616
One pin bumped in place; pip check passes and pip-audit finds nothing.
2026-10-27 21:46:16 +01:00
peter e2b725dba7 test(e2e): the Files sort round trip starts from a known sort and puts the account's back
A header press cycles the sort (asc, desc, none), so the case depended on what an earlier run left
in the account: the third run in a row pressed it to none.
2026-10-27 16:34:17 +01:00
peter 6d57d62ac8 fix(nav): a linked player, file and search survive what the app does by itself (review round 2)
- Playlists: a link over a playlist the account does not hold plays alone; the rail's own
  fallback no longer counts as a pick that gives the player up.
- The linked opener opens or clears nothing once the address names another player, and stops
  looking (and fetching its row) once it settled.
- Files: a list the server's cap cut — the sorted or searched one too — keeps the link, with a
  notice of its own (EN/HU).
- History: an in-place view rewrite (search refine, count, channel to channel, sub-view)
  drops a reloaded entry's overlay marks (replaceView); a landing on a channel opened from a
  search keeps the search; the Feed press on a search opens the plain feed.
- Files/Playlists: an emptied search is written at once.
- Client log: plf/cf (column text filters) are typed keys; the shell has none of its own.
- Contract: writers outside lib/ take a literal or a lib/ function's result; every URL key is
  sorted typed or not; the writer regexes are held by a fixture; sub-views named; the bookmark
  net reads page tags.
- Feed defaults in one constant; one live-search queue helper; Scheduler writes through the
  Activity codec; notRun copy (EN/HU) names Forward and a plain "Search YouTube" button.
2026-10-27 16:08:52 +01:00
peter 0d5c6273f1 fix(nav): the address follows the view through overlays, reloads and Back (review round 1)
- A URL mirror names the view again on every entry Back or Forward lands on, so a change made
  under a dialog, a drawer or a sub-view is not undone by the next reload.
- A view pushed from the entry a reload under a dialog started from no longer inherits its overlay
  mark; a reload under two overlays closes onto the page in one step.
- The linked player reopens over a live search, gives itself up when its list moves first, and
  plays alone over an emptied playlist; a capped Files list keeps the link it cannot resolve.
- The Feed press over a channel ends the search; leaving a search ends its asked mark; Files and
  Playlists searches end with their page; the feed always names its view and debounces its search.
- Search-box keys are declared per module and kept out of the client log below TRACE.
- Writers outside lib/ go through codecs; the URL contract scans computed keys, overlay writers and
  two-line reads, and holds every page to a bookmark round trip (Config, Activity, Playlists and the
  Files sort and viewer added).
2026-10-27 13:52:42 +01:00
peter 7b4e007680 feat(nav): the live YouTube search and Home's budget are named in the address
A YouTube search is `?yt=<term>` (+ `ytn` for a non-default count), so a reload, a link
or Back/Forward reopens it. It runs by itself only where it costs no quota: the free page
scrape, a search asked for in this document, or results still cached. An API search
reopened any other way shows its term and waits for "Search YouTube again".

Home's time budget is `hb`; the question's draw stays on the history entry.

An e2e spec opens every page's bookmark in a fresh browser context; CLAUDE.md and
docs/development.md name the module URL codec as the place for view state.
2026-10-27 11:51:29 +01:00
peter 3d26ca0efa feat(player): a bookmark reopens every player as it was, once
The Plex player names the playlist it plays through (v=player:<id>:playlist:<pid>), so a bookmark
rebuilds the queue from the playlist's own order while the item is still in it.

Found in the browser and fixed:
- after a reload the shell's player opened a second one over Home and the feed: it read the catalog
  row the page had fetched (a disabled query still reads the shared cache); it now waits for ready.
- every reload under an open overlay left a dead history step: the first overlay to open takes over
  the entry the document was loaded on.
- the feed's address was implicit, the same as the start-page landing, so a reloaded or bookmarked
  feed opened as Home: the feed now names itself (page=feed) like every page.
- a linked file or video that is gone was said, but stayed in the address and was said again on
  every reload: its keys are taken off.

The overlay address is written in one place (the entry is named once it is current), and the player
keys only the codec reads are no longer exported.
2026-10-27 07:33:55 +01:00
peter fa893c73a3 feat(player): the YouTube player and the file viewer are named in the address
The player names what it plays in its own history entry (watch, wrel, wtab): a reload or a
bookmark reopens it on the page that hosts it, with that page's list as the queue, or alone
from the catalog (GET /api/videos/{id}/row); a page that hosts none gets the shell's. Back
lands on the page's address without it (useBackToClose's url; ui/Modal passes it on). The
Files viewer and player do the same (fview, fplay). The Plex and channel-manager views are
always written in full, as their fallback is remembered state.
2026-10-27 05:29:49 +01:00
peter d4f24de4ba feat(nav): tabs, list views and name searches are named in the address
Settings, Configuration and Users name their tab (stab/cfgtab/utab); a link's tab wins and a tab pre-selected
while its page is on screen now switches it. A playlist's sort, filters and grouping (plsort/plf/plgroup), the
Files sort (fsort, always written) and the name searches of Channels, Files, Playlists and Plex (cq/fq/plq/pq+pqf)
reopen from the address; the searches are written once the typing pauses.
2026-10-26 01:13:39 +01:00
peter 91d36b3357 fix(channel): the channel page's address names the channel and its own view, not the page under it
Opening a channel kept every key of the page under it (`?q=music&sort=oldest&chan=…`, a sort the
channel did not show), and the channel's own tab, sort and content types lived in its state alone, so a
reload or a bookmark lost them. Now the channel's address is `channelUrl`: the shell's keys and `chan`,
shared by `openChannel` and the channel link's href. The page writes its view (`chtab`, `chsort`,
`chtypes`, lib/channelPageUrlState) and reopens it; these are shell keys, cleared by a page switch and
by closing it in place. While a channel stands over a page, no module store writes its keys (`urlPage`);
Back finds them on the page's own entry. A link written by an older build opens without the page's keys.
2026-10-26 00:19:02 +01:00
peter 9bd7304bb9 fix(plex): a reload during a binge reopens the episode playing now
The player stepped to the next item in its own state while the history entry still held the item it
opened with, so F5 restarted the first episode. The player now says every item it moves on to, and the
page rewrites its entry with it (useHistorySubview's `follow`: no navigation, the overlays over the
player stay open, and the entry under an overlay is rewritten when a step lands back on it). The
player's URL is the item playing (`?v=player:<id>`), so a bookmark opens it; the list it plays from
stays in the history entry.
2026-10-25 23:24:29 +01:00
peter b30c60d1a5 feat(nav): each module declares its URL keys; a page switch clears them and a page opens without another's
The shell cleared a hand-kept key list on a page switch (it missed the Files and feed keys) and cleared
nothing when a page opened from a link carrying another page's keys. ModuleDef.urlKeys now names every
key a page's view writes, the shell derives both from the registry, and moduleUrl.contract.test holds the
declarations true: one owner per key, no shell key, exactly what each codec writes, and no undeclared
key literal in the app.
2026-10-25 20:48:14 +01:00
peter 4aff19a9a4 Merge branch 'feature/e45-related-videos' 2026-10-25 17:35:21 +01:00
peter dc50b856fd chore(security): gitleaks knows the public InnerTube key and a fixture's page cursors are no secrets 2026-10-25 17:35:20 +01:00
peter ee6e37f032 release: 0.112.0 (feature/e45-related-videos) 2026-10-25 17:31:11 +01:00
peter 4c2fbf9b8d chore(release): 0.112.0 2026-10-25 17:22:29 +01:00
peter 9d6d7525e5 fix(feed): a live search whose first page failed offers Retry where trying again can help
Search YouTube again with the same term read the failure from the cache and sent nothing, so a busy or unreachable YouTube left the search on its error until the term changed. The first page's failure is now the error state with Retry, offered only where a retry can answer (not the day's cap, the feature turned off or the shared budget).
2026-10-25 14:05:07 +01:00
peter ee4a5b46f8 refactor(player): the watched toggle is stamped at each answer, with no hold while a write is out
The hold changed nothing a user could see (each answer stamps the toggle right after its own echo), and it would have outranked a mark made elsewhere after the click while the write was still out.
2026-10-25 13:56:40 +01:00
peter 0bc1024287 fix(player): marks are weighed when answered, lit and pressed hold under the pointer, and Restart is one glyph
The watched toggle weighed its own click at click time against marks stamped when the server answered, so the late answer of its previous click overruled the newer one; it now holds its word while any of its writes is on the way, is stamped at the answer, and drops a refused write. The hosts return the write. In a live search the feed's own answered writes stand down over the rows, which carry the marks, so a newer mark from the related list reaches a card the feed touched earlier.

A lit mode cycle got the hover layer, and a variant's hover colour outranked the pressed and lit colours: the state layer stands down on [data-lit] and those colours are important. The menus' Play from the beginning wears RotateCcw like the card's Restart, Reset wears Eraser and Redownload the retry arrow (DESIGN decision log). Tests pin the toggle's orders, the Watch later mark at its call sites, the live-search overrides, the hover exemptions and the menu glyphs.
2026-10-25 13:51:26 +01:00
peter 13f65b4844 test(db): lock probes read only their own database's locks
pg_locks is the whole server's: under the parallel lane a worker's own advisory claim on the same playlist id, or its own lock wait, was counted by another worker's probe (a leak reported where there was none, a commit made before the second tab blocked).
2026-10-25 13:51:26 +01:00
peter 2e1934689f fix(search): a search that never reached YouTube gets its slot back, and a busy one waits a bounded time
The slot was kept as soon as search.list was about to be called, so a search refused before it went out (no connection, no token) cost the user a slot for nothing. It is kept now only once the client recorded the call's units, or the request went out and its answer never came (YouTube may have counted it): YouTubeError says whether a transport failure was sent, and the client counts what it spent.

The process lock around the budget check and the call is waited for at most ten seconds; past that the search is refused as busy (search_busy, a retry can help) and gets its slot back, instead of parking a request worker behind every slow call before it. The lock test now pins one hold around both steps, and the busy test fails on a wait without a bound.
2026-10-25 13:51:25 +01:00
peter 84db570951 fix(player): marks give way to newer lists, Watch later items mark, and auto-advance has its own glyphs
A session's marks are weighed per field against when each page of a never-refetched list was read:
a page read before the mark (a Show more / Load more on its way) takes it, one read after keeps the
server's word. The live search lays them over its rows too, not only the related list.

Items of the built-in Watch later list ARE the bookmark, so adding, removing and moving them go
through features/video/videoWrites and mark the video; the contract test covers those writes and
catches a write that is named without being called on the spot.

The player's watched toggle follows a mark confirmed elsewhere since it opened (the related row's
menu, a notice's Undo); Restart on the related row after a description link goes back to it.
The file viewer tells the help sheet its keys wait under a layer; refusalReason replaces the last
two open-coded reads.

Auto-advance wears list glyphs of its own (the skip arrows are Previous / Next beside it), and the
auto-advance and loop cycles are `lit`, not toggles: no aria-pressed, the name says the mode. Both
places render from one list of controls, so the docked tooltips name the mode too. The English cap
message and admin hint quote "Show more"; the hint says refunds are limited.
2026-10-25 04:01:43 +01:00
peter b25179c06a fix(search): a paid search keeps its slot, refunds are few, and the budget check holds the call
An API-sourced live search that reached search.list has spent 100 units (the client records them
before it reads the status), so it keeps its daily slot: refunding it let a bogus cursor spend the
shared budget without ever counting against the cap. A failure before the call still gives the slot
back.

Every refund is rationed too (5 an hour per user and list): a cursor is a free string, and a request
built to fail would otherwise scrape YouTube from the server without end.

The API source asks the budget again right before search.list, under one process lock with the
call: checked apart, searches sent together at the last 100 units all passed and all spent.
2026-10-25 04:01:42 +01:00
peter 620dd8178f feat(player): the side panel's list controls and watch toggle are glyphs on one line
The queue row (previous, count, next, auto-advance, loop) wrapped onto three lines in the
narrow side panel; glyph-only, it fits one. Mark watched joins the action icons as a toggle
(check / double check). Every control keeps its full name and shortcut as its tooltip and
accessible name; the docked bar keeps its labels. The related switch moves up to the toolbar
rung (12px), and its second side reads just "Trending".
2026-10-25 02:43:01 +01:00
peter 5d009964d8 fix(i18n): the related cap message reads right at any limit
"1 requests" at a limit of 1: the number stands alone now, and the
sentence after it says what counts.
2026-10-25 02:16:20 +01:00
peter 4b5b66343d fix(player): related marks survive, Restart restarts, and refused pages offer nothing to click
- Every state and Watch-later write goes through features/video/videoWrites,
  which marks the never-refetched lists (related lists, live searches) once
  the server has it; a contract test holds it. The Notification Center's
  undo, the feed's reset and the Watch-later toggles skipped it before.
- Confirmed marks are also kept apart (qk.videoMarks) and laid over the
  related rows: a Show more page landing later rebuilt them from the pages
  it read when it started, and dropped a mark made meanwhile.
- Restart on a related video starts at 0 (the session's stored position
  won before), and on the one already playing it seeks.
- The watched toggle remembers each video's mark: Back showed the opened
  video's launch-time state after a related one was marked.
- The help sheet leaves out the player's keys while a layer above it has
  them waiting.
- After a refusal a retry cannot answer, neither Show more (related) nor
  Load more (live search) is offered; a failed live-search page keeps the
  results already read and says why under them.
- The related cap message counts requests, as the cap does.
2026-10-25 02:50:34 +02:00
peter ed51f5be7e fix(search): the API-sourced live search takes its slot under the lock too
Checked first and logged after search.list, requests sent together at the
last free slot all passed, and each spent 100 units of the shared budget.
Every live search now claims its slot (quota.claim_action) whatever the
source, and gives it back when it showed nothing. The cap counts the
zero-unit claims only, so search.list's own spend event is accounting,
not a second slot. log_action, the unlocked writer, goes.

Tests: the API source reads one page per request, is refused before the
call when the budget cannot pay, counts one slot per search and refunds a
failed one; the lock test now proves the count happens after the lock.
2026-10-25 02:50:34 +02:00
peter 005784e291 test(player): a related video's end leaves an auto-advancing queue where it was 2026-10-25 00:42:34 +02:00
peter b5045165c0 fix(player): a related video played is tracked, and its list is honest about its state
A related video played in place is the player's item now: its title and its own watched /
playlist / download controls, its progress checkpointed and resumed, marked watched at its end —
and its end does not step the queue. Back returns to the opened video, where it was left.

The related list:
- its marks announce through the shared notice (the Notification Center finds them), join the
  host's player session (the feed refetches on close), and stay per write, so an earlier answer no
  longer undoes a later mark still on its way;
- every host's state write patches the cached related lists, which are never refetched;
- kept for the session (gcTime), a retry asks for the page that failed, a list read so far with
  nothing unseen says so, a refusal a retry cannot answer offers none, and the followed mark names
  itself on hover.

Turned off on the instance (a limit of 0), the tab, the drawer button and the live-search buttons
are left out.

Keys: the player's keys and its focus reclaim defer to a layer above it (the drawer); going wide
moves the drawer's list into the panel's tab; `j` goes on from the slot of a row removed inside the
top layer; the help sheet lists the list keys over a layer that holds a list.
2026-10-25 00:32:28 +02:00
peter d53729a63d fix(youtube): a related page is empty only on YouTube's own word
The watch frame alone read as "nothing related" — so a related column YouTube moved, or a list
drawn by renderers this does not read, answered an empty list instead of an error. Now only the
"isn't available" promo YouTube draws for a video it no longer has is an empty page; anything else
it cannot read is a ScrapeError.

The flattening of a video renderer and the read of the continuation token are one each, in
innertube, for the search and the related scrape.
2026-10-25 00:32:27 +02:00
peter b04b51c909 fix(search): every related and live-search request takes a slot, and a failed one gives it back
A related list's cursor cost nothing, and a cursor is a free string: a replayed one read and
enriched without end. Every request now takes a slot of the user's daily cap — taken under a lock
before the work (`quota.claim_action`), so requests sent together cannot all pass the last free
one, and given back when the request fails outright (`quota.refund_action`). The live search's
scrape source takes its slots the same way.

A limit of 0 turns the feature off: a distinct 409 `related_disabled` / `search_disabled`, and
`/api/me` says `related_enabled` / `search_enabled` so the UI can leave it out.

A videos.list that fails no longer reads as an empty page: `ingest_candidates` raises, and the
search and the related list answer the refusal (the shared budget spent says "quota") instead of
"no results". The two routes' page loops are one (`_gather`).

`clear` deletes only what the caller found: a related list's rows carry no find, and any user
could name them.
2026-10-25 00:32:26 +02:00
peter 7d45c4f59f fix(keys): a list inside the topmost layer walks by keyboard
The list keys are the page's and an open layer silences them, so the
player's related videos could only be reached by Tab. A layer now names
its root element; with the focus on a row inside the topmost one, j/k,
the arrows and Enter walk that list, the walk stays inside the layer,
and the player's seek and volume arrows leave a focused row alone.
2026-10-24 22:45:26 +02:00
peter 5205091116 refactor(player): the related-region pref key stays private to its list 2026-10-24 17:03:01 +02:00
peter 686c673fb2 feat(player): a video's related videos, on demand beside it
The player shows what YouTube lists beside the video, without what the
user has watched or hidden. Wide, a Related tab beside the description;
stacked, a button beside the info opens them in a drawer. A row plays in
the same player (Back returns) and the list stays the opened video's.
A switch picks on-topic or Hungarian-trending lists, kept on the account.

Each related row says whether the user follows its channel; the row is
the video card's narrow form with its whole menu. VideoMenu moves to
features/video, where the player can reach it.
2026-10-24 16:57:52 +02:00
peter 8b37c470ba feat(catalog): a video's related videos, without what the user has seen
GET /api/search/related/{video_id} reads the list YouTube shows beside a
video from the website's InnerTube "next" endpoint (no quota), and brings
it into the catalog the way live-search results come in. It leaves out
what the user watched or hid, here or in their Takeout history (unless
taken back to "new" since), blocked channels and the video itself, and
pages on until 20 are gathered (at most 4 pages). region=topical|hu picks
the locale the list is filled in. Its own per-user daily cap
(related_daily_limit_per_user, one event per opened list); a failed
scrape answers 422 youtube/unavailable.

The InnerTube request (live key, consent cookie, errors) moves into
youtube/innertube.py, shared by the search scrape and the related one.
2026-10-24 16:00:37 +02:00
peter 4695e1a62f release: 0.111.0 (feature/e48-watched-channels) 2026-10-24 13:19:55 +02:00
peter 1adf59ea1b chore(release): 0.111.0 2026-10-24 13:11:09 +02:00
peter bc8fe2cc6c fix(catalog): a follow from YouTube keeps an explored channel, and the list counts only what can come
A subscription pulled in from YouTube left an explored channel flagged as an exploration's, so its
later videos came in hidden from the Library with their watched marks waiting for good; a subscribe
landing while an exploration's videos came in left those flagged too. One step, settle_explored,
now holds the rule by who follows the channel now: the subscribe here, the subscription import, the
watched-channel videos and the explore page all end with it. Migration 0116 clears what the gap
left behind.

The list's "may still grow" count no longer counts a channel it already shows from a stub row, and
the comments on the cleanup's keep rule and on the in-memory daily shares say what they do.
2026-10-24 11:29:14 +02:00
peter 33dd9ff054 fix(reader): the watched-channels list keeps its channels, its shares and its retries
- A channel someone follows: the videos its exploration brings in (the list's
  dialog, the channel page's explore) come in as the catalog's own, not
  flagged via_explore, which subscribing would never clear and which kept
  their watched marks waiting for good.
- The explore cleanup keeps the row of a channel a Takeout history shows
  watched (it still reclaims the videos), so a looked-at channel no longer
  drops off the list.
- The Shorts job takes up only users whose list still has lengths to ask or
  channels to name, by each user's own follows, blocks and dismissals; a
  naming that failed is tried again, and the list says it is not complete.
- The length lookups have a daily share of their own: a large history no
  longer leaves the export's Watch later file waiting a day.
- Only a channel of the user's own history can be put out of the list.
- The dialog's videos have a query root of their own: the list's
  invalidations no longer re-ask (and re-explore) under an open dialog.
2026-10-24 01:10:29 +02:00
peter b5f9cf14db feat(channels): a "Discover from your history" tab for the channels watched but not followed
The Channel manager lists the channels the Takeout history shows watched
often but not followed, the most watched first: twenty at a time, the
channel's About on hover, its name to its page, Follow, and Not interested
with an Undo. A row's count opens the videos it counts, ready to play one
or all. Before a history is uploaded it points to the upload; while the
import still asks YouTube it says more channels may join.
2026-10-24 00:00:50 +02:00
peter bd2340c204 feat(reader): the channels a Takeout history shows watched but not followed
The import asks YouTube how long the watched videos of the candidate
channels are (five or more watched, none the user follows, blocked, put
out or owns), keeps the length on the Takeout row and names the channels
that make the list: five videos of five minutes or more. The shorts job
asks what an import could not. A channel's counted videos come in as an
exploration's when its list is opened, ready to play; a channel can be put
out of the list and back.
2026-10-23 23:45:54 +02:00
peter 84af92aa29 release: 0.110.0 (feature/e41-history-import) 2026-10-23 22:48:06 +02:00
peter 420baff463 chore(release): 0.110.0 2026-10-23 22:39:20 +02:00
peter 9c5f81fd94 fix(reader): the Takeout import keeps a later upload, an un-mark and its memory bound
- An ad-hoc request made while its target's run is under way is the run after it: the scheduler
  dropped a second fire of a running id, so a second Takeout file was never imported and its key
  stayed queued until a restart.
- Un-marking or resetting a video is remembered (video_unmarks): an import of an older YouTube view
  no longer marks it watched again; un-marks take the user's Takeout lock.
- One user's failure in the Shorts job's Takeout pass is theirs: the job keeps its own result, the
  others are settled, and a failed lookup spends none of the daily share.
- The upload reads the user's id before giving the read connection back.
- A watched row waits on a discovery video (search or explore) until the catalog keeps it, so a
  mark never pins it against the cleanup jobs.
- The history is decoded a window at a time, a Watch later file over 2 MiB is refused unread, and
  two uploads at once across all users (takeout_server_busy).
- A Watch later list goes in as one presence read and one insert.
2026-10-23 22:07:11 +02:00
peter 9f580fd8c8 fix(reader): the Takeout import holds up under abuse, races and a run it cannot see
- Uploads: one at a time per user and ten an hour, each refused only after its body arrived;
  the history read an entry at a time (32 MiB cap), a Watch later list capped at YouTube's 5000,
  a user's kept videos at 200k; any unreadable shape is a 422, not a 500.
- Quota: Watch later lookups stop at the backfill reserve and at a per-user daily share; what is
  not asked about waits, known videos still go in, and the shorts job retries the rest.
- Staging and settling share one per-user advisory lock (no deadlock); a run settles only the
  rows it read, so a row staged meanwhile is never called gone.
- A failed run's notice names a translated failure, never the upstream's text.
- The status says when an import is queued; the page watches it to its end even when left, then
  reads the feed, its counts and the playlists again.
- Settings: one live region, the zone kept (unavailable) while a run goes, focus handed back,
  a missed drop kept from the browser, an oversize file refused before it is sent.
- Scheduler and run lists show Takeout runs by the importing user.
2026-10-23 20:48:23 +02:00
peter e0b02a0c14 fix(inbox): a Takeout import's notice opens the user's own import
The import's notice reaches a user who may not see the Scheduler, an
admin page, yet its link opened it. It now opens Settings › Account,
where the import's status is; any other run still opens the Scheduler.
The job id is held to the server's by the backend-ids contract. The
notice's Hungarian names the list as the app does elsewhere.
2026-10-23 19:15:06 +02:00
peter 3553eac8c4 feat(settings): upload a Google Takeout export in Settings › Account
A "YouTube history" section under YouTube access: the watch history and
the Watch later csv are dropped or picked (one request each, each file's
outcome said on its own), and the section then says what the import did
- only the counts it produced - or that it is running, failed or was cut
off by a restart. While it runs it asks again until it ends; a failed
re-read shows the failure, not the previous answer.

The drop zone the Files upload had inline is now ui/DropZone, used by
both. The e2e seed clears the account's kept Takeout videos.
2026-10-23 18:51:05 +02:00
peter 5d9952ab1e feat(reader): a Takeout video the catalog learns later is marked watched then; the import's status
A watched video the catalog did not know at import time waits in `takeout_items`. The Shorts job now
settles those rows right after its probe (the only place a video's Short-ness becomes known), every
user's, a commit each: `reader.takeout.apply_waiting`, its count in the job's result.

`GET /api/takeout` is what the settings page reads: the user's last import run (`jobruns.last_run`)
and the kept videos by kind and outcome, "waiting" for one not known yet.
2026-10-23 17:30:28 +02:00
peter f486a69ef2 fix(deps): brotli 1.2.1 (PYSEC-2026-4322) 2026-10-23 17:10:38 +02:00
peter 7b8505af95 feat(reader): a Google Takeout export becomes watch state and Watch later
The YouTube API reads neither a user's watch history nor their Watch later list, so the Takeout
export is the only way in. `POST /api/takeout` takes one file of it (watch-history.json or
Watch later-videos.csv, told apart by content), keeps its videos in `takeout_items` and queues the
user's `takeout_import` run, whose outcome reaches their inbox.

The run marks watched the known videos the Shorts probe has cleared, at the Takeout time; a Short
is not marked, and a state the user set here (watched, hidden, or changed after that time) wins, by
the upsert's own condition. A video the catalog does not know yet waits with no outcome. Watch later
videos are fetched if unknown and appended oldest first.

An ad-hoc run can now ask for a completion notice (`run_ad_hoc(..., notify=True)`).
2026-10-23 17:04:54 +02:00
peter bf5ceec8a1 release: 0.109.2 (bug/plex-player-server-away) 2026-10-23 15:00:27 +02:00
peter e788326247 chore(release): 0.109.2 — a server restart mid-episode is waited out by the Plex player 2026-10-23 14:52:04 +02:00
peter 38c71b7b4f fix(plex): a server that restarts mid-episode is waited out, not reported as a stream that cannot play
During a deploy the gateway answers 502 (or nothing answers at all) until the app is back. The
player took the first such answer for a broken stream: it said "Playback couldn't start" over a
picture still playing from its buffer, stopped loading, and never asked again, so the episode froze
where the buffer ended with the error still up. Now a 502/503/504 or no answer is the server being
away: the buffer keeps playing undisturbed, the session is asked for again where the viewer is
(2, 4, 8, then every 15 s), and the error is said only after three minutes away. A seek or another
item replaces a pending retry. Measured on localdev: a 45 s API stop now resumes at the viewer's
position, where it froze with the error before.
2026-10-23 14:42:17 +02:00
peter cd5b26efa7 release: 0.109.1 (improvement/e44-fe-shell) 2026-10-23 14:21:45 +02:00
peter 0892a1a400 chore(release): 0.109.1 — the frontend in one folder per page with lint-held layers; an edited saved view survives a visit away 2026-10-23 14:13:44 +02:00
peter b443ac0c84 fix(frontend): the layer gates cover what the move left outside them, and each page keeps its own logic
Gates the folder move had narrowed or left open: the MenuCheck contract scans the whole app again (it
saw only components/); local/layers refuses an import of an entry (main.tsx, the /watch page) from
any layer, and reads a backtick import() as the path it is; the /watch contract forbids the shell,
the modules and the features by folder (its one path to App had gone stale); chunkGuard's own test
gets the vitest focus/skip rules. The root tests' exemption from local/layers is in the gate
register, with its three imports.

Placement: Plex's chip builder moves into lib beside the feed's, the folder names into
features/folders, and three files only one page uses into that page. ui/FilterRail takes its reset
from useRail instead of a page id. The channel page reads the feed's view and builds "Show in feed"
itself, and the feed's scroll key is the feed's hook, so the shell holds no feed logic.

The floating header re-measures when one page's pills replace another's in the same commit (the
claim count stayed equal, so only the observer caught it). A Plex view applied and edited now stays
across a visit to another page, as the feed's does: pinned by a test. docs/development.md says what a
page with a rail adds in lib.
2026-10-23 13:45:40 +02:00
peter ad05932217 docs(frontend): the canonical table and the ui README name the module shape; adding a page is written down
CLAUDE.md gains the header-content, module-store and saved-views rows. docs/development.md says what a
new page is and which tests hold it. components/ui/README.md names every file in the directory, and a
contract test keeps it that way: seventeen had slipped in unlisted. copyItems' comment still claimed it
was a feature file.
2026-10-23 12:52:04 +02:00
peter 6ae7825a14 refactor(frontend): the module stores mount from the registry, and the feed and Plex each own their saved views
A module's definition names its store (feed: filters, view, saved views; Plex: browse state, saved
views; channels, files, playlists their provider), and ModuleStores mounts them at the root in
registry order instead of a hand-nested tree in main.tsx. A contract test holds that every store's
read side is in reach and that no store's static imports leave its own module, so the order cannot
matter.

Saved views were one provider that switched between the feed and Plex by the active page and reset
on each switch. The logic is now one hook (useSavedViewsStore) with an instance per module; the
shared chip, popover and modal read the views their page or rail hands down. A feed view applied
and edited now survives a visit to Plex. Plex's views are still asked for only on its page, and a
module's manage modal still closes when its page is left. The feed's URL-override check applies
only on the feed page, where its filter params live.

The Channels store's tab type moves from the page into the store's context.
2026-10-23 04:59:10 +02:00
peter cde5a98fd2 refactor(frontend): each module defines itself in its folder; rails load lazily and keep their own state
A page used to be listed three times (lib/modules for its label, icon and gate; shell/moduleRegistry for
its page and rail; lib/pageMeta for its title), and App held every rail's layout and collapse flag.
Now each module's `module.ts` says it all — label, icon, gate, its lazy page and rail, its width —
handed out through its index.ts, and features/navigation/moduleRegistry lists them in nav order for
the rail, the palette, the tab title, the reachable set and the shell's render. A contract test holds
the registry to the page ids and each definition to its own folder, with no static import of a page.

Every rail is lazy now, loaded together with its page so a pinned rail never arrives after it and
shoves the content aside; the feed's and the playlists' rails took @dnd-kit off the boot path (boot
428.5 → 393.9 KiB gzip, the App chunk 122 → 32 KiB). A rail reads its own layout and collapse flag
(lib/useRail) and writes a layout change to the `me` cache too, since it now mounts anew on each
visit. App's session effects, its one-shot entry-URL handlers and the admin nudge move into shell
hooks; App is layout. The registry's three layer suppressions are gone: none is left.
2026-10-23 04:00:58 +02:00
peter e83c5e2d97 fix(frontend): the content is padded by the filled header before paint, not when the observer catches up
The header's pills now arrive by portal after the shell has mounted, so the shell's one synchronous
measurement saw an empty header, and the ResizeObserver brought `--hdr-h` up a frame or more later:
meanwhile the content sat under the pills (the feed toolbar jumped 55px on a tablet; e2e caught it).
The measurement re-runs before paint whenever what fills the header changes.
2026-10-23 03:24:23 +02:00
peter 6a0c1d377f fix(frontend): the channels, files and playlists faces stop exporting the stores' read hooks only the old header used 2026-10-23 00:30:14 +02:00
peter 4ef963b005 refactor(frontend): each page fills the floating header itself; the header knows no page
The header used to pick its pills by `page === …` and import five module controls, and Plex's
Discover/Browse toggle needed a context bridge to reach the sub-view that lives in PlexBrowse.
Now the header is an empty band: PageShell hands its root box out as a slot, a page portals its
own pills into it (PageHeader), and the palette's Jump-to pill shows where no page does
(PageHeaderDefault, rendered inside the page's Suspense so a loading page shows no stand-in).
Plex's toggles take the sub-view as a prop; PlexViewBridge and its context are gone, and with
them the header's five layer suppressions.

No visible change: every header pill's box measured identical against the previous build on eight
pages, in Hungarian and English.
2026-10-23 00:23:38 +02:00
peter 0b24f2d8fa fix(frontend): the chunk guard's logic lives in the guard, so no file sits in two TypeScript projects
The guard's logic was a src/lib file the config project also listed. The app project, which
references the config project, then refuses that source on a clean build (TS6305) — while every
local run reused the referenced project's cached output and stayed green; only the Docker image
build failed. The logic is back inside chunkGuard.ts, its test beside it at the frontend root
(vitest now also collects `*.test.ts` there), and a gate contract fails when the config project's
include reaches into src/ again. The guard was re-proved on a build with a leaking re-export.
2026-10-22 23:48:16 +02:00
peter d5867bacca feat(frontend): the folders are layers — an import only goes down, a module is reached through its index.ts
`local/layers` (eslint.config.js) resolves every import to its file and refuses one that goes up
(lib/ → components/ui/ → features/ → modules/ → shell/) or that reaches into another module's own
files. A module's index.ts is its public face: the stores, mounted at the root and read app-wide,
which features/ may read too; a page is still opened lazily by import(). Its contract test probes
every edge, and three mutants of the rule (off, no face check, a door for everyone) each turn it red.

Six modules got that face; the shared channel UI (about, tags, tag manager) moved to
features/channel, the filter chips and the saved-views context to features/filters, and the prefs
store's controller type moved next to the store instead of importing it from its own page. The
header's per-module controls and the eagerly loaded rails keep eight inline suppressions, each
naming why: they go when the header gets slots and every rail loads lazily.

`chunkGuard` fails the production build when a lazily imported module lands in a boot chunk (the
entry, the shell, what they import) — e.g. a page re-exported from an index.ts, which makes its
import() a silent no-op. Proved on a build with that mutant; today's build has none.
2026-10-22 23:30:27 +02:00
peter 63a0fecc8b refactor(frontend): the flat components/ folder moves into shell/, modules/<page>/ and features/<name>/
A pure move (git mv, history kept): 333 files leave components/, which now holds only the platform
layer (ui/) and the legal pages. The shell is the frame (App, header, nav rail, palette, boot); each
page lives in modules/<page>/ with its rail, store and tests; what several pages share sits in
features/<name>/; the shared-link page is watch/. Domain-free helpers went straight to ui/
(ViewSwitcher, FilterRail, the confirm dialog, the language switcher, the copy rows).

Every relative specifier was rewritten (imports, dynamic imports, vi.mock, new URL), and the
contract tests that scan sources now scan the whole app source, not only components/ — each glob
was counted before and after so none went green over fewer files. The locale-format rule matched
`/src/components/` only, so the move would have taken ~300 files out from under its i18n-singleton
check; it now covers every folder that holds components, with a test per folder.
2026-10-22 23:12:45 +02:00
peter fdf8076013 release: 0.109.0 (improvement/e43-identity-diet) 2026-10-21 03:05:48 +02:00
peter 6287fc9cf8 chore(release): 0.109.0 — one account per browser, settings that follow the account, a tab that names its account 2026-10-21 02:55:46 +02:00
peter 3c6c23507b fix(frontend): a page opened at a //path boots, and the account-switch copy reads right in Hungarian
Every address-bar writer glued `pathname + query`, so a path of `//evil.com/x` named another host:
history refused it (SecurityError) and the first nav mirror threw the app into its crash screen. One
builder, `hrefHere`, now makes the address from the URL itself and stays on this origin.
2026-10-21 02:51:48 +02:00
peter ef99f7b5c7 fix(auth): a Google connect names its tab's account too, and a refused tab says why
- /auth/google/connect is a navigation and carries no headers: the tab names the account it
  was built for in `expect`, checked by the same comparison as the request header
  (built_for_another_account); a mismatch lands on /?account=changed
- the link marker names the OAuth state of its own flow: an older flow coming back no longer
  spends the marker of the connect still open (which then returned as a plain sign-in)
- a 401 account_changed reloads the tab with a one-time flag, and the reloaded tab says it
  now shows the account signed in
- a refused switch to a suspended or not-yet-approved account says so in the signed-in
  app's words, naming the address a request was recorded for
- the prefs merge locks FOR NO KEY UPDATE, so a transaction that only added a row referring
  to the account does not hold the save up; a test pins the lock
- giving up a lost page closes a channel page open over it
- e2e README names the chromium-only specs
2026-10-21 02:05:33 +02:00
peter 94418a087e fix(auth): a tab acts only as the account it was built for, and the review's other findings
A tab names the account it was claimed for on every request, beacon and upload
(X-Siftlode-Expect-Account); current_user and /auth/logout refuse a mismatch with
401 account_changed and keep the session, so a tab left showing the account
signed in before reloads instead of writing to the new one.

/auth/login clears a connect marker left by an abandoned connect, and a link
completes only on a session that still holds. The prefs merge reads the row
under a lock. A tab's sessionStorage records its own owner. A page the account
loses is given up, address bar and remembered page included. The panel layouts
are preferences; the storage contract also refuses a key handed to the Storage
API or to usePref outside its kind. A refused sign-in as someone else and a
failed sign-out say so. The chromium-only e2e specs are in the gate register,
the member walk waits for its requests, and devices.spec checks the save itself.
2026-10-21 01:05:12 +02:00
peter 28b77fed1e docs: storage kinds in the canonical patterns, a password beside Google sign-in, performance mode is this device's
CLAUDE.md's per-account state row becomes the browser storage rule: a key
in storage.ts with its kind, a preference through usePref. The
self-hosting guide says to set a password even when signing in with
Google, so a broken Google client does not lock the account out.
Performance mode's hint says it applies on this device only.
2026-10-20 23:34:43 +02:00
peter 50764a916d fix(deps): certifi 2026.10.19 (GHSA-6q48-6c7w-9gw2) 2026-10-20 23:14:37 +02:00
peter f21cdfa156 refactor(frontend): every stored key declares its kind, and preferences follow the account
storage.ts's registry gives each key a kind (pref, device, view, cache,
public); a key without one does not compile. A contract test refuses a key
spelled out elsewhere, an account preference read or written straight from
storage instead of through usePref, and an e2e seed of a key nobody reads.
The keys the next account's wipe keeps are the public ones.

The seven preferences kept only in the browser move onto usePref: start on
Home, the onboarding dismissal, the release banner's last version, the
channel manager's layout, the Plex playlist layout, the playlist rail's sort
and channel discovery's sort. usePref reads an older bare-string copy, and a
value only this browser held is saved to the server once, so the account's
other devices get it.

Performance mode is the device's: it is neither read from the account nor
saved to it.
2026-10-20 23:04:48 +02:00
peter d6dada7fe5 refactor(frontend): one account per browser, its local state under plain keys
The per-tab account pin, the X-Siftlode-Account header and ?account= go: the session cookie alone
says who is asking. localStorage keys lose their account suffix; the browser records whose state it
holds (siftlode.owner), BootGate hands it to the account the server answered as, and a different
account starts from a clean slate (the public /watch keys stay). The first boot moves the account's
old per-account copies onto the plain keys and drops other accounts' copies. Until the claim, and
after another tab claimed the browser for someone else, the account helpers read and write nothing.

The account panel lists no other accounts: "Add another account" becomes "Sign in as someone else"
(Google's account chooser; the pick replaces this account, a cancel keeps it), shown where Google
sign-in is configured.
2026-10-20 15:29:41 +02:00
peter 6645caa157 refactor(auth): one Google connect route for a sign-in and for YouTube access
/auth/link and /auth/upgrade become /auth/google/connect?access=identity|read|write: one handler, one
table of what each access asks Google for, the same callback markers. An unknown access is refused
(422) instead of quietly meaning read, and an instance without Google answers ?link=error for a
grant too, not a silent return to the app. The frontend builds the URL in lib/googleConnect, and a
contract test holds it to the backend route and its access values.
2026-10-20 12:49:11 +02:00
peter 59ec4a28ba refactor(auth): one account per browser session
The session held a wallet of up to six accounts, and a request picked one by a header or an
?account= parameter. It now holds one: user_id and its epoch. A sign-in replaces whoever was
signed in, sign-out and account deletion end the session, and an account refused mid-session
(revoked, suspended, deactivated) clears it. /api/me/accounts and /api/me/switch are gone, and
with them their refusal codes and the UI copy for them.

A cookie the multi-account build wrote carries no single epoch and is refused: its holder signs
in once more. A Google link finishing in a browser now signed in as someone else is refused, as
before for an account outside the wallet. A test holds that no app code reads the wallet keys or
the account header again.
2026-10-20 04:36:26 +02:00
peter dc4f4f5143 test(e2e): a plain member's shell is held end to end, and a page it cannot open lands on the feed
The suite ran only as an admin, so nothing end to end held what a member sees. A second seeded
account, a member with its own session, now walks every rail page (no admin entry, no admin call,
no error state) and follows links to the admin pages.

It found one: a member's link to an admin page showed the feed with no rail row current, and the
address bar kept naming the admin page — only the content fell back. The navigation state now
derives the page on screen through each module's gate (reachablePage), so the rail, the content,
the tab title and the address bar agree, on a link, a Back, and a role taken away mid-session.
2026-10-20 04:08:11 +02:00
peter 909732b3c1 release: 0.108.1 (improvement/e42-backend-units) 2026-10-20 02:12:49 +02:00
peter 215f022fe3 chore(release): 0.108.1 — the backend in four units, each owning its tables; a first Watch later and a first resume point survive a write beside them 2026-10-20 02:04:56 +02:00
peter 6ea750a698 test: the commit check starts every request clean, and the fence docs claim what it holds
The test's own writes are committed before a request, so a handler's Core DML can no longer hide
in a transaction the test had already opened. The entry-point probe answers on a marked line the
API's logging cannot mix into. A listed commit whose function no longer commits fails the fence.
CLAUDE.md and the backend README now name what an AST cannot see; the route-commit tests no longer
claim their read-back holds the commit.
2026-10-20 01:56:31 +02:00
peter c5f121139b fix: every first watch state survives a write beside it, and a folded Watch later keeps its pin
A season marked watched, a pull from Plex and a YouTube resume point made their first row by a read
then an add, and failed on the unique key when a first checkpoint or the watched mark made it in
between. They go through the same insert-then-read as the checkpoint (the resume point is one
upsert); a pull that meets a local change still waiting for its push leaves it.

Folding duplicate Watch later lists moved their videos but not a Home pin, which the cascade then
deleted; the first pin of a dropped copy moves onto the kept list.

Marking one notice read trimmed before its own flag was flushed, so the trim left one past the cap.
The route test now crosses the cap and reads the result from a session of its own.
2026-10-20 01:56:30 +02:00
peter b04b2cbd62 test: the rewritten write handlers commit, and three more owner filters hold
Each handler that now calls a service and commits is driven through HTTP once.
A second account's watch state, Watch later list and private download profile
are never read or changed. The autotag job's orphan prune is seen committed.
2026-10-20 00:45:22 +02:00
peter 3a5212cd8a fix: a first watch state and a first Watch later survive two requests at once
The first checkpoint of an item and the final flush beside it both read no
state and both inserted; the loser failed on uq_plex_user_item. Two first saves
each made a Watch later list. Both are created by an ON CONFLICT DO NOTHING
insert and then read; a partial unique index holds one Watch later per user, and
the migration folds an existing second list into the first. Plex keeps one
per-item state map (rollup.item_state_map).
2026-10-20 00:45:22 +02:00
peter b376373af5 fix(files): a folder's kind is answered before its new name, as before
A blank name on a source folder answered 400 name_required; it answers 409
not_own again.
2026-10-20 00:45:22 +02:00
peter 8206384c91 test(fence): owner filters in any form, statement writes under any name, core commits by list
The owner-filter check missed an aliased() model, an import alias, .in_() and
filter_by; four live inline filters on aliased VideoState go through owned_by now.
Cross-unit writes are seen under any import name, through sa.update and via
__table__. Core modules may commit only where CORE_COMMITS names the reason;
trim_read and audit.gc flush now and their callers commit. The pin target, the
debug-session header and a link's target fetch through owned_row. The docs
claim what the fence holds, and say what is beyond it.
2026-10-20 00:45:21 +02:00
peter e27656ee7e test: the request commit check sees Core DML and row locks
A db.execute(update/delete) fires no flush and stages nothing, so a handler whose
only write was Core DML could forget its commit and stay green. The check now
asks Postgres whether the request opened a writing transaction. It found one
real case: a repeated issue send returned with the session row still locked.
2026-10-20 00:45:20 +02:00
peter d73cdcee97 fix(capture): every process registers every table before its first query
The models now live in their units' packages, and a core model names a unit's by
string (User.subscriptions -> Subscription). The download child imports neither
catalog nor app.tables before db.get, so mapper configuration failed and every
download, clip, edit and upload job died with child_exit. Each entry point now
imports app.tables, and a test imports each one in a clean interpreter.
2026-10-20 00:45:20 +02:00
peter 5ce292cd28 test(fence): the unit fence has no exemptions, and the owner filter is held in every module
The per-file pin list and the unit-by-unit switch go: every route file is held
to zero database work, every unit's package to its edges, and core code says
whose a row is through userscope too. CLAUDE.md, the backend README and the
gate register describe the units as they now are.
2026-10-19 23:36:01 +02:00
peter ad6ccbac05 refactor(core): the core's owner filters go through userscope, each held by a second account
Tokens, notifications, quota, debug sessions and the reset-link burn filtered
their owner inline. Ten of the fifteen filters had no test that noticed one
going missing; each now has a test where another account's rows must not
count, show or change.
2026-10-19 23:36:01 +02:00
peter 531eb2eef1 refactor(plex): playlists, playback, Discover and admin through the unit; the plex package is fenced
The last Plex HTTP files do no database work of their own, so ALLOWED is empty: no route builds
SQL, touches the session or a unit's table. Discover's rails move whole into plex/discover.py; a
user's playlists, watch-state writes (the race-safe finish, an early checkpoint, a status, a
season or show at once) and the watch-sync link go through plex/queries.py and plex/service.py.

The plex package joins FENCED: it reads no other unit's tables, says whose a row is through
userscope (the rollup and the watch sync's 8 inline filters moved to owned_by), and commits only in
its named jobs — the library sync, TMDB enrichment, the watch sync and its background pushes.
A playlist's has-item probe keeps the plain key lookup, so a NUL in it is still the 422 a test pins.

Pins 178 -> 0; basedpyright baseline 118 -> 106.
2026-10-19 22:42:46 +02:00
peter 2cf1b688d9 test(plex): a collection's editable flag and the mirror's reads, committed through HTTP
The editable endpoint had no route test: dropping its commit left every test green.
2026-10-19 22:11:57 +02:00
peter 3b0e365e26 refactor(plex): the browse, detail, people and collection pages read through queries and write through the service
The HTTP files browse, detail, people, collections and _shared do no database work of their own
now. New plex/queries.py (the rows a page shows, a user's watch state through userscope),
plex/service.py (the info pages' enrichment, a collection's editable flag, the row kept after a
failed follow-up read, the person bio's background fetch; none commits), plex/library.py (the
unified browse feed, its search and facets) and plex/collections.py (a collection's source,
editability, card, certificate and member order). Every watch-state join in the moved code names
its owner through owned_by.

Pins 354 -> 178 in 4 files; basedpyright baseline 132 -> 118.
2026-10-19 22:11:20 +02:00
peter 533571f572 refactor(plex): the unit's tables move into plex/models
PlexLibrary, PlexShow, PlexSeason, PlexItem, PlexPerson, PlexCollection, PlexState, PlexLink,
PlexPlaylist and PlexPlaylistItem leave app/models.py for app/plex/models.py; app.tables registers
them. Every import is re-pointed; no schema change.
2026-10-19 21:42:07 +02:00
peter 4b6db47083 test(catalog): the channel manager's local writes, committed through HTTP 2026-10-19 02:36:19 +02:00
peter 5229fd96be refactor(catalog): the HTTP layer reads through queries and writes through the service
Channels, search and sync do no database work in their routes any more: reads go through
catalog/queries, writes through catalog/service, and a live search's ingest is catalog/search.
What a catalog route needs from Reader (a playlist's channels, a user's tag links, the feed
projection of results) it asks reader/queries and reader/service. The scheduler and the runs page
read channel titles and the work queue through catalog/queries. Catalog is fenced below its HTTP
layer: its owner filters go through userscope, it reads no other unit's tables, and its ingest
jobs commit only where they are named.
2026-10-19 02:32:20 +02:00
peter 238e6f8a61 refactor(catalog): the unit's tables move into catalog/models
Channel, Subscription, Video, SearchFind, ExploredChannel and BlockedChannel (and LIVE_OR_UPCOMING)
leave app/models.py for app/catalog/models.py; app.tables registers them. The User↔Subscription
relationship names its other side as a type only. No schema change.
2026-10-19 02:03:14 +02:00
peter a35eece652 fix(e2e): the seed imports Reader's tables from reader/models 2026-10-19 01:33:32 +02:00
peter b703234c17 test(reader): the service tests hold each rule their mutant broke
A reorder's left-out item, a playlisted video beside one not asked about, another user's tag, and
the user's own channel link committed before the auto-tagger reads the table.
2026-10-19 01:28:38 +02:00
peter dbb4065e7b refactor(reader): the HTTP layer reads through queries and writes through the service
Feed, playlists, saved views, tags and the Home pins do no database work in their routes any more:
reads go through reader/queries (and feedquery for a feed page, its counts and facets), writes
through reader/service and discover/pins, which flush; the handlers commit. Reader is fenced below
its HTTP layer too: its owner filters go through userscope, and the catalog's jobs ask Reader
instead of joining its tables (the auto-tagger's system tags, the cleanup jobs' keep rule, the
maintenance notice). A playlist sync's catalog ingest moves to sync/videos.
2026-10-19 01:20:14 +02:00
peter e82f87acb0 refactor(reader): the unit's tables move into reader/models
VideoState, Playlist, PlaylistItem, SavedView, ComfortPin, Tag and ChannelTag leave app/models.py
for app/reader/models.py; app.tables registers them. No schema change.
2026-10-19 00:53:44 +02:00
peter e9c60f72cd refactor(capture): the HTTP layer reads through queries and writes through the service
Capture's routes (downloads, files, public watch links) and the token list no longer build SQL, call
the session or set a model's fields: reads are downloads/queries, writes the unit's service (service,
folders, links, quota), and each write flushes for the handler to commit. Whose a row is goes
through userscope (owned_by, owned_row), which owned_or_404 now uses too.

The fence holds Capture below its routes as well: it reads only Catalog's and Plex's tables, writes
no other unit's, filters owners through userscope and commits only in its jobs and the upload
ingest. A test per service use case, among them that the tray lists only the caller's files.
2026-10-19 00:19:03 +02:00
peter 3a688fa0f5 refactor(capture): the unit's tables move into downloads/models
The nine Capture tables (files, jobs, folders, shares, links, quota, cookies, profiles, library
resume points) leave app/models.py for the unit's own module; app.tables registers every unit's
tables for the migrations, the test database and the ownership check. No schema change.
2026-10-18 23:45:53 +02:00
peter d66174ed16 test(arch): a unit's HTTP layer does no database work beyond its pin
The fence is switched on unit by unit: each route file's database work (SQL built, session data
calls, model attributes set, unit tables imported) is pinned at today's count and may only shrink.
A core route may not reach into a unit's tables. Every model has an owner.
2026-10-18 23:01:12 +02:00
peter c85ae42b31 release: 0.108.0 (feature/e40-job-history) 2026-10-18 21:48:23 +02:00
peter cb9a89682a chore(release): 0.108.0 — one Activity page for runs, admin actions, the server log and quota; every background run has a history 2026-10-18 21:40:52 +02:00
peter b842e4e871 fix(ui): a looping pulse stops for reduced motion; the e2e waits for skeleton bones, not for any pulse
- Seven looping pulses (a running job's dot, the scheduler's progress bar, a queued download, the
  download tray, a playlist upload, a channel's full-history action, the setup wizard's test mail)
  ran regardless of prefers-reduced-motion, which docs/DESIGN.md promises to respect: each is
  motion-safe now, held by a contract test.
- The a11y suite's "settled" wait took every aria-hidden pulse for a loading skeleton, so the
  Scheduler legend's running dot, which pulses for good, kept it waiting until it timed out. A
  skeleton bone now says it is one (data-skeleton) and the wait looks for that.
2026-10-18 21:40:52 +02:00
peter 5082636f15 fix(activity): a run nothing here has open reads as interrupted, a rollback sums a stopped run's spend
- The scheduler keeps one registry of the runs this process has open, for the interval jobs and the
  ad-hoc runs alike. A row the table still says is `running` but no run here has open (its outcome
  failed to write) reads as interrupted, as the next start's sweep will close it: no pulsing dot
  beside "No run recorded yet".
- Migration 0109's downgrade sums a still-running run's quota events into its spend before the link
  to them is dropped, as the startup sweep would.
- A test holds that a dialog closing onto the same address leaves an open log line open.
2026-10-18 21:21:19 +02:00
peter 13b7577d9f fix(activity): an ad-hoc run's row is one run's, a rollback keeps a run whole, Back reads one entry's filters
- The channel-backfill row says "running" from the runs this process has open, not from the table:
  a row whose outcome failed to write no longer reads as running until a restart. The run still
  going leads the row, so a newer run's outcome never sits under the older one's channel.
- Migration 0109's downgrade carries every run's trigger, spend and start in the audit row, and the
  upgrade restores them: a manual run whose admin was deleted stays manual.
- Back/Forward is read by the page's one popstate listener, both halves in one render; the server
  log adopts its half during that render, so no read goes out with one entry's filters and the
  other's range.
- The server log's failed facets use the other tabs' words (the menus may be short); the accounts
  message shows only when the menu really lacks them.
2026-10-18 17:32:22 +02:00
peter fad656988a fix(activity): runs keep their true outcome and spend, the page reaches every kept row, a closing dialog leaves the server log alone
Backend
- A run's outcome write sits outside the job's own try: a database blip after a run that
  succeeded no longer records it as failed (the row waits for the next start's sweep).
- An interrupted run keeps what it spent (the sweep sums its quota rows, as finish does).
- The Scheduler's ad-hoc row says running while any backfill runs, not only the newest.
- recent_runs reads each job's last runs off its index instead of ranking the whole table.
- An interval change is stored with its audit row in one transaction, then rescheduled.
- One API instance per database is the stated contract (.env.example, jobruns).
- Turning full history on for a channel already complete records no run.
- Migration 0109 round-trips interrupted and ad-hoc runs; the run words' DB CHECKs are
  test-held to the enums; the account filter is one helper for the four activity reads.
- An audit row says how many of its request's log lines are still kept.

Frontend
- An "All" period reaches every row still kept (runs and admin actions outlive 90 days).
- Back/Forward is followed in place by the server log: a closing dialog no longer turns
  Live off; a pivot still opens a fresh log.
- Runs moved over from the audit log read their one-liner in the reader's language.
- Filter choices or accounts that failed to load say so beside the filters, with a retry.
- An old server-log link keeps its account, text and range; a remembered retired page
  opens its tab.
- A request whose lines are gone says so instead of linking to an empty list.
- The log lines and the history rows share one row frame, and the server log one list
  frame with the other tabs; an empty list is announced; each day's quota is readable.
2026-10-18 15:38:32 +02:00
peter 3021be5166 fix(activity): a run's quota reads with its unit, its outcome keeps room at a large text size, a token's audit rows read in Hungarian
The run list showed the quota spend as a bare figure and, at the largest text size on a 1024-wide
screen, its fixed columns left the outcome column no width at all. The outcome now keeps 10rem and
the trigger column gives way; the audit list's summary keeps the same floor. A run that stored no
log line says so instead of linking to an empty list. An API token revoke and a token's file work
rendered the server's English line under the Hungarian UI; both are rebuilt in the reader's
language, and the test holds the server's wording.
2026-10-18 04:22:52 +02:00
peter c288ba147b fix(activity): an old audit/log link opens its tab, times read relative, one name for admin actions
Seen in the browser, not in the page's own tests: the navigation provider rewrites a retired page id
to `activity` before the page's chunk reads the URL, so `?page=audit` landed on the Runs tab. The
retired-page map now carries the params that open the successor's part (`at`), and the provider
writes them as it rewrites the page — on load and on Back alike.

A list time is relative with the exact second on hover (the DESIGN copy rule); a run moved over from
the audit log, whose start and end are one instant, shows no length instead of an invented "1 sec".
The audit log is called "admin actions" everywhere it is named (the clear button and its confirm, the
Configuration group, the cleanup job, which also prunes the run history its description left out).
2026-10-18 04:00:38 +02:00
peter fe6308d844 refactor(activity): the window's hours stay private to its URL state 2026-10-18 03:48:30 +02:00
peter 24448e7c6f feat(activity): one Activity page — runs, admin actions, server log and quota under one filter bar
The audit log, the server log and the admin quota view were three pages (the quota one a Stats tab),
each with its own filters, and the background runs had no list at all. Now one page ("Napló" /
"Activity") reads all four, each still its own table on the server: a shared bar (text, a period of
1 h to 90 days defaulting to 7, an account) and each tab's own filters, the whole view in the URL.
The old page ids open their tab.

Server side: the audit log and the new run list (`/api/admin/runs`) filter and page like the server
log, through one keyset helper (`routes/_keyset`) the log now shares; an audit row keeps the request
it was made in (migration 0111), so its log lines are a pivot away, as a run's are (`lrun`). The
quota breakdown takes since/until and one account. The Scheduler rows get a history button to their
job's runs.

Also fixed on the way: re-saving the value in force on the Configuration page staged its override
without committing it (the commit sat inside the audit branch), so the save was lost.
2026-10-18 03:45:18 +02:00
peter d13483a593 feat(config): job intervals live in sysconfig, every setting says when it applies, the unread SponsorBlock default goes
An interval override was a row of its own table beside the one store every other admin knob uses; it
is now the `job_interval_<job>` sysconfig key (migration 0110 copies them over and drops the table).
The Scheduler page stays its one editor: the Configuration page and the setup wizard neither list nor
write those keys, since a write there would skip the reschedule.

Each ConfigSpec now states `applies` (required, no default). Every key is `live`, held by a scan that
finds no read at import time; a `restart` key is refused until the page can say so. A second scan
requires a reader for every key on the Configuration page — `sponsorblock_default` had none since it
was added, so it leaves the registry, the translations and the store.
2026-10-18 02:23:15 +02:00
peter 3bab2e7dc7 docs(scheduler): the snapshot docstring names the ad-hoc runs it now carries 2026-10-18 01:29:46 +02:00
peter 1b1e3cdcf2 feat(scheduler): each row shows its last three runs, an interrupted run as such, and the ad-hoc runs beside the jobs
A row's one dot said how the LAST run went, and an interrupted run fell through to "idle" — a grey dot
that read as "no run recorded". The row now leads with its last three runs, newest first, each named
for a screen reader and in its tooltip (how it went, when, who started it); an interrupted run is a
ring, set apart from a skipped run and a failure, and its line says the server restarted under it.

A channel's full-history backfill gets its own "Ad-hoc runs" block: its runs, its channel and what
it fetched, no interval and no run button — it needs its target. AD_HOC_JOBS names every such job;
the backend-id contract holds their names in both languages, and run_ad_hoc refuses any other id.
2026-10-18 01:20:53 +02:00
peter 27c2fcbd2e feat(scheduler): a channel's "get full history" backfill is a recorded run, and a row knows its last three runs
The one-channel deep backfill ran as a FastAPI BackgroundTask: what it fetched, what it spent and
how it failed reached only the log. It is now a one-shot run on the scheduler (`channel_backfill`,
trigger ad_hoc, actor = the user, target = the channel) with its quota tagged to the run. Its
per-channel exclusion with the scheduled backfill and the two-run cap are unchanged, and a run
they turn away says why in its result instead of returning nothing. A failure is no longer
swallowed: the run records it.

The scheduler snapshot carries each job's last three runs (one windowed query), so the page can
show an interrupted or skipped run instead of the latest outcome alone.
2026-10-18 01:05:24 +02:00
peter 911527a548 fix(scheduler): the dashboard no longer says a job has not run "this session" — runs are kept across restarts 2026-10-17 23:22:13 +02:00
peter 8fd91b7b15 fix(scheduler): keep run history 180 days, as long as the audit log kept the runs it moved over 2026-10-17 23:15:17 +02:00
peter 45c0ee7d58 feat(scheduler): every background run is a durable row
What a job last did lived in the API process's memory and was gone at
every restart; the audit log kept a run only when it was manual, failed
or changed something. Now each run is a job_runs row written at its
start and its end in its own session: who started it, its outcome as
data, what it spent of the YouTube quota (quota_events.job_run_id) and
the lines it logged (log_events.run_id, filterable in the log console).

- a run a stopped process left open is marked interrupted at startup
- "run now" is a one-shot job on the scheduler, which now always starts
  (interval jobs only with SCHEDULER_ENABLED); the manual actor is reset
  after the run, since pool threads are reused
- the dashboard's last run comes from job_runs, so it survives a deploy
- runs leave the audit log: migration 0109 moves the old job_run rows
  over (and back on downgrade); the audit page's run folding goes
- job_run_retention_days (default 30) prunes old runs in audit_gc
2026-10-17 23:08:55 +02:00
peter 4b1f6d7ec2 release: 0.107.3 (improvement/dev-tooling) 2026-10-17 21:34:53 +02:00
peter 51bf780100 chore(release): 0.107.3 — text on glass holds 4.5:1; the e2e suite runs axe, the gate lints every shell script 2026-10-17 21:21:37 +02:00
peter 3ada34073b fix(theme): text on glass holds 4.5:1 — light --muted kept at 6:1 on the page, tonal fills opaque
Pixel-measured over every scheme x mode x backdrop: the item count fell to 3.8:1 (dark too: its 8%
fill let the toolbar glass and the thumbnails under it through), Channels' tabs to 4.1:1 and the
current Files row to 4.4:1. Light --muted moves 15% toward --fg in every scheme; the badges, the item
count and the current rail row paint opaque bg-tint* fills mixed into the surface. The token contract
now asks 6:1 of --muted on the bare page and checks those three sites use the tints.
2026-10-17 20:06:54 +02:00
peter d74f2a9ca1 fix(deps): source-map-js 1.2.2 — the indexed-source-map denial-of-service advisory (dev tooling only) 2026-10-17 17:51:10 +02:00
peter a8dddc8c24 test(e2e): axe-core's WCAG A + AA rules over every page and the player (chromium only); contrast stays with the visual check 2026-10-17 17:48:53 +02:00
peter 460bdec406 chore(tooling): drop scripts/lint-changed.sh — the shared gate runner's Stop hook lints the turn's files, shellcheck included 2026-10-17 17:22:35 +02:00
peter 66181d1394 chore(gate): shellcheck joins the gate — every tracked shell script, the host binary held to a pin 2026-10-17 15:31:34 +02:00
peter 0a1fef28ac docs(claude): the YouTube write-error helper, the overlay and z-scale rule, and px-frozen sizes join the canonical patterns 2026-10-17 14:47:57 +02:00
peter fe572d9207 docs: a copy-only change runs the Conformance check's step 4 alone, in both languages 2026-10-17 14:27:45 +02:00
peter c8247c0dde release: 0.107.2 (improvement/gate-runner) 2026-10-17 04:37:30 +02:00
peter 722c37a6d0 chore(release): 0.107.2 — the gate lists and runs single lanes, the backend tests use every core 2026-10-17 04:30:33 +02:00
peter 5ba5679239 test: tests that assumed an idle machine wait for what they assert, or carry a budget for their work
Under a busy machine (the gate's lanes side by side, 12 busy cores) these went red at random: a press landed while the button was still busy, an answer was settled before it was asked for, a page grew past one wait's budget, a lazy chunk was transformed inside a 1 s wait, focus was taken a frame after the press. They now wait for the condition they depend on, preload the chunks they open, and the tests that do heavy work (whole-repo scans, the type-aware linter, module resets) carry an explicit budget naming that work. The whole suite four times under 12 busy-loop processes: red 4/4 before, 0/4 after.
2026-10-17 01:08:15 +02:00
peter 926e5e9899 chore(gate): check.sh can list its lanes and run one of them
--list prints the lane names, --only "<lane>" runs that one lane, so a runner can schedule the lanes itself and a red lane can be re-run on its own. An --only run reports that lane, never the gate, and an unknown name exits 2 instead of passing with nothing run; the contract test holds both.
2026-10-16 02:53:30 +02:00
peter 9251918e08 chore(test): the DB lane runs on every core — each pytest-xdist worker gets its own database 2026-10-16 01:13:31 +02:00
peter 2b6d968833 release: 0.107.1 (improvement/gate-timing) 2026-10-16 00:45:45 +02:00
peter 95e6b2c3d9 chore(release): 0.107.1 — a failed download's explanation says press Retry 2026-10-16 00:33:59 +02:00
peter 80c55fbf1b merge bug/download-retry-copy into improvement/gate-timing 2026-10-16 00:33:38 +02:00
peter 011f269152 chore(gate): every lane reports how long it took, and a run can log the times for comparison 2026-10-16 00:13:47 +02:00
peter dcb244dd04 fix(downloads): a failed download's explanation says press Retry, not retry it from a list that is gone 2026-10-15 22:56:40 +02:00
peter 08a9f4e255 release: 0.107.0 (feature/e39b-machine-access) 2026-10-15 06:20:35 +02:00
peter 1673134c36 chore(release): 0.107.0 — API tokens for your machines, share a whole folder by link 2026-10-15 06:09:23 +02:00
peter 27427eef32 fix(files): ending every session and making a token lock the user without holding up rows that refer to it 2026-10-15 06:05:51 +02:00
peter a248ca6c13 fix(files): the token list is fresh after the sessions end, a copy again reads Copied again, the Copied button says so 2026-10-15 05:47:00 +02:00
peter 620f84dba4 fix(files): a token made as every session ends goes with them, and a file being shared holds its job 2026-10-15 05:47:00 +02:00
peter 2efa7d5a75 fix(files): an upload is said once in any order, copy buttons speak, a link list that failed says so
- The queue holds an upload it sees land done while this tab's upload still waits for its answer, so
  the dialog's notice is the only one whichever comes first; another device's upload follows after.
- One `useCopyFeedback` for every Copy beside a value (token, share link, set-password link): Copied
  for a moment, a notice a screen reader hears, a warning when the clipboard refuses; the token
  dialog's two Copy buttons are named by what they copy.
- The share dialog's link list shows loading, and a load error with Retry, never "no links" — and no
  New link beside a list it has not seen.
- The settings texts say that signing out elsewhere or a password change revokes the API tokens.
- DESIGN.md logs the folder-link decisions; the design lab keeps only the chosen renderings of the
  Links, Tokens and Files rounds.
2026-10-15 04:58:17 +02:00
peter 827df7494c fix(files): ending every session ends every API token, and the share links hold their folder
- A password reset or change, "log out everywhere" and a revoked untrusted password revoke all of the
  account's API tokens too (audited): one minted from a stolen session no longer outlives the answer.
- A bearer token or link grant with a non-ASCII character is refused (401/403), not a 500.
- Making a token's grants or a folder link holds the folder, so a racing delete cannot fail the
  foreign key; a link's view count is one atomic UPDATE.
- A folder link's card counts exactly the files its page lists (one `links.folder_files`).
- A file with no extension is served without a trailing dot.
- 0106's downgrade names a Hub folder so it never collides with one of the user's folders.
- The speed baseline reads share tokens through the app (the database keeps them hashed/encrypted).
- Tests: the token scope on the real session stack, the folder poster, the Hub refused for a link,
  0106's downgrade data step, 0107 without a key; docs/comments follow (three folder kinds, folder
  links, key change, the cleanup job's token warnings).
2026-10-15 04:58:16 +02:00
peter 86712b6b0e feat(files): share a folder by link — its page lists what is in it, plays a video above the list
A /watch link may now open one of its owner's own folders (migration 0108: folder_id beside job_id,
exactly one set; a folder's links go with it). The page shows what is in the folder when it is
opened, newest first: a video or a song plays in the same player a file link has, above the list,
each keeping its own resume point; a document only downloads, never opens as a page. A file moved
out stops opening through the link at its next request. Password, expiry and stream-only work as on
a file link; the unfurl names the folder and counts its files, never their names. Source folders
and the Hub cannot be shared.

The folder menu gets "Copy link" and "Share with a link…" (the Share dialog's link half); the rail
marks a folder a live link opens.

Two fixes in the Share dialog on the way: revoking the last link no longer mints a new one at once,
and the new-link form goes back to downloadable after making a link.
2026-10-15 02:23:24 +02:00
peter 90567718b2 feat(files): share-link tokens are kept only hashed and encrypted, and every link change is audited
A /watch link's token is the credential, and it sat in the database as written. Migration 0107
replaces the column with the token's SHA-256 (the lookup) and a Fernet-encrypted copy (so the owner
can copy the same link again); every existing address keeps working, and the downgrade decrypts
them back. Without a copy this key can read, a link still opens but its address cannot be shown:
"Copy link" then takes the newest link that has one or makes a new one, and the Share dialog says
so in that row instead of a broken URL. Making, changing and revoking a link are audited, never
with its token or password.
2026-10-15 01:02:29 +02:00
peter c31d7f9d65 fix(files): an upload is announced once, as an upload, and a finish shows in Files at once
An upload that came back done was announced twice: by the upload dialog ("File uploaded") and
again by the tray, which saw a job turn done ("Downloaded: …"). The dialog now marks its job as
announced, so the queue keeps it for the tray but does not say it again. An upload the worker
converted first is still the tray's to announce, now as "Uploaded: …".

A finished job's notice offered "Open" while Files still listed the folder without it (its own
poll runs every 15 s): the tray now refreshes the library the moment it sees a finish.
2026-10-14 23:54:19 +02:00
peter db56f25387 feat(files): API tokens in Settings, the Hub folder in Files, a notice before a token runs out
Settings gains an API tokens tab: a row per token (name and prefix; each
folder with its rights; last use and expiry, "expires soon" marked),
revoked from its menu behind a confirm. A new token is the Hub exchange in
one click, or folders and rights picked per folder (no Write where an
upload cannot land); expiry 30/90 days, 1 year or never (warned). The
token shows once in the same dialog, with Copy and a ready curl line.

The Hub folder closes the Sources in the Files rail, takes uploads and
moved-in files. A token about to run out warns its user once, a week
before (the Files GC's sixth pass); the notice opens the tokens tab.
2026-10-14 23:33:34 +02:00
peter 4f58d15d3f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 21:19:14 +02:00
peter 2c66ce1cf4 release: 0.106.2 (improvement/thumbs-and-apt-refresh) 2026-10-14 21:17:39 +02:00
peter ce45445d17 chore(release): 0.106.2 — pictures that fail to load retry instead of breaking 2026-10-14 21:06:18 +02:00
peter 383d394e32 fix(ui): a picture that fails to load shows the card's own frame, then retries
A bare <img> sent a Referer that Google's image CDNs refuse and had no retry and no fallback, so a
Home shelf card showed the browser's broken-image mark. ui/RemoteImage gathers what Avatar, the
channel banner and the download rows had each hand-rolled: no Referer, up to three retries past the
browser's negative cache, the caller's fallback (by default the card's empty frame) instead of the
broken mark, the failure remembered by URL. Every picture now goes through it; a lint rule refuses
a bare <img>.
2026-10-14 20:51:08 +02:00
peter 06e8291896 build(image): rebuild the apt-get upgrade layer when APT_REFRESH changes
A build cache kept the upgrade layer for as long as nothing above it changed, so a publish shipped
curl and librsvg a release behind fixes already in the Debian archive (the CVE gate refused it).
The release build passes the day; empty, the default for a self-hosted build, leaves the cache alone.
2026-10-14 20:51:07 +02:00
peter db6b60029f Merge branch 'main' into feature/e39b-machine-access 2026-10-14 20:17:05 +02:00
peter 02134799cc release: 0.106.1 (bug/plex-settings-outside-click) 2026-10-14 20:03:38 +02:00
peter c8bc843517 chore(release): 0.106.1 — the Plex player survives a click outside its Settings panel 2026-10-14 19:44:25 +02:00
peter 759a7536b9 fix(history): an overlay opened while a back() is in flight waits to push its entry
The Plex player's ⋯ → Settings closed the menu (its entry went back) and opened the panel a task
later, before that back() had landed. The panel's push was the entry the landing back() took, so
the books counted one entry more than the history held, and closing the panel by a click outside
it stepped back over the player's own entry: the player closed.

A push now waits while a back() of ours is in flight; the popstate that lands the last one
reconciles again.
2026-10-14 19:43:51 +02:00
peter b2d34e6e6c feat(files): API tokens — a machine reaches the library folder by folder
A user can make a token for a machine of theirs (`/api/tokens`, signed-in only): `sfl_` + 32
random bytes + a checksum, only its SHA-256 kept, shown once. Sent as a Bearer header it is
honoured on six `/api/files` routes only — folders, list, file, hash, upload, delete — and there
only in the folders its grants name, with the rights they give (list / read / write / delete).
One check, `FilesCaller.may`, in every route; a folder or file out of reach answers like a
missing one; the files shared with the user are never in reach. A token beats a cookie on the
same request.

The Hub preset makes the user's one Hub folder (a new folder kind) and grants it every right.
Refused tokens are slowed down per client, a runaway machine per token; last use is stamped at
most once a minute; making and revoking a token is audited, and a token's uploads and deletes
name it in their audit rows.
2026-10-14 18:40:23 +02:00
peter c335d54929 release: 0.106.0 (feature/e39-files) 2026-10-13 03:34:56 +02:00
peter 7188ad7d71 chore(release): 0.106.0 — the Files library 2026-10-13 03:24:14 +02:00
peter ebbae6794c fix(files): review round 3, the page — a search waits for the server, the viewer keeps its file, a drop ends the drag
- past the list's cap a search waits in a skeleton, never showing the newest files' 'no match' for a
  file the server is about to find; a sort keeps its stand-in rows
- the viewer keeps the file it was opened on when a server answer replaces the rows under it
- a drop another zone took still clears the page's drop overlay
- 'The first N items' only for a server sort; the past-the-end notice no longer claims the file exists
- a failed server search is tested to say so
2026-10-13 03:15:59 +02:00
peter c9baf68b16 fix(files): review round 3, the server — a fetched locked re-read, a finish that follows its file, an eviction that counts again
- set_retention and delete_own re-read the folder under its lock and use that row (an unfetched ORM
  result never refreshes the object); a folder gone meanwhile is refused (409), and the route audits
  the retention actually replaced
- a finishing download is stamped only while it is still in the folder (and keep) it was read with:
  a move or a folder delete landing meanwhile sends the read round again, so no file is done with no
  expiry
- the eviction pass counts the disk before each candidate and skips one asked for since it began
- docs: source builds get their own volume-copy snippet
- race tests assert the lock wait; one built on a committing delete never raced
2026-10-13 03:15:59 +02:00
peter 5bb101e493 fix(files): review round 2, the page — the viewer's Escape, the dialog's drop, the capped list's honesty
- Escape closes the file viewer again: its layer sits above its dialog's, so it takes the Escape.
- The page's drop guard leaves a drag the open Upload dialog's zone accepted alone (it ran last and
  refused that drop).
- A capped list's server sort or search keeps the rows on screen until its answer is in (no skeleton
  under the pressed header), says "The first N items" when that answer is cut too, and an arrival is
  decided on the rows shown: one past the end of a cut list is said to be so, not gone.
- The search an arrival clears is taken at once.
- Tests: confirm names asserted per confirm, a fitting text pages under the arrows, a deleted file
  leaves the tray's finished list.
2026-10-13 02:32:50 +02:00
peter 54a6bbf07f fix(files): review round 2, the server — locks the GC and folders read under, the mount check, the old mount name
- GC: the expiry warning marks each file conditionally on the expiry it saw, one commit each (a file
  deleted meanwhile no longer fails the run; one re-timed keeps its fresh expiry unwarned); eviction
  counts the disk again when a candidate was freed meanwhile, so it never evicts for space already free.
- Folders: a retention change locks the folder and its files, and a finishing file reads the retention
  under a share lock, so neither misses the other; a folder delete locks the files it takes home (a
  file moved out meanwhile stays put, one deleted meanwhile no longer fails the delete).
- Upload: the folder is checked before the probe and held only once the job names it.
- Dedup enqueue locks the cached asset, so a release meanwhile yields a fresh one instead of a 500.
- Mount check reads the kernel's mount table instead of comparing devices (the vfs storage driver gave
  a mounted volume the device of /), and only inside a container; the API also logs a fresh, empty
  volume in the library's place.
- Compose files fall back to DOWNLOAD_HOST_PATH; the upgrade notes cover the from-source pull.
- Folder routes call owned_or_404 directly.
2026-10-13 02:32:49 +02:00
peter 8b2260c9a8 fix(files): review round 1, the page — honest states, the keyboard, the tray's words
- A library past the list's cap is searched and sorted on the server; the count says "the newest".
- Each folder and each search opens on its first page (DataTable pageScope, useCardPager scope).
- A file dropped where the page takes none never reaches the browser (which would unload the app).
- The viewer's arrows wait while a layer opened from it is on top, and scroll a wide text line;
  a text shows its first 5000 lines and says where the rest is.
- Confirms name the file (no more "your downloads"); the asking menu rows end in "…".
- The tray names a running step beside its bar, shows a pause mark for paused work, polls slowly
  while only paused work waits, and forgets a finished file that was deleted.
- An unchanged rename sends nothing; a paste opens the download dialog only with nothing on top;
  the link box never replaces what was typed.
- Honest states: folders that failed to load, a file no longer there, the storage figure; the rail
  says "nothing downloaded" only when there is nothing.
- Focus after deleting the open folder and when the format form opens; the format name's label.
- ui/Meter (one danger line for both quota bars) and ui/RenameField in the platform layer.
- Copy fixes, stale comments and the speculative hub key; a contract test for the list's
  parameters; e2e: the Files page and the paste intake.
2026-10-13 00:14:09 +02:00
peter 815083bcc4 fix(files): review round 1, the server — the tray's window, the GC's locks, a file's origin, the upgrade path
- The queue keeps a done job by when it became done (ready_at), not by its last write: renaming,
  keeping or moving an old file, or the GC's expiry warning, no longer brings it back as just finished.
- GC expiry and eviction re-read each file under the asset→job locks the user routes take, and commit
  one file at a time: a delete, a re-download or a keep made while the GC runs is honoured, never a
  second release that frees a file another user still holds.
- A file records the origin it was placed under (download_jobs.source_key, migration 0105): an edit
  whose parent expired still goes back to its folder, not to Uploads.
- A folder a file is being put in is held (FOR SHARE) until it lands; a delete of it waits and takes
  that file home too, instead of failing on the foreign key.
- The hash route reads at most two files at once (429 past that) and holds no connection meanwhile.
- 0104's backfill dates old files by their request date only: no notice prod wrote names its job.
- The old DOWNLOAD_ROOT still finds the files; a files root that is not a mounted volume stops the
  worker and is logged by the API; the volume-copy recipe looks the project-prefixed names up.
- Stale comments about the /downloads mount and the Downloads page.
2026-10-13 00:14:09 +02:00
peter 96509fe356 fix(files): a file is added when it is done, not when it was asked for
A download that failed for days and then went through sat days down the library, under its
request date, and read as missing. A job now records when it became done (`ready_at`), stamped
where the expiry already is (`folders.stamp_done`, `finalize_if_running`) and on an insert that is
already done; a check constraint holds every path to done to it. The library sorts and dates its
"Added" by it; the queue keeps the request order.

The migration dates a file from before by its "ready to watch" notice where the inbox still holds
one, else by its request date, where it already sat.
2026-10-12 22:19:37 +02:00
peter 132fb56428 fix(downloads): a site the downloader no longer understands says so, and offers a retry
yt-dlp asks for a bug report on an extraction failure it did not expect: the site answered in a way its extractor does not read (measured on a TikTok link: fails on every attempt under yt-dlp 2026.07.04, downloads under 2026.08.19). That failure read as "Something went wrong" with no retry; it is now its own category, site_changed, with a Retry in the failure window. A contract test holds every failure category to its copy in both languages.
2026-10-12 20:54:47 +02:00
peter f6bce23a94 fix(files): the Storage tab drops its user picker — every user already has a row and a menu
The list holds every user, files or not, so the picker above it only repeated a row's Edit quota. The empty text names what the list holds: users.
2026-10-12 20:06:30 +02:00
peter fb8fb980e5 feat(files): a file that is not media opens in a viewer — an image, a PDF, text as text
- GET /api/files/{id}/preview: an allow-list on the stored extension (downloads/preview.py) picks
  image / PDF / text; served inline under `CSP: sandbox; frame-ancestors 'self'`, the type from the
  table; HTML and SVG arrive as plain text; a text is its first 1 MiB (X-Preview-Truncated), sent
  as UTF-8 whatever it was written in (BOM, UTF-8, else cp1250); a NUL means binary: 415
- the list says which preview each file gets (asset.preview); FE/BE contract on kinds and header
- the viewer (shared dialog shell): ← / → through the list's such files in its order, Save, open in
  a tab, the file's menu; one stage height for every kind; a no-preview card offers the file
- the name, the card's picture and the menu's Preview open it; an image up to 8 MB is its own
  thumbnail
2026-10-12 15:28:31 +02:00
peter 6707550513 fix(files): a steady list header, every user on the Storage tab, the name gets the row's room
- The Files list starts at the scroller's top: the padding above it let the sticky header ride up
  12px on the first scroll, then stop.
- Users -> Storage lists every user with their quota, not only those already holding a file — a
  quota is set before anyone downloads.
- On a wide Files list only the "…" holds a place; Copy link, Share and Save come up over the date
  while the row is hovered or holds the keyboard (a card keeps them in its line). Keeps is narrower.
2026-10-11 20:33:48 +02:00
peter 078194731b feat(files): the Downloads page goes — the tray, a Storage tab and a Downloads settings tab take its place
- The Downloads module, its rail icon and badge are gone. A bookmark, the remembered page and an old
  history entry that name it land on Files (urlState.resolvePage); `g d` goes to Files too.
- Users page: a Storage tab — the disk in numbers, every user's use against their own limit with
  their file count, every user's downloads; aligned tables, one menu per row (edit / reset quota,
  open source, copy, the owner's quota). The admin storage rows carry quota + file count, the admin
  job list its owner.
- Settings: a Downloads tab — the download formats in one list (duplicate a built-in, edit or delete
  your own) and the site sign-ins, moved from Account.
- "View" on an added download, a clip or a converting upload opens the tray on that row (unfolded,
  back if closed; a finished one opens as a file; under a player it opens without taking focus).
- The download dialog opens on the format used last.
- DataTable: `pager={false}` for a short list, `cardRow` to give a lone "…" its own card line.
- Welcome and README speak of Files; a new Files screenshot.
2026-10-11 19:37:13 +02:00
peter 1c4f0042d2 feat(files): download from anywhere — a link box, a pasted link, the palette; the running downloads in a tray
- Files toolbar link box (paste button, Enter); a lone http(s) link pasted onto any page opens the one
  app-wide download dialog, which now names the link; the palette's "Download from a link…" focuses
  the box on its history entry.
- The download tray, bottom right on every page: what waits, runs, is paused or failed (reason +
  Retry, the queue's right-click / "…" menu), and what finished in this tab as "Done · Open" until
  closed; folds to a pill (per-account pref); closed, it stays shut — across a reload — until the
  work changes or something finishes. Back to top steps up over it, the page end keeps clear.
- A finish is announced once: "Downloaded: …", whether a phone plays it, Copy link + Open (toasts
  and notices may carry a second action).
- The inbox's ready notice opens the file in its folder: download_done carries job_id + folder_id.
- GET /api/downloads is the queue only (in flight, failed, finished in the last 10 minutes), with
  each row's source key.
2026-10-11 16:56:13 +02:00
peter d22c64f4dd feat(files): folders managed in place, a grid per folder, drop to upload, open on one file
- Own folders are made, renamed and deleted from the rail (right click, long press,
  Shift+F10) or the "…" beside an open folder's name; any folder's retention is a
  submenu (7/30/90 days, until deleted). Deleting confirms where its files go.
- List or grid per folder, sharing one sort; the grid is the one media card.
- A file dropped anywhere on the page opens Upload with it, bound for the folder
  on screen.
- A link (?item=) or the upload notice opens Files on the file: its page turned
  to, the keyboard on it, a ring round it.
- An in-place name field left by Enter or Escape hands the focus back to what
  replaced it (the row, the entry, the "…") instead of dropping it on <body>.
- The table's saved page size is per account.
2026-10-11 15:59:12 +02:00
peter 85aa917b38 feat(files): the Files page — the library in one aligned list, its folders in a rail
Files is the one library: every done file, the ones shared with me, or one
folder's, picked in a side rail (sources, own folders, the storage meter) and
mirrored to ?folder=. One table with every kind of value in its own column;
Copy link, Share and Save in sight, every action on a right click: play,
cut, rename in the row, move to an own folder or back home, keep, details,
re-download, delete.

The Downloads page keeps only the queue until the tray replaces it. The old
/api/downloads library routes are gone: list shared, upload, the details
PATCH, delete, the file and the usage now live under /api/files, and a
listed file says which source it came from. The job's "expired" flag is gone
with the state it described.
2026-10-11 05:34:25 +02:00
peter 1a1665b8ec feat(files): the Files library API — folders, moving and keeping files, upload into a folder, the file's hash
/api/files is the library's home: list a user's done files (by folder, searched, sorted, or the
ones shared with them), rename / move / keep / delete a file, make, rename and delete own folders
(a deleted folder's files go back to their source folders), and set how long any folder keeps a
file — done files shift by the difference, never closer than the GC's warning window.

An upload can name its folder (an own folder or Uploads), checked before the body is read and again
before the file lands; it is hashed while it streams, and any file can be hashed on first ask. The
download says the hash (Repr-Digest) once it is known; a new file at an asset's path forgets it.

Uploads, deletes, keeps, moves and folder changes are audited (ids only). The fileless "expired"
rows of the old retention are deleted by migration 0103: an expired file now leaves the library.

The shared route helpers moved to routes/files.py; the old /api/downloads library routes delegate
to the new ones until the client moves over.
2026-10-11 04:42:55 +02:00
peter bf99444ded feat(files): every file sits in a library folder, and keeps its own expiry
The Files library starts here. A user's every download, clip and upload now
lands in a source folder made per origin (YouTube, Plex clips, uploads, one per
other site), decided once in downloads/folders.py at every place a job is made.
How long a file stays moves from the shared asset to the user's job: stamped
from its folder when it is done, never for a file the user keeps. The GC warns
each user once, then takes the expired file out of their library; the cache cap
never evicts a kept file, and an evicted one leaves every holder's library.

The media mount is /files now (FILES_ROOT, FILES_HOST_PATH, volume
siftlode_files); docs/self-hosting.md says how to carry an old volume over.
Migration 0102 places every existing job and carries each done file's expiry
over from its asset.

safe_abs_path takes a missing path and answers None, which drops one baselined
type error.
2026-10-11 03:30:34 +02:00
peter 07e0990ecf design(files): the one-library round in the lab — aligned list columns, the tray, the done notice
The Files page replaces the Downloads page: list rows with one column per value kind and a
right-click menu, an app-wide download tray, a done notice that copies the link. Picked
variants are logged in DESIGN.md.
2026-10-11 02:59:51 +02:00
peter e2ed2bf874 merge: chore/e2e-retry-connect into E39 (rides the next release) 2026-10-11 02:30:16 +02:00
peter b9eb8458b3 test(e2e): a localhost connect the machine dropped is tried again on the test side
On a Windows host a burst of test traffic can drop a loopback connect now and then (every port,
any program — measured on the dev desktop). The browser resends it and only waits; Node (libuv)
sends a loopback SYN once and fails at ~300 ms with "connect ETIMEDOUT". `e2e/retryConnect.ts` wraps
Playwright's request context so that one error is tried again (0.5 / 1 / 2 / 4 / 8 s, the system's
own retry waits) while every other failure goes straight to the test; global.setup's health check,
login and /api/me and speed.measure's share probe go through it. Unit-tested; e2e 61/61.
2026-10-11 00:45:13 +02:00
peter 891507dc75 release: 0.105.0 (feature/e38-ad-hoc-clips) 2026-10-10 23:38:13 +02:00
peter e6894ac0fc release: 0.105.0 — ad-hoc clips 2026-10-10 23:13:22 +02:00
peter 663a7e8012 fix(player): the Plex player's fills hold on its chrome: an amber head on a white played part
A light theme's accent as a fill on the always-dark chrome was ~2.8:1 (Slate), under the 3:1 a
control needs. A native checkbox and slider tint with `accent-chosen` (the accent elsewhere, the
media amber under `media-marks`), and so does the skip button's countdown. The progress bar plays
white with an amber head: amber there marks a place, as the intro / credits markers do, and an amber
played part made a marker ahead of the head read as already played (owner's choice of three).
2026-10-10 22:56:09 +02:00
peter 93192dadd8 fix(player): a chosen tab's underline and an open or pressed icon read the scope's chosen colour
The underline named `--color-chosen`, which is resolved once on :root, so it stayed the accent under
`media-marks` (seen in the browser: fuchsia on Slate light); it reads `--chosen` where it is used
now. An icon button pressed or with its menu open tinted its glyph with the accent; it takes the
chosen colour — the accent everywhere but the player chrome, where it is the media amber.
2026-10-10 22:26:22 +02:00
peter 5b48018e92 fix(downloads): a produced file is held to the space left by its real size
yt-dlp's `max_filesize` only stops a plain HTTP file that announces its size: YouTube's fragmented
formats never meet it — measured on localdev, a 718 MB download ran to the end with 1 MB left and
put its owner four times over the cap. The download now re-reads what is left once its file is made
(after the phone-safe pass) and refuses one larger than that as the owner's refusal; an editor clip,
which nothing checked, is held the same way, as a player clip already was.
2026-10-10 22:10:56 +02:00
peter 8312c6d592 fix(player): what is chosen on the player chrome is marked amber, not written in the accent
A light theme's accent as text on the always-dark chrome was 2.84:1 (Slate). `--chosen` is the
accent unless a scope names its own; `media-marks` (was media-focus) points it, with the focus
ring, at the media amber. A chosen track stays white with an amber tick (MenuCheck reads
`text-chosen`); a chosen settings tab or toggle is white with an amber underline (MEDIA_CHOSEN).
The theme contract test holds the token, the tick and the player components free of accent text.
2026-10-10 21:56:47 +02:00
peter e1d325b942 fix(downloads): a download larger than the space left says so, not "processing failed"
yt-dlp refuses a file announced over `max_filesize` by logging one line and returning with no file
(measured, and pinned in test_formats against a local server): the worker found no output and
failed the job as `postprocess`. `formats.watching_size_cap` now sees that line through our yt-dlp
logger, and the download (and, should it ever happen, a YouTube clip's range fetch) fails as the
owner's refusal with DOWNLOAD_TOO_BIG_MESSAGE, classified `quota_exceeded`. The download's quota
copy says it does not fit in what is left — true at a full cap too — instead of "Storage full".
A pre-existing type error the change touched is fixed (baseline down by one).
2026-10-10 21:56:47 +02:00
peter 977ba244fa fix(clip): a time's error line sits in the flow, the remove button off the cut, and every player surface rings amber
- useTimecodeField holds a time field's draft for its caller, which places TimecodeError (role
  alert) in its own flow under the fields' row: it no longer lies over the crop row taking its
  clicks, runs out of the editor's card, or joins the field's <label> name.
- The editor's remove-cut buttons sit in a row under the track at their cut (trackLeft), off the
  cut's grab area, where a press drags it; RangeTrack's adornment slot is gone.
- media-focus re-points --focus to the media amber on the Plex player, the YouTube stage and every
  bg-media-chrome surface; a contract test holds each such class string to media-focus or
  media-scope, the field's focus edge to the focus token, and --color-focus to the inline theme.
- An upload's quota headline says the converted file does not fit, not "Storage full".
- DESIGN.md: the tokens paragraph says what is re-pointed where; the decision row measures all
  three colours on the chrome.
2026-10-10 21:29:55 +02:00
peter 89d18b46a6 fix(downloads): a converted upload too big says so, and a suspended owner's waiting rebuild is not "preparing"
- A converted upload over the space left (not zero) fails with CONVERTED_TOO_BIG_MESSAGE, not
  "storage is full"; it classifies as quota_exceeded by the phrase it shares with the clip's.
- The public watch page says "preparing" only for a job a worker will take: a suspended owner's
  queued job waits for as long as the suspension lasts, so the link says the video is gone, as
  for a paused rebuild.
- The worker's claim docstring names the restart case: a running job put back in the queue waits.
2026-10-10 21:29:55 +02:00
peter c27cfdcb9c fix(clip): typed times belong to their value, and every mark on the player chrome can be seen
TimecodeField drops a draft when the value moves under it (another segment selected, a nudge), so
leaving the field never commits it to another cut; it is invalid only once offered, with a line
under it saying how to type a time. On the always-dark chrome media-scope points every focus ring
at --media-mark and errors at the new --media-danger (a light theme's accent and danger were under
3:1 there). A track whose dragged handle unmounts ends the drag on the pointer's release instead of
locking up; a remove refused while clips are queued leaves no focus to take later; each remove-cut
button names its cut and is a 24px target. DESIGN.md records the media tokens; stale comments fixed.
2026-10-10 20:28:52 +02:00
peter a867145bcb fix(downloads): a suspended user's queued jobs wait, and a clip too big for the space left says so
A suspension now blocks the user's downloads as quota.resolve already advised: the worker leaves
their queued jobs unclaimed until it is lifted (the confirm says so). A cut clip larger than the
remaining storage is refused with its own message (not "storage is full"), and a clip's quota
refusal reads as a clip's in both languages. The cancel poll compares to JobStatus.RUNNING.
2026-10-10 20:28:52 +02:00
peter 4d4eea2c54 fix(clip): marks on the player chrome keep their amber in a light theme, and a removed cut hands the focus on
- A media amber (`--media-mark`), the same in both themes, for what marks a place on the always-dark
  chrome or over a video: the clip strip's playhead and focused end, the crop frame's ring, Plex's
  intro / credits markers. A light theme's warn was a dark amber there, under 3:1 (contract-tested).
- The editor's remove-cut button rings inside itself (the track clipped its ring), and once it goes
  with its cut the focus moves to the cut beside it, or to Split here, not to <body>.
2026-10-10 15:57:02 +02:00
peter c489b60c8b fix(clip): clip mode keeps its range, its focus and its keys through the player's own moves
- A mark on the very end seeks a moment short of it; while a clip is marked the end of the item
  neither closes the player nor steps the queue, and Plex's auto-skip waits.
- Plex: Escape and Back peel what is over the picture (info, up-next, the countdown) before the
  clip; Back under a dialog is the dialog's; no range opens while the length is unknown.
- YouTube: the focus goes back to the scissors the user can see, else the player; the panel keeps
  its name, Loop and focus when the view is maximised or restored under it.
- A typed time stays in its field until taken, then shows where it landed; unreadable stays marked
  invalid; a decimal comma is read (ui/TimecodeField, the editor's fields too).
- RangeTrack: a drag is its pointer's, a cancelled drag reports final, an adornment sits beside the
  slider, a caller-chosen focus ring (amber on the clip strip). CropFrame: a drag is its pointer's.
- The editor's remove-cut is a real button at the cut's foot; a segment's keep toggle sits over the
  cut's hit area.
2026-10-10 15:35:21 +02:00
peter b4bf71ea64 fix(clip): a clip is held to its owner's storage cap, and its limits and failures say the server's truth
- The cut file is measured against what its owner has left, for a YouTube range (whose fetch never
  reads yt-dlp's size ceiling) and for a Plex clip (which fetches nothing), before the fetch and
  again before it is placed; a refusal leaves the asset pending for a Retry.
- A silent Plex file is cut silent; a file the probe cannot read says so.
- A second job on a failed or renamed clip gets the same YouTube diagnosis and the fetched title.
- The "too long" refusal names the server's cap; a contract test holds the client's copies of the
  clip limits to clip.py; the shipped player clients are pinned against the installed yt-dlp.
- One HDR transfer set and one rating_key lookup; job status words from JobStatus; the display-name
  guard sees subscript writes and the clip route; stale comments and types follow the clip job.
2026-10-10 15:35:12 +02:00
peter 72df11f2c0 fix(ui): a small crop frame takes smaller handles, so it can still be grabbed to move
The handles reach 10 px into the frame from every side; a frame at its minimum on a small picture
(the Downloads editor at 800 px: 41 x 23) was then nearly all handle - 74 of its 920 px still moved
it. Below 64 px on its shorter side the frame takes 12 px corners and 8 px edges: 262 px move it.
2026-10-10 13:54:04 +02:00
peter 0a16bc40ae fix(ui): a crop frame grabbed with the pointer takes the focus, so the arrows go on from there
The press is held back from the page (no text selection, no play / pause underneath), which also
kept it from focusing: after a drag the arrows still went to whatever had the focus before, and
Escape did not reach the frame. The press now focuses the frame or the corner it grabbed (an edge
focuses its frame).
2026-10-10 13:38:49 +02:00
peter 520b0afb90 feat(clip): crop a clip to a frame drawn on the picture, and give the clip panel the room it needs
An optional crop in clip mode: a Crop toggle puts a frame on the picture (free, 16:9, 9:16 or 1:1;
the whole picture stays the default). The frame is sent in fractions of the box it was drawn in,
with that box's shape; the worker fits the real picture into the box by its display aspect (sample
aspect and rotation aware), drops the part over the bars, and crops first in the filter chain. A
frame covering the whole box is no crop at all, and one off the picture is refused by name.

The crop frame is a ui primitive (CropFrame) shared with the video editor, keyboard-operable.
YouTube: the wide side panel shuts while clipping and comes back after; maximised, the clip panel
docks in the stage. Plex: the picture sits above the clip panel so the frame lies on it.
2026-10-10 12:52:59 +02:00
peter e301632765 fix(clip): a clip refusal speaks the user's language, and a choice in the clip form no longer sends it
The backend refuses a clip with a structured reason, but the client showed every one as the generic
"couldn't make the clip": each reason now has its own message in both languages, held to the
backend's reasons in both directions by the contract test. A segmented toggle's and the priority
stepper's buttons had no type, so inside a form one of them submitted it.
2026-10-10 12:52:12 +02:00
peter 50157cc2fd feat(player): clip mode - mark a range of what is playing and cut it into a download
Both players get a scissors in their control row (Plex: the control row; YouTube: the actions and
the docked bar), `c`, and `[` / `]` to mark at the playhead. The clip panel lives in the player, so
the watched video is the preview: an overview of the whole item and a zoomed strip around the
selection whose ends are sliders (arrows step a tenth, Shift a second), Start / End fields with
nudges and "to the playhead", the length, a looping Play selection, a name, Cut. Escape closes the
panel before the player and the focus goes back to the scissors. A cut is a toast with a Downloads
link plus the usual Ready notification.

- ui/RangeTrack + RulerTicks: the shared, domain-free time track (regions, keyboard-steppable
  handles with a 1.5rem hit area, a playhead, a background slot, over a window of the item). The
  Downloads editor's cuts moved onto it, so they are keyboard-steppable now.
- lib/clipRange holds the marking rules (1 s to 10 min, the server's), lib/timecode the editing
  timecode the editor's private tc/parseTc became.
- keymap: a symbol typed through AltGr answers its bare-symbol row - on the HU layout `[` is
  AltGr+F, which arrives as Ctrl+Alt; letter and Mod rows still never match Ctrl+Alt.
- The YouTube stage sits in a wrapper with the panel, so the maximised stage's blur escape matches
  at any depth.
- media-scope: fields on the always-dark player chrome read white in a light theme too.
2026-10-10 03:58:27 +02:00
peter f221c1962a fix(downloads): a YouTube clip goes with the user's own cookie and is diagnosed like a download
A clip fetched its range signed out, so a video behind YouTube's per-video "not a bot" wall could
never be clipped while its download went through with the user's cookie - and the failure copy told
the user to add a cookie the clip would not use. The clip asset is private by construction
(`clip.asset_ref` names its user), so `usable_clip_cookie` attaches the user's YouTube cookie on that
identity, and a failed range fetch now carries the download catalog's diagnosis
(`youtube.bot_wall` / `youtube.cookie_rejected`) instead of a bare category.
2026-10-10 03:58:27 +02:00
peter d466ae915b fix(downloads): a YouTube clip outside the catalog takes its title from the fetch
It was queued as "Clip (clip)" and stayed so: nothing renamed it once yt-dlp had answered.
2026-10-10 02:43:13 +02:00
peter 14598ec320 feat(downloads): ad-hoc clips cut straight from a YouTube video or a Plex item
POST /api/downloads/clip takes a source (a YouTube id or a Plex rating key), a range of at most
ten minutes and, for Plex, the audio track being heard. The clip becomes a per-user asset in the
library like an editor clip, shareable as soon as it is ready. A YouTube clip fetches only its
range (yt-dlp download_ranges: 3-7 s whatever the video's length); a Plex clip reads the library
file in place. Both then go through one re-encode: frame-exact at the marks, H.264 at most 1080p
with HDR tonemapped to SDR, stereo AAC, no chapter track, .mp4.

Job kinds are a JobKind StrEnum now, and Re-download refuses every kind but a download.
2026-10-10 02:31:38 +02:00
peter b16cf7668c fix(downloads): YouTube downloads reach their profile's height again
The primary player client, android_vr, was still offered 1080p and 2160p, but every fetch of
those formats died with HTTP 403: it now needs a stream PO token the bgutil sidecar cannot mint.
The fallback set then served 360p only, so every download finished at 360p without failing.
tv_simply is offered the full ladder and its fetch completes (measured: 1080p, and a localdev
download through the pipeline reached the default profile's 720p). yt-dlp 2026.8.19 marks
the clients that need such a token, so it stops fetching into the 403.
2026-10-10 02:08:43 +02:00
peter 3438aeb695 fix(player): the player's own focus grabs leave a tabbed-to control alone
Turning ready, the pointer leaving the video, the window regaining focus, a wheel over it and
leaving fullscreen each put the focus back on the card unconditionally, so a viewer who had
tabbed to a control right after opening started over from the first button on the next Tab.
They now take it back only when it sits in the iframe or off the card.
2026-10-10 01:14:31 +02:00
peter c003a48244 release: 0.104.0 (improvement/adhoc-batch-oct) 2026-10-10 00:24:27 +02:00
peter 8a2334e198 release: 0.104.0 notes and version 2026-10-10 00:15:23 +02:00
peter 1f010328df fix(plex): plain text survives a decimal reference too long to parse
int() refuses a decimal run past 4300 digits, so one such reference in a summary raised out of the sync and the text migration. Any number past seven significant digits is beyond U+10FFFF, so it becomes U+FFFD like any out-of-range reference; leading zeros are dropped first. The rails test's comment now says what it guards: both rails agree with the rollup.
2026-10-10 00:03:08 +02:00
peter 77638f7a0b fix(player): a link that opens the cut text is scrolled back into view
Focus had scrolled the clamped box itself; opened, the link dropped by that much, out of a scrolling panel's sight. DESIGN states where Close sits in the stacked layout; a stale test comment goes.
2026-10-09 23:46:40 +02:00
peter 2cff4ad954 fix(feed): an override yields only to a refetch that set out after its write was answered
Send order let a refetch read before an unanswered mark committed and drop it. A reset's position 0 is written again after a refetch that set out before the server took the reset. The player-session refetch also waits for the player's closing checkpoint, which the player now tracks (progressFlushes); refetched first, the list put the older position back.
2026-10-09 23:46:39 +02:00
peter 8776586ca3 fix(plex): a matched episode and a collection member offer Details in their menu
Their titles open the item's page, a pointer-only door; the card menu is the keyboard's.
2026-10-09 23:46:39 +02:00
peter 92a17aba83 fix(plex): a reshuffle survives a refresh landing mid-deal; the ↻ spins for a deal only
The ↻ intent was spent as its fetch set out, so a watched toggle's resync cancelling that fetch re-ran it as a re-read of the old draw. It now lasts until the latest deal answers. The button spun on every re-read of the same titles; it now follows the deal. Comments that said the Surprise draw is never refetched are brought in line.
2026-10-09 23:46:39 +02:00
peter fc53bc03bc fix(plex): the rails' Restart comes from the one rollup; plain text decodes whole entity names only
Both Continue-watching rails built a show's card state inline; they now take it from rollup.show_state, so a show part watched with nothing to resume offers Play alone there too. A named reference counts only when its whole name is an HTML5 entity: html.unescape still decoded the legacy "&not" inside "&notice;". Tests cover the grid's in-progress-but-not-resumable case through its SQL and both rails.
2026-10-09 23:46:39 +02:00
peter 6df257d7df fix(plex): Restart only where an episode has a resume point; the Surprise draw is re-read, not redrawn
Show cards carry resumable (ShowState, from the same rollup) so a part-watched
show whose next episode is untouched offers Play, not a Continue and a Restart
that do the same thing; plexStarted is the one rule for cards and menus. The
Surprise rail's key sits under the discover root: whatever refreshes the rails
re-reads the draw (keep=) with its watch state now, kept past the 5-minute gc,
and only the reshuffle deals. Closing the player refreshes the Discover rails.
A show card's play acts on the last click only, and only where it was made.
2026-10-09 22:01:28 +02:00
peter 5572499b56 fix(player): the shut panel keeps the dialog's name; Tab onto a cut link opens the text
With the side panel shut the title heading was gone, and the dialog it named
went unnamed: the strip carries an sr-only copy. A link below a clamp's cut
stayed tabbable out of sight; reaching it now opens the text, measured from the
text's top since focus scrolls the clamped box. DESIGN describes the modal as
built, with log rows for its chrome and the wheel rule.
2026-10-09 22:01:27 +02:00
peter 76738f8ea9 fix(plex): plain text decodes only well-formed character references
html.unescape also decodes the legacy forms without a semicolon, so prose like
"AT&notice" came out as "AT¬ice". The show and collection sync paths are
now tested too.
2026-10-09 22:01:27 +02:00
peter 568bf93510 fix(feed): an optimistic write outlives only the server answers older than it
A reset wrote a position mask over the cached list, so a replay's saved
position stayed hidden: a reopen resumed at 0 and its first checkpoint wrote
over the progress. The reset now writes 0 into the cache, where the player's
closing patch writes the next one. Overrides carry their write order, and a
landing refetch drops only those written before it set out. The player
session ends after the modal unmounts, so a clip auto-marked watched as it
closes still refetches the list.
2026-10-09 22:01:27 +02:00
peter 942154cf49 fix(plex): a Surprise draw stays put until its reshuffle
The rail's query was always stale, so Back from a card's player or page remounted it into a new
draw, as did a return to the window: the card just looked at was gone. It now stands until the
reshuffle (or another scope, language or filter) asks for a new one.
2026-10-09 01:29:12 +02:00
peter 3eb87e7108 feat(plex): every show card plays the episode it would put on
A show card's picture opened the show page everywhere but Home, whose shelves carried the
resolved `next`. Now every Plex picture plays: a show asks `GET /api/plex/show/{id}/next`
(the in-progress episode, else the first unwatched, else S1·E1 — the rule Home and the show
page's Resume follow) on the click, fresh each time. A started show shows Continue + Restart;
its title, an ⓘ and the menu's "Open show" row open its page; one with no episodes opens there.

The question is asked per show, never carried on every card: three probes a show would cost a
grid page up to 180 queries. The hover preview asks it once it opens and names the episode
("Next: S6·E16 — Title"). Related shows on a show page play the same way.
2026-10-09 01:22:00 +02:00
peter 851b97b09d fix(ui): a card's hover chrome answers keyboard focus, not any focus
Back from a channel page hands the card its focus again after a mouse trip;
revealed on focus-within, its hover row stayed up under no pointer until a
reload (owner). The row, the Play layer, the Continue + Restart pair and the
global .reveal rule now show on hover or on :focus-visible on the card or
inside it, so Tab and the arrow keys still reveal it.
2026-10-09 00:44:46 +02:00
peter ee94d1949a fix(feed): a video marked watched in the player stays out of an Unwatched view
Marked watched in the player (by hand, or by the player at its end), a card
left an Unwatched list at once, then came back as "new" when the player
closed, until a reload. The player's closing position patch writes into the
cached feed; the feed took the bumped dataUpdatedAt for fresh server data,
dropped its optimistic overrides, and nothing refetched. The overrides now
clear only when a refetch lands, and a player session that changed a state
refetches the list and its count by the view's filters on close, once its
writes have landed.
2026-10-09 00:44:46 +02:00
peter 4f076b5d45 feat(feed): a card's menu filters the feed by its channel, and back
Under "Open the channel" a video card's menu (the right-click on its
channel name included) adds the channel to the feed's channel filter, or,
when it is already in, takes it out. Channels OR together, so it widens a
filter that names others, as "Show in feed" does. Not on a channel page,
whose view is that channel already, nor in a live YouTube search or on Home.
2026-10-09 00:44:46 +02:00
peter ece74a42ab feat(player): a roomier YouTube player, its panel shut on demand
- The full-height previous/next strips beside the card are gone: the panel's
  transport row and Shift+arrows step on their own, and the video gains the
  7rem the strips' gutters took.
- Close is one icon in the card's top-right corner (the wide panel's head,
  the stacked card's title row).
- The panel's description is cut to three lines with More / Less
  (ClampedText now takes the linked text as children).
- Wide, the panel shuts to a 2.75rem strip (close, open, previous, next) for
  a bigger video; it stays shut while the queue steps on, and a video opened
  from a card starts it open. Focus moves to the counterpart toggle.
- The wheel has one meaning per place: over a panel (or the stacked card)
  with something to scroll it only scrolls, never falling through to the
  volume at the end; over the video, or a panel that fits, it is the volume.
- A long channel name is cut on the panel's line instead of running the
  20rem panel into a sideways scroll.
2026-10-09 00:27:29 +02:00
peter 600d5c39da feat(ui): a button over a picture answers the pointer with an accent ring
The media variant's hover darkened its body from 60 to 75% black, which no
one could see on a poster; a rail's scroll arrow over a card could not be told
from the card's own Play under it. Its hover is now a 2px accent ring over an
80% body (owner pick B), on every card control, Plex and YouTube, and the rail
arrows. DESIGN.md B3 names it beside the edged pills, with its log row.
2026-10-09 00:27:28 +02:00
peter 4c7f73af9d feat(plex): a title opens its page, the picture plays; an episode links up
A card title lit like a link but played the item. Now the picture plays and
the title opens the page of what it names: a movie's or an episode's info
page, a show's page, and for an episode standing as its show (Recently
watched) the show. Every poster and episode card carries it (grid, rails,
person, collection, season, related, Home), a card with no page keeps a plain
title, and the season page gains the menu's Details row too.

The episode page had no way up: the detail now serves the show's and the
season's rating keys, and on the page the show title opens the show and
"S1·E25" the season (Back returns to where the page was).
2026-10-09 00:06:18 +02:00
peter 5f42ff61c8 feat(plex): a Surprise reshuffle deals the previous draw last
The rail draws three unseen titles per kind under the sidebar filters; on a
small filtered pool every reshuffle dealt the same few again. The redraw now
sends the draw on screen (exclude=), the server deals the rest of the pool
first and tops up from the excluded titles only when the pool is short, so a
small pool still fills the rail. (The two discover.py baseline entries went
with the rewritten draw.)
2026-10-09 00:06:11 +02:00
peter 6dd82ef5e2 fix(plex): summaries and taglines are stored as plain text
A Plex agent can hand a summary over as HTML (<p> paragraphs, &aacute;
entities); the info page and the hover card showed the tags. One cleaner,
app/plex/plaintext.py, now runs on every summary and tagline the sync, the
Hungarian overlay and the rich-detail fetch store, and migration 0101 cleans
the rows written before through the same function. Paragraph breaks survive
as newlines, which the info page and the hover card now keep.
2026-10-09 00:06:02 +02:00
peter 75f4efde98 release: 0.103.0 (bug/yt-fullscreen-button) 2026-10-08 22:50:52 +02:00
peter c0059a0c4b release: 0.103.0 notes within the item length cap 2026-10-08 22:36:15 +02:00
peter a29cadf03e release: 0.103.0 notes and version 2026-10-08 22:30:56 +02:00
peter 83886494ce fix(ui): a membership menu re-homes focus a frame later, once its new rows are on the list
Rows that just mounted join the menu's navigation list a render after the status effect ran: a row
focused that early was unknown to it, and the arrows did nothing from it (seen live after a keyboard
Retry). The retry test now arrows on from the row it lands on.
2026-10-08 22:16:47 +02:00
peter 4eaa55385f fix(plex): a collection kept after a failed refresh holds its seed title
When Plex made the collection and only the read after it failed, the kept row had no members: the
title showed unticked in the collection it had just been added to, the collection empty, and the
name stayed in the field for a retry that could only be refused as taken. The seed's row now names
the collection (child_count 1), and the panel clears the field on that refusal as on success. Also
held: the panel's lines sit beside its list, and a keyboard Retry lands on the first row.
2026-10-08 22:12:43 +02:00
peter 7541b1b016 fix(ui): membership menus keep their lines beside the list and focus in the panel
The loading / failed-load lines and their Retry sat inside role=menu, which may own its items only:
the three surfaces now open with `listInside` and the rows sit in a PopoverList. A keyboard Retry
unmounted the focused button and left focus on <body> with the menu open, the arrows dead: focus is
put back on the first row, else the field, else the new Retry. The channel tag editor said a failed
tag load as "no tags yet"; it reads the catalog query's state now. The playlist panel's name and the
tag editor's error state are held by tests.
2026-10-08 22:12:43 +02:00
peter 24ad17fc7d test: hold the show collection's type, an unknown title's 404 and Brave's fullscreen stand-in 2026-10-08 21:18:48 +02:00
peter 48bd038c5f fix(plex): a collection created on Plex keeps its editable row when the refresh read fails
The rollback dropped the new row: the next sync recreated it read-only, out of every 'add to' list, and the duplicate-name check let a retry make a second same-named collection on Plex.
2026-10-08 21:18:48 +02:00
peter 0fce732497 fix(plex): a playlist row's menu adds its title to a collection too
The row menu called the title's action list without the anchor the panel opens beside, and the optional parameter dropped the row silently. The anchor is now required, so every host menu decides.
2026-10-08 21:18:48 +02:00
peter 7e94da72b9 fix(ui): one membership menu for playlists, collections and tags
Three hand-copied panels had drifted: Add to playlist said a failed load as 'no playlists yet'. ui/MembershipMenu now holds them: a loading line and an error with Retry instead of a blank or an empty list, the up/down arrows step from the name field into rows that arrived after the menu opened, and the panels opened from a menu carry their own accessible name. A write that landed flips its tick at once, so a second tap before the refetch undoes it instead of repeating it; a collection edit also refreshes a show's own page.
2026-10-08 21:18:48 +02:00
peter fadb4f17a9 fix(plex): subtitles draw only the player's own track, lifted in the player's own pixels
A direct-played file's own subtitle stream (WebKit lists an mp4's mov_text) was switched to hidden and drawn under any choice, Off included: only the player's <track> is kept hidden and read, other subtitle tracks are switched off. The lift was measured in screen pixels and applied inside the x1.25 windowed player, and re-measured off a box carrying its own (or a mid-slide) transform: an untransformed outer box is measured, the distance divided by the scale, and a cue that just appeared is placed, not slid from where the last one was left.
2026-10-08 21:18:48 +02:00
peter 57a81c5643 fix(plex): subtitles survive hls.js and a remounted track
hls.js emptied the cues of every text track on the element when a manifest
loaded or the media attached, our own <track> included, so a VTT that loaded
first showed nothing for the session; its subtitle and caption controllers are
no longer built. A remounted default <track> was switched to showing by the
browser after the hide had run, drawing the cues natively under ours; the hide
now follows every added track and mode change.
2026-10-08 19:35:47 +02:00
peter dae0b80f6a feat(plex): add a movie or show to a collection from its card menu
"Add to collection…" opens a panel beside the card: the title's library's
manual collections tick it in or out, and a name + Enter creates a new one with
the title in it. It used to take the Collections page, a new collection, its
name and a title search, then another search per title. A lean endpoint lists
the collections with whether each holds the title. Admin only, as on the
title's page.

Also: a collection seeded with a show is made as a show collection (Plex type
2). Measured: as a movie one it came out empty and refused the show after.
2026-10-08 03:09:55 +02:00
peter 75093458c9 feat(plex): subtitles rise above the control bar while it shows
On an iPad the bar covered the text whenever it was up: the browser places a
native cue by its line, which nothing animates. The track now stays loaded as
hidden and PlexSubtitles draws the active cues, measured against the bar and
lifted above it in 200 ms, back down when it hides. The viewer's size, colour,
background and outline settings style the text directly. Also fixes a test
typing slip from the search pill change.
2026-10-08 02:59:13 +02:00
peter 5db6dc8ee6 docs(design): log the touch header search 2026-10-08 02:52:39 +02:00
peter 45a9ceae25 fix(ui): the header search has room on a tablet
On touch the 44 px key cap and Go filled a 44 px pill edge to edge and the
placeholder was cut. The key cap is left out on touch (⌘K still works), the
pill grows to 52 px so a ring shows round Go, the box's floor counts the
controls beside the text, and Go drops to its icon by the column's width,
not the screen's.
2026-10-08 02:52:32 +02:00
peter 5249c41edc fix(player): a fullscreen button the browser cannot honour is not shown
Brave on iPadOS switches element fullscreen off, patches requestFullscreen with
a video-only stand-in and fakes fullscreenEnabled, so the Plex fullscreen button
and the YouTube docked bar's did nothing. Support is now read off
Element.prototype and the buttons are left out without it (owner). A stand-in
that returns a boolean no longer throws. A test's bare matchMedia stub no
longer outlives its describe.
2026-10-08 02:50:51 +02:00
peter c3664826fe fix(ui): a caption never opens under a finger
On an iPad a tap's synthetic mouseenter opened the Tooltip; WebKit then took the
tap as a hover and swallowed its click, so every rail button needed two taps,
and with no mouseleave from a finger the caption stayed up. Tooltip now opens
for a mouse and a key's focus only (owner: no captions on touch).
2026-10-08 02:45:48 +02:00
peter c9b8fe15a3 fix(ui): a scroller with no edge to fade carries no mask
iPad WebKit kept a scrolled container's mask stuck to its content: at the
bottom of a scrolled page the last strip, a table's pager, sat under a dark
band. With nothing to fade there is no mask; isolation keeps its layer.
2026-10-08 02:43:58 +02:00
peter 1044eaf8a9 fix(plex): the player holds the page still and covers it whole; the row drops the countdown
On an iPad a drag on the video scrolled the page behind the player, and in Brave
dvh did not follow the toolbar, so a strip of the page showed under it. The
page is now locked without overscroll on a black root while the player is open,
and the player is sized from its fixed box. The time row keeps the position,
the length and the end clock; the countdown is gone (owner).
2026-10-08 02:42:31 +02:00
peter ca949d22de fix(player): the channel line wraps instead of squeezing the meta into a column
On a narrow card (a tablet, a zoomed window) the unwrapped row of name, meta and
actions left the meta a one-word column. The row now wraps: the meta keeps at
least 12rem, and the actions move down together.
2026-10-08 02:25:58 +02:00
peter 20178f0ec4 release: 0.102.0 (improvement/e37-player) 2026-10-08 01:58:01 +02:00
peter 49a9319f84 fix(player): the click layer clears only the fullscreen button's column, not a full-width band
Measured: across the full width YouTube needs only its seek bar (up to 96 px
from the bottom); the 140 px is the fullscreen button's, at the right end. A
full-width 140 px band handed clicks on the picture to the iframe, which then
takes the keyboard focus and our shortcuts with it. The layer is now two boxes,
hidden from assistive tech (a mouse surface; the keys are its shortcuts), and
the tests render the big view too, where the docked bar sits under the iframe.
2026-10-08 01:42:39 +02:00
peter 70d56ab939 fix(player): YouTube's fullscreen button stays clickable on a short player
The click layer over the embed left the bottom 22 % of the stage free. YouTube's
new embed keeps its fullscreen button 80-128 px above the iframe's bottom, so on
a stage under ~580 px (a small window, a browser zoomed to 150 %) the layer lay
over it and a click paused instead. The cutouts now have px floors (56 top, 140
bottom) and are measured against the iframe's own box.
2026-10-08 01:36:36 +02:00
peter 3865689c26 chore(release): 0.102.0 2026-10-08 01:33:39 +02:00
peter c32429ba69 fix(plex): an episode with no number is named by its title on every card
The cards, the up-next card and the mark-next label printed a question
mark where Plex has no number; the player's header already showed the
title alone. One helper (episodeNumbers/episodeName) for every place.
2026-10-08 01:16:30 +02:00
peter a504550196 docs: the superseded fallback answers its own session; who uses beacon() 2026-10-08 01:16:30 +02:00
peter 5095047c81 test(player): pin the stop URL's ids, the beacon's settle, the kept point's edges
The stop-URL contract passed with the two ids swapped and read the backend
from the working directory; nothing called a real beacon's onSettled or its
library pass-through; the kept-point rule was held by its constants only.
2026-10-08 01:16:30 +02:00
peter 4e70f168b1 fix(player): a layer that opens by itself takes no history entry
The auto-skip countdown and the up-next card pushed an entry without a
user gesture (Chrome's Back button skips those), and one pushed under the
open shortcuts sheet sat above it, so Back closed the hidden layer.
2026-10-08 01:16:30 +02:00
peter 7f17f488ab fix(nav): a sub-view step closes the overlays standing over it first
Leaving the Plex player while one of its layers was up (the info panel when
the film ended, the countdown under the back arrow) landed as a user Back:
the overlay listener spent it closing the top layer and the player stayed.
useHistorySubview's open/back now go through closeOverlaysThenNavigate,
like setPage. The RECLOSE comment no longer cites the shared Plex entry.
2026-10-08 01:16:30 +02:00
peter d1704a5519 revert: the Modal Escape guard for a fullscreen own-content video
Chrome does not hand the page the Escape it spends on leaving fullscreen (checked with a real
keypress: the first Esc leaves fullscreen, the second closes the player). The guard defended a case
that does not happen, under a comment saying it does. This reverts 121ebeeb.
2026-10-07 23:59:32 +02:00
peter 121ebeebc1 fix(ui): an Escape while a dialog's own video is fullscreen does not close the dialog
The page gets the Escape keydown with the video still the fullscreen element. When that element is
the dialog's own content (the library player), closing the dialog took the player away with the
fullscreen. A dialog opened over something else that is fullscreen still closes.
2026-10-07 23:57:04 +02:00
peter ee5290ba01 refactor(plex): the item progress body extends the shared checkpoint
ItemProgressIn repeated ProgressIn's config, fields and bounds; it now adds only `final`. The
threshold comment points where the constants live now, and plexProgressBeacon's comment is back
on it.
2026-10-07 23:16:38 +02:00
peter dd6a666ad1 test(player): hold the session stop and the kept-point rule to the server
- The client's resume thresholds equal the server's (read from both sources).
- The session stop's URL matches the route the server serves it on.
- The player stops its server session by its item and its own id on an unload and on close,
  and keeps it when the page only goes into the back/forward cache.
2026-10-07 23:16:38 +02:00
peter 392099770a fix(plex): an episode Plex has no number for is named by its title alone
The OS media title printed the missing number as "Enull", the player and info headers as a bare
"E". One helper for the three lines.
2026-10-07 23:16:38 +02:00
peter 2aa3ac6e85 fix(player): the browser Back peels each layer over the Plex video, one press at a time
All the layers shared one history entry, so the first Back spent it on one layer and the second
left the player with the stats readout (or the countdown) still up. Each layer now holds its own
entry; Back closes the one opened last.
2026-10-07 23:16:38 +02:00
peter 891dc628f3 fix(library): a listing poll can no longer put back an older resume point
A poll already out when the player closed answered with the point the server held before the
closing beacon, over the row's patch; Continue then opened there and wrote it back. The close now
cancels the listings' fetches before patching and reads them again once the beacon has landed.
2026-10-07 23:16:25 +02:00
peter 0a35f2a3b1 fix(player): a stale session answer can no longer stop the playback that won
- The engine tracks the request out per item: A, then B, then back to A sent a second request
  for A beside the one still out.
- A hardware transcode superseded during its fallback answered with the NEWER session; the losing
  request's player then stopped it as stale. It now answers with its own (dead) session.
- A replaced hls.js instance's late manifest no longer marks the player ready or plays the
  outgoing media.
- A pause given after the answer stands: the manifest and the first canplay read the live intent.
- The debounced A/V-offset restart is dropped on an item change instead of restarting the
  previous item under the next one's title.
2026-10-07 23:16:25 +02:00
peter d25a132495 fix(ui): a dialog whose own video goes fullscreen no longer crashes the app
The portal host followed the fullscreen element into it. When that element is the dialog's own
content (the library player's <video>, by its f key or its own button), the move is into the host's
own descendant, which the DOM refuses: HierarchyRequestError in a layout effect, and the crash
screen replaced the app. The host now stays where it is then — the content already shows.
2026-10-07 23:16:25 +02:00
peter 61d3a1a189 chore(db): drop plex_items.markers, a column nothing writes or reads
Markers are read live from Plex for the item the player opens; the cached copy was never filled.
2026-10-07 02:23:37 +02:00
peter d482a5e8e2 chore(dev): the dev proxy answers a failed connect with 502, as the prod proxy does
Vite's own answer was a bodiless 500, which the app reads as a real server fault and shows the blocking error dialog; a 502 is the passing connection loss it is.
2026-10-07 02:23:37 +02:00
peter 2064373ac2 fix(history): a step back never leaves the app for an older page
Chrome keeps 50 entries a tab, and when the list is full a new entry drops the
oldest one the page left without a user gesture, which can be the page's own
entry under a sub-view or an overlay. A programmatic back() then left the
document; once it landed on an older entry whose restored Plex player started
playing on its own. Every programmatic back() now asks the Navigation API first
and, with no entry of this document below, settles in place: the sub-view turns
to its root, an overlay's entry drops its mark, the channel page and the YouTube
search clear.
2026-10-07 02:23:37 +02:00
peter 0cda5b6dd4 fix(player): only the credits that end an item move on to the next one
An item can carry several credits blocks with mid- and post-credit scenes between
them, and Plex flags the one that really ends it. The player left at the first
block. Now the Plex-final block moves on (by the auto-skip or the Skip button), an
earlier block is only skipped to its end by the button, and with no final flag the
last block ends the item by the button alone.
2026-10-07 01:48:10 +02:00
peter f3c94641a7 fix(player): a library item played from the start keeps its point
A cached file that autoplays from 0:00 fires its one playing event before the resume hook
subscribes, so the hook never counted it as played and reported nothing - not on pause, not on
close. An element already playing when the hook subscribes now counts as played.
2026-10-07 00:36:28 +02:00
peter 59b27467ab feat(player): the OS media controls, the basic keys and resume for the library and shared-link players
Plex, library and /watch tell the operating system what plays (title, line, picture) and take its
play / pause / skip / seek through one hook (lib/mediaSession); the Plex player drives it with its own
seek and clock, so the lock screen shows the film's position, not the stream's. The YouTube player is
left alone: its iframe already publishes all of it, and the OS routes to the frame that plays.

The library and /watch players get the basic keys (lib/mediaKeys, the keymap's mediaPlayer rows) and
open where the viewer stopped (lib/resume): the library keeps the point on the server, per user and
library item (library_states, POST /api/downloads/{id}/progress, listed back on both listings), the
shared link in this browser. A started library row continues, its menu can start it over, and its
picture carries the cards' progress bar. The resume rule (too early / near the end = from the start)
is one function shared with the YouTube route.

The library dialog opens focused on its panel (Modal initialFocus "player"): on the ✕, Space closed
the film. While the media element itself has the focus, Space and the arrows stay its own controls'
— the browser acts on them whatever a listener does, and one Space played and paused.
2026-10-06 23:55:41 +02:00
peter 02feaa9a1c refactor(player): one panel at a time over the Plex video, and one Back cascade
The ⋯ menu, the tracks menu, the tuning panel, the info overlay and the up-next card were a flag
each, so two could stand on top of each other, and Back/Escape were written out three times. They
are one `panel` value now; the stats readout stays a separate toggle beside it. One `back()` peels
the countdown, the panel, then the stats, then leaves — for the arrow, ⌫, Escape and the mouse
Back alike. The mouse Back now also closes the ⋯ menu instead of leaving the player.
2026-10-06 23:04:35 +02:00
peter c1058f1d5a refactor(player): the Plex session machine is a tested engine, one request per item in flight
PlexPlayer's session logic (generation, pending play intent, clock ownership, session offset, reclaim
budget, listener teardown) moves into lib/hlsEngine.ts, a plain class with hls.js behind lib/hlsPort.ts
and its own unit tests.

- A seek, track change or reclaim asked for while a session request is out replaces the queued
  target; it is sent once the in-flight answer settles. A burst of 20 presses past the buffer was 20
  requests and 20 ffmpeg spawns; it is now 2-3, and plays again ~2 s sooner.
- The player stops its server session when it leaves the item, closes or the page unloads
  (POST /api/plex/stream/{rk}/session/{sid}/stop, identity-aware). A left transcode used to run to
  the end of the film first.
- A restart asked for after a new session attached but before it played no longer reads the emptied
  media as the viewer's pause.
2026-10-06 22:31:48 +02:00
peter 23f16b69df release: 0.101.0 (improvement/e36-delivery) 2026-10-06 21:14:35 +02:00
peter c813814c3c chore(release): 0.101.0 2026-10-06 21:04:19 +02:00
peter ed6facb703 fix(downloads): a conversion's commit, a stopped upload, the fast join and a clip's copy hold up
- The fast join gives each ffconcat entry its own media-only format whitelist, so a library file
  can no longer be opened as a nested concat list (the list's whitelist was handed down to it).
- A file the guarded probe cannot read is a download-only file, not a guess from yt-dlp's info.
- A conversion's or an ingest's commit that raises is re-read in a fresh session before any file
  is removed: a commit that landed keeps the file its row now names.
- An original that cannot be removed after conversion is logged; the upload stays ready.
- An upload whose conversion is cancelled or deleted midway is freed once nothing holds it; a
  paused one is kept.
- A shared link says "preparing" only while its video's job is queued or running.
- The admin storage view counts every file on disk ("Files on disk"), not only ready ones.
- A clip's failure title, unknown failure and cancel dialog, and a clip's and an upload's delete
  dialog speak of what the job made; an upload's quota refusal explains that the original is kept.
- /watch announces "preparing" as an alert and moves focus to the new Retry after an action.
2026-10-06 21:02:07 +02:00
peter d44d8d6dda fix(gate): knip sees the watch entry, the build must emit watch.html, and two hand copies are held
- The raw-source contract tests read files from disk (rawSources.testkit)
  instead of import.meta.glob, which knip counted as importing every src file —
  its unused-files lane could never name an orphan. knip now names
  src/watchMain.tsx as an entry (registered in docs/gates.md).
- vite build fails when index.html or watch.html is not emitted.
- The watch-entry import walk follows dynamic import() too.
- audit_metrics.py's plan-code patterns are held equal to the guard's.
- CrashBoundary is domain-free: it moves into components/ui.
2026-10-06 03:19:33 +02:00
peter b20e9a3947 fix(player): the stacked layout's minimum width no longer clips the wide one
The 36rem floor guards the stacked layout's width/height feedback loop; the
wide side-panel card has no such loop, and on a short wide window the floor
made the video taller than the card, cutting off YouTube's control bar. The
floor now applies to the stacked layout only; e2e covers a 1400x380 window.
2026-10-06 03:19:33 +02:00
peter 6073a006b8 fix(downloads): an upload's cancel, progress, failure and notice speak of an upload
- Cancelling an upload asks "Delete upload?" and says the uploaded file is
  removed from the server with it.
- The phone-safe conversion's real ffmpeg % shows as a bar ("Converting for
  phones"), not an indeterminate pulse.
- A failed upload is titled "The upload isn't ready", with its own reasons
  (convert_failed, retryable; upload_missing) instead of download copy.
- The inbox says "Upload ready to share" for a converted upload.
2026-10-06 03:19:32 +02:00
peter dba4758c59 fix(watch): a link whose video is being made again says so, in the page's language
The public watch route answers video_preparing for an asset that is pending or
downloading, and the page shows "being prepared — try again" with a Retry
instead of "this link is gone". watch.html ships one lang; the page now sets
documentElement.lang to the language its strings were picked in.
2026-10-06 03:19:32 +02:00
peter 68cfc7e25d fix(downloads): an upload's conversion is guarded, counted, named and cleaned up
- Every ffmpeg/ffprobe over a library file names the demuxers it may use
  (edit.INPUT_GUARD; a downloaded thumbnail edit.THUMB_GUARD): a playlist-type
  demuxer would read local paths named inside a user's upload.
- A re-encode ends on an even frame (edit.EVEN_FRAME): an odd 4:4:4/RGB source
  failed its conversion; a clip stores the frame its file has.
- An upload counts against its owner's cap from the moment it lands (queued, or
  kept after a failed conversion) — the footprint, the GC's cap and the admin
  view read one ON_DISK predicate — and a conversion that outgrows the cap is
  refused, keeping the original.
- The converted name keeps _a{id} (a dotted bare name once lost it and two
  uploads shared a file); a converted file placed but not committed is removed,
  and the orphan sweep enters .uploads; the LRU cap never evicts an upload.
- Ingest judges the extension the file is stored under; a failed conversion is
  convert_failed / upload_missing, reports the "converting" phase, and notifies
  as an upload. Job statuses in routes/downloads use JobStatus.
2026-10-06 03:19:32 +02:00
peter 00886d92fe fix(player): the YouTube player keeps its size on a short window and when maximised
Two layout bugs since the player grew a side panel, both reported from prod.

Maximised (F) on a wide window, the video was a thin line across the screen: the side-panel layout
aligns the stage `self-start`, and Chrome applies a flex item's self-alignment to an absolutely
positioned box too, so the fixed stage took its content's height — 0 px. The maximised stage now
stretches between its insets whatever the layout asks of it.

On a short window under 1280 px (a laptop at 100%, a 1080p screen at 175%) only the side strips were
left: the card narrows so the chrome under the video fits, but that chrome grows as the card narrows
(its text wraps), and the loop ran the card down to 2 px. The video now never goes narrower than
36rem to make room; past that the card scrolls.

Measured before/after (localdev, real YouTube iframe): 1097×520 open 0×0 → 610×343; 1920×911 maximised
1920×0 → 1920×911; 1440×760 and 1366×650 maximised fill the window; leaving maximise restores the card.
Held by an e2e test at both sizes (red on the old code: 0 px wide, and 911 ≠ 0), green on Chrome,
WebKit and iPad.
2026-10-05 23:40:13 +02:00
peter e7d2eb8cf6 test(downloads): what a share link serves passes the phone-safe rule
A link serves an asset once it is ready, so the contract is pinned on both sides. The code: every
function in app/ that marks something ready must run the rule (`_ensure_phone_safe`, or the
upload's own `playable.plan`) — an AST scan names any that does not, and fails blind if it stops
finding the four paths known today (download, editor clip, upload conversion, upload kept as is).
The bytes: the files an iPhone + iPad refused (HEVC tagged hev1, AV1, Matroska, MP3 inside an
.mp4) and the ones other browsers cannot be trusted with (VP9 + Opus, 10-bit H.264, Opus in Ogg)
go through the upload route, the worker's conversion and the public file route; what the link
hands out must be served as mp4 / m4a / mp3 and pass the rule.

Seen red: a planted ready-marking function in service.py is named; with the conversion step
short-circuited, all seven files fail (on the rule, or on being served as quicktime / webm / ogg).
2026-10-05 21:05:22 +02:00
peter e886890d6e perf(watch): a shared link loads the player, not the app
A /watch link is opened by whoever it was sent to — signed out, often on a phone on mobile data —
and it booted through main.tsx: every app provider, the app's i18n (every locale of every area),
the flag sprites, ~392 KB gzip before the player could start. It now has its own HTML entry,
watch.html → watchMain.tsx, which imports WatchPage and what it renders; the server hands every
/watch/ path that page (the OG block injected as before), and vite dev does the same.

Measured on the build: /watch JS 290 → 87 KB gzip, CSS 104 → 19 KB; the app's own bytes unchanged.
Two leaks had to go for that: lib/format imported the app's i18n for one "just now" string (the
relative-time formatters now take `t`, like the file's other t-formatters), and the boundary's
crash copy came from the same i18n (CrashBoundary takes its copy; ErrorBoundary is the app's
wrapper, the watch entry gives it the page's own strings). The page's popover tree went too: it
renders no popover.

Held by: a contract test walking the entry's runtime imports, naming the path to anything only the
app may load (red on a planted i18n import in lib/format), and holding watch.html's head equal to
index.html's; a DB-lane test that every /watch/ path gets the player's page with the link's OG tags
and app paths get the app (red on the old routing). Screenshots at 375 px, EN and HU, video / audio /
password / gone / wrong password / unlocked: identical to the old entry's, bar the video's buffered bar.
2026-10-05 20:51:39 +02:00
peter 5561d9a379 feat(downloads): every ready file is phone-safe — uploads and clips too
A share link is opened on whatever the friend holds. Measured on an iPhone and an iPad: AV1,
Matroska, HEVC tagged hev1 and MP3 inside an .mp4 fail there; Chrome and Firefox add HEVC and
10-bit / 4:4:4 H.264. The old check judged the codec NAME only and ran on downloads only, so a
10-bit H.264 passed, an upload went out exactly as it came (an iPhone's HEVC .mov, a webm, an mkv)
and a fast trim kept its source's codecs.

One rule now, `downloads/playable.py`: video is H.264 8-bit 4:2:0 + AAC in .mp4, audio-only is
.mp3 or AAC .m4a; `plan` says what must change, `build_cmd` re-encodes only the failing stream (or
remuxes). The worker applies it to a download, an editor clip and — new — an upload: media that
fails the rule is queued (asset pending, job queued) and the worker converts it, keeping only the
converted file (owner's choice). Until then the original sits at rel_path, so Retry starts from it
and a delete removes it; an upload that lost its last holder mid-conversion is reaped by the GC.
Every libx264 run (edit, concat, conversion) now pins `-pix_fmt yuv420p` — without it a 10-bit
source came out High 10. The download cache signature moves to v5 so a cached 10-bit asset
re-derives. The upload toast says when the file is still converting (HU/EN).

Tests: the plan for every measured class; real ffmpeg turning a 10-bit Matroska + Opus and an
mp3-in-mp4 into files the rule accepts (red without the pixel-format pin); the queued upload's
delete, cancel-then-GC, failed conversion and real worker conversion (the cleanup tests red
without their rules); the toast (red without the branch). Live on localdev: 9 fixture uploads
converted, every served file probed H.264 High yuv420p + AAC, or mp3 / m4a.
2026-10-05 03:16:27 +02:00
peter 2768ab5bdd fix(gate): the plan-code guard catches decision codes, and the purge takes them
A plan's decision ids (`D1`, `D4`, `D6-A`) cited in 11 comments and test titles named a document
no reader of this repo has, and the guard had no pattern for them. It has now (mirrored in
audit_metrics), with cases for each form the citations took and for the look-alikes it must leave
alone (a JPEG marker, "3D"); the 11 lines say their why without the id. Case test seen red without
the pattern, the scan red on a planted `D9:`.
2026-10-05 03:16:13 +02:00
peter 8c79590165 release: 0.100.0 (improvement/e35-tooling) 2026-10-05 01:42:39 +02:00
peter eef3ac9c5c chore(release): 0.100.0 2026-10-05 01:19:51 +02:00
peter 4ba7c966d7 fix(nav): the dead-page rewrite does not restore a spent focus stamp
It spread the event's snapshot, which still carried the sfNav stamp lib/history's own listener had
just taken off the entry, so a later landing there (an overlay closing) refocused a stale item. It
now spreads the live entry (test red on the snapshot). Also: format.test's comment named a test
image that never runs vitest; it is the gate host's locale.
2026-10-05 01:13:06 +02:00
peter 513dd8175a fix(auth): becoming admin through ADMIN_EMAILS drops the password, on any sign-in
The promotion revoke guessed from today's SMTP state whether the password was trustworthy, and ran
only when the account was adopted. Both guesses failed: a row verified while SMTP was off stays
"verified" after SMTP is configured, and a later sign-in matched by Google identity promoted the
account with no revoke at all. Either way a stranger's password and session could become admin.

Now one rule, where the promotion happens: when ADMIN_EMAILS turns a non-admin account into an
admin and it holds a password, the password goes and every session ends; the owner, signed in
through Google at that moment, sets a new one in Settings. An account that already is admin keeps
its password. Tests: the activated squat, the SMTP case, the later sign-in (all three red without
the rule) and an admin's own sign-in.
2026-10-05 01:13:06 +02:00
peter c4298b0a73 docs(comments): no WebSocket and no CI in the present tense
The chat cut removed the only WebSocket and CI went before it, but comments still leaned on both:
`?account=` was justified by the socket, job polling by an in-process WS registry, the e2e flows as
"proxy-safe (none use the WebSocket)", and pyrightconfig named a CI container the engine contract
test never checked. Each now says what holds today (the wallet gate, no push channel, the two
contract test halves, the image build's `npm ci`, the gate).
2026-10-05 00:50:36 +02:00
peter 5ca2a0e6ee fix(gate): the plan-code guard catches bare sprint and phase codes, and the purge takes the P ones
The sprint alternative matched only `(S6)` or `S3e2`, so the commonest purged form (`# S6:`,
`S3's`, `"""S6 —`, `S1 justified`) and every phase code (`P1`) would come back unflagged; the phase
codes had in fact survived the purge in 16 lines (Plex client sections, the upload's header shape),
now reworded around them.

One pattern now covers epic, roadmap, sprint and phase codes; the named exceptions are TV episode
notation, a fixture's season or episode title (`title="S1"`) and ruff's `E4/E7/E9`, and a ruff rule
like `S110` does not match. The case test lists the purged forms and no longer claims every form.
audit_metrics copies the definition (ASCII, as JavaScript's \b is) and reports it as
plan_code_lines / plan_code_files.
2026-10-05 00:50:36 +02:00
peter 5257d57196 fix(nav): Back onto a removed page also rewrites the entry to the feed
The fallback rendered the feed but left the history entry and the address bar on the dead page, so
a Feed click was a no-op (the same-page guard), filter mirroring kept `?page=messages` and the nav
diagnostics logged a page that does not exist. The entry is now replaced with the feed's, keeping
every other URL key (test red without it).
2026-10-05 00:50:35 +02:00
peter 58ad95263c fix(auth): a Google sign-in that makes an account admin drops a password nobody proved
The never-activated rule left one order of events open: on an instance without SMTP, a stranger
who registered an ADMIN_EMAILS address (or a sibling spelling) and was then activated by another
admin kept their password and live session, and the owner's first Google sign-in raised that row
to admin, the stranger's cookie with it. Activation vouches for access, not for who chose the
password, and without SMTP nothing else ever proved the mailbox.

When ADMIN_EMAILS promotes an adopted account that holds a password on an instance without SMTP,
the password goes and the session epoch is bumped. With SMTP the verified flag is a proven mailbox,
and an account the sign-in does not promote keeps its password, so the "your password was cleared"
detour stays off the default path. Tests: the activated squat (red without the rule; the squatter's
session is signed out) and the SMTP case that keeps the password.
2026-10-05 00:50:35 +02:00
peter 96ad1ef306 docs(comments): correct comments the cuts left stale
The registration language is read today (preferences.language, the mail language); the Modal and
the players order against popovers through layerStack, not the FloatingTree; the upload's auth
dependency is current_user; the module of access tests covers onboarding and access, not only the
canonical email. The dev proxy's WebSocket upgrade served the removed messages socket and goes; the
test compose header says a pure-logic run needs SIFTLODE_TEST_REQUIRE_DB=0; the speed measure no
longer reports an unreachable database as "no share link".
2026-10-05 00:19:50 +02:00
peter d1bbe4524c fix(gate): close the gaps in the plan-code guard, the button lists and the tooling
- planCodes: the quote and slash lookarounds let the commonest form through (a docstring or test
  title opening with a code, a slash pair); the two real exceptions (an episode titled "E1", ruff's
  E4/E7/E9) are now named and cut first, with cases that fail on the old pattern. audit_metrics
  counts by the same definition.
- button-primitive: both file lists live in the rule body, out of sight of the snapshot and the
  register; each is now pinned to its exact set of ui/ files (red when either is widened).
- check.sh --brief shows a short log whole and keeps the line before each hit, so ruff's rule and
  eslint's file path survive the excerpt.
- lint-changed runs prettier over every changed frontend file, as the gate does, not only code.
- CLAUDE.md and gates.md no longer claim the basedpyright baseline was found harmless: it was never
  triaged, and the reportOptional* entries are the likeliest real bugs.
2026-10-05 00:19:50 +02:00
peter eecc195b7e fix(migrations): dropping the demo also takes its reset job's interval-change audit rows
0098 deleted the reset job's run rows but not the scheduler_interval rows that changing its interval
wrote; with the job's label gone they rendered as a raw id. The delete now takes every job-targeted
row for it, and a test seeds both kinds below 0098 and upgrades (red on the old filter). The test
has its own file so the removed-features guard can name it.
2026-10-05 00:19:50 +02:00
peter 2f901ad643 fix(nav): Back onto a page this build no longer has lands on the feed
A history entry pushed while the Messages page existed still names it after a reload onto the new
build; popstate took sfPage unchecked and the shell's module lookup threw. It is now validated with
isPage, like every initial-load path already was.
2026-10-05 00:19:50 +02:00
peter 1f357b0c21 fix(auth): Google adoption revokes the password of a never-activated account
Without SMTP, registration marks the address verified with no proof, and an ADMIN_EMAILS /
ALLOWED_EMAILS address gets no invite row to gate it. So a stranger who registered such an address
(or a sibling spelling) before its owner kept their own password when the owner's first Google
sign-in adopted the row, activated it and made it admin, and could then sign in with it.

The adoption now treats a never-activated row like an unverified one: the password goes,
password_revoked_at is stamped and the session epoch bumped. An account already let in keeps its
password. The helper is renamed to say what it now judges (revoke_untrusted_preset_password).

test_google_callback.py drives the callback's account-matching branches through the real route,
none of which a test ran before: the squat (red on the old rule), adoption by mailbox, the
sub-collision and unverified-address refusals, the refused sign-in recorded in Google's locale, and
the link flow's conflict, mismatch and attach answers.
2026-10-05 00:19:50 +02:00
peter fb962682eb chore(blame): skip the plan-code purge in git blame
It touched 671 files' comments and nothing else; blame should keep naming the commit that wrote each line.
2026-10-05 00:19:50 +02:00
peter 4e71723a07 test(comments): no plan code lands in the code again
planCodes.contract.test.ts scans every tracked text file and every new one not yet added (comments,
docstrings, test titles, lint messages alike) for an epic or roadmap code (`E28`, `E9.5`, `R14`) and
for a sprint or review round cited on its own (`(S6)`, `S3e2`, `round 4`, `deep-review`). Not
`E2E`, not a ruff rule family (`E4/E7/E9`), not a TV episode (`S2 · E5`, a test episode titled
"E1"), not `S3h` (the media-card decision docs/DESIGN.md is headed by). Markdown and the Alembic
migrations keep their history and are not scanned.

One sweep test rather than an ESLint rule plus a ruff one: the codes sat in Python, TS, CSS, shell,
TOML, YAML and lint messages, and no single linter reads all of them. Seen red on a planted
backend/app file (all three plan-code lines named, its allowed line not).

audit_metrics.py counts epic codes by the same definition (it used to miss `E9.5` and `R6`), and
CLAUDE.md's "comment the why" rule names the test.
2026-10-05 00:19:39 +02:00
peter 4aaaaa7860 chore(comments): drop epic, sprint and review-round ids from the code
The code cited its plan instead of saying why: `E28 review J4`, `(R6 S3b)`, `S1 justified…`,
`deep-review round 5`. That plan lives in the private ops repo, so for every reader of this one the
citation was noise, and a sentence whose subject was the plan step ("round 2 showed…") hid its
lesson behind an id. Each one is gone and the "why" kept: a bare citation is deleted, a sentence
carrying a lesson is reworded around it ("a later attempt showed…").

Measured (`scripts/audit_metrics.py`, product code): 652 lines in 270 files carried an epic code by
the definition the next commit holds (507 / 241 by the old, two-digit-only one) -> 0. Across all
code, tests, scripts and configs: ~1100 epic-code lines plus ~320 sprint/round ids.

Behaviour unchanged, checked mechanically: every changed .py file has the same AST once docstrings
are blanked; css/sh/toml/yml/json files are identical once comments are stripped; the only
non-comment changes in TS/JS are test titles, one description string in a contract table, and five
ESLint messages, whose text is all the gates snapshot diff holds. Not touched: Markdown (docs and
the changelog keep their history), Alembic migrations, TV episode notation (`S2 · E5`), DESIGN.md
ids (`C1`, `B4`, `S3h`), tracker ids (`SEC-16`, `DX-17`).

One stale claim fixed on the way: test_hls_reaper's docstring said the player checkpoints "every
10 s whether or not it is playing", which plex/stream.py calls wrong; it now says what the code does.
2026-10-05 00:19:39 +02:00
peter c73d8be094 docs(welcome): retake the landing screenshots on today's UI
The seven gallery shots dated from 2026-07 and still showed the old
labelled sidebar with the removed Messages item. Retaken at the same size
(1200x675 @2x, dark) from the localdev stack with the E2E account (no
personal account name on screen), filled for the shoot with a copy of a
real account's subscriptions, playlists and finished downloads, removed
again afterwards.
2026-10-05 00:19:39 +02:00
peter 30545d0d88 fix(gate): --brief leaves the test database's own log out of an excerpt
The DB lane provokes constraint errors on purpose, and their ERROR: lines
from the testdb container came first in a red pytest excerpt, ahead of the
failing test (seen in E35 S3). The excerpt now skips the testdb-N lines.
2026-10-05 00:19:39 +02:00
peter ac702192c9 docs(claude): drop the canonical-pattern rows a lint rule already holds
Buttons, text fields and native controls, glass surfaces, the native
<select> and network calls are each refused by a lint rule whose message
names the primitive to use (button-primitive, input-primitive,
surface-role, no-restricted-syntax, no-restricted-globals). Their table
rows repeated that, so a change had to be made twice. One paragraph now
says they are lint-held and lists the primitives. The rows nothing
mechanical holds stay, as do ORM and current time, now test-held.
2026-10-05 00:19:39 +02:00
peter 24c47d12bd refactor(backend): select() and an inline now everywhere, held by a test
CLAUDE.md's canonical table names SQLAlchemy 2.0 select() and an inline
datetime.now(timezone.utc), and both had drifted: five db.query(...) calls
remained in routes/auth.py and routes/me.py (seven before the demo went),
and four private now helpers stood in sync/videos.py, utils.py (imported
as _now by explore and maintenance), scheduler.py (_now_iso) and the e2e
seed. All are the canonical form now.

Ruff cannot ban a method call on an instance or a function by name, so
tests/test_canonical_patterns.py reads every app module's AST and fails on
a .query(...) call or a now/_now/now_utc helper — seen red on a planted file.
2026-10-05 00:19:39 +02:00
peter ab76166878 docs(gates): basedpyright keeps new-errors-only, without a burn-down mandate
E34 read the standard-mode findings and none was a real bug, so working the
baseline down as work of its own buys nothing. The lane is unchanged: it
fails on a new error, and on a run that tightened the baseline without the
commit. CLAUDE.md and the register row say so.
2026-10-05 00:19:39 +02:00
peter a42f346a32 test(gates): the snapshot holds a config per scope, not a path per file
gates.snapshot.json listed ~800 file paths under their config, so every
new or deleted frontend file regenerated it (31% of commits, E34 S2) and
the regen diff buried the gate changes it exists to show. Now a scope is
named by path (scopeOf: src, src tests, ui, lib tests, …), the snapshot
holds one effective config per scope (2345 → 1456 lines), and a new test
fails when a file's config is not its scope's.

Not narrowed: a new file in a known place changes nothing (it gets its
scope's config, checked), a glob that moves one file into another config
fails (seen: an override on Feed.tsx → "src: src/components/Feed.tsx"), a
scope whose rules change is a snapshot diff, and a file no config lints
is still a register miss.
2026-10-05 00:19:39 +02:00
peter f2fe0fc0df build(gate)!: coverage leaves the gate and runs on demand
Both lanes printed a coverage total with no threshold (owner decision E32
D4). No real hit as a lane: the one time the number found anything (5 bugs,
E32) it was read by hand, and since then it was a line nobody acted on. It
was also the most expensive part of the vitest lane, and the run that
flaked in E35 S1. The vitest lane runs without --coverage, the pytest lane
runs plain pytest; docs/development.md gives both on-demand commands, and
the register row says why.
2026-10-05 00:19:39 +02:00
peter ca707fd390 build(eslint): focus-safe-disabled folds into button-primitive
Two rules looked at the same <button>: button-primitive refused a raw one
outside the control primitives, focus-safe-disabled refused a raw
`disabled` one everywhere but FocusSafeButton and AnchoredPopover. They are
one rule now, with the same reach: outside the primitives a raw button is
refused (a disabled one with the focus message); inside a primitive and in
a test only a raw `<button disabled>` is, and the two native-disabled owners
are named in the rule rather than in a config override. One override block,
one rule entry and two register rows go; the snapshot loses a config group.
focusSafe.contract.test.ts pins every case, the primitive and test ones new.
2026-10-05 00:19:39 +02:00
peter 2eb20ca227 build(eslint): one px-frozen box selector for box utilities and grid minimums
BOX_PX and GRID_MIN_PX were two esquery patterns with two messages for one
rule (DESIGN.md: a px box does not scale with --font-scale). They are one
alternation now, with one message that names both fixes. Nothing it refused
before passes now; pxBox.contract.test.ts pins both shapes unchanged.
2026-10-05 00:19:38 +02:00
peter adaa7dc23c build(eslint): drop the Z_RAW and EDGED_LABEL selectors
Both came out of the E28 claim audit (0521149c named z-levels, e13d93a3
static labels as flat Badges) after the audit itself had fixed the sites.
No real hit since: 0 matches in every lint run from E28 to E34 (S2/S3
measurement). Each still cost a probe test (zLayer.contract, the edged-label
half of hoverAndLink.contract) and a place in the gate snapshot. The design
rules stay in DESIGN.md (Tokens: no ad-hoc z-index; Static vs pressable),
with a decision-log row saying the selectors went and why.
2026-10-05 00:19:38 +02:00
peter 91f6a77fa3 build(eslint): drop the three React Compiler rules that cannot fire on 7.1.1
react-hooks/exhaustive-effect-dependencies, memoized-effect-dependencies and
rule-suppression were switched on in E32 "for the plugin version that wires
them up". Read in the plugin source and probed (E32 review): the first's
validation is compiled off, nothing emits the second's category, and lint
mode hands the third no suppression rules. 0 hits by construction — a rule
that cannot report reads as coverage it does not give. The comment keeps
their names and why, so a plugin upgrade can re-check them.
2026-10-05 00:19:38 +02:00
peter 36653886d5 build(frontend)!: knip drops --production; plain knip stays
The production pass (7ed2ac4e, E32) asked "is this export used by what
ships?". No real hit: its only findings were 37 exports a test imports,
answered with 36 `@internal` JSDoc tags (E34 S2) — test-only exports, not
defects. Since then it has only ever asked for another tag. The tags go
with it (36, no longer read by anything), and so do the production-only
`!…!` exclusions in knip.jsonc and their register rows. Plain knip (unused
files, exports and dependencies) keeps running in the gate.
2026-10-05 00:19:38 +02:00
peter 7e681e112e test(gates)!: the size/complexity ratchet leaves the gate; audit_metrics reports it
sourceSize.contract.test.ts, its test kit and backend/tests/test_source_size.py
held 13 FE / 9 BE file ceilings and 64 FE / 32 BE function ceilings. No real
hit (E34 S2/S4): its history is its own birth (751e1ae5), two rounds of
fixes to its own logic (e432dc27, d1df76e8), the snapshot move (182ec311)
and ceiling edits inside feature fixes (5806adda, df4a79a1, and the S2/S3
cuts here). It caught no defect, went red 8 times on its own limits and drew
5 review findings about itself. The numbers stay visible: audit_metrics
reports file size and complexity, and an epic compares before and after.
docs/gates.md: the two size rows go; the C901 row says where it is read now.
2026-10-05 00:19:38 +02:00
peter 3642397f63 fix(tooling): audit_metrics counts the demo footprint by the banned word
Its pattern also matched "demote" and the Hungarian "Demokratikus", so
the footprint read 5 after the demo was gone (and 56, not 53, in the S1
baseline). It now uses the removedFeatures contract test's pattern.
2026-10-05 00:19:38 +02:00
peter 6d8b7be6d8 feat!: remove the shared demo account
The demo account — one shared user that whitelisted emails entered from
the login page, reset on a schedule — goes end to end (owner decision,
E34): demo.py and the reset job, users.is_demo and demo_whitelist
(migration 0098 deletes the demo user, its audit rows, the reset job's
override and notices, then drops the column and table; the downgrade
restores the empty schema), the /auth/demo entry, the admin whitelist and
reset, the Demo tab, the banner, the login page's "Try the demo", and
every is_demo branch in routes, userscope, scheduler and the UI.

require_human only ever refused the demo account, so its 87 uses become
current_user (owner-approved); require_write_scope keeps its own check.
The login page's "or" divider now shows only beside the Google button:
without the demo entry under it, an instance without Google would have
said "or" to nothing.

A contract test (removedFeatures) fails on the word anywhere in the
backend, frontend, e2e or scripts — seen red on a planted hit. Size
ceilings tightened for the modules that shrank; admin.py is back under
the limit.
2026-10-05 00:19:38 +02:00
peter cca3315836 feat!: remove direct messaging (chat)
E2EE direct messages, the chat dock, the Messages module, the message
WebSocket and its realtime registry, and the messages / message_keys
tables (migration 0097; the downgrade restores the empty schema). Prod
held two welcome notes, no message between people and one key — never
used (owner decision, E34). WebSocket now has no owner module: the lint
bans it everywhere. Privacy policy updated: it no longer promises E2EE
for a feature that is gone.
2026-10-05 00:19:38 +02:00
peter 7f08c510bf feat(auth)!: remove the public access-request endpoint
POST /auth/request-access had no caller left: the login page's form was
gone, and registration and a refused Google sign-in already land in the
same admin approval. A third public write path with no protection of
its own (owner decision, E34).

Its one guard moves to where every path passes: upsert_pending_invite no
longer opens a pending row for an address the env already lets in. The
registration path lacked it — registering b.ob@gmail.com beside an
env-granted bob@gmail.com opened a pending row, and is_allowed answers
from the invite rows once any exist, so the real owner was locked out.
The tests that drove the endpoint now drive registration or the shared
function.
2026-10-05 00:19:19 +02:00
peter a499f00209 refactor(compose): localdev is an override of docker-compose.yml, not a copy 2026-10-04 20:09:58 +02:00
peter c2bf8fb7f6 feat(tooling): audit_metrics.py — size, complexity, gate weight and a speed baseline 2026-10-04 20:09:58 +02:00
peter 11d35910fd feat(tooling): lint-changed.sh — fast lint lanes on the uncommitted files, usable as a Stop hook 2026-10-04 20:09:58 +02:00
peter 27f48b7a8b feat(gate): check.sh --brief prints a red lane's first error lines only 2026-10-04 20:09:58 +02:00
peter bc856ab2af test: warm cold dynamic imports before the timed cases
Under a full coverage run on every core, a test file's first cold import
(0.6-0.8 s alone) outran the 5 s case timeout and the half-done import
failed every later case in the file: the vitest lane went red in 2 of 5
runs with no defect. A beforeAll warm-up in the 4 files of that pattern;
5 of 5 full runs green after.
2026-10-04 20:09:58 +02:00
peter 0dfb4a984b release: 0.99.2 (bug/b1-e34-defects) 2026-10-04 18:28:53 +02:00
peter 5bcd1d8d8a chore(release): 0.99.2 2026-10-04 18:19:37 +02:00
peter f49789665e fix(i18n): label the history cleanup's quota_reaped result
The scheduler row read 'quota reaped 0' — the raw key, humanised — because the new result key had no
label (found in the browser check; nothing tests result-key labels).
2026-10-04 18:03:19 +02:00
peter 0e077b84c8 fix(search): the live-search tooltip says what a search costs
It always said a live search spends the shared API quota, while the default source (the website's
own endpoint, scrape) spends none — the owner avoided a free search because of it. /api/me now names
the running search_source, and the tooltip names the cost under it: none for scrape (the enrich step
costs a unit or two), 100 units a page for the official API.
2026-10-04 17:53:19 +02:00
peter ce8ccfc3c0 fix(quota): prune quota history past a retention window
quota_events had no retention: one row per API spend plus a zero-unit row per live search, kept
forever. The daily history cleanup (audit_gc) now also prunes it past quota_retention_days (a new
Quota setting, default 365 so the admin usage view's longest window stays whole; 0 = forever). The
floor of two days keeps today's rows, which the per-user search cap counts.
2026-10-04 17:53:19 +02:00
peter 33dec56e45 fix(worker): obey the stored concurrency and log level, live
The admin page's download_worker_concurrency and log_level were stored but never reached the worker:
it read the env once at boot (worker.py) and never looked again. The worker now re-reads both every
10 s (_sync_knobs): extra loop threads start at once, surplus ones retire after their current job, and
a changed level is applied to the process. An unreadable database keeps what already runs.
2026-10-04 17:53:19 +02:00
peter 2a699e26ba release: 0.99.1 (improvement/drop-ci) 2026-10-04 06:45:48 +02:00
peter 13c8c05661 chore(release): 0.99.1 2026-10-04 06:36:06 +02:00
peter cc414de952 build: the gate is scripts/check.sh, and CI goes
The Gitea Actions gate repeated, after the merge, what a release already runs
before it, and its runner shared the git host's container: one run there
starved Gitea of memory until it stopped answering. The lanes now live in
scripts/check.sh, run the same by the release dispatcher and by hand, and the
gate-contract test holds its lines (the ruff and pip-audit checks read it
instead of the workflow). The two audits only CI ran join the gate: pip-audit
over the lock (pinned in the test image, blocking at any severity) and npm
audit of the production dependencies. The engine contracts drop the CI
containers; the register records that pushes have no server-side gate.
2026-10-04 06:20:58 +02:00
peter 0528b35f2d release: 0.99.0 (feature/e32-gate-integrity)
gate / backend (push) Successful in 8m5s
gate / frontend (push) Successful in 8m32s
2026-10-04 05:19:05 +02:00
peter cb46774851 chore(release): 0.99.0 2026-10-04 05:09:33 +02:00
peter f1cc6ce5dc fix: the reconcile scan's return type names its settled sections; the composer stays at its size ceiling 2026-10-04 05:04:14 +02:00
peter 4efe799eda fix(ui): the issue composer opens on its title, not on the Edit switch
A form dialog opens on the body's first control, and the composer's first
control is the Edit / Preview switch above the form: the focus landed there,
at open and when a retried load brought the form, and typing went nowhere. A
form can now mark the field to open on (data-initial-focus); every other
dialog keeps its first control. The Modal's comment on the re-armed watch now
says where the focus goes when the focused control is removed but others stay.
2026-10-04 04:56:57 +02:00
peter 40b27f2982 fix(sync): a channel with no uploads backfills as done, not as a failure
YouTube answers the uploads list of a channel with no uploads as
playlistNotFound. The recent and deep backfills raised on it, so the channel
was never stamped and was picked again, charged a quota unit and logged an
ERROR on every run. Both now read it as an empty last page, as explore already
does; any other failure still raises.
2026-10-04 04:51:25 +02:00
peter 0314c4cce2 fix(plex): the watch reconcile settles each library on its own walk
A mirrored library that answers "0 items" is not believed (its rows stay), but
that verdict stopped the whole reconcile: one library emptied on Plex with its
mirrored items kept made every run incomplete, and un-watches in every other
library never synced again. Each section now settles on its own walk, and only
the settled sections' rows are cleared; the unsettled ones are named in the
warning.
2026-10-04 04:49:55 +02:00
peter 182ec31120 test(gates): hold the lint and lane configs to a committed snapshot
The register test detected narrowings by modelling config shapes, and each
review round found one more shape it did not model (a basePath global ignore,
a rule src never declares, a widened glob filed under a narrow row, a substring
check on a CI line, knip.json beside knip.jsonc). It now compares instead:
gates.snapshot.json holds ESLint's effective config for every code file and the
lines that run each lane, and any difference fails until it is regenerated.
The register is read off the snapshot's configs, which surfaced the ui/ and
lib/ test scopes the sampling never reached.

The complexity ratchet also names a wrapped function by its binding
(const onSave = useCallback(() => ...)), not as "useCallback callback".
2026-10-04 04:47:10 +02:00
peter 12a39f5291 fix(plex): the watch reconcile clears only what it walked, and doubts a mirrored 0
A library left out of the allowlist is never scanned, so its watch states were cleared on
every reconcile; clearing is now limited to the sections walked. A section we mirror items
from that says it holds 0 is not believed complete. The page size is no longer read as the
section total when totalSize is missing (measured: the server always sends totalSize on a
paged request, 0 for an empty one).
2026-10-04 04:16:52 +02:00
peter 79704b0001 fix(youtube): "this playlist no longer exists" only where the user picked that list
playlistNotFound was mapped to "gone" everywhere: exploring a channel without uploads (its
uploads list answers not found) and a list deleted mid bulk pull both said a playlist no
longer exists. The reason is now "gone" only on single-playlist routes; explore reads it
as an empty back-catalogue, and the bulk pull skips that list and syncs the rest.
2026-10-04 04:16:52 +02:00
peter 4191189b4c fix(ui): a late form takes a dialog's focus once, even if it arrived elsewhere
The watch stayed armed when the form arrived while the focus was off the close button, so a
later change in the body pulled the focus off a close button the user went back to. It now
disarms on arrival and re-arms only when the focus is taken away.
2026-10-04 04:16:52 +02:00
peter 7dbda60815 fix(ui): an undo in the same tick as a rebase steps back to a rebased snapshot
rebase updated the state only, so the mirror the callbacks read still held the removed
row and an undo right after it wrote that row back for a moment.
2026-10-04 04:16:52 +02:00
peter d1df76e8e5 test(gates): the complexity ratchet names a function by what it is bound to
ESLint leaves an arrow bound to a const or a property unnamed, so three ratchet keys were
(anonymous) and ranked by position again. A tiny rule reports each unnamed function's
binding (const, property, assignment, else <callee> callback) at the head complexity
reports, and the keys line up by location.
2026-10-04 04:16:52 +02:00
peter 3f6a9efd57 test(gates): the register test parses ruff's TOML and holds pytest.ini line by line
Round 2 of the E32 review found the key whitelists blind: a line regex saw neither a
[lint.per-file-ignores] table nor a quoted key, and pytest.ini's continuation lines and
key: value form slipped past. Ruff's config is now parsed (smol-toml) and held to its
key paths; pytest.ini to its whole lines; the CI ruff steps to --config backend/ruff.toml,
which is what makes that file the whole configuration.

ESLint: every glob of a block must reach a sample (reactPlugin.ts and devOnlyGuard.ts
rode along unseen), a block's own severity is held to src, a no-restricted-* list is held
to the src block's declaration rather than Feed.tsx's effective config, named global
ignores count, an ignored sample is named, and linterOptions are checked. tsconfig.node
must keep allowJs + checkJs while it lists a .js file; knip's positive glob is pinned.
2026-10-04 04:16:52 +02:00
peter 64e8f1da3d fix(lint): E402 is on — the ignore meant for main.py had let 11 late imports in elsewhere
ruff.toml ignored E402 everywhere for main.py's logging setup. Measured: 28 hits, of which 11
had nothing to do with it — runner.py's logger line between its imports, models.py importing
canonical_email mid-file, a test constant above its imports, and main.py's own third-party and
mimetypes imports. Those are moved up. What stays after configure_logging in main.py is marked
per statement (noqa E402, logging is configured first), its comment now says why the order
matters (a line logged while importing, not the logger's creation). The register row goes, the
ruff key whitelist loses ignore, and a late import is red again. The T20 comment counts the
third deliberate print (scripts/e2e_seed.py).
2026-10-04 03:14:24 +02:00
peter a1224e9a8c test(gates): the repo-root scripts are linted and format-checked
scripts/check-node-modules.mjs and gen-changelog.mjs run in check and release but sat outside
every lane: the eslint and prettier lanes run from frontend/, and an ESLint config cannot reach
past its own directory. A contract test now holds them to the js-configs bar (recommended rules
under Node, the frontend's Prettier style); an unused variable and the old formatting are each
red. 0 lint hits; both files reformatted. Registered in docs/gates.md.
2026-10-04 03:00:49 +02:00
peter da4b19a39a docs(lint): the config comments say which rules can report, and count what is suppressed
eslint.config.js called every zero-hit React Compiler diagnostic a tripwire. Under
eslint-plugin-react-hooks 7.1.1 three cannot report anything: exhaustive-effect-dependencies (its
validation is compiled off), memoized-effect-dependencies (nothing emits its category) and
rule-suppression (lint mode passes no suppression rules). Probed: a missing effect dependency is
reported by exhaustive-deps only, and an inline disable is named by nothing. The header's
"33 suppressions" of exhaustive-deps is 0. reactPlugin.ts said one react-hooks rule was still
off; none is (three run at warn, which --max-warnings 0 fails).
2026-10-04 03:00:49 +02:00
peter 5a74cf86b7 test(api): the wire-key checks send every filter field, and tsc says when one is new
The Plex and feed filter fixtures were typed as the filter type itself, whose scalar filters are
optional: a new optional field compiled without a test value, so its wire key went unchecked.
Both are Required<…> now (a probe field fails typecheck on each). The feed set gained the fields
it omitted (year, librarySource, the scope and show switches) and goes out with and without year,
which wins over the other date filters; a misspelt year key is now red in the wire check, which
passed it before.
2026-10-04 02:57:09 +02:00
peter e432dc2780 test(gates): the complexity ratchet holds each function by name, and its scope is registered
The ceilings were a per-file ranked list, so one function could grow by what another in the same
file lost: Home.tsx's [34, 32] passed with Home at 32 and Hero at 34. Both twins now key every
offender as module::function (an unnamed one is (anonymous); a repeated name gets 2, 3 in source
order) and re-baseline at today's measured values. Swapping two ceilings is red on both sides
(Home 34 > 32; feedquery filtered_query 28 > 14).

The ratchet measures shipped code only. docs/gates.md said C901 covered backend + scripts; it
covers backend/app. The exclusion of tests, test kits, e2e and scripts/ is now a row with its
measured hits, and the stale counts are replaced (9 BE modules over 800, not 11).
2026-10-04 02:55:07 +02:00
peter 3741d1a46d fix(playlists): a list reopened from the cache keeps its rows
Leaving the Playlists page and coming back painted the selected list empty ("0 videos") while
the rail counted its rows. The page re-seeds its undoable order from the cached list in an
effect; StrictMode runs that effect twice on the remount, so a reset was followed in the same
tick by a rebase. Since E32 S3 a reset is state only and the rebase read the mirror ref, which
still held the empty initial history, and wrote it back over the reset.

rebase is now a state updater, so it builds on whatever an earlier call in the tick installed.
Both tests seen RED (hook: [] for [10, 20]; page: no rows), and the bug reproduced in the
browser on the old hook.
2026-10-04 02:48:10 +02:00
peter 98ec1abf25 test(gates): the register test reads every config block, option and key, not eight samples and severities
E32 review round 1 (T1–T5, s1). The test that holds docs/gates.md to the effective configuration
could itself be passed by a narrowing it did not look at:

- ESLint: besides each scope's sample, every config block is read on its own. A block that targets
  files no sample reaches fails as an "unsampled block"; a no-restricted-* list re-declared without
  an entry src has, an option that lets something through (allow…/ignore…/except…), a block-level
  `ignores` and EVERY global ignores object (not only the first) count. A scope switching a src rule
  at warn off now counts too (it skipped them). Two live narrowings surfaced and are registered:
  no-floating-promises' allowForKnownSafeCalls (src and e2e) and the three network owners'
  no-restricted-globals.
- TypeScript: a strict sub-flag set false beside `strict: true`, an extends/exclude/files, a changed
  include, and a frontend-root config file no project checks. eslint.config.js is that file: listed,
  with the 43 errors strict checkJs reports in it (measured, none a defect).
- Ruff, basedpyright, pytest and knip are read as key whitelists (an unknown key fails until someone
  decides whether it narrows); pyright's information/hint count as weak; ruff's extend-select is
  pinned; pytest's three keys are pinned exactly; knip's script must run both modes.

Each new check was probed with a planted narrowing (an unsampled block, a block ignores, a second
global ignores, a warn rule off in a scope, strictNullChecks false, a ruff extend-exclude, a pytest
-m) and failed on it.
2026-10-04 02:37:33 +02:00
peter e5b155ada9 refactor(ui): reuse the store, media-query and deleted-tag rules the review found rebuilt, and name a loading pick
E32 review round 1 (F3–F6).

- Channels' open-row store and the chat dock use lib/store's createStore instead of a hand-rolled
  listener set (the chat dock gets a test of its own).
- The Plex backdrop reads prefers-reduced-motion through useMediaQuery; the query string is one
  constant (REDUCED_MOTION) for the hook and the carousel's one-off read.
- A link-restored collection filter showed its raw rating_key in the picker while the titles
  loaded: usePlexCollectionTitles now says it is pending, the picker draws a placeholder
  (labelsPending, as the channel picker does) and so does the chip row.
- The "deleted tag" rule was written twice with two i18n keys and had drifted: the channel manager
  showed NO chip for a pick while the tag list loaded, or ever after a failed load, while the pick
  still narrowed the table. One rule now (lib/tagPicks), one key (common.deletedTag); the manager's
  row shows every pick, as a placeholder while loading and as its id after a failure.
2026-10-04 02:29:03 +02:00
peter e09d613b04 fix(ui): a reviewed log takes the focus back once, and a dialog's focus survives a Retry
E32 review round 1 (F1, F2).

- Issue composer: the review's focus-return token became state in S4 and was never spent, so every
  Preview → Edit round trip remounted the attachments and moved the focus from the toggle to the
  log's Review button. The token is cleared once it has been used, as the ref was.
- Modal (initialFocus "field"): the late-form watch moved the focus to the first control once and
  stopped. A failed load's Retry then gave way to the spinner, its focus fell to <body> inside the
  aria-modal dialog, and the form's first field never got it. The watch now re-arms when the focus
  falls to <body>: it waits on the ✕ and moves to the first field when the form arrives. Focus in a
  dialog stacked on top is left alone.
2026-10-04 02:29:03 +02:00
peter 38943fa93e fix(maintenance): one clock per run, so a video phase 1 recovers is not fetched again in phase 2
E32 review round 1 (B7). Each phase took its own now; phase 2's "not checked this run" test then
let a just-recovered video through and spent another videos.list call on it.
2026-10-04 02:14:04 +02:00
peter 887b4e546c fix(feed): a malformed saved view cannot break the autotag cleanup, and a foreign private tag reads as missing
E32 review round 1 (B5, B6).

- The autotag orphan cleanup reads every user's feed views. A view whose tag pick was not a list
  ("tags": 5) raised for everyone's job, True passed as tag id 1, and ~66k ids pushed the NOT IN
  past psycopg's 65535 bind parameters. The views are now read defensively and the kept ids
  filtered in Python; the saved-views route holds filters.tags to a list of at most 500 ids, on
  create and on update.
- tags_by_category resolved any tag id, so another user's private tag landed in its real category
  and the feed summary answered differently than for an id that does not exist, which told the
  caller which foreign ids exist. It is owner-scoped now (pre-existing; S2's missing-tag branch
  made the difference visible).
2026-10-04 02:14:04 +02:00
peter df4a79a1f1 fix(sync): an empty answer the remote's own count contradicts is not believed, one level down
E32 review round 1 (B1, B2, B3, B8).

- Plex watch reconcile: a section the server SAYS is empty (totalSize 0, nothing listed) is a
  complete listing. It read as "cut short" forever, so one empty library stopped un-watches from
  ever syncing. A listing with no total at all stays incomplete (the client now returns None for
  it instead of 0), and a whole scan that listed nothing still clears nothing. Tests now cover
  completeness across two sections, both orders.
- Plex collections: no members from a collection whose listing counts some (childCount) keeps
  the membership we hold, in the full sync and in the targeted resync after a write.
- YouTube playlists: no items for a playlist whose listing counts some (itemCount) keeps the
  mirror in the bulk pull, and is refused as retryable in "reset to YouTube" and in the push
  plan/merge, where it read as "YouTube deleted them all". The rule sits beside vanished().
- A playlist deleted on YouTube answered "reconnect your account"; it now says it is gone
  (errors.youtube.gone, HU+EN).
- The pure 3-way merge moved to sync/playlist_merge.py; playlists.py is back under 800 lines and
  off the size list.
2026-10-04 02:13:53 +02:00
peter 5d10a521a8 docs(gates): react-hooks/refs is on, the compiler skips one function, and the async-cleanup idiom that keeps it so 2026-10-04 00:05:37 +02:00
peter d7fb7075b8 refactor(ui): every function compiles under the React Compiler but VirtualGrid, and react-hooks/refs is on
E32 S4 — the burn-down from 29 skipped functions to 1:

- try/finally (26 sites): the cleanup runs after a try/catch; steps that return early moved into
  their own function (push, the setup wizard's next, the share page's unlock).
- A conditional, loop or optional call inside a try skips the function too, and react-hooks/todo
  does not see it: the value is computed before the try, or the try lives in a module-level helper
  (config writes, clip queueing, clipboard, frame counts, seekable end, YouTube player times).
- The two exhaustive-deps suppressions are gone. useYtProgress takes its per-item snapshot through
  one effect event, read once when the player is built; PlexPlayer's unmount relies on the
  leaving-the-item cleanup for the generation bump it duplicated.
- What the skips hid: PlexPlayer's 17 render-phase ref writes (useLatest), its tracks popover alias,
  the loadSession self-reference (declared early, set after commit); PlayerModal's session-position
  read during render (now where the player starts), navRef/modeRef writes, and a description helper
  that ran a ref-carrying callback during render (now PlayerDescription); SavedViewsProvider's
  latest-cfg ref, and its setState in the seed effect (preferSticky now prefers the starred default
  among same-filter views, so the seed needs no sticky write — test seen red on the old helper).
- A store's hook was called `use()`, which the compiler does not know as a hook: inside the newly
  compiled usePlaylistActions it was memoized, ran on the first render alone, and the playlist rail
  threw "rendered fewer hooks" on its second render. Renamed to `useValue()` everywhere; the old
  contract test claimed the call was safe inside any `use…` function, which was false — replaced by
  a render test that is red under the old name.
- PlexPlayer's palette ref was written and never read since the Backspace guard moved to the layer
  stack: removed.

react-hooks/refs is back at error with no exception. The ratchet lists only VirtualGrid
(IncompatibleLibrary) and pins both players as compiled. No size or complexity ceiling was raised.
2026-10-04 00:05:37 +02:00
peter a486c7521a docs(gates): the compiler burn-down after E32 S3, and the two idioms that keep it
react-hooks/refs 245 -> 31 hits and 93 -> 29 skipped functions in the register and the eslint
comment. The canonical-patterns table gets the latest-value row (useEffectEvent; a handler or timer:
useLatest / useStableCallback, never a render-phase ref write) and the popover / scroll-fade row
(destructure the ref setter).
2026-10-03 23:18:33 +02:00
peter 76358a0d77 refactor(ui): the popover engine and every render-phase ref write compile under the React Compiler
E32 S3. The compiler skipped 93 functions; it now skips 29 (try/finally shapes, two suppressions,
VirtualGrid). react-hooks/refs hits 245 -> 31.

- Popover engine: `ref={pop.refs.setReference}` made the compiler treat the whole popover handle as
  a ref, so every component holding one was skipped (21 call sites, the engine and its surface
  among them). The setter is destructured now; the same for useScrollFade and useReturnFocus.
- Render-phase `ref.current = x` writes moved to `useEffectEvent` where only effects read them,
  else to the new `useLatest` (written in a layout effect) / `useStableCallback`.
- FocusSafeButton (under every Button) read the focused element during render; it reads it through
  useSyncExternalStore, which React asks on every render instead of a cached memo.

Bugs this surfaced, fixed:
- Combobox: a chip could keep showing its raw id. The label cache was a ref an effect filled after
  the render that needed it; it is state merged during render now (test seen RED).
- useUndoable rendered from a ref plus a forced re-render; compiled, that memoizes a stale value.
  The history is state now, mirrored in a ref for the stable callbacks.
- ConfigPanel and the saved-views editor re-seeded their drafts in an effect, painting the stale
  draft for a frame; both adjust during render against the remembered key.
- Toaster's handover is written after commit instead of during render; a new test pins that the
  inherited set is taken once, at the takeover (seen RED on a per-render re-read).
- IssueComposer mutated a ref prop as a one-shot focus token; it is plain state.
- PlexBrowse wrote its info-scroll memory from an effect declared before the ref, past its own
  "one writer" rule; the writer comes first now and the effect goes through it.

The compiler ratchet drops to the measured 29 and now also requires FocusSafeButton and the popover
engine to compile.
2026-10-03 23:18:26 +02:00
peter dc00b55feb test(auth): warm the Welcome import once so the first case cannot time out under load 2026-10-03 21:53:06 +02:00
peter fb6b56f121 test(api): every filter key the client sends is one the backend reads
FastAPI ignores an unknown query parameter, so a misspelt or renamed key silently stops a
filter while its chip still shows. Contract tests read routes/feed.py _feed_params and
plex/query.py LibraryFilters. The superseded feedCount/facets calls and keys are removed.
2026-10-03 21:53:06 +02:00
peter 2e2d6f5746 fix(downloads): the quota editor opens at once and says it is loading or failed
It rendered nothing until its quota arrived and nothing at all when the load failed, so the
click looked dead. The first field still takes focus when the form arrives.
2026-10-03 21:53:06 +02:00
peter 2ab9ff6a42 fix(plex): a server without a machine id refuses the collection write by name
The missing identifier became server:/// in the write URI. Client tests over a MockTransport
(errors, paging, cutoff, write URIs, image transcode) and admin-route tests (connection test,
enabled-section sampling, no server URL in a 422, the one-time watch import).
2026-10-03 21:53:06 +02:00
peter 378c4daf0f fix(youtube): the scrape search never pins a stale key, and an unknown shape is an error
A failed config fetch or a keyless consent page cached the fallback InnerTube key for the life
of the process; once it went stale, search returned nothing with no error. The fallback is now
used for that call only. A 200 JSON without a result frame returned an empty page that read as
no results; it is a ScrapeError now. Parser, cursor and Shorts-probe tests added.
2026-10-03 21:53:05 +02:00
peter 8f1208fdfe fix(feed): a filter on a deleted tag matches nothing and its chip says so
A saved view or the stored filter outlives its tag. feedquery dropped the unknown id, so the
view showed the whole feed under its old name. The id now lands in a MISSING_TAG_CATEGORY that
filters to nothing in the SQL and in feed_summary's mirror alike; the feed chip reads "Deleted
tag" (a placeholder while the tag list loads, the id if it failed). Channels' persisted tag
filter gets the same chip, first in the row so the +N overflow cannot fold it away.
2026-10-03 21:53:05 +02:00
peter 52f12e5331 test(e2e): click only a channel link that is the element hit at its centre
A card inside the viewport can still sit under the chrome above the scroller (the update
banner); the covered click made Playwright scroll and retry, moving the feed the scroll-restore
test then measured. webkit failed 3 in 24 before, 0 in 48 after.
2026-10-03 21:10:27 +02:00
peter c4abbe48c1 refactor(sync): the empty-answer rule for a mirror is one function, vanished()
The YouTube playlist guard from the previous commit put sync/playlists.py over its size ceiling;
the rule now lives beside should_prune as vanished(existing, listed), which the mirror calls in
place of its own list. The pyright baseline also drops by one (187), from the plex/overlay split.
2026-10-03 19:45:14 +02:00
peter 4fc307c426 fix(autotag): the orphan cleanup keeps a system tag a saved view filters on
Feed views store tag ids. A system tag no channel carried for a while was deleted and, once a
channel earned it again, recreated under a new id — the view's filter then pointed at nothing, the
feed dropped the unknown id, and the view showed the whole feed under its old name. The cleanup now
skips tags a saved feed view references (test seen red without it).
2026-10-03 19:37:24 +02:00
peter 328d415cfe test(e2ee): the DM crypto round-trips, refuses the wrong key, and keeps its nonces apart
lib/e2ee was 14% covered and fails silently in every way it can break. Real WebCrypto and an
in-memory IndexedDB: a message decrypts for its partner and nobody else, every message gets its own
IV, a new device unlocks with the passphrase and refuses a wrong one, the key survives a reload
until reset, a partner's rotated key is used only after forgetConv, and the key check never shares
the wrap IV (GCM nonce reuse). Mutations seen red: the shared IV, a no-op forgetConv, no persist.
2026-10-03 19:35:36 +02:00
peter 43e4929b45 fix(sync): an empty answer no longer empties a mirror; maintenance checks each video once a run
The catalog prune and the subscription import have long refused to reconcile against an empty
listing that contradicts what we hold (should_prune: a 200 with no items is a blip far more often
than a user who removed everything). Two mirrors did not: the YouTube playlist mirrors (dropped
every mirror) and the Plex collections (dropped every collection with its user-set `editable`
flag, and every member's keys — the guard landed in plex/sync.py with the previous commit). Both
now hold the same rule; test_empty_listing_guards covers each way (seen red).

sync/maintenance (19% covered) gets a lifecycle test: recover, grace, delete + one notification
per user, live stream kept, abandoned premiere. It found the rolling revalidation re-fetching the
same videos until `checked` reached the batch (15000 by default) whenever fewer videos were
available — quota spent on answers it already had; a run now skips what it has checked.
2026-10-03 19:33:38 +02:00
peter 5806adda7e fix(plex): a listing cut short no longer clears watch state or Hungarian text
The daily watch reconcile reads "absent from Plex's scan" as "Plex un-watched it" and deletes the
row — but it walked the sections without asking _paginate whether the walk reached the server's
total, so a Plex hiccup mid-walk (an empty page early) un-watched every title it failed to reach
until the next complete run. The scan now reports completeness; an incomplete one still pulls in
what it saw, re-pushes dirty rows, and clears nothing (logged as plex.watch.reconcile_incomplete).

The HU overlay had the same shape: clear every row's HU text, then refill from the listing. It now
reads the listing first and clears only after a complete one. The overlay moved to plex/overlay.py
(plex/sync.py drops under the 800-line limit and off the size ratchet's list).

Tests: plex/paths (23% -> 100%: prefix boundary, longest prefix, dead mount), the reconcile's
clear/keep/re-push rules, the cut-short scan and listing (each seen red with the guard removed).
2026-10-03 19:33:30 +02:00
peter c73761ae39 refactor(plex): one cast row for a show's page and a title's info
CastStrip (a show's page) and PlexInfo (a title's page and the player's overlay) each drew the
cast carousel by hand, and the copies had drifted: the gap, lazy pictures, the name's tooltip, the
initial's size. Both now render CastRow, which takes the overlay's white-on-art palette as a tone;
each caller keeps its own frame and heading. A DOM test holds the palette per variant (seen red
with the tone wired to "panel"). jscpd over the shipped code: 1 clone >= 10 lines -> 0.
2026-10-03 19:12:17 +02:00
peter ca9ca85b18 test: no test throws a response away; the stream session is narrowed, not baselined
The 7 baselined Optional accesses in plex/stream.py were one fact the type did not carry: the
initial spawn (replace_only=None) always supersedes, so only the fallback respawn can get None.
An assert at the call site says so; the pyright baseline drops 195 -> 188.

test_abuse_limits: the public-URL and YouTube enqueue tests stopped at the profile lookup
(400 no_profile) and passed on that unrelated refusal. They now send an inline spec and assert
the job is queued. The same shape — a request whose response is discarded — sat at 53 more
places across the suite (all returned what they should, measured with a TestClient probe);
each now asserts its status, and test_response_checked keeps it that way (seen red).
2026-10-03 19:06:47 +02:00
peter 614d728561 test(e2e): the nav spec opens the admin pages from their rail panel
E31 put the admin pages behind one rail row ("Administration", nav-admin)
that opens a panel. The spec still clicked nav-scheduler, nav-config,
nav-users and nav-audit as rows of their own, and timed out on every engine.
That went unseen from E31 to now because e2e ran in no required gate, the
exact hole E32 closes: the first run of the new release e2e step found it.

The rail test keeps the rail rows. A new test opens each admin page from the
panel and checks that the group row is current and that the reopened panel
marks the page. Seen red before the fix; the whole suite then passes, 52/52.
2026-10-03 17:41:26 +02:00
peter 783f6408f9 docs: the gate register, held to the effective config by a test
docs/gates.md lists every place a gate deliberately checks less than it
could, each with its reason and the count measured when the row was
written:
- an ESLint rule off, at warn, or not applied, by scope;
- a weak compiler flag, a ruff ignore, the basedpyright mode and baseline;
- a knip exclusion, a lane that does not run everywhere;
- an accepted CVE.
gates.contract.test.ts reads the EFFECTIVE configuration and fails when a
narrowing has no row:
- ESLint's computed config per scope;
- the three tsconfigs, ruff.toml, pyrightconfig, pytest.ini and knip.jsonc;
- .trivyignore.yaml, where an entry also needs a statement and an expiry;
- CI's pip-audit ignores.
It also pins `--max-warnings 0`, which is what makes a warn-level rule fail.
Seen red on a missing row, a new `off` rule and an allowlist entry without
an expiry.

CLAUDE.md gains the rule E32 exists for: a gate with any real hit is never
switched off or narrowed, and an "enforced by" comment is checked by running
the check. The gate descriptions there, in docs/development.md and in
backend/app/README.md name the lanes as they now are, and the "directly"
recipe points at the full list.
2026-10-03 17:26:42 +02:00
peter 751e1ae5ed test: a size and complexity ratchet; coverage reported on both sides
The "class size" item from the quality checklist, measured:
- A shipped file over 800 lines may not grow past its line count today
  (13 frontend, 11 backend). No other file may cross 800.
- A function over the complexity limit may not rise, and no new one may
  appear. The limit is eslint `complexity` 20 on the frontend (64 functions
  over it today) and ruff C901 10 on the backend (32).
- Each is a per-file ceiling. Shrinking is free; an entry back under the
  limit must be deleted, so the lists only shrink.

C901 is measured in a test rather than switched on in ruff.toml: ruff has no
per-function baseline, and a noqa would let a function grow without limit.

Coverage (owner decision D4) is printed with no threshold:
`vitest --coverage` (v8) gives 80% of lines, and the pytest lane ends with
the total for app/, 78%. The bar stays "every regression test seen red".
.coverage is ignored by git and kept out of the image's build context.
2026-10-03 17:18:53 +02:00
peter da554df7c7 build(image): the image passes a Trivy scan: OS security updates, no runtime pip
The first Trivy scan of the shipped image (0.98.0) found four fixable HIGH
advisories. One was pcre2: a Debian fix newer than the base snapshot, now
taken by `apt-get upgrade` at build time. The others were urllib3 (bumped in
the lock), and msgpack, setuptools and urllib3 vendored inside pip itself.
pip is removed once it has installed the lock, since nothing runs it at
runtime.

A local build of this Dockerfile with --pull scans clean: 0 HIGH/CRITICAL
with a fix, no secrets. .trivyignore.yaml is the publish gate's allowlist,
empty today. Each entry must carry a statement and an expiry date.
2026-10-03 17:06:28 +02:00
peter b1ff4058ac build: CI builds the bundle and audits what ships; the toolchain advisories are cleared
The server gate adds `vite build` (and with it devOnlyGuard, which until now
first ran inside the image build, after main was pushed) and
`npm audit --omit=dev --audit-level=high`. pip-audit is blocking: when it was
finally read it held three urllib3 advisories with a fix available.
urllib3 goes 2.7.0 -> 2.8.0 as a single in-place pin, passing pip check,
rather than a regeneration that would move every package.

The dev toolchain's 18 advisories are gone:
- vite 5 -> 6.4.3 (path traversal, fs.deny bypass, and esbuild <= 0.24.2
  through it);
- vitest 3 -> 4.1.11 (mocker path traversal);
- typescript-eslint 8.46 -> 8.71;
- `npm audit fix` for the rest.
tsc, eslint, knip, all 2269 tests and the build pass unchanged.

scripts/check-node-modules.mjs checks that frontend/node_modules holds
exactly what package-lock.json pins. The local gate's frontend lanes judge
the host tree, and nothing tied it to the lock the image installs.
2026-10-03 16:50:56 +02:00
peter 7ed2ac4e95 build(frontend): knip also checks what ships; strict indexing in every project
`npm run knip` now runs a second, --production pass, which asks whether the
code that ships uses an export, not just whether anything does. Of the 47
exports only tests reached:
- 34 are exported for their test and say so with @internal, naming the test.
- 11 were module-private names with a stray `export`. Two of them were not
  used at all and are deleted.
- The unused alreadyReported re-export is gone.
Test kits, the e2e suite and the two build-tooling modules are excluded from
the production project, each named in knip.jsonc.

noUncheckedIndexedAccess is now on for the node and e2e tsconfigs too (0
errors), matching the app project. tsconfig.node.json listed a
tailwind.config.js that has not existed since Tailwind 4. That entry is now
playwright.config.ts, which the typed lint finds through this project.
2026-10-03 16:44:28 +02:00
peter cff8fe9a6a test(frontend): a React Compiler ratchet over everything the build ships
compilerCoverage.contract.test.ts used to check two files. It now runs the
real compiler over all of src (minus tests) and holds the exact set it skips
today, file -> function -> category: 93 functions. A new skip or a category
change is red, and so is a listed skip that has gone, so the list only
shrinks. Channels and DataTable must also show up as compiled, so a harness
that compiled nothing cannot pass.

Measured against the audit's scan: no function this sprint touched started
to skip. PlexPersonView and useOverlayOrder now compile.

reactPlugin.ts claimed the compiler's rules were linted, which was false
while react-hooks/refs was off. That comment and the refs block in
eslint.config.js now say what holds the line and why refs is still off:
245 real hits, pinned by the ratchet, burned down from S3.
2026-10-03 16:38:46 +02:00
peter 471846d784 build(frontend): close the lint gaps the E32 audit measured
eslint now also enforces:
- the React Compiler diagnostics that `recommended` leaves out (todo,
  memo-dependencies and ten zero-hit tripwires);
- a description on every suppression, and no blanket disables;
- no focused or skipped tests (vitest and playwright);
- the typed pair no-floating-promises and no-deprecated.

Lint also covers e2e/ and the TypeScript build configs, which `eslint .`
had been skipping in silence.

Every hit is fixed or suppressed inline with a named reason:
- The 31 try statements the compiler cannot lower yet carry a suppression
  until S3 rewrites them. Five other compiler skips were plain fixes:
  a module counter bumped inside a hook, a ref read in a default
  parameter, and a local named like the <section> it renders.
- The real defects behind floating promises are fixed. ChatDock threw at
  nobody when the device key store could not be opened. VideoEditor's
  play() rejection went unhandled. loadYouTubeApi never rejected, so a
  blocked IFrame API left the player on an empty stage forever; it now
  fails on a script error or after 15 s, shows its own error with the
  way out, and lets the next open try again.
- MutableRefObject, FormEvent and lucide's deprecated brand icon are
  replaced. The YouTube glyph now lives in lib/brandIcons.
- The lint-probe harness lets its virtual file names into TypeScript's
  default project, so the contract tests still run the real config.
2026-10-03 16:34:20 +02:00
peter 3c16012524 build(backend): close the lint gaps the E32 audit measured
ruff gains RUF100, PLW1510 and flake8-bugbear, every hit fixed rather than
ignored. B008 stays on with FastAPI's Depends/Query declared immutable.
basedpyright moves to standard mode, pytest to --strict-markers, and a hand
run of pytest now requires the database unless SIFTLODE_TEST_REQUIRE_DB=0.

PLW1510 found a real bug: the three ffprobe probes in plex/stream.py never
looked at the exit status, so a probe that failed was read as the file's
truth (an HDR source sent down the HW-safe path, a multi-audio file collapsed
to one track, a subtitle offset taken from garbage) and the warning each
except branch promises never fired. They now run with check=True.
2026-10-03 16:04:06 +02:00
peter cdfe7a8986 perf(channels): the row menu opens and closes without re-rendering the table
The React Compiler silently skipped Channels and DataTable (a ref written
during render; the refs lint is off), and even once compiled it dropped the
memo of the rows and columns (they reach filterRows/optionsFromColumns past
the page's hooks) and the columns read the open row. Every open, close and
reposition of the right-click menu rebuilt the whole table: on "All" rows
~0.9 s to open and ~0.6 s to close (dev). Now: effect events instead of the
render-written refs, explicit memos for rows, columns and their inputs, a
module-scope keyOf, and the open row reaches its "…" through a small store.
Measured on 397 rows (dev): ~75 ms to open, ~35 ms to close.

A contract test runs the real compiler over both files and fails on any
skipped function. The dropped memos have no automated guard: they need
the full page rendered with its providers; the measurement above is the
evidence.
2026-10-03 05:10:52 +02:00
peter 3e96211db0 fix(plex): the backdrop crossfade toggle wins over reduced motion once set
prefers-reduced-motion hard-disabled the rotation whatever the toggle said,
and Windows reports it whenever its animation effects are off — so the
owner never saw the crossfade. Untouched, the setting now follows the
system (and the toggle shows that state); a pick made on the toggle is
stored and wins on every device.
2026-10-03 04:38:50 +02:00
peter e9c15baae0 fix(feed): a channel filter's chips name their channels from the first paint
After a reload the chips (filter panel and the row above the feed) read as
raw UC… ids until the subscription list landed. The names seen last time
are now kept per account (useChannelNames) and name the chips at once; the
list then replaces the copy, so a renamed or dropped channel heals. A
channel never named on this browser draws a placeholder bone while the
list loads; a failed list still falls back to the id.
2026-10-03 03:29:54 +02:00
peter 6edb22988c release: 0.98.0 (feature/e31-design-review-2)
gate / frontend (push) Successful in 8m29s
gate / backend (push) Successful in 9m54s
2026-10-03 02:58:46 +02:00
peter f9c3714025 release: 0.98.0 notes — design review 2 2026-10-03 02:58:07 +02:00
peter a4775e1cd0 fix(ui): the filter rail shows no tag counts until the facets land
It printed the tag's whole-library count as the current filter's while the
facets loaded, and for good when they failed: live, "Music 238" beside a
search that matched 3 of its channels. Ordering still falls back to it.
2026-10-03 02:43:52 +02:00
peter b40440b825 fix(ui): review round 2 — reveal floor, failed next page, count states, row Play
- DataTable: the reveal floor holds only in the view it was made for (Channels
  never clears its follow, so every later sort mounted rows up to it)
- Feed + Plex browse: a failed next page stops the read-ahead and shows a row
  that retries, instead of re-requesting it once a round-trip
- the YouTube Load more click uses nextPageOnClick; a contract pins NEXT_PAGE
  to the read-aheads and the click sites to nextPageOnClick
- ItemCount: a failed count says so; a stale one says updating to a screen reader
- the narrow-picture Play lift applies only where a hover row is there to clear
- PlayerModal: one description ladder for the popover and the panel, both tested
- plexImage contract reads every img src, any attribute order
- Tabs comment names who persists the sub-tab
2026-10-03 02:25:15 +02:00
peter c4c442bc5b fix(feed): a hand-made relevance cursor is refused, not a crash
The page halves compare the cursor's timestamp with the window's in Python;
a null or naive one raised there, past the route's 400. decode_cursor now
refuses a null non-nullable key and a naive timestamp.
2026-10-03 02:25:14 +02:00
peter c74ceb73e3 fix(feed): the count pill dims while a new search's count loads
keepPreviousData held the last filter's count, and a broad word's count
takes seconds: the pill showed the old number as the new search's. It now
dims and is aria-busy until the new count lands, as AdminLogs does.
2026-10-03 01:01:46 +02:00
peter e0e28c006c perf(feed): relevance ranks the newest 2000 matches, the rest follow newest first
A broad word matched most of the library and ts_rank read every match:
"a" took 4.4 s a page on prod. The window's start is read once (0.1 s) and
rides the cursor, so a video ingested mid-scroll cannot repeat a row; the
page is read as its two halves, each on an index (80 ms + 5 ms).
2026-10-03 00:28:29 +02:00
peter f8f22fc985 test(i18n): a language test unmounts its reader before switching the language back
useTags reset the language in a finally while its hook was still mounted;
the switch scheduled a render, and in a loaded gate run it landed after
the jsdom environment was torn down ('window is not defined', two
unhandled errors). auditColumns and comfortPin had the same shape.
2026-10-02 23:48:37 +02:00
peter 90e0baa870 style(db): ruff format the re-runnable index migrations 2026-10-02 23:43:21 +02:00
peter dfd43ae89c docs: the folding channel header in its e2e, and three comments brought up to date
The e2e asserted the channel chrome never moves; it folds once now. The
ui catalog lists scrollFold and progressiveRows, Tabs names the
per-tab memory its callers use, and the welcome note says a removed
language orphans its stored copies too.
2026-10-02 23:42:32 +02:00
peter 608fde2d55 fix(db): re-running 0095/0096 rebuilds an invalid leftover index
A concurrent build that failed part-way leaves an INVALID index under its
name; IF NOT EXISTS kept it on the re-run, and 0095 then dropped the old
channel_id index, leaving none usable. The leftover is now dropped and
built again, and the old index goes only once the new one is valid.
2026-10-02 23:42:32 +02:00
peter 0302c466e8 fix(i18n): a failed RSS poll over one channel reads as one, and keeps its stops
rss_all_failed is a plural pair counted on the raw total (the values are
formatted strings, which pick no plural form); both sentences end with a
stop. The Hungarian 'show N more' uses the house's megjelenítése.
2026-10-02 23:42:32 +02:00
peter a50055577a fix(plex): every small Plex picture asks for its card-size cut
The playlist rows, the search box and the collection picker still
fetched the 400px poster for a 32-80px picture. A contract test pins the
set of bare Plex pictures, and the srcset widths to imgcache.SIZES.
2026-10-02 23:42:32 +02:00
peter e16cdbd16f fix(ui): a card's Play clears its hover row at a large text size; Downloads links and the player description
At text size 1.3 a 242px feed thumbnail put the 63px Play 13px into the
hover row (the Play grows with the root font, the picture's height does
not). A picture narrower than 14rem centres the Play and the resume pair
above the row; the picture box is a size container for it.

The Downloads title and channel links were 21px tall inline boxes; they
stand on a 24px line now (B1). The player's description says a failed
load, with a retry, instead of 'No description' (both the side panel
and the popover).
2026-10-02 23:42:31 +02:00
peter 15ec67b029 fix(ui): a read-ahead waits out a background refetch, and a revealed row is mounted before it is focused
NEXT_PAGE joins ANY fetch in flight: a read-ahead during an invalidation
or a poll joined that refetch and appended no page, and nothing asked
again until the reader moved the range (the channel page during a
backfill). The read-ahead now waits while the list fetches anything
(mayReadAhead, re-run on isFetching); a Load older click cancels a poll
instead (nextPageOnClick).

A page turn to a revealed row restarted the progressive mount at 30
rows; Channels focuses that row in the next commit, and at index 40 of
a 50-row page it was not there, so focus fell to <body>. The mount now
reaches through the revealed row.
2026-10-02 23:42:31 +02:00
peter ac53a3b757 perf(search): a one- or two-letter last word is matched whole, not as a prefix
A 1-2 letter prefix matched most of the library and ts_rank scored every
match: "m" took ~4-8 s on prod against ~150 ms now. The prefix match
starts at three characters (MIN_PREFIX_LEN); shared by the YouTube feed
and the Plex library search.
2026-10-02 21:52:47 +02:00
peter e6de18b8c0 perf(plex): the facets ask the episode rollup once per call, not once per show group
A watch-state scope (watched / in progress / unwatched) joined the per-user episode rollup into
each show facet query: five passes over the user's episodes per call, half of a filtered facets
request. The shows in scope are now resolved once and every group filters on their ids; the
grid-parity test holds the counts. Localdev: 99 -> 61 ms.
2026-10-02 21:32:49 +02:00
peter fb2e3d8490 perf(plex): a card fetches its picture at a card's size, not the detail page's
Cards fetched the full cuts: a 400x600 poster (~38 KB) for a ~144px picture and the 1280x720
backdrop (~162 KB) for a ~212px shelf picture. The image proxy now also cuts thumb_sm (240x360)
and art_sm (480x270), cached apart; card pictures (grid, shelves, collection tiles, Up next) carry
both in srcset with sizes=auto, so the browser picks by laid-out width and DPR. Localdev:
17 KB a poster, 56 KB a backdrop. The detail page keeps the full cuts.
2026-10-02 21:27:13 +02:00
peter d928d50686 perf(db): each feed sort rides an index, and an ICU upgrade rebuilds the title ones
The views, duration and title sorts had no index: every page read the whole videos heap and kept a
top-N (prod 350 ms a page, 550-780 for title). Migration 0096 adds one per direction and, for title,
per reader collation, in the exact ORDER BY feedquery builds. Localdev prod copy: plain sorts
180-310 ms -> ~1 ms, page 20 3-5 ms, narrow filters unchanged. view_count/duration are only written
by the enrich, which already rewrites indexed columns, so no HOT update is lost.

The title indexes are the first on an ICU collation, and postgres:16-alpine floats: collation.heal_drift
rebuilds them (CONCURRENTLY) at startup when Postgres reports the ICU version moved.
test_feed_sort_indexes holds every videos sort to an index and checks the rebuild.
2026-10-02 21:17:41 +02:00
peter 540822d2b0 perf(ui): a next-page fetch joins the one in flight instead of sending the cursor again
react-query's fetchNextPage cancels a next page still loading and sends it again by default.
Our requests ignore the abort signal, so the server ran both: the feed's read-ahead effect
re-runs on scroll frames before isFetchingNextPage reaches it, and on prod every feed page
went out 2-3 times (a 0.8 s title sort queued to 2.3 s). Every call now passes NEXT_PAGE
({cancelRefetch: false}), lint-enforced, with a contract test.
2026-10-02 20:46:30 +02:00
peter b4ef07d28e perf(ui): a long table page paints its first rows at once and the rest in steps
The channel manager's All page (~400 rows) blocked one frame for 0.7-0.9 s on prod to mount,
and every sort and search key re-rendered every row (300-400 ms). No single cell is to blame,
measured column by column. DataTable now mounts the first 30 rows at once and 40 more per
background transition; a new page, size, sort, filter or caller-side search (narrowedBy)
starts over from the first rows. A row is its own component, so the compiler keeps the rows a
step already holds.
2026-10-02 03:16:37 +02:00
peter e995c664c6 perf(db): the channel manager and the feed sidebar read an index, not the video heap
GET /api/channels summed every subscribed channel's videos by reading each video's heap page
(681 MB on prod): 51-61 ms warm, but 2.25 s on a first open and 6.4 s under EXPLAIN once the
table had left memory. /feed/summary and /feed/years did the same (~140 ms warm, ~800 ms cold).
Migration 0095 replaces ix_videos_channel_pub_desc with the same key INCLUDING the columns
those queries read, so they are Index Only Scans over a 25 MB index. A test holds the queries
and the index's column list together.

The admin Configuration page did a db.get per setting (two for an unset one): it now reads
every override in one query (~40 ms of ~55 on prod).
2026-10-02 03:16:37 +02:00
peter 20010178b0 fix(ui): the Plex detail poster keeps its 2:3, and a long crew shows its first names
The hero's grid row is as tall as the crew column beside the text, and the poster, a flex child,
stretched to it: a 2:3 picture drawn ~175x590 and cropped. It now aligns to the top, and a wide hero
gives the movie and show poster a bigger rung (w-56). The crew, the hero's tallest part, shows its
first 6 names and a 'Show N more' toggle; a list of 8 or fewer shows whole.
2026-10-02 02:37:20 +02:00
peter 78fdcda6bb feat(ui): E31 S4c — the channel header folds, the Plex pages fill their hero, the player gets a side panel
- Channel page: the banner and identity block fold away once the grid scrolls
  (ui/scrollFold: folds past 32px only if the page can stay scrolled, unfolds
  at the very top) and the name rides in the tab row. One Back button in both
  states keeps its focus; folded at a tablet width the actions wrap instead of
  pushing the page sideways.
- Plex movie and show pages: the TMDB facts over the key crew are the hero's
  second column (plexDetailAside), the keywords sit under the text; the show
  page shows them for the first time, with the same three toggles. A
  collection's hero is sized to its text.
- YouTube player, 1280px and wider: the title, channel, actions, description
  and queue sit in a side panel beside the video, which takes the window's
  height (1438x809 at 1920x911, was ~1008 wide). The wheel scrolls that panel
  while it can and turns the volume elsewhere.
- DESIGN.md decision log: the keywords' place under the text.
2026-10-02 02:13:17 +02:00
peter 98d0ca6eeb fix(i18n): every counted string is a plural pair, so one reads singular
"1 seasons", "Synced 1 subscriptions", "1 video(s) were skipped": 21 English
strings carried {{count}} in one plain key and read plural at one. Each is a
_one/_other pair now (Hungarian keeps one form for both), and so is every other
counted string, so the next wording change is a plural too. A contract test
holds the rule for every locale file.
2026-10-02 02:13:17 +02:00
peter e4c80fb7bb fix(ui): a page's sub-tab is remembered per browser tab, not forever
Settings, Configuration, Statistics and Users kept the last sub-tab in localStorage, so every
fresh tab opened Configuration on "Plex" as if that were its home. useTabPersistedState keeps it
in sessionStorage (account-scoped): a reload and a trip to another page keep it, a new tab starts
on the first sub-tab. The pre-selects (a Review link, the Google-link return) write the same store;
a value an older build left in localStorage is cleared on first read.
2026-10-02 00:49:46 +02:00
peter be156fc52b feat(ui): E31 S4b — the admin pages behind one rail row, Configuration as a sectioned list, Statistics as tiles and bars
- NavRailGroup: one "Admin" rail row opening a panel of the admin pages (links, so a
  middle-click still opens a tab); the rail fits 133% zoom without a hidden scroll
- sysconfig.SECTIONS orders the config groups under four headings (every group in exactly
  one, test-held); the page shows them as Settings' side list (Tabs `group` headings);
  the lone registration group reads "Registration" under "Access"
- StatTiles / StatTile / StatBars: the Statistics overview's figures as tiles, the
  per-operation quota use as horizontal bars (a zero draws none)
2026-10-02 00:07:00 +02:00
peter 25905aa786 fix(audit): fold a job's successful runs whatever ran between them
The owner read a fold that broke at every other job as the same row five
times. A job's successful runs now fold into one row at the newest run's
place across other jobs and admin actions; only that job's own failed or
skipped run breaks the fold (it stays its own row, and older successes
start a new fold below it).
2026-10-01 23:05:38 +02:00
peter 87d71e3762 feat(ui): E31 S4a — rail rows are links, the channel page opens on unwatched, users and audit tidied
- Nav rail page rows are real links (/?page=<id>): a middle-click or
  Ctrl-click opens the page in a new tab; a plain click stays in-app.
- A fresh channel page shows the unwatched videos; a picked watch state
  is a remembered preference (channelShow).
- Users: the role, suspend and delete move into the row's one menu
  (the "…", right-click, Shift+F10); locked ones keep their reason.
- Audit log: back-to-back successful runs of one job fold into one row
  that opens to the earlier ones; a failed run reads in the danger colour.
- DESIGN.md decision log: the E31 picks.
2026-10-01 22:51:38 +02:00
peter ab146211fa fix(scheduler,i18n): a failure a job names reads in the reader's language
The audit log showed a failed RSS poll as the server's English sentence
under a Hungarian UI. A job can now raise progress.JobFailed(reason,
message, **params): the scheduler stores the reason and params as the
run's data (dashboard, manual-run notice, audit row) beside the English
line, and the client renders scheduler.failure.<reason>. The RSS outage
names its first channel and the reason code (HTTP 404 vs unreachable).
An upstream's own error text stays as raised (diagnostic).
2026-10-01 22:51:37 +02:00
peter ec22eefd31 fix(ui): E31 S2 — the tab names the object, the channel header wraps, the player fits a short window
- The browser tab names the place and the object: PageHeading reports the
  view's own title (lib/documentTitle), so a channel, playlist or any Plex
  view reads "<object> · <module> · Siftlode" however it was opened; the
  Feed is named too; a season adds its show (tabTitle).
- The channel page's tab + action row wraps and lets the tag lane fold into
  "+N": at 1024 it ran past the edge and the page scrolled sideways.
- The YouTube player card's width is capped from the window height less the
  chrome measured under the video: at 133% zoom (685px) it cut its channel row.
- Plex certificate words (Not Rated, Approved, Passed) are named through the
  vocabulary on the info badge, show page, collection, filter and chips.
- DESIGN.md: the zoom target is 125% and 133%, not 200% (decision log).
2026-10-01 21:17:20 +02:00
peter 2543d44860 fix(messages): the welcome reads in the reader's language, not the one it was created in
E31 S1: the welcome is rendered once, at account creation, so an account that
switched language kept it in the old one (the owner's, English under a
Hungarian UI). A stored copy is recognised by its opening line (derived from
WELCOME) and served in the reader's language; other system messages stay as
stored.
2026-10-01 21:17:09 +02:00
peter 29c0715237 release: 0.97.0 (feature/e30-app-in-hungarian)
gate / frontend (push) Successful in 6m16s
gate / backend (push) Successful in 7m19s
2026-10-01 10:44:40 +02:00
peter 6b86f99249 release: 0.97.0 notes — the app in Hungarian 2026-10-01 10:44:39 +02:00
peter 3563858fd1 fix(ui,i18n): E30 review round 3 — the stored language follows the screen, a fresh text starts collapsed, Retry waits again 2026-10-01 10:43:59 +02:00
peter 08e62bbb16 fix(plex): E30 review round 3 — a dead headshot breaks the Plex-failing streak, 410 held by a test 2026-10-01 10:43:59 +02:00
peter 9454717f47 fix(ui,i18n): E30 review round 2 — honest bio failure, a full show summary, one HU name per thing
- Person page: a bio ask that fails says so with a retry instead of reading as 'no biography'.
- Show page: the summary is a ClampedText (5 lines + More), like the person and collection pages;
  one More/Less pair (plex.textMore/textLess) for all three.
- Channels: a re-slotted row is placed in the language the server orders by (the stored
  preference, else English), not the UI's guess.
- A skipped run's body is only the reason (its title already says skipped).
- HU: Behúzási kvótatartalék; Eseménynapló ürítve / az eseménynaplót; indok; közvetlenül
  játszható / közvetlen közvetítés in the Plex hints.
- Tests: ClampedText re-measures on a resize, the show summary clamp, serverOrderLang.
2026-10-01 08:31:04 +02:00
peter 6ac9ffd45e fix(plex,audit): E30 review round 2 — only a missing file is a dead headshot, one spelling per portrait
- warm_portraits: only 404/410 mark a headshot dead (measured: Plex answers a dead CDN file 404);
  a refused token or a 5xx is Plex failing, ends the run after three in a row and marks nothing.
  'failed' counts every failure, not only the current streak.
- is_portrait_url: plain path segments only — no dot or empty segment an HTTP client folds onto
  one file; person_thumb_url uses the same predicate (27 852 stored headshots all pass).
- Tests: Plex refusing (401/403/429/5xx), failures between successes (portraits and bios), the kept
  original served from cache, the audit_clear / demo_reset / discovery_purge rows' counts as data,
  and the 0094 backfill on real rows.
2026-10-01 08:31:03 +02:00
peter fa55e01ef0 fix(ui,i18n): E30 review round 1 — audit rows in the reader's language, one HU name per thing
- audit log: a job run named from its result data (nested phases too), a skipped run says so, a
  secret's reset reads as a reset, choices named as Configuration names them, interval / batch /
  counts localized; the inbox shares the same jobRunText
- HU: remux is "Közvetlen közvetítés" (it shared "Közvetlen lejátszás" with direct play); backfill is
  behúzás, enrich is dúsítás, collection is kollekció, the allow-list is fehérlista; the last
  "jobnak" / "transzkódolás"; a test holds the options of one choice to distinct names
- contract: every skipped / reason word a job returns is named in both languages
- person page: the bio wait is timed from the open and a failing ask stops the polling; ClampedText
  (ui) shared with the collection page, whose short summary lost its dead More
- a playlist names season 0 the Specials; the channel re-slot compares in the reader's alphabet; an
  invite's date is in the admin's language; the language-ordered lists are asked again once a
  language switch is saved (saveLanguage)
- poster grids' column minimum in rem; lint refuses minmax(Npx,…)
2026-09-30 03:26:58 +02:00
peter ab963c81d2 fix(plex,audit): E30 review round 1 — the people job keeps going, HU genres everywhere, results as data
- people_enrich: a failed TMDB lookup or a dead headshot is skipped, not the end of the run (a dead
  file is marked and left for a week; three failures in a row end it); portraits wait for the Plex
  module switch; a no-Plex original keeps its own cache key, so the resize still replaces it; the job
  entry lives in the plex domain (run_people_enrich)
- person-image takes only a headshot-shaped CDN url (no query variants of one file)
- every synced row gets Hungarian genres, and a HU genre filter matches the word as picked too;
  migration 0094 fills the rows no overlay reached
- a job run's audit row carries its result as data; audit_clear / demo_reset store their counts
- one credit-column list for the prune and the enrichment order; _overlay_pass typed
- tests: batch skip + stop, HU bio stored, dead headshots, module switch, the person-image route,
  the job body, genre filter + facet write + twin fallback, tagline/summary fallbacks, audit data,
  the reader's alphabet at every remaining collated() call site
2026-09-30 03:26:57 +02:00
peter 59e6cc027d fix(plex): a short "Known for" rail sits right beside the text (E30 S6)
With one to three titles (about one person in five) the rail sat at the far edge of the hero with a gap
between it and the bio. The person's column is now capped at the bio's reading measure and a short rail
follows it directly, so the leftover is the panel's end (owner pick "b"). Four or more keep half the hero.
2026-09-30 02:27:28 +02:00
peter ca94a716d8 docs(design): composition and the reader — a chapter and a Conformance step (E30 S6)
The rules covered each piece; nothing asked whether a screen works as a whole, and the person page passed
every rule with half its panel empty. New chapter: the space is used, one focal point, worst real data,
where am I, a click answers at once (no outside service on open), zoom and text size first-class (the
owner reads zoomed in). Conformance step 5 writes one composition line per screenshot and checks 200%
zoom and the largest text size. Anti-patterns and a decision-log row to match.

HU: the download worker setting reads "Párhuzamos letöltések" (a leftover from the S5 sweep).
2026-09-30 02:19:54 +02:00
peter 63d01babe5 feat(plex): the person page in two columns; a bio on its way holds its place (E30 S6)
The biography sat at reading width in the left half of a full-width panel with nothing on the right.
Now the person (headshot, facts, the bio clamped to eight lines with More / Less when it is cut) is on the
left and "Known for" on the right; with fewer than four titles the rail takes only its posters' width.
One column where the panel is narrow (200% zoom, the largest text size).

While the server fetches a bio (`enriching`), the page shows a paragraph skeleton (a new "text" shape of
LoadingState) and asks again every 1.5 s for up to 20 s. The view is keyed by the person's name, so
another person starts fresh.
2026-09-30 02:19:54 +02:00
peter 705377c797 feat(plex): the people are local — bios and portraits fetched ahead, not on the click (E30 S6)
A person's TMDB bio was fetched only when their page was first opened, off the request path, so the first
open showed no bio and the page never asked again (2 of 18,202 linked people were ever enriched).

- people_enrich scheduler job (every 10 min, 300 per run): linked people's bios from TMDB and their
  portraits into the image cache, the most-credited people first. A failed TMDB lookup ends the batch.
- Portraits are resized through Plex's photo transcoder to a 320 box (~15 KB instead of ~170 KB; the
  person-image proxy uses the same cache key, and a full-size original is removed as its small twin lands).
- The person response says `enriching` while a bio is on its way (a person the job hasn't reached yet).
- "Known for" is ranked by vote count, without guest visits as themselves (talk shows had crowded out the
  films: Tom Cruise had 14 of 15), with directed works, from a 40-deep pool; the page shows up to 15.
  Migration 0093 un-stamps enriched people so the job refetches them.
- A phased job result (backfill, people_enrich) now counts as a change for the audit trail; before, an
  automatic backfill run that added videos was never audited.
- HU: the watch reconcile job reads "egyeztetés" (a leftover from the S5 sweep).
2026-09-30 02:19:54 +02:00
peter 2819049133 feat(audit): job runs and config changes read in the reader's language (E30 S5)
The audit log showed the server's fixed English lines ("rss_poll: ok — new=5, failed=0",
"log_level = DEBUG", "Background sync paused") to a Hungarian reader. The details are rebuilt from the
row's data on the client: a job run names the job and its result as the inbox does, a config change
names the field and its values (on/off, grouped counts, "default"), a pause/resume adds nothing the
action does not say. A failed run keeps its error text (diagnostic). parseSummary reads enrich's bare
count ("50") back as `enriched`, as the server's `_result_data` names it — the audit and old notices
keep only the one-liner. No backend change; old rows render too.
2026-09-30 01:08:29 +02:00
peter 95b94a1d72 fix(ui): a longer label fits its control — pill value truncates, headers get their room (E30 S5)
Found in the Hungarian UAT: a toolbar SelectTrigger's text value wrapped out of the pill in a narrow
place (the playlists rail) — it now truncates in its own span; the Channels "Prioritás" and "Utolsó
feltöltés" headers were clipped by their fixed columns — sized for the label, funnel and sort arrow;
a sortable header is at least 24px wide (Hungarian "Ki" was a 12px target). PRE-EXISTING, any
language: the messages unlock button and "Forgot passphrase" sat flush — one gapped row.
2026-09-30 01:08:29 +02:00
peter 12077fbebc feat(i18n): Hungarian words for the English loanwords, one name per thing (E30 S5)
The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info,
Frontend/Backend, Backfill and transcode get the Hungarian word (közvetítés, Prioritás, Feladat,
Eseménynapló, Demó, Automatikus, Infó, Felület/Szerver, videók behúzása, átkódolás). The final sweep
also took out reset/sync/reconcile/setup/logs/whitelist/email, and three features that went by two
names: access requests (kérés/kérelem), backfill (betöltés/behúzás), and the connection test
("Check the connection" beside "Test connection", in both languages). Two quota actions shared one
Hungarian name. The playlists rail title and two of its sort labels are shorter so they fit.
DESIGN.md Copy: the rule and its decision-log row.
2026-09-30 01:08:29 +02:00
peter 519fbe2a18 feat(i18n): the person page's biography in Hungarian, from TMDB's translation (E30 S4)
The /person fetch appends translations and keeps TMDB's Hungarian biography beside the instance-language
one (new plex_people.biography_hu, migration 0092). A Hungarian reader is served it, the English one when
TMDB has none. Already-enriched people are un-stamped, so their next open refetches them once.
2026-09-29 23:44:48 +02:00
peter a39504e617 fix(stats): the full-history ratio counts the requested channels on both sides
"Full history: done / requested" counted every subscribed channel whose
back-catalog was complete in the numerator - also the ones another user asked
for, or short enough to finish on their own - against only the channels this
user requested, so the Stats page read "342/329". The numerator now counts the
requested channels that are done; "channels without full history yet" still
counts every subscribed one.
2026-09-29 22:46:04 +02:00
peter aac79ffa53 fix(i18n): numbers, units, dates and order in the UI language, and after a switch (E30 S3)
Every locale formatter takes the language as an argument. Under the React
Compiler a formatter that read the i18n singleton was memoized on its other
arguments, so a card kept "2 hr 4 min" after a switch to Hungarian; plexLang()
had the same flaw and left Plex queries in the old language. Components now
take the language from useTranslation, and the signature makes forgetting it
a type error.

- ratings, file sizes, stream stats: the language's decimal mark ("8,3")
- view and subscriber counts compact in the language ("1,2 E")
- seconds as the language's unit ("6 mp") in the sliders and the stats
- a person's birth and death dates as dates, not ISO strings
- A-Z lists by the shown name in the language's alphabet (compareText),
  the tag catalog re-sorted by its translated names
- lint local/locale-format (was date-format) also refuses Intl.NumberFormat
  outside lib/format.ts, localeCompare, and a component reading the language
  off the i18n singleton
2026-09-29 22:39:57 +02:00
peter 67c9f10153 feat(i18n): every server A→Z list in the reader's alphabet (E30 S3)
The Postgres image is Alpine, whose en_US.utf8 is byte order: "Életrajzi" and
"Ötödik" sorted after "Zorró" in the feed, the channel lists, the Plex grid,
its genres and collections. `app/collation.py` names the ICU collation of the
reader's language (hu-x-icu puts "Ö" after "O" and "Cs" after "Cu"), wraps
every text ORDER BY with it - the feed's keyset cursor compares with the same
expression, so paging walks the order it sorted in - and orders the lists built
in Python (genre and studio facets, collection members, the discovery re-sort)
through the same collation. Plex title tiebreaks use the shown (HU) title.
2026-09-29 22:39:57 +02:00
peter 584d082b6b feat(inbox): a run notice from before the result rode along reads its one-liner back into one (E30 S2) 2026-09-29 21:32:29 +02:00
peter f3a85916ec feat(i18n): the server's closed English vocabularies named in the reader's language (E30 S2)
- lib/vocab + the `vocab` locale area: TMDB crew jobs and departments, auto-tags (D1 A: the data
  stays English, the catalog is named at its one source, useTags), built-in download profiles,
  YouTube topics; held key-for-key to the backend's own lists by backendIds.contract.
- Countries by ISO code: TMDB parse stores {code, name}; rows stored before are coded from TMDB's
  own country list in one request (run_enrich); Intl names them, historic codes (YU, SU, XI…)
  from the catalog.
- Player tracks named from the language tag, forced/SDH flags and codec tail, not Plex's
  server-language displayTitle; no invented "Audio N" on the server.
- Download lifecycle notices (shared / expiring / expired / evicted) named from data; the share
  notice carries the title.
- Scheduler: the job result rides along as data and is named per key (dashboard + run notice).
- Invite status named; a render crash's notice no longer shows the engine's English message.
- Fix: YouTube links …/wiki/Humour, the tagger looked for "Humor" — Comedy never came from a topic.
2026-09-29 21:21:19 +02:00
peter f6298d29af fix(plex): the recs rail's anchor and a held recommendation are named in the reader's language (E30 S1b)
Found in the S1b UAT: 'Mert megnézted: Batman Forever' kept the English title, and
the info page's TMDB recommendations kept TMDB's English titles even for titles we
hold. A held recommendation is now named as the library names it, in the reader's
language; one we lack keeps TMDB's title.
2026-09-29 02:25:36 +02:00
peter 632e39d458 feat(plex): every card rail in the reader's language; an episode in a poster rail stands as its show (E30 S1b)
The grid served Hungarian titles, but every other card surface (Discover rails, the
show page's episodes, search, collections, the person page, playlists, Home's resume
rail, hero and comfort shelf) served English. The card serializers now take a
required keyword-only lang (a forgotten call site fails instead of going English), and
each of those reads takes ?lang=hu, with the language in its query key. A Hungarian
search finds an episode by its Hungarian title too.

An episode on a poster rail showed its 16:9 still cropped into a poster, titled only
with the episode name; it now shows the show's poster and name, with S·E · title under
it (option A). Plex's generic season names (Season 1, Specials) are rebuilt from the
number in the reader's language; a season with a name of its own keeps it.

Home's comfort-shelf move wrote its optimistic order to the key without the language,
which nothing read any more; it writes the query's own key.
2026-09-29 02:12:32 +02:00
peter 13d95ef70e feat(plex): episodes, taglines and genres in Hungarian from the HU overlay (E30 S1a)
The HU TV section lists every episode with its Hungarian title and summary, but the
overlay read only movies and shows, so every episode stayed English. It now overlays
the episodes too, and the HU tagline (new tagline_hu column, served on ?lang=hu).

Plex's HU agent returns a mixed genre vocabulary (Action beside Dráma), and a title
with no HU twin kept its English list, so the HU genre list showed Drama and Dráma.
plex/genres.py maps every genre word to one Hungarian word; the overlay writes it
into genres_hu for every row, and a HU genre filter is read through the same map,
so a saved filter still holding Action keeps matching.
2026-09-29 01:52:02 +02:00
peter 0f9ed7cdbf test: a list key waits for the list's registration, not only its rows
gate / frontend (push) Successful in 6m5s
gate / backend (push) Successful in 7m10s
The server gate's frontend job went red on 0.96.0: PlexPlaylistView's j/Delete
test pressed j after the rows painted but before useListNav's passive effect
registered the list, so the key did nothing on the slower runner. Every test
that presses a list key after an async load now waits for [data-nav-list] on
its list first (PlexPlaylistView, Playlists, Home, PlexDiscover). Checked
against a 30 ms delayed registration: the old tests fail, the new ones pass.
2026-09-27 23:01:57 +02:00
peter ffcea0cb07 release: 0.96.0 (feature/e28-design-constitution)
gate / frontend (push) Failing after 6m12s
gate / backend (push) Successful in 7m22s
2026-09-27 22:46:39 +02:00
peter a3f8314c9a chore(release): 0.96.0 — the design constitution
E28: one action list per object, keyboard reorder, touch drawers and player
controls, the table pager, plus four deep-review fix rounds (playlist sync
locking, player restart and focus behaviour).
2026-09-27 22:46:39 +02:00
peter d5ccb1091a fix(player): a held key on faded bands stays one blind press; a play asked for mid-restart is the new session's
The repeats of a Space/Enter that landed on faded bands pressed the control once the bands were up;
they are swallowed with it. A play or pause while a session restart is out now sets that session's
intent instead of playing the outgoing stream. Tests: Skip intro outside the bands keeps its keys,
every restore takeover (wheel, finger, scrollbar, key) is covered, and the chip handoff test tells
the chip before from the row's first control.
2026-09-27 22:34:14 +02:00
peter 528774d640 fix(playlists): a push stops at a list deleted mid-way, and a rename made during it stays unsynced
A delete landing while a push read YouTube left a plan over no items, so the push emptied the
YouTube playlist a local-only delete keeps: the push now stops at a vanished list (PlaylistGone,
404), and a delete is refused while a sync of the list runs. The closing baseline fingerprints the
title YouTube got, not the one its lock re-reads, so a rename made during the writes stays dirty.
The other-tab test helper bounds its lock wait, so a lock-order regression fails instead of hanging.
2026-09-27 22:34:14 +02:00
peter 3a9ea9e330 test: the side panel's focus step is held in its commit; the MenuCheck contract counts chosen roles 2026-09-27 21:56:24 +02:00
peter 25f2d9d0e4 fix(ui): a removed chip's focus outlives its neighbour; a scroll restore yields to the reader
A chip handed its focus to a Clear all or +N that went in the same removal, and the focus fell to
the body. The restore now stops when the reader scrolls, and finishes the way once the Back-to-top
spacer brings the saved spot within reach.
2026-09-27 21:56:23 +02:00
peter 5ab6778b97 fix(player): a Restart of the paused film plays the new session; a key on faded bands only shows them
The same-film restart read the paused state, restarted the session without play, and resumed the
outgoing stream beside it. Space or Enter on a control the bands had faded from pressed it unseen;
now it only brings them back, and Space plays or pauses.
2026-09-27 21:56:23 +02:00
peter 26697472cf fix(playlists): every sync path locks the list before its items, and holds it to the write
The mirror refresh re-ingested inside its row lock, and that ingest's commit let go of the lock
before the replace: a remove made then was undone and the list read as synced. Push, re-pull and a
delete touched items before the list row, a deadlock beside the playlist-first route edits. A push
now re-plans from the items under the lock and derives dirty after its YouTube writes.
lock_playlists moves to the sync package. Tests pin both ids of a move's lock and the claim's
own connection.
2026-09-27 21:56:23 +02:00
peter 2abd4ffcb6 fix(player): a session superseded before it played is reclaimed, not "couldn't start"
Two session requests for one item in flight at once (a quick double seek; the
StrictMode replay of the start effect) let the server's last spawn be the older
request's. The newer request's answer then named a session already gone; its
playlist 404'd before a manifest parsed, and the player stopped with "Playback
couldn't start". The 404 reclaim now covers a session that never played too,
bounded by the same budget, and keeps the start's own autoplay.
2026-09-27 20:22:43 +02:00
peter a9ac303d3e fix(side-panel): the key's focus steps into the peeked card in the commit that opens it
A frame later never came in a page the browser was not painting (measured: a hidden page runs no
requestAnimationFrame), and the focus stayed on the inert tab under the card.
2026-09-27 19:58:18 +02:00
peter 740602b9bf fix(ui): focus, paging and state that a key or a finger can reach
- A key landing on a side panel's tab steps into the card it peeks open, not onto the hidden tab.
- The Channels cards keep their status / pager row at the list's end while the band is folded.
- A menu row's note shows on an enabled row too: the download state under 'Download…', not a title.
- A removal chip hands the focus to its neighbour before it goes, for every caller (the feed's and
  Plex's active filters); the hidden-filter cue's own copy is gone.
- Back to a spot within a page's end spacer lands at the page's end, not its top.
- A running push / revert is tracked per list: another list's push is no longer refused unseen.
2026-09-27 19:43:04 +02:00
peter e31a588f19 fix(menus): one tick for every choice; a listbox holds its options only
- The saved views, the view switcher, the year filter and the table's single-choice filter mark the
  chosen row with MenuCheck, as DESIGN asks, not a trailing tick, a dot or a filled row; a contract
  test holds every checkable menu row to it. keepSelectedPaint had no caller left and is gone.
- A combobox's panel carries its listbox role on an inner PopoverList (listInside), so the failed
  load, Loading, No matches and the 'n more' line sit beside the options, not inside the listbox. The
  Plex search box's Loading line likewise.
- The Plex sidebar's playlist picker offers Create only once the lists have loaded: over a failed load
  it pushed the error and its Retry out at the first keystroke.
2026-09-27 19:43:04 +02:00
peter cda626be10 fix(player): the playing film restarts in place; arrows stay the volume; the bands' holds end
- The info overlay's collection strip listing the film itself: Play/Restart restart or resume it in
  place instead of pausing it on a frozen 0:00.
- Up/Down on the closed ⋯ and tracks buttons stay the volume (openOnArrowKey: false).
- A keyboard focus in the bands holds them for 10 s after the last key, not the whole film; a tap
  leaving a tapped button no longer re-shows the bands its own toggle then hid.
- Next leaving under the focus hands it to its neighbour (the player keeps its focus).
- The chosen audio / subtitle track wears the MenuCheck tick.
2026-09-27 19:42:53 +02:00
peter b7ce1d467c fix(playlists): item writes lock the list first; one bulk pull per user; the claim on its own connection
- Every playlist item write (add, remove, move, reorder, rename; Plex remove and remove-bulk) row-locks
  the playlist before it touches an item, a move both lists in id order: remove beside reorder was a
  deadlock, and two adds of one video a unique-key 500.
- The scheduled mirror refresh re-reads dirty under that lock, so an edit made during its YouTube calls
  is not replaced by YouTube's order.
- A user's bulk pull holds a per-user advisory claim: two at once created duplicate mirrors. The manual
  sync answers 409 sync_all_busy instead of '0 synced', and like the re-pull spends its cooldown only
  on a pull that runs.
- Deleted mirrors are committed at once, not only when a later list commits.
- The claims open an unpooled connection of their own instead of a third pooled one per push.
2026-09-27 19:42:53 +02:00
peter 52fb999af6 feat(pickers): a list that failed to load offers Retry
A picker whose options failed to load said so, but the only way back was closing its dialog.
Combobox takes `onRetry`: a Retry beside the message for the pointer, and Enter in the field for
the keyboard (the panel is a listbox, which may hold options only, so the alert names Enter for
assistive tech). Every caller with an error state passes its query's refetch: both user pickers,
the channel picker and the two Plex pickers.
2026-09-27 17:47:01 +02:00
peter e0453fa07d fix(playlists): one YouTube sync of a list at a time, across tabs and the scheduler
The client's in-flight flag covers the menus of one page only. A push reads YouTube, then writes
both sides for seconds and commits step by step, so a row lock ends at its first commit: two tabs,
or a push and the scheduled mirror refresh, could interleave into a doubled YouTube item or a
baseline recorded over the other's writes. `sync_claim` takes a session-level advisory lock on a
connection of its own (it survives those commits; a crashed worker's lock dies with its
connection, so no claim column is left stuck). Push and revert answer 409 `sync_busy`; the
scheduled refresh skips a list being synced and picks it up next run.
2026-09-27 17:47:01 +02:00
peter 2525a12264 fix(keys): a key a focused widget already handled fires no shortcut
The player's settings tabs switched tab on an arrow and prevented its default, yet the player's
window listener still sought the video 10 s. dispatch() now skips a defaultPrevented key for
every scope; listNav's private copy of the same guard goes.
2026-09-27 17:28:10 +02:00
peter 8e09dd85f8 test: the store contract reads the last match without Array.at (the node tsconfig's lib) 2026-09-27 16:38:33 +02:00
peter addb3d1b32 fix(a11y): every side-panel pin names its panel; the tab under an open card leaves the tree
The card's own pin and the drawer's still said 'Unpin panel' / 'Pin panel open' after the edge tab
learned the title (F6). Naming them too made two buttons share a name, because the invisible tab
under an open card was still in the accessibility tree: it is now aria-hidden as well as out of
the Tab order. The generic sidebar.pin / unpin keys are gone.
2026-09-27 16:30:40 +02:00
peter 86884e368e fix(ui): the page's end scrolls out from under Back to top
The button floats over the bottom-right corner, and at the end of a long table it covered the closing
pager's '100' and 'All'. While it is up, PageScroller ends the page a button's height later (only
then, so a full-height layout gets no scrollbar). Its shown state is a shared store read through a
use… wrapper: a store's .use() called straight from a component is memoized by the React Compiler
and never re-renders (contract: store.contract.test.ts).
2026-09-27 16:28:18 +02:00
peter 51f881681b feat(tables): name a filter whose column is off screen, and let its chip clear it
A tabletHidden/narrowHidden column below its width, or any column in a card
layout, took its header funnel with it: the rows were narrowed by a filter
nobody could see. The cue is shown by the inverse of the column's own
container query, so it agrees with the table at every width and text size.
Wired into DataTable, the playlist table and the channel cards (E28 F2).
2026-09-27 16:22:26 +02:00
peter 27483ae1d6 fix(focus): a removed control's focus skips a neighbour the CSS has hidden
A display:none button refuses focus(), so handing the focus to a hidden
column's header button dropped it to <body>.
2026-09-27 16:22:26 +02:00
peter 595c29e17c style: prettier 2026-09-27 15:17:02 +02:00
peter acf93a1f36 test: the video menu's download row is 'Download…'; drop home.pin.menu (its menu is gone) 2026-09-27 15:13:19 +02:00
peter 84278293df docs(ui): the raw-button rule is 'outside the control primitives', and the README table is whole
CLAUDE.md, DESIGN.md and ui/README said a raw <button> is allowed anywhere in components/ui/; the
lint rule allows it only in BUTTON_PRIMITIVES (and refuses a bare FocusSafeButton elsewhere too).
The focus-safe rule's message no longer points feature code at FocusSafeButton. The README's
'actions & inputs' row was split in two by a stray line break.
2026-09-27 15:09:22 +02:00
peter dc24d80ed4 fix(player): a description link keeps its sentence's punctuation and its own parentheses
The trailing .,;:!?)] was trimmed off the href and dropped from the text too, and a URL ending in a
balanced ")" (a Wikipedia article) was cut short. The trim keeps a closing paren that has its
opener, and the trimmed tail is put back after the link.
2026-09-27 15:08:27 +02:00
peter 3810af66da fix(channels): every subscribe and unsubscribe refreshes the same reads
Each of the four sites kept its own invalidation list and each missed something (the channel menu's
unsubscribe left the feed showing the channel's videos; the channel page's left the counts stale).
lib/subscriptionCache invalidateSubscription is the one set; a contract test holds every call site
to it.
2026-09-27 15:07:39 +02:00
peter eb339195c7 feat(channels): the band's tag chips and status row fold on a short window (B4)
Every other page band folds its filter row below ~600px of height; the Channels band never did, so
its tag chip lane and status / pager row kept a short laptop window for themselves. They fold behind
one "Filters · n" (n = picked tags + other active column filters); the tabs and the TAGS editor stay,
and a folded table keeps its controls at the list's end.
2026-09-27 15:06:32 +02:00
peter 5d1065c1bc feat(playlists): a playlist's menu copies its name and its YouTube address
Every object's action list ends its middle group with Copy name / Copy link (E28 C1, the rows a
finger cannot select); the playlist's had neither.
2026-09-27 15:04:47 +02:00
peter 6ffb5bd8c1 refactor(pins): drop the unused ComfortPinMenu and the comment that described it
Every surface composes the pin row into its object's one action list (E28 C1); the pin-only menu
had no caller left. Also un-exports isStandInFocus, which only its own file reads since H1.
2026-09-27 15:03:28 +02:00
peter dddf79ec2f refactor(ui): FilterSelect and Combobox draw their checks with MenuCheck
They still painted their own 14px box beside the menus' 16px MenuCheck. A contract test holds the
filled-box paint to MenuCheck alone.
2026-09-27 15:02:42 +02:00
peter 5ebb4cb506 fix(a11y): chips, grips, the panel tab and Back to top keep the focus somewhere real
- F5: a chip's × pressed from the keyboard hands the focus to the next chip (else the previous one,
  else what follows) before it unmounts — in DismissibleChip, so all five chip rows get it
  (`focusNeighbour` in lib/useFocusKeeper).
- F4: a drag grip names what it lifts (`common.order.dragNamed`: panel groups, personal views);
  the two generic "Drag to reorder" keys are gone.
- F6: the side panel's edge tab is out of the Tab order while the card covers it, names its panel,
  and a keyboard pin moves the focus into the card.
- F7: Back to top is no Tab stop while hidden, and a keyboard press leaves the focus on the page.
2026-09-27 15:01:22 +02:00
peter 0b1d905866 fix(ui): Enter on a selectable title opens it even with its words selected
endsTextSelection takes the click event and exempts a keyboard click (detail 0): Enter / Space
ends no selection, so a title selected and then tabbed to opened nothing. All three callers
(FocusSafeButton, the Plex caption, the video card title) go through it.
2026-09-27 14:55:50 +02:00
peter 52c15adaf2 fix(downloads): a row whose remote picture fails too falls back to its glyph
The remote thumbnail was the last resort with no fallback of its own, so a dead CDN link drew the
browser's broken-image mark. A failed remote URL is remembered like a failed poster, and the row
shows its film / music glyph.
2026-09-27 14:54:28 +02:00
peter 3372626e34 fix(notifications): "Find in feed" drops every narrowing that could hide the video
locate() spread the reader's filters and cleared only the tag, search and age cap, so a year, a
custom range, a length, a content type (a hidden Short under "no Shorts") or the scope could still
leave the video out. It builds the filters from nothing now, keeping only the sort and tag modes.
(Also types the C1 test's pending plan.)
2026-09-27 14:53:24 +02:00
peter 98bb841a6a fix(playlists): one push or revert at a time for a list, whichever menu started it
The in-flight flag was per hook instance, and the rail row's menu and the list page header's "…"
are two at once: a push from one left the other offering a second push. It is one app-wide store
(`lib/store`), read with `get()` at the start so a second start refuses before a render; a revert
re-reads it after its confirm dialog.
2026-09-27 14:51:37 +02:00
peter 98bf0a9d66 fix(i18n): the video menu's download row says "Download…", and counted copy is pluralized
- The row opens the download dialog, so it is labelled so; the download's state ("Downloaded",
  "In queue") is its hint, not its label.
- The share dialog's expiry ("1 days") and a link's view count ("1 views") go through i18next's
  plural forms (`count`).
2026-09-27 14:50:04 +02:00
peter 17bb14fb18 fix(ui): a picker whose list failed says so, and takes no pick while one is being written
- Combobox gains `error` (said in the list, not "No matches") and a soft `disabled` (keeps the
  focus, announced, opens and changes nothing).
- UserPicker passes both: the share dialog's recipient picker is off while a share is pending, and
  the share and quota pickers say when the user list failed to load.
- ChannelPicker and the Plex playlist/collection pickers say a failed load too.
- The channel menu's Tags row says loading / failed instead of "no tags yet".
2026-09-27 14:47:46 +02:00
peter 70f98b352f fix(plex): a collection sibling is a full Plex card, and the player's info overlay plays it
- PlexInfo's collection strip gives each card the related strip's action set: the "…" menu (with
  Restart for a touch reader), the watched toggle and the playlist add.
- `onPlayItem` is required, so no variant draws a Play that does nothing; the player's info overlay
  plays the sibling in the running player (`go(id, _, fromZero)` for its Restart).
2026-09-27 14:44:38 +02:00
peter efa4aeeeda fix(plex): a card offers Restart by the menu's own "started" rule
The picture read "started" off the rounded progress %, so a resume point under 1 % showed Play
while the card's menu offered "from the start". Both use `plexStarted` now.
2026-09-27 14:38:27 +02:00
peter a7d41eb5f3 fix(plex): the player's settings panel is a tab list with named, touch-sized rows
- Its section switch is `ui/Tabs` (new `tone="media"` paint for the dark chrome) with a wired
  tabpanel; the panel opens with the focus on the selected tab.
- On/off rows are `CheckRow` (the whole row, 44 on touch), the "no background" checkbox too.
- Every slider and the four colour wells carry the name their row shows.
- The clock's date-style buttons reach the touch target.
2026-09-27 14:38:27 +02:00
peter 76eead976f fix(plex): the player's stage follows the tablet toolbar, and its bands stay out of a double-tap
- The stage is sized in dvh (80dvh × 1.25 windowed, 100dvh fullscreen); the viewport-units
  contract no longer exempts the player.
- A tap that reveals the bands keeps them pointer-events-none for the double-tap window, so the
  second tap of a skip at the bottom is not a seek.
- An arrow on the volume slider wakes the bands, and a keyboard focus in a band holds them
  (`holdOnFocus`, `:focus-visible` only, so a clicked button does not pin the bar).
- Next stays mounted (disabled) while the next episode's detail loads, keeping the focus.
2026-09-27 14:34:31 +02:00
peter 922d589b09 fix(ui): Tabs is an ARIA tab list with roving focus and a wired panel
The rail was a row of aria-pressed toggles: every tab a Tab stop, no arrows,
nothing tying a tab to its content. tablist/tab/aria-selected, one Tab stop,
arrows on either axis, Home/End, and useTabPanel for aria-controls/labelledby.
2026-09-27 14:22:37 +02:00
peter a975ab627a fix(plex): the player's ⋯ and tracks lists are real menus, and every close returns the focus
They were hand-built divs: no menu roles, no arrow keys, and Escape left the
focus on <body>. Both run on the popover engine now (media-tone rows), and the
tuning panel is a labelled dialog that takes the focus and hands it back to ⋯.
2026-09-27 14:22:37 +02:00
peter 9bedd96774 fix(nav): Back restores the focus after a click on the page's blank area
The page region is <main tabIndex=-1>, so a background click focused it and the
Back restore took that for the reader's choice. A resting place is marked and
one predicate (focusIsUnclaimed) decides "the focus is nowhere" everywhere.
2026-09-27 14:06:16 +02:00
peter 383c85231a fix(plex): a Restart does not replay on a reload or a Forward onto the player
fromStart rode in history.state._sub, so reloading the player entry restarted
the item from 0 again and the progress checkpoint overwrote its resume point.
A sub-view config can now say what the history entry keeps of a view.
2026-09-27 14:03:48 +02:00
peter de62035889 fix(playlists): an Undo puts a row back after its old neighbour, not at a stale index
A raw index recorded at removal drifted once anything changed before the Undo.
Both lists now read the list fresh and re-insert each row after the row that
preceded it (lib/reinsert). A failed watched toggle in a Plex playlist also
puts the row's state back: its local copy was never re-seeded.
2026-09-27 14:02:15 +02:00
peter b456232d00 fix(playlists): lock the playlist row before reading and writing item positions
Two adds read the same max(position) and a reorder racing an add collided with
it. Every add/insert/reorder takes the parent row FOR UPDATE; the YouTube list
breaks position ties by id and a Plex reorder renumbers rows it was not sent.
2026-09-27 13:58:23 +02:00
peter c9e95ecea6 fix(channels): a channel YouTube no longer returns is not asked for on every view
channels.list simply omits a terminated channel, so nothing was stamped and the
channel page spent a quota unit on it every time. The miss is stamped like a
fetch; the published_at sentinel that kept it stale is gone.
2026-09-27 13:58:23 +02:00
peter 7987bd8a37 fix(plex): a stray byte keeps a UTF-8 subtitle UTF-8; untagged and Serbian read as cp1250
One invalid byte sent a whole UTF-8 sidecar to a legacy code page, an untagged
file fell to Windows-1252 (ő/ű became õ/û) and Serbian was read as Cyrillic.
The cache key moves to sub3_ so a track the old rules mis-decoded is read again.
2026-09-27 13:58:23 +02:00
peter 3411265dc1 docs(design): tablet columns, glyph pager, match switch, missing picture, Copy 2026-09-27 07:12:51 +02:00
peter 098b8dbe43 feat(downloads): a bare link is titled once, a missing picture shows a glyph
A job with no name yet is titled by its short link, as the link itself;
the channel line no longer falls back to the ref and the link line does
not repeat it. No picture at all draws a film (or note) glyph instead of
an empty box.
2026-09-27 07:12:51 +02:00
peter f48f16830e feat(filters): the any/all switch sits on the panel above its picker
ui/MatchToggle replaces the Combobox popover footer (Feed tags, topics,
languages; Plex collections) and PlexSidebar's own toggle (genres,
actors): shown from two picks, one wording. A closed picker had hidden
that "all" was on.
2026-09-27 07:12:51 +02:00
peter 955c23d321 feat(tables): a glyph pager with first/last, and tablet-width columns step aside
The pager is four named arrows round the page box (first, previous,
next, last). A table hides a tabletHidden column below 64rem of its own
width, as narrowHidden does below 42rem, through one shared helper that
also hides the column's <col>. The Channels table drops Prio, About and
Length there: at 1024 the channel name had 60 px.
2026-09-27 07:12:51 +02:00
peter d3e1d8219c docs(design): Fjord stays a fifth scheme beside Midnight
The trial is over: the owner kept it. The trial wording goes from the
tokens, the scheme list, the lab and DESIGN; B2's own-hue rule still
binds any new scheme.
2026-09-27 06:46:52 +02:00
peter 4480839db0 merge: bug/channel-details-refresh into E28
# Conflicts:
#	frontend/src/components/ChannelPage.tsx
2026-09-27 06:42:37 +02:00
peter 8e25567ff2 merge: bug/hu-subtitle-encoding into E28 2026-09-27 06:42:08 +02:00
peter 8f65bd184e fix(i18n): a locale edit hot-swaps the texts instead of re-running main.tsx
The locale JSON's importer did not accept HMR, so the update climbed to
main.tsx, whose re-run called createRoot on the live container again; the two
roots tore each other's DOM down ('removeChild ... not a child' by the dozen),
each a fatal toast evicting an old notification. Dev only.
2026-09-27 06:35:40 +02:00
peter ae9c248987 fix(downloads): Retry draws the refresh glyph, not Play (E28 S4)
The first UI-review run (Downloads) found a failed row whose Retry wore the
Play glyph in the row and its menu, promising a video. DESIGN.md gains
'a glyph means one thing' and a log row.
2026-09-27 06:35:40 +02:00
peter bd6aaa1e43 feat(design): the Copy rule — names, verbs, dates and units (E28 S3g)
The owner's picks (all the recommended options) become DESIGN.md's Copy
chapter and a sweep of both languages (742 values, no key drift):

- One name per feature: the rail's label, which may shorten it (Alerts
  -> Notifications, Fiókok -> Felhasználók, Download Center -> Downloads,
  Channel manager -> Channels); one "sync from YouTube" verb.
- Four verbs, four meanings (Delete / Remove / Dismiss / Discard, and
  their Hungarian pairs), checked against what each action does.
- Sentence case, the app's places capitalised; "…" on a command that
  asks first (a popover or form title got its own key without it);
  one-sentence messages without a stop or a tick, "Couldn't …"; curly
  quotes.
- Dates only from lib/format.ts, in the UI language: formatDateTime
  names the month and stops at the minute (it printed "9/27/2026,
  8:30:14 AM"), formatDateTimeSec for the log, the audit and the
  recorder; ui/RelativeTime pairs "3d ago" with the exact minute on
  hover. local/date-format refuses toLocale*String and Intl date
  formatters elsewhere.
- Units from Intl, short (formatUnit): runtimes, totals, the Scheduler
  countdown, the stats range, the player's tap skip. "2h 13m" was
  English in both languages.
- src/i18n/copyRule.test.ts holds the machine-readable part (quotes,
  "...", ticks, bare unit letters).
2026-09-27 05:46:17 +02:00
peter 2c65edfffd fix(downloads): the failure link keeps the 24px floor on a mouse (E28 S3g)
The S3g measure found the row's failure link 20px tall (the meta line's
text height): DESIGN B1 puts nothing under 24. Coarse pointers already
grew it to 44.
2026-09-27 05:13:32 +02:00
peter 6c2c01e67b fix(design): the S3g conformance sweep — primitives at their rung (E28 S3g)
The probe re-run over the 15 pages at 1920x1080 found every button, icon
button, chip, field, tab, segment and menu row drawn by its primitive
except the S3h Plex picture, a div with the button role. It is a
Pressable now, and the button-primitive lint rule refuses an element
with role="button" outside the control primitives.

- SegmentedToggle defaults to the toolbar rung (28, text-xs): the old
  small default had put the pager's page sizes and the YouTube result
  count on 10px type beside 28px buttons. The Plex match toggle asks
  for the small rung explicitly.
- The language badge on the rail uses text-3xs (was an 8px one-off);
  the Plex playlist's watched mark and the Home hero's length use
  MEDIA_BADGE instead of hand-copied classes.
- A media picture shows the pointer whether it is a link or a button.
- DESIGN.md: a Conformance chapter (the four-step check before a UI
  change is done, with the S3g baseline), the segmented rung, two log
  rows. CLAUDE.md points at the check.
2026-09-27 05:11:34 +02:00
peter ef343dbec0 feat(design): side panels open as a drawer on touch and below 768 (E28 S3f3)
C2 as decided: on a coarse pointer, or a window below 768, the panel's
tab (shown at every width, 44 wide on touch) opens the panel as a
start-edge drawer over the content, with a scrim. It is ui/Modal's new
drawer-start placement, so Esc, Back, the scrim, the focus trap and the
scroll lock are the dialog's own. A tap never peeks; pinning is a button
in the drawer's header, offered only at 1024 and up; a pin saved on a
desktop is left alone below that. A mouse on a wide window keeps
hover-to-peek and click-to-pin.

DESIGN: C2 as built, rows answer to their own width, phone-only layout
deferred to its own epic (principle 2).
2026-09-27 04:38:42 +02:00
peter a58a398427 feat(design): the three 375 rows (E28 S3f3)
- Download row: answers to its own width (container query); below 40rem
  it stacks: picture beside a two-line title, the meta line across the
  row (one line, never wraps: zoomed on the iPad its pieces broke apart),
  then the source link and the actions on one wrapping line. The paste
  field takes a line of its own when the buttons would squeeze it.
- YT playlist row: below 42rem the length rides on the picture and the
  title runs two lines over the channel.
- Channels tag row: the suggestions take a line of their own below
  40rem (the chip lane was 0 px and the row's menu off the screen).
- Pager: on a phone the arrows drop their words (names kept), and the
  size label stays one line.
- Lint: a class split where the interpolation is the variant itself
  (${NARROW}:w-24) is refused too; it had shipped with no CSS.
2026-09-27 04:15:08 +02:00
peter d510c8a534 fix(plex): the show and season restart names its object and starts at 0 (E28 S3f2)
The header button said only "Play from start" and resumed the first
episode at its saved point. It now reads "Season from the start · S6 · E1"
or "Show from the start · S1 · E1" and asks the player for 0.
2026-09-27 03:30:04 +02:00
peter dc5206f9d3 docs(design): C3 + C5 as built, and its log row (E28 S3f2) 2026-09-27 02:12:54 +02:00
peter 89e65ca331 feat(design): players on touch and the C5 control set, restart on the card (E28 S3f2)
Plex player: a finger's tap on the picture only shows or hides the controls,
the hide timer starts with playback, a double-tap on a side third skips 10 s,
the seek bar is a 2.75rem touch-none hit area with its head shown on touch.
The row is play/pause, Back by position, Next only with a next item, mute,
time, tracks, a fixed ⋯ (settings, media info, stats, shortcuts, download)
and fullscreen; Stop, Restart and the ±10 s buttons are gone.

A started card shows the video card's Continue + Restart pair (one shared
MediaResumeButtons, beside the picture's button), and its menu lists the
restart; the player opens that item at 0 (fromStart). YouTube modal: the
description opens from an ⓘ, the queue arrows sit inside the screen.
2026-09-27 02:12:54 +02:00
peter 17916b411a docs(design): C4 as built, and its log row (E28 S3f1) 2026-09-27 01:33:37 +02:00
peter 3ef2c38eb4 feat(design): dialogs open by purpose and keep their commit row pinned (E28 S3f1)
Modal takes a required initialFocus (field / close / action), a field that
loads late takes the focus while it still rests on the close button, and
ModalFooter portals a dialog's commit row under the scrolling body from any
depth. The header and footer fades follow the body's scroll timeline, and the
viewport's interactive-widget=resizes-content keeps the footer above an
on-screen keyboard.
2026-09-27 01:33:37 +02:00
peter 4c63d6660d docs(design): the media card chapter and its log rows (E28 S3h) 2026-09-27 00:54:28 +02:00
peter a453b77f04 feat(design): one media card, and Home's hero and shelves as cards (E28 S3h)
- ui/MediaCard + mediaCardStyles: one frame for every video, Plex title and
  Home tile; the meta line pinned to the bottom edge, the length a badge on
  the picture, the media Play only where the click plays, the hover row at
  the picture's bottom-right (the owner's pick C).
- Shelf density: on any strip a one-line title over one meta line; mixed
  Home shelves are landscape; rails turn content-visibility off (a card's
  reserved height had become the strip's). Home slot 232px, so the Play
  and the hover row no longer touch; a landscape shelf takes the small Play.
- Home hero is a card (hover row, the whole menu, a touch "…"), three
  quarters wide and centred; channel / playlist tiles get Play, open and
  "…"; a show tile acts on its next episode, ⓘ opens the show.
- Plex preview: a ~0.9 s mouse rest opens a passive panel with the whole
  title, facts and synopsis (popover `pointerOnly`); the synopsis and title
  tooltips are gone.
- Compact list row B: the title leads, "…" at the end, toggles over the
  date on hover.
- Media-info layout B (collections, crew, cast beside the facts sidebar;
  an empty column is not drawn); filtering values are chips there and on
  the show page; "Cast" over the cast-only strips.
- Browse tiles hover like a card; prose capped by the `measure` utility.
2026-09-27 00:54:19 +02:00
peter 8e1004fe6f fix(channel): load the banner without a referrer
Google's image CDN refuses a banner request that carries a Referer from our origin - every
time, not only under load (measured: 6/6 refused with it, 8/8 served without). The banner
<img> now sends none, as ui/Avatar does for the same CDN; the comments that blamed a
transient burst throttle are corrected.
2026-09-26 21:45:46 +02:00
peter 663f30feeb fix(sync): refresh a channel's details once they are a week old
Channel metadata (banner, avatar, description, counts) was fetched only the first time a
channel was seen: the subscription import asked only for channels with no details, and the
channel page only for ones missing a join date. 394 of 397 subscribed channels had July
metadata. Both now also refresh details older than DETAILS_MAX_AGE (7 days), through one
shared staleness rule; channels.list costs 1 unit per 50 channels.
2026-09-26 21:45:46 +02:00
peter 6653d81a80 fix(channel): no broken-image glyph while the banner retries (E28)
The banner is retried on error; each failed attempt showed the browser's
broken-image glyph for as long as the retry waited. The <img> now stays
invisible until its source has actually loaded.
2026-09-26 20:55:34 +02:00
peter 21df7c8348 docs(design): the background menu as built, and the log row (E28 S3e4c) 2026-09-26 20:48:53 +02:00
peter 97381e65d0 feat(design): a page's background menu on the five list pages (E28 S3e4c)
A right-click on the EMPTY area of a page opens a menu of that page's own
toolbar: the feed (sort, reshuffle, view, watch state, content, source, clear
filters), the Plex library grid (sort, watch state, clear), the channel manager
(layout, the cards' sort, sync status, edit tags, clear the tag filter, read
subscriptions), a playlist (play all, group by, reset view, save order,
undo/redo, the list's own menu) and the notification center (mark all read).
No destructive row and no backfill.

PageShell listens on the page area; lib/backgroundTarget decides empty (not
text, a control, a link, media, or a row/card with its own menu). Shift+right-
click, a touch long-press and a right-click on selected words keep the
browser's menu. ui/PageMenu registers while mounted, so a loading or error
state keeps the browser's menu too. The rows mirror each control through
ui/pageMenuItems (sortSubmenu via pickSort, choiceSubmenu, multiSubmenu).
pageMenuScope.contract.test.ts pins the five pages.
2026-09-26 20:48:53 +02:00
peter f8ced6d819 docs(design): the flush-row ring, the fade at rest, the narrow table, and their log rows (E28) 2026-09-26 20:01:04 +02:00
peter f254588409 fix(playlists): hide '#' and Channel by the table's width, not the window's (E28)
Beside the playlist rail an 800 px window left the table 432 px: the fixed
columns took all of it and the title column was 0 px wide, its thumbnail
drawn over the channel. The table is an @container now and the columns hide
under 42rem of its own width (narrowHidden, was phoneHidden).
2026-09-26 20:01:04 +02:00
peter 3e7c79fa69 fix(design): the keyboard ring of a flush row stays visible (E28)
A list item's ring sat 2 px outside it. On a table row it ran under the
sticky head and past the table's sides; on a Plex playlist show header, flush
in its clipping card, it was cut away and the focus was invisible. A table row
(by its tag), a log line and the Plex show header and episode rows now ring
inside themselves, the show header with its card's corners.

The sticky header's fade-below was drawn at rest, over the first row's top
16 px, hiding the ring's top edge: it now fades in with the first 1rem of
scroll (a scroll-driven animation; no fade where unsupported).
2026-09-26 20:01:04 +02:00
peter 47c12f9405 docs(design): right-click on a selection, and the log row (E28 S3e4b) 2026-09-26 19:03:34 +02:00
peter bd0914b7ed fix(design): a right-click on selected words opens the browser's own menu (E28 S3e4b)
The row menu took every right-click, so selected text could only be copied
with Ctrl+C. On the selection (its laid-out line boxes) the event is left to
the browser; beside it the object's menu opens as before.
2026-09-26 19:03:34 +02:00
peter 079cee4e9c docs(design): titles in click surfaces stay selectable, and the log row (E28 S3e4b) 2026-09-26 18:46:24 +02:00
peter edf2f22ef9 feat(design): titles in click surfaces stay selectable (E28 S3e4b)
A button whose text is a title or a name is now `selectable` (FocusSafeButton,
under Pressable and Button): Chrome selects inside a button once it is
user-select: text (measured), and the click that ends a selection, or a press
dragged past the slop, opens nothing. 23 surfaces carry it (cards, rows, people,
channels, playlists, users, views, messages, log lines, description links); the
Plex caption div gets the same guard. A contract test pins every plain Pressable
with its reason, so a new one has to choose.
2026-09-26 18:46:24 +02:00
peter e1ef56cdbc docs(design): the non-drag reorder forms, and the log row (E28 S3e4a) 2026-09-26 18:14:28 +02:00
peter 4d0abc61d5 feat(design): reorder without a drag everywhere a grip drags (WCAG 2.5.7, E28 S3e4a)
One rule, two forms: useOrderItems (Top / Up / Down / Bottom rows of a row
menu) and OrderButtons (four buttons with the place, or a compact Up / Down
pair). A move that would go nowhere is shown off.

- YouTube playlist rows: the four moves now come from useOrderItems.
- Plex playlist rows: a block among the blocks, an episode within its show,
  off while a write is in flight; drag and menu share moveBlock/moveEpisode.
- Saved views editor: the selected view's four buttons (owner: A), and the
  grip's keyboard pick-up (its instructions promised one it did not have).
- Side panel groups in edit mode: an Up / Down pair (owner: A); a group the
  stored order does not hold yet moves too; the name wraps there.
- The grip's touch-action: none is now test-held.
2026-09-26 18:14:28 +02:00
peter 7c2b22bafa docs(design): the channel menu's Tags and Priority, MenuCheck, and the log row (E28 S3e3b) 2026-09-26 17:34:47 +02:00
peter 5e5c8a42a9 feat(design): the channel menu's Tags and Priority submenus (C1, E28 S3e3b)
Tags A-Z as in-place checks (the menu stays open for the next); Priority
as the fixed steps 10 5 3 2 1 0, an off-scale current value ticked in its
place. A pick re-sorts the list at once, moving only that row
(placeByPriority, the server's order), and the table / card pager and
the focus follow it (DataTable and useCardPager take a reveal seq,
listNav.focusNavItem lands on the row's own "…", a virtualized card
revealed first). ActionMenu rows take check: many | one. The menu reads
the row fresh from the cache and keys its target by id.
2026-09-26 17:34:47 +02:00
peter 2b3094ffc5 refactor(design): one drawn check for every checkable row that cannot be an input (E28 S3e3b)
ui/MenuCheck: a box filled with the accent for a set (a minus for a group
only partly in), a bare tick for one choice, in the row's leading slot.
Replaces seven hand-drawn marks: three box sizes, a tinted box, a cross
for "on" in the table filter, a trailing tick in the language menu.
2026-09-26 17:34:46 +02:00
peter 9c91eb5d22 feat(design): a library row keeps Share (the link copy) in sight, ahead of save (E28 S3e3)
The owner's call: sharing a download's link is the library's main errand, more
important than saving the file, so it does not hide in the row's "…". Manage
sharing stays in the menu. The "…" tests press as a mouse, so a stray long-press
timer no longer fires after teardown.
2026-09-26 07:21:21 +02:00
peter 2df60fbf07 docs(design): how the download, notification and Scheduler action lists are built, and the log row (E28 S3e3) 2026-09-26 07:10:02 +02:00
peter a2d17d693c feat(design): the download, notification and Scheduler action lists (C1) — the rows' "…", the band's and the strip's "…" (E28 S3e3)
A download row keeps play (pause / resume / retry in the queue) and the save link; the editors,
sharing, re-download, the copies and cancel / delete / remove moved into its "…"
(useDownloadActions). A link inside the row keeps the browser's own right-click menu. A
notification row keeps its link and its unread mark and shows its dismiss on hover; an activity
notice's Find in feed and Undo are in its "…". The inbox band keeps Mark all read and puts Clear
everything, now behind a confirm, in its "…". The Scheduler strip keeps Run all and Pause and puts
Purge discovery in its "…". Polled lists key their menus by id (keyOf). ui/menuGroups draws a
rule only between two non-empty groups. Find in feed also drops the tag, search and age filters
that could hide the video it looks for.
2026-09-26 07:10:02 +02:00
peter c4917619ee fix(design): a row's "…" closes its own menu when its row object is rebuilt (E28 S3e2)
The "…" told its own menu from another row's by payload identity. The Plex playlist page builds
each row's menu target afresh on every render, so a second press on the same "…" reopened the
menu instead of closing it. useContextMenu takes an optional `keyOf`, and the page keys its
targets by the row key.
2026-09-26 07:10:02 +02:00
peter 760bdae278 docs(design): how a channel's and a Plex playlist's action lists are built, and the log row (E28 S3e2) 2026-09-26 05:56:08 +02:00
peter 1b946113c6 feat(design): a channel's and a Plex playlist's one action list (C1) — the row's "…", Delete + undo, an action toast's 8 s floor (E28 S3e2)
- Channels: the row's full-history, hide and unsubscribe buttons become its "…" at the row end
  (useChannelActions: open, pin, YouTube, copy name / link, full history or the admin reset,
  hide with an undo toast, unsubscribe behind its danger confirm); the card layout docks the "…"
  beside the name (ChannelLayoutGrid moreKey). The admin reset's confirm now names the reset.
- Plex playlist page: a title, a show group and a season each open the page's one row menu (a
  title's is its usePlexCardActions list plus the removal); no X at rest; removals take an undo
  toast whose Undo puts every row back at its own index in one request (bulk add `positions`);
  j/k walk the visible rows, Delete removes the walked one; rename and delete in the header's "…".
- notify: a toast with an action lives at least 8 s (the owner's call), whatever the toast setting.
2026-09-26 05:56:07 +02:00
peter b0e51c6f89 docs(design): how a playlist's action list is built, and the log row (E28 S3e2) 2026-09-26 04:05:39 +02:00
peter 7455fff401 feat(design): a playlist's one action list (C1) — the row's "…", Delete + undo, the header's "…" (E28 S3e2)
- Playlist rows: the X column became the row's "…" (play, pin, copy title/link, move, order,
  remove). Remove runs at once, with no confirm; its toast's Undo puts the row back at its index.
  The Delete key removes a keyboard-walked row, and focus moves on to the next.
- POST /api/playlists/{id}/items takes an optional `position`, so an undo is one request that
  renumbers the list around the row.
- A new `listDelete` key scope. Only a list with `onDelete` registers it, so the help sheet does not
  offer Delete where nothing handles it.
- A playlist has one list, `usePlaylistActions.items`: rename, pin, YouTube open/push/revert,
  delete. The rail row's "…" and the page header's "…" both open it. Push and revert moved into
  the hook; the header keeps Play all.
- Phone-width playlist table: "#" and Channel are hidden below sm (`Column.phoneHidden`). Their
  fixed widths had left the title no width.
2026-09-26 04:05:39 +02:00
peter aaf1843b09 docs(design): how a Plex title's action list is built, and the log row (E28 S3e2) 2026-09-26 01:55:29 +02:00
peter a4737b92e4 feat(design): a Plex title's one action list (C1) — the poster's "…", details and the pin in its menu (E28 S3e2)
Every Plex poster list (library, Discover rails, person, show, season, collection, Home's rails)
now opens one menu per list from the right-click, the long-press, Shift+F10 and the card's "…":
play/open, watched, add to playlist, media info, copy title, the pin (usePlexCardActions +
PlexMenu, replacing the pin-only PlexPinMenu). The card keeps its state at rest and shows
watched / add to playlist / "…" on its picture on hover (ui/reveal.ts, shared with the video
card); on touch the toggles are gone and a 44px "…" sits beside the title, while the watched pill
stays. The hover ⓘ left the card; an episode's add button moved from its caption to the hover
row. Seasons get their own short menu (open, watched, add, copy), labelled "Open season" (the
card's overlay said "Open show"). Related shows are ordinary poster cards now. A collection's
admin "Remove" X moved into its menu, still behind the confirmation. The collection grid is two
columns on a phone. The unused .reveal-idle rule is gone.
2026-09-26 01:51:53 +02:00
peter 75ae810866 feat(design): a video's one action list (C1) — the card's "…", the menu holds the rest (E28 S3e1)
- useContextMenu gains buttonProps (a row's visible "…" opens the same menu under the button; a
  second press closes it) and anchor (the element the menu came from, for a panel a row opens).
- ui/MoreButton; ActionMenu rows can run inGesture (a clipboard write inside the click, Safari).
- useVideoActions: one builder for a video's menu (play / restart, channel, watched, save, add to
  playlist, download, details, copy title / link, hide, reset, pin), composed by the feed and by
  Home's rails; the playlists and details panels open anchored beside the card (VideoFollowUp).
- The card keeps watched / save / add to playlist and the "…": on the picture on hover (the
  duration steps aside), in the rows' action column; on touch only a 44px "…" beside the title.
  Download, hide, reset and the hover info card leave the card.
- The title is selectable: a click ending a selection neither plays nor follows the link.
- DESIGN.md: how C1 is built + the log row.
2026-09-26 01:23:44 +02:00
peter 9d7526ee57 fix(feed): a reset clears the card's resume position at once, not only its status
The optimistic layer overrode only the status, so after "Reset — clear watch progress/status" the
card kept its progress bar and its Continue button until the next refetch. Found in the E28 S3e1
UAT.
2026-09-26 01:23:44 +02:00
peter fa0b0edd63 docs(design): how B4, B9 and the viewport rule are built, and the log rows (E28 S3d4) 2026-09-26 00:39:10 +02:00
peter bfc6d199c0 fix(a11y): a video card's thumbnail link is named by the title, not the duration badge (E28 S3d4)
Its only text was the duration, so every card's first Tab stop was announced as "44:10".
2026-09-26 00:39:10 +02:00
peter 4d6e59b80a feat(design): the public watch page on the app's surfaces (B6) — the scheme backdrop, the title and actions on a glass panel, a glass Download (E28 S3d4) 2026-09-26 00:39:10 +02:00
peter 4646aae320 fix(design): the table header takes the B9 fade utility, and every size is dvh, not vh (E28 S3d4)
The table header carried an arbitrary-value copy of fade-below; a contract refuses a hand gradient of a surface colour. 100vh is the viewport with a tablet browser's toolbar retracted, so h-screen / min-h-screen / static vh hid the last rows under it; a contract holds dvh (the Plex player stage changes with S3f, where it is played on the tablet).
2026-09-26 00:39:10 +02:00
peter eddeca82d3 feat(design): the page band (B4) — one h1 per page, section headings, skip to content, filters fold on a short window (E28 S3d4)
- PageShell names the page with a visually hidden h1; a view with its own title renders it as
  PageHeading and the hidden one steps aside (contract: no hand <h1> inside the shell).
- Section captions are SectionLabel headings (h2, h3 in a dialog), ui/form Section included.
- "Skip to content" is the first Tab stop and focuses the scroller's <main>.
- Below 600px of height the band's filter row folds into "Filters · n" (Feed, Plex, server log),
  the same row shown in place.
- The band holds controls only: Downloads and Notifications lose their title cards, the Config and
  channel-discovery intros scroll with the content, the Stats tabs and the audit pager join the band,
  and the server log's toolbar sticks with the fade under it.
2026-09-26 00:39:01 +02:00
peter 69edeecc45 fix(a11y): a labelled rail row says its label and the exact unread count, not the capped badge first (E28)
With the labels on, the row's name was its text, badge first: 15 unread was heard as "9+ Alerts". The
name is now the visible label plus the count, as the icon-only row's is, and the drawn badge is hidden.
2026-09-25 23:59:20 +02:00
peter 6088e476a7 fix(design): the command palette's active row takes the menus' roving highlight, not an invisible bg-card; the Plex typeahead's hand copy of it goes (E28)
The palette row was ~1.07:1 on its glass (B3 names bg-card hovers a violation). ROVING_HIGHLIGHT is
exported from AnchoredPopover for the listboxes that are not PopoverItems; a contract holds that no file
paints the fill itself and every hand-built option row takes the recipe.
2026-09-25 23:51:39 +02:00
peter ce171fa954 fix(feed): a custom date bound applies once its year is a plausible upload year, not per typed digit (E28)
A native date field reports a complete date on every keystroke of its year (0002, 0020, 0202, 2026),
so one typed year refetched the feed four times. The field holds a draft while focused; leaving it
empty clears the bound, leaving it half-typed restores it.
2026-09-25 23:43:26 +02:00
peter 113da76c36 fix(design): a user row answers as a whole card — the hover layer under its badges and buttons, a click anywhere opens the account (E28)
The layer was the name button's own and ended at the row's first badge or button. The card now paints
the B3 layer under its content (isolate + a new z-under level), the name button's hit layer spans the
card, and the actions sit above it.
2026-09-25 23:43:26 +02:00
peter 4ef11901db fix(design): the nav rail's footer rows (language, About, Sign out) share the rail row and glyph (E28) 2026-09-25 17:21:01 +02:00
peter 29abf609f3 fix(design): the lab draws with the primitives' own classes, not a height-ladder copy; DESIGN no longer claims accent-color tints the colour well (E28) 2026-09-25 17:15:03 +02:00
peter e13d93a3ed fix(design): a toggle is pressed, never a bare aria-pressed; static labels are flat Badges with a tone, edged label spans lint-refused (E28) 2026-09-25 17:04:08 +02:00
peter b9850b1710 feat(release-notes): terse bilingual notes as data files — new/fixed only, the dialog lists the latest ten (E28) 2026-09-25 15:49:16 +02:00
peter 905a9f2856 feat(design): Fjord, a trial fifth scheme — Midnight's navy with a steel-blue accent and steel rules (E28) 2026-09-25 15:39:49 +02:00
peter 1d8b83c37c fix(plex): a sidecar subtitle in a legacy code page rendered its accents as U+FFFD
A Hungarian .srt beside the video (Windows-1250) was decoded as UTF-8 with
errors=replace: one episode served 1391 U+FFFD and no accented letter.
decode_subtitle now lets a BOM decide, keeps valid UTF-8, and otherwise
reads the legacy code page of the stream's language (cp1250 for Central
European, cp1251 for Cyrillic, else cp1252) — on the ffmpeg path too.

The cache has no eviction, so its key is versioned (sub2_): a track cached
broken is read again once instead of being served for good.
2026-09-25 15:20:31 +02:00
peter 62dbf35ea3 fix(player): the Plex bars hold while hovered, volume is the native slider, tooltips no longer re-layer the row (E28)
- F22: the control bar faded out from under a mouse parked on it; the hide timer
  now waits while the pointer is on either band and re-arms on leave (test seen
  RED by disabling the hold).
- The hand-drawn volume bar broke DESIGN.md A6 (native sliders): it had no hover,
  focus, keyboard or screen-reader state. It is the Slider primitive now; a
  pointer drag blurs it so Space and the arrows keep driving the player.
- The control tooltips are composited from mount (will-change: opacity): a fade
  that promoted one on hover re-layered its neighbours under the player's
  scale(1.25), and the whole row shimmered.
2026-09-25 14:48:45 +02:00
peter e9ccd9bc98 fix(player): the YouTube player's full window filled only its card — a blurred ancestor trapped the fixed stage (E28)
E28 S3d1 moved the modal's card from `.glass-card` (no blur) to the `.glass` panel role, whose
`backdrop-filter` makes the card the containing block of the fixed "full window" stage: F showed a
894×161 strip inside the card with a scrolling, wrapped control bar. While the stage is maximised the
card now drops its blur (`html .glass:has(> .player-stage--max)`, outranking the dark theme's rule);
measured live: the stage 1268×861 = the viewport, the blur back on exit.
2026-09-25 14:20:23 +02:00
peter cc8e0648db fix(design): the button and chip hover colours were never generated — an act: variant instead of a runtime prefix (E28)
buttonStyles/chipStyles glued the hover guard on at runtime (`${ACT}hover:text-fg`). Tailwind emits CSS
only for a class it finds whole in the source, so all 22 guarded hovers shipped with no rule: no text or
edge colour on hover for ghost/outline/danger buttons, no white layer on the player's control row, no
underline on a link button, no accent hover on token/suggest/ghost chips. The stylesheet had 0 rules
for `not-aria-disabled`.

- `@custom-variant act` (index.css): the same guard as the state layers; `:disabled` never matches an
  <a>, so one variant serves buttons and links and the per-element variant maps go.
- ui/tailwindScan.contract.test.ts: every class the primitives can emit stands whole in their source.
- a lint rule refuses a variant right after an interpolation; the `enabled:hover:` rule now points at
  `act:hover:`.
2026-09-25 14:13:51 +02:00
peter ff80cc96fd fix(design): rating and live colour tokens, the player's menus over its notices, the info overlay on a light theme (E28)
- F16: --rating (theme-scoped gold, 4.5:1 on the page and its chip), --rating-media, --live/--live-fg,
  --media-chrome replace 14 palette literals; a lint rule refuses palette classes; the theme contract
  measures them.
- The Plex player's notices drew over its open settings/tracks popovers: notices z-base, popovers z-menu.
- The player's info overlay rendered theme tokens on its always-dark panel: on a light theme the score
  (2.6:1), collection years (3.2:1) and cast initials (1.05:1) were unreadable; onDark variants now.
- The lint-rule contract tests share one ESLint with a one-off warm-up (a cold first lint took 8 s,
  past vitest's 5 s).
2026-09-25 13:55:26 +02:00
peter 295c1db5f1 fix(design): claim audit part 1b — whole-row switches, the field-family edge, state-layer hovers, icon links, reason-disabled controls, 24/44 targets (E28)
- F12 (owner: A): SwitchRow, the whole settings row toggles; a Config boolean toggles from its title
  and hint (a label pointed at the switch).
- F14 (owner: B): the form dropdown trigger wears the field frame (see-through, the >= 3:1 edge);
  labelled pills keep the faint edge and the accent-edge hover. DESIGN says so, with log rows.
- F6: ~15 bg-card / bg-border / opacity hovers onto the B3 state layer; a lint rule (HOVER_SWAP)
  refuses the swap, contract-tested.
- F10: bare 14px icon links onto IconLink / LinkButton; a local lint rule refuses a glyph-only <a>.
- F13: reason-disabled controls are aria-disabled (Users, Channels backfill, Discovery, Scheduler
  runs, Config tests); FocusSafeButton now swallows the click of a caller's aria-disabled; the
  Config tests say why while dirty (testDirty, HU/EN).
- F5/F11: every control measured live at 24 (mouse) and 44 (coarse) across every page; touch-target
  is a utility that wins over a caller's min-h-6; link variants carry it too; SegmentedToggle,
  Tabs, NavRail, menu rows, PriorityStepper, sliders, CheckRow, InnerInput, card links and the rest.
- F9: DismissibleChip's label reaches the chip's 24; the Combobox chips are DismissibleChips.
- F15: the "+N" / "More..." open state worded as it renders.
2026-09-25 03:58:17 +02:00
peter 0521149cf1 fix(design): claim audit part 1 — a closing key no longer re-presses its trigger, the Switch box and name, named z-levels (E28)
- In-place editors (Playlists heading + rail rename, Plex collection + playlist rename, PriorityStepper,
  Scheduler interval): a real Enter closed the editor, returned the focus to the trigger, and the same
  keystroke's activation pressed that trigger again - re-opening the editor, or in the rail selecting
  the renamed playlist. useReturnFocus.arm(e) now cancels the key's default (type-enforced); the rail and
  Scheduler editors do the same. The earlier tests dispatched a synthetic keydown, which has no default.
- Plex playlist rename: an unchanged or empty title closes with no write, as the collection rename does.
- Switch: a 36x24 box around the track (the 24 floor), 44 on a coarse pointer; `label` is required, and
  the Plex two-way-sync switch had no name.
- z-index: numeric z-10/z-20 and a literal 40 moved onto the named layer scale; a lint rule + contract.
- The /watch file card's Download is the filled primary: the page's one action.
- Stale comments: the 1024x768 floor, the retired glass tuner.
2026-09-25 02:25:54 +02:00
peter b4a6569d0d fix(design): the saved-view chip on one line, one Escape per layer, the /watch action row (E28 S3d)
- SelectTrigger lays its toolbar value out as a row; the saved-view chip passes its glyph and
  modified dot through the icon/trailing slots (an <svg> is display:block and broke the chip in two).
- One Escape in a dialog's open list closed the list AND the dialog: the list's close flushes
  synchronously and pops its layer before the dialog's window listener runs. takeEscape() is the one
  close-on-Escape test (topmost AND not yet taken); Modal, the Welcome lightbox and useLayer use it,
  and keymap.dispatch ignores a taken Escape.
- /watch: the Download sits on the toolbar rung, level with the Repeat chip.
2026-09-25 01:40:45 +02:00
peter 1833f594cb fix(design): an audit of the S3d claims — the float shadow, filter hints, one action size per row (E28 S3d)
- A glass role's shadow is one token (shadow-float); the header search's drop-shadow utility had
  replaced its top highlight, and the debug pill's shadow-lg did the same. The surface-role lint
  rule now refuses a shadow size beside a role.
- Plex: every value that filters carries a hint (genre, certificate, studio had none), and the facts
  panel's year and certificate filter like the header's. filterKeyword -> filterValue.
- Downloads: the row resized its buttons with a [&_button] override that the save link escaped
  (28px beside 32px); the size is the primitive's md rung now.
2026-09-25 01:16:20 +02:00
peter 1615838f33 fix(design): the rail's active edge bar was clipped by the scrolling list (E28 S3d3)
An overflow-y container clips on both axes; the bar sits in the rail's padding, outside the row.
NavRailList spans that padding (-mx-1 px-1) so the bar stays inside the list's box.
2026-09-25 00:59:23 +02:00
peter 7fa8f4ebde feat(design): the NavRail primitive with optional short labels, the flat surfaces onto roles (E28 S3d3, B7/B5) 2026-09-25 00:47:46 +02:00
peter 6354454cc9 feat(design): skeleton first loads by shape on the shared reveal clock (E28 S3d2, B8) 2026-09-25 00:37:45 +02:00
peter c47c193ad1 feat(design): glass surfaces by role, the static Badge, LinkButton, user pickers on the Combobox (E28 S3d1) 2026-09-25 00:34:35 +02:00
peter db10bfbbca feat(design): the header search is a 40 glass pill; a wrapper field dims for its input only (E28 S3c) 2026-09-25 00:05:31 +02:00
peter dac9d89f91 feat(design): the field family, the dropdown trigger, native controls, the no-raw-input lint rule (E28 S3c)
- ui/Input, Textarea, InnerInput + fieldClass (A4): the >= 3:1 edge, a 2px accent
  focus edge, the danger edge for invalid, md/sm/xs/heading rungs, 1rem and 44 on touch
- ui/SelectTrigger (A5): SortControl, FilterSelect, the view switcher and the date
  filter on one trigger, a pill in a toolbar, r8 in a form
- ui/NativeControls (A6): Checkbox, CheckRow, Slider, ColorInput
- every raw input swept (~100 sites); inputCls retired; local/input-primitive
- useReturnFocus: an in-place editor hands the focus back to its trigger on
  Enter/Escape (the playlist and Plex title renames, the priority number);
  the Plex playlist rename gains Escape and an accessible name
- back to top: a round 48 FAB (owner decision); DESIGN.md A2 and the log
2026-09-24 23:48:22 +02:00
peter 81936e9f56 feat(design): every button on its family, the chip primitive, the lint rule (E28 S3b2)
The second half of the button sweep: the remaining hand-rolled buttons in
feature components, the design lab and the components/ui composites now
stand on Button, IconButton, Pressable, Chip or DragHandle.

- Chip + DismissibleChip (filter / token / tag / suggest / ghost, 24px
  floor, 44 on touch) replace five chip geometries; FilterChipGroup
  renders Chips; overflow lanes measure a mirror painted by chipClass.
- The Plex control row is IconButton's player rung (36, 20px glyph,
  round, overlay; 44 on touch); the YT modal bar is on the sm rung with
  pressed cycle toggles and names that survive a folded label.
- New variants: IconButton ghost-accent, danger-soft; Button warn on a
  new state-warn utility. The .btn-{tone} recipes and btnCls are gone.
- Fixed on the way: membership rows (collection editor, Plex playlist
  add) announce checked/mixed; a one-letter tag chip was 21px wide;
  descriptionLinks moved out of lib/ (it renders components).
- local/button-primitive: a raw <button>/<FocusSafeButton> outside the
  control-primitive files is a lint error (contract-tested, seen red).
2026-09-24 23:03:23 +02:00
peter 86f467bd65 feat(design): the button families on the decided ladder, first half of the sweep (E28 S3b1)
Button is rebuilt to A1/B1/B3: explicit heights (xs 24 / sm 28 / md 32, lg 40 for a hero CTA), r8 (a
pill on media), state layers instead of opacity and bg-card hovers, a `pressed` toggle, `overlay`,
`media` and `link-muted` variants. The icon-only sizes move to the new IconButton (28 / 32, 48 only for
the Play on a picture; a required label). Pressable (a card, row or tile that is the control) and
DragHandle join them. The state-* utilities now stand down on a disabled, busy or pressed control, held
by theme.contract.test.ts.

Swept onto them: the Button icon-size callers and 16 files (Plex playlist, Plex detail, the YT player
modal chrome, Welcome, Plex and video cards with AddToPlaylist and DownloadButton, the video editor,
Feed, personal views, Scheduler, Users, the nav rail, Plex detail parts, Settings): 268 raw buttons are
153. Fixed on the way: FocusSafeButton overwrote a caller's aria-disabled (the "unavailable, here is
why" state; adminInvites caught it), a season toggle and a Plex settings switch had no name/state, and
the video card's overlay follows the C5 card (Resume pill, From the start pill, a round Play).
2026-09-24 05:25:55 +02:00
peter 7d480e2ad0 feat(design): Matrix toned down to moss, text on an accent fill held to --accent-fg
The lime accent was the loudest of the four (OKLCH L 0.85, C 0.21) and read as neon; the owner chose the
moss variant at Midnight's lightness with near-neutral greys, and the Matrix backdrops follow it. Fixed
white/black parts on a solid accent fill (two switch knobs, three play buttons, the Up next pill) now use
--accent-fg, and theme.contract.test.ts refuses a fixed white or black next to a solid bg-accent.
2026-09-24 04:49:07 +02:00
peter ad136c3fe8 feat(design): the scheme set, the colour contract and the token utilities (E28 S3a)
Four schemes (Midnight; Forest cyan, Slate fuchsia, Matrix lime), YouTube
and Starship retired with no data migration: normalizeTheme maps a stored
retired scheme to Midnight. A vitest contract measures every scheme x mode
(text 4.5:1, control edge 3:1, status text, the accent's 30-degree hue gap);
it caught light-mode danger at 4.43:1, now red-700. The state layers, the
strong edge, the touch target and the fades are index.css tokens and
utilities the lab now reads. Backdrops recoloured to the new accents, with
a grey overlay grain that actually breaks the banding (longest flat run
15-26px -> 4-8px). The glass tuner is retired.
2026-09-24 04:33:13 +02:00
peter 9c4a999e54 feat(design): C5 recorded, every E28 decision taken
The owner's lean Plex control set with both refinements (Back by position,
a touch double-tap for the 10 s skip) goes into DESIGN.md's Players
chapter and the log. The lab is now the reference only.
2026-09-24 04:12:19 +02:00
peter 3f5634a44f feat(design): C5 redrawn around the owner's reworked Plex controls
Stop, Restart and the 10 s skips leave the row, Back restarts or steps to
the previous episode, Next shows only when there is a next item, and the
card carries Resume + From the start. The lab asks back two points: how
Back tells restart from previous, and a touch path for the 10 s skip.
2026-09-24 04:05:09 +02:00
peter f4bc60d659 feat(design): batch C recorded, the player row open as C5 (E28 S2d)
C1-C4 as proposed: the action classification, overlay-drawer side panels,
touch-first players, dialog focus by purpose with a pinned footer. The
players were then played live at tablet size, which found the Plex
control row running off-screen below 1024; how it fits is open as C5.
CLAUDE.md now points at DESIGN.md.
2026-09-24 03:27:26 +02:00
peter 7055548260 feat(design): batch B recorded, batch C in the design lab (E28 S2c)
DESIGN.md: B1-B9 decided (compact density, the four-scheme set with the
accent-vs-status rule, state layers, the sticky band, surface per role, the
watch page, the slim NavRail with optional labels, loading by shape, the
fade edges) with their decision-log rows. The lab keeps the chosen A and B
renderings as reference on shared tokens (labTokens.ts) and adds batch C:
the action-placement classification for review, side panels on touch and
below 768, the Plex player on touch, and the dialog conventions.
2026-09-24 03:10:00 +02:00
peter 477271907e feat(design): batch A recorded, batch B in the design lab (E28 S2b)
DESIGN.md: the Controls chapter decided (A1-A6) with the decision-log rows;
density stays open as B1. The lab keeps only the chosen A renderings as
reference and adds batch B: density, the scheme set (measured accent hue
gaps and light contrast), state layers, the page-chrome model, surface per
role, the public watch page, the nav rail, the loading form and where the
header + fade edge goes.
2026-09-24 02:21:20 +02:00
peter 50923bb684 fix(build): keep JSX line-end spaces in dev and vitest
plugin-react recognises the React Compiler only by its package name, and
reactPlugin.ts passes the imported plugin, so plugin-react left Babel's
retainLines on in dev. With the compiler's re-printed JSX that trimmed the
space at the end of a text line (the privacy page read
"for example/api/feed" on :5173). Production builds never had it on.
Turn retainLines off through generatorOpts, which wins over the plugin's
top-level option, and pin it with a DOM test seen red without the fix.
2026-09-24 02:21:20 +02:00
peter d2e40dcffb feat(design): design lab + DESIGN.md skeleton (E28 S2a)
- docs/DESIGN.md: the written source of truth for the UI. Decided chapters
  (principles, device adaptation, action placement, text stays selectable,
  interaction rules, copy, anti-patterns, decision log); controls and
  surfaces are marked Pending until the owner picks from the lab.
- /design-lab (dev only): renders each open decision's variants with the
  app's real CSS, themes and font scale. Batch A = six control decisions.
- devOnlyGuard: a production build fails if anything under src/design-lab
  reaches the bundle (seen red with an unguarded import).
- Size contract: tablet portrait + landscape and up, replacing the
  1024x768 floor with its never-built "too small" message.
2026-09-24 01:34:12 +02:00
peter 58878958d8 merge: chore/e2e-password-from-store (E2E password from the secret store, no committed default)
gate / frontend (push) Successful in 4m37s
gate / backend (push) Successful in 6m29s
2026-09-23 23:05:17 +02:00
peter b47d22cf3b chore(e2e): no committed default for the E2E account password
The localdev e2e account's password was a hard-coded default in this public repo, so it
was never a credential. E2E_PASSWORD is now required; the ops wrapper exports it from
the secret store, and the seed/setup fail with a clear message when it is missing.
2026-09-23 02:35:19 +02:00
peter e804ff3b8e release: 0.95.0 (improvement/p1-upload-auth-first)
gate / frontend (push) Successful in 4m35s
gate / backend (push) Successful in 6m18s
2026-09-23 01:32:45 +02:00
peter cd6f119883 chore(release): 0.95.0
The P1 epic: the upload route reads its own body after `require_human`
resolves, so an anonymous caller can no longer make the server spool 500 MiB
before the auth check runs. Three deep-review fix rounds on top of it.
2026-09-23 01:32:32 +02:00
peter 321776480e fix(uploads): make the drift guards fail for the reasons they claim
Deep-review round 3, targeted at the surfaces round 2's fix commit changed.
Five of the eight findings are the same shape: the guards written to stop
half-done sweeps were themselves half-done.

test_upload.py's display-field guard walked only `ast.Assign`, so the one
writer with an unbounded REQUEST HEADER behind it — `spool_upload`'s
`display_name=display_text(...)` keyword — was invisible to the guard meant
to protect it. It also stopped at `app/routes/`, missing the four writers in
`downloads/` that actually construct the row, and asserted `not offenders`,
which enforces nothing once a refactor moves the writers out of its view.
It now walks assignments AND keywords across `app/`, and holds exact set
equality against a registry of all twelve write sites with each one's value.

test_body_limit.py's release guard used `any(... ast.walk(fn))`, which
accepts a `release_read_connection` that runs AFTER the await it exists to
precede — satisfying the letter of the rule while pinning a connection for
exactly the transfer it was supposed to free. Position is checked now, and a
late release is its own, louder failure. The registry keyed sites by file
basename while scanning recursively, and `admin.py`, `discover.py` and
`playlists.py` each exist both at the top level and under `plex/` — the same
entry-collapse the registry was widened to close. Keyed by relative path now.
And all three guards recognised a body read only by the `capped_` naming
convention, so a route spelling `await request.body()` inline declared no
ceiling, released no connection, and was invisible to every one of them.

`release_read_connection`'s precondition claimed the caller "must have
NOTHING PENDING, and that is checked rather than described" while checking
only UNFLUSHED ORM work. Measured: after `add`+`flush` all three collections
are empty and the rollback loses the row, and Core DML never enters them at
all — both of them this codebase's canonical write shapes. It now also asks
the transaction directly via `pg_current_xact_id_if_assigned()`.

`errors.tooLarge` was reworded to "try a smaller file", but that key answers
every route's body cap: the issue composer's over-size JSON draft got advice
about a file it does not have. Back to payload-neutral; the file-specific
sentence keeps its own key.

The upload dialog's new danger border outranked `dragOver`, so a user
replacing an over-cap file got no sign the drop zone was live — and with a
file picked the label shows its name rather than "drop it here", so the
border was the only feedback left.

Four further findings were refuted 3/3 and are not fixed: `aria-invalid` on
the drop zone (measured in Chromium — the AX node does carry it on a native
button), a bidi-override spoof through `display_text` (the served filename
already strips those categories, and the proposed filter would have broken
every ZWJ emoji), and `capped_body` not draining (the screenshot route's edge
ceiling equals its own cap, so that refusal is unreachable, and the log sink
is sent by beacon with nobody reading the answer).
2026-09-23 01:20:22 +02:00
peter 7c0647f87a fix(uploads): finish the sweeps round 1 left half-done
Deep-review round 2, on round 1's own fix commit. It found MORE than round 1
did (28 raw, against 13), and most of it was mine -- a new piece of UI attracts
findings a backend-only diff never does. The pattern in almost all of them is
the same: a principle applied to the site that prompted it and not to its
siblings.

* `display_text` reached three of the FIVE writers of a user-supplied name.
  `POST /api/downloads` and `POST /api/downloads/edit` still handed Postgres an
  unbounded string, so a 300-character name pasted into the Add-download dialog
  or the editor was a bare 500 (main.py re-raises every DataError but the NUL
  case). Four of the seven reviewers found it independently. The bound now lives
  in `models.py`, the three columns are DECLARED with it, and the guard for the
  next writer is parsed rather than remembered: every `display_name` /
  `display_uploader` assignment under `app/routes/` must go through the helper.
  That guard immediately found two more -- the Google sign-in callback writes
  the profile name into the same 255-wide column -- so it is six, not five.

* The connection release reached one of the THREE routes that read their own
  body. The screenshot route (10 MiB, from a phone) and the client log sink both
  still held a pooled connection idle-in-transaction for the length of the
  upload, which is the invariant db.py sizes the pool on. All three now call
  `db.release_read_connection`, whose precondition -- nothing pending on the
  shared session -- is CHECKED rather than described, because a whole-session
  rollback discarding a caller's staged write is a mistake this repo has already
  made once. `test_body_limit.py` holds the rule for the next such route.

* The AST drift registry keyed its sites by (module, limit expression), so a
  second route copied from the first collapsed onto the same entry and went
  uncovered -- the exact hole the registry exists to close. Keyed per call site
  now, and it carries the release requirement too.

* The dialog's own refusal was announced once and never again: two over-cap
  picks leave an identical node with identical text, and an untouched live
  region says nothing. It is keyed per pick now, the drop zone reads as invalid
  and points at the sentence, and the reason is reachable from the control
  rather than only from a one-shot announcement.

* Two tests rebuilt the blind spot they were written to remove: the dialog's cap
  test sat nowhere near the boundary (a `>` loosened to `>=` survived it, which
  is the mutation round 1 fixed on the server side), and the header-precedence
  test only covered an exact-case duplicate while a differently-cased one would
  have been COMBINED onto the wire rather than replaced. Both fixed, and the
  merge now dedupes case-insensitively because header names are.

Also: the upload dialog's unstubbed usage query was firing a real request from
jsdom in a neighbouring suite and leaking a delayed "connection lost" toast
across test boundaries; `GET /api/downloads/usage` had no test at all though the
whole client-side check hangs off it; and the comment claiming the edge 413 is
readable now says what was measured -- readable to 200 MiB over loopback, a bare
`onerror` through a proxy -- so the pre-send check is named as the mechanism and
that branch as the backstop it is.

Every test added or changed here was mutation-checked. One mutation SURVIVED and
is recorded rather than papered over: the `tooBig` term inside `submit()` cannot
be reached, because `ui/Button` swallows the activation itself, so it is marked
as belt and braces instead of being described as the guard.
2026-09-22 23:31:55 +02:00
peter efd7b217ca fix(uploads): make both caps answerable, and stop holding a connection
The epic-end deep review found three defects in P1 and four tests that did not
hold what they claimed. They land together because they share the upload route
and its test module; splitting by hunk would have produced commits that do not
individually run green.

* The edge ceiling and the route's own cap are the SAME number at the default
  configuration, so the edge always answers first: a declared Content-Length
  over the cap is refused before a byte is read, and an undeclared body trips
  the middleware on the same chunk the route would have tripped on. The route's
  numbered 422 -- and the drain written for it last week -- are reachable only
  once an admin LOWERS the knob, which is the one shape the browser UAT tested.
  Measured: headroom does not fix this, because the declared-length check fires
  first for any body big enough to matter and the drain pushes the count back
  over the edge for the rest. So the edge's own refusal is made readable
  instead: `errors.tooLarge` localizes `too_large` the way `upload_too_large`
  was already localized, and the dialog refuses an over-cap file BEFORE sending
  it, off the `upload_max_bytes` that GET /api/downloads/usage now reports.
  Measured in Chrome against the API directly: that 413 arrives as `onload`
  with a readable body in 43 ms.

* Reading the body behind the auth dependency put the whole transfer inside the
  request's database session -- the auth SELECT autobegins a transaction, so one
  of the twenty pooled connections sat `idle in transaction`, with nothing to
  time it out, for as long as the client cared to keep sending. That is the
  invariant db.py sizes the pool on and plex/playback.py records an incident
  for. The read transaction now ends before the await. Measured on the dev
  server: an 1818 ms upload shows no idle-in-transaction sample while the bytes
  move, only afterwards, when the handler actually writes.

* The display name moved into a request header, where nothing bounds its length,
  while the column stayed varchar(255) -- so a long enough name was a DataError
  500 after the whole file had already transferred. One `display_text` helper
  now serves all three writers (the metadata PATCH already truncated, the upload
  ingest did not), and the dialog stops at the same number.

* The gate invariant asked `route.dependant.body_params`, which holds only the
  route's OWN callable's parameters: a body declared in a SUB-DEPENDENCY -- the
  upload route's own idiom -- leaves it empty while FastAPI still reads the body
  before auth. It asks `route.body_field` now, which is what FastAPI itself
  consults. The AST registry of body-reading call sites had also never followed
  `capped_stream_to`, so the one route the whole invariant exists for was the
  single site the drift guard could not see.

Four tests that passed over their own mutation are fixed: the cap boundary (a
`>` loosened to `>=` survived the entire suite), the percent-decoding (every
filename the suite sends is ASCII, where quote() is the identity function), the
display-name header (no test at all), and the staging cleanup (deleting the
unlink left the suite green while every refused upload leaked its partial bytes
onto the volume). Every test added or changed here was mutation-checked.
2026-09-22 22:41:22 +02:00
peter 3d7ef9e945 fix(uploads): drain an over-cap body before refusing it
Found in the browser UAT, not by the gate. Moving the cap onto the arriving
bytes meant it fires while the client is still sending, and answering + closing
mid-upload is what XHR reports as a network failure: the upload dialog said
"the server couldn't be reached" while the access log showed a correct 422
carrying the limit. A refusal the client cannot read is an outage as far as the
user is concerned.

`capped_stream_to` now reads the remainder to nowhere before raising, so the
client finishes sending and reads the real answer -- re-measured in the
browser, which now names the limit. Nothing past the cap is written, and the
request is still bounded by bodylimit's ceiling, so the cost is bandwidth an
AUTHENTICATED caller was sending anyway. The anonymous path is untouched:
auth still resolves before a byte is read.

The regression test sits on the helper, not the route, and the reason is worth
keeping: the route-level version was written first and PASSED with the drain
deleted. httpx's ASGI transport pumps the request body whether or not the app
reads it, so "drained" and "abandoned" are indistinguishable through
TestClient. Its own mutation check is what exposed it.
2026-09-22 02:11:21 +02:00
peter 7fce7a7ce5 chore(deps): drop python-multipart -- nothing parses a multipart body now
Its own entry in requirements.txt said the download center's local-file upload
was "the only multipart endpoint in the app". Since that route reads a raw
body, the claim is false and the package is dead weight: no route declares
File/Form, and nothing calls `request.form()`.

Removed from requirements.txt and from its single pinned line in
requirements.lock. The lock is generated and normally regenerated wholesale,
but a full regeneration would bump every transitive dependency, which has no
business riding inside a security fix -- the next legitimate regeneration
reproduces exactly this state, since the package is gone from the source.

The `multipart` entry in logs.py's third-party level map goes with it: a level
for a logger that can no longer exist.
2026-09-22 01:59:09 +02:00
peter f5018bdcbc fix(uploads): read the body after authenticating, not before it
`POST /api/downloads/upload` declared `file: UploadFile = File(...)`, and
FastAPI parses and spools a declared body before the first dependency runs --
`require_human` included. An unauthenticated caller could therefore make the
process write the whole per-file allowance (500 MiB by default) to temp, once
per request. SEC-20's ceiling bounded that number but could not stop the
writing, which is why E27 parked it rather than closing it.

The route now takes no body parameter. `spool_upload` names `require_human`
first and only then streams `request.stream()` to the staging temp file, so a
stranger is refused before a byte moves -- the shape the screenshot route has
used since E16. Filename and display name travel as percent-encoded headers
(latin-1 on the wire, so a Hungarian filename needs the encoding) instead of
multipart parts.

`capped_stream_to` joins `capped_body` in routes/_common.py: the same counting
rule, written to a file descriptor rather than accumulated, because holding
500 MiB would trade a disk problem for a memory one. Its refusal is the
caller's, so the upload route keeps answering 422 `upload_too_large` with the
admin knob's own number.

With no envelope to allow for, the edge is now exactly that knob, and
MULTIPART_ENVELOPE_ALLOWANCE is gone.

The invariant stops being prose: test_body_limit.py asks FastAPI itself
(`route.dependant.body_params`) and fails the gate if any route with a raised
ceiling declares a body. A comment could not have prevented the defect it
described; that test would have.

Both new tests were seen red for the right reason -- before the fix the
anonymous probe got its 401 only after the server had read 4236 bytes of a
multipart body, and re-declaring the File parameter turns the invariant test
red naming the route and the parameter.
2026-09-22 01:59:03 +02:00
peter b82c22b888 release: 0.94.0 (improvement/e27-contract-hardening)
gate / frontend (push) Successful in 4m32s
gate / backend (push) Successful in 6m8s
2026-09-22 00:34:12 +02:00
peter f839a461e6 chore(release): 0.94.0 — framework & contract hardening 2026-09-22 00:34:06 +02:00
peter 8525666913 test: round 4 — an assertion that could not fail, and a comment that did not add up
Two findings, both in the review's own work; nothing in the product code. That is
the shift that ends the loop: 22 → 13 → 6 → 2, and the last two are a test and a
comment rather than a defect.

The "an override that is fine is still honoured" leg of the bounds-fallback test
stored `spec.max` — which, for this key, IS the env default, because the spec's
max is derived from it. So the assertion was satisfied by the fallback it existed
to rule out: mutating `_coerce` to reject EVERY int override left all three legs
green. It stores `spec.max - 1` now and asserts the value differs from the
default, so the leg can actually fail.

And the shrink test's comment claimed `3 - 5 = -2` for an expression that computes
`chips.length - visible` = `2 - 3 = -1` — naming neither operand nor the result,
in a comment that was rewritten one round earlier precisely for miscounting.
2026-09-22 00:22:30 +02:00
peter 3f24248e01 fix: round 3 — a vacuous ordering test, a clamp that was a silent rewrite, a regex that missed
Round 3 over round 2's fixes. The trend is 22 → 13 → 6, and the character has
shifted: these are mostly defects in the REVIEW's own work, not in the epic.

The ordering test I added to hold the middleware's position could not fail for
either arrangement it names. All four `@app.middleware("http")` layers — the
access log, the two header decorators AND the setup gate that must be inside —
register as the same `BaseHTTPMiddleware`, so asserting on the class passes for
the pre-move stack and for a cap moved back outside CORS. It names the layers by
their dispatch function now, which is the only thing that distinguishes them.

The `sysconfig` bounds check CLAMPED, and clamping is a silent rewrite — the thing
this module refuses to do everywhere else, where an unusable override falls back
to the env default and logs once. On the max side the difference is cosmetic; on
the min side it is a live bug. `download_upload_max_bytes` had `min=0` before E7,
and a stored 0 means "no per-file cap" to the route that reads it. Clamped to 1 it
becomes a ONE-BYTE cap: uploads dead, nothing in the log, the admin page showing
"1". It raises `OutOfRange` now and takes the existing fallback, with its own
event code.

The `capped_body` sweep grepped for an identifier as the limit argument, so a site
written `capped_body(request, 4 * 1024 * 1024)` — or with the parameter not named
`request` — was MISSED, and a miss reads as "no new sites". Parsed with `ast`
instead: every call is seen whatever it looks like, and an unfamiliar argument
shape fails the equality instead of vanishing.

Frontend: the shrink test drove the mock in a way the real hook cannot produce
(it moved `visible` in the same step as the shrink, and the comment miscounted the
tags), so it reproduced the right END STATE by accident rather than the sequence.
`visible` is now left alone across the shrink, which IS the precondition.

Ops: the baseline guard answered "did this run rewrite it?" and therefore went
green on the second run over an already-tightened, uncommitted baseline — the
exact state its own comment says marks the localdev stack stale ever after. It
asks both questions now. And `.dockerignore` joins BAKED_PATHS: it decides what
the COPY lines actually take, so it is an input to the build context exactly as
the copied trees are.
2026-09-22 00:12:30 +02:00
peter 6b86673af4 chore: untrack tsconfig.tsbuildinfo, and hold the CI baseline the way the local gate does
`frontend/tsconfig.tsbuildinfo` was swept into the previous commit by a `git add
-A`. It is a machine-local incremental artifact carrying absolute Windows paths,
no lane reads or regenerates it, and it was already stale in the commit that added
it — it does not list the contract test that same commit created. Untracked, and
`*.tsbuildinfo` ignored, because the day anything does start rewriting it a green
gate would leave a tracked file dirty: the exact hazard the basedpyright baseline
just got a guard for.

And CI ran bare basedpyright while the local lane gained that guard, so a
burn-down would have been green on the server and red on the developer's machine
over the same tree. Two gates, one tree, one answer.
2026-09-21 23:59:59 +02:00
peter d924c665f8 fix(frontend): round 2 — the popover latch reset read a stale, negative count
The reset I added guards on `hidden === 0`, and `hidden` can be NEGATIVE.
`useChipOverflowFit` keeps `visible` in state and re-decides in a layout effect,
so the render in which the chip set SHRINKS still carries the previous
measurement — delete three tags under an open "+N" and `chips.length - visible`
goes to -2, the equality never matches, and the flag survives the one window the
reset exists to close. The next re-fit brings the menu back on its own, which is
the original defect with an extra step. Clamped at all three lanes, with a test
that drives the shrink through the stale value.

Also: the Scheduler characterization banner said "the four tests below" and now
sits above six, the last two of which are genuine regression guards for the
range-refusal fix. A future mutation check would have read it, seen them go red,
and concluded the harness was broken rather than that the guard worked.
2026-09-21 23:59:59 +02:00
peter aa51644f5f fix(security): round 2 — the SEC-20 fix had four defects of its own
A second review round over the first round's fixes, which is where the remaining
real bugs usually are. All four are the same failure as the round it reviewed: a
claim stated in a comment that the code does not hold.

The middleware moved inside CORS, but the setup gate is registered LATER and is
therefore OUTSIDE it — so the comment's "the setup gate never runs for a body that
is already too large" was false the moment it was written. On an unconfigured
instance that gate opens a DB session per request, which is exactly the anonymous
pre-auth work SEC-20 exists to remove. The cap now sits between them, and the
order is asserted against `app.user_middleware` rather than described: prose about
a stack that `add_middleware` builds inside-out is how this got backwards.

The edge ceiling and the route's own cap were pinned EQUAL, but they measure
different things — the edge counts raw body bytes, the route counts the decoded
file. A file of exactly the advertised maximum was refused one layer before the
route that would have taken it. The edge now allows for the multipart envelope,
and the test asserts the relation and the gap instead of equality.

`sysconfig`'s new `max` was enforced on write only. That bound did not exist
before this epic, so a value that was legal when it was written can be sitting in
the database — served unclamped it produces exactly the drift the comment says is
impossible, with the admin page showing a number outside its own range. Int
overrides are clamped on read now, which also survives an env default being
lowered later.

And `DOWNLOAD_UPLOAD_MAX_BYTES=0` meant "no per-file cap" to the route but a hard
zero to the edge: every upload refused, including the admin page's own way back,
recoverable only by editing env and restarting. The setting takes `ge=1`.

Two test gaps from the same round: the header assertion that is the ONLY check of
the middleware's position was DB-gated and skipped on a host without Postgres, and
the headroom "ratio" the docblock calls load-bearing was held as `<=`, i.e. no
ratio at all. Both fixed, plus a sweep that every `capped_body` call site in the
routers is known and fits under its ceiling — the point of that list is the third
entry, not the two that exist today.
2026-09-21 23:59:45 +02:00
peter 378b7559e4 build(frontend): finish the React 19 move, and hold the versions that were left in prose
react-i18next was still on ^14.1.2. The E27 S1 plan named it as "the one peer that
actually needs moving — React 19 support is the 15 line", and then it did not
move. The 14 line's <Trans> reads `element.ref`, removed in React 19, and its
withTranslation types are built on the GLOBAL JSX namespace @types/react 19 no
longer declares — both latent only because this app imports neither. 15.7.4 fixes
both at the source (`c.props.ref ?? c.ref`, `React.JSX.LibraryManagedAttributes`).

`reactEngine.contract.test.ts` joins its Node and Python twins and holds what was
only a comment: the compiler's `target` against the installed React major, react
and react-dom together, the types against the runtime, and react-i18next's floor.
A stale target is not a corrupt bundle — the plugin validates the option — but it
is a silent one, since tsc never sees Babel's output.

pythonEngine gains the seventh spelling nobody held: `requirements.lock`'s own
regeneration recipe names a python image, and wheels resolve against THAT
interpreter, so a recipe left behind yields a lock that does not match the image
installing it.

The type lane now covers `alembic` too, for the reason the ruff lane was widened
to it — migrations run on every container start, so one that raises crash-loops
prod. It came back at zero new errors, so the baseline is untouched. `tests/` is
deliberately still out, and the config says why.

vitest.config's docblock claimed the React transform is "a no-op for the plain .ts
node tests". It is not: the compiler rewrites src/lib's hooks in that lane too,
which is the point of sharing one factory with the build — and worth knowing
before ruling the compiler out of a divergence.

Tests: useCardPager had none, in either the hook or its two callers, though its
page clamp moved to render in S2 — the transformation where a non-convergent
condition is a crash rather than a glitch. Four cases now, including termination
on an empty list. Combobox's and Avatar's close/failed-URL cases are relabelled
CHARACTERIZATION: both pass against the pre-sweep code, and the tests beside them
are the ones that discriminate.
2026-09-21 23:33:02 +02:00
peter 05121fbc99 fix(security): SEC-20's exemptions become raised ceilings, and the 413 is dressed
The module admitted its two opt-outs under a stated rule — "both enforce their own
limit on the bytes they actually read, which is what makes opting them out safe" —
and that rule is false for one of them. `POST /api/downloads/upload` declares
`file: UploadFile = File(...)`, so FastAPI parses and spools the whole multipart
part to a temp file BEFORE the first dependency runs, `require_human` included:
its in-handler cap measures bytes that have already landed. Measured on the
installed fastapi 0.141.1 / starlette 1.6.0 — starlette's max_part_size guards
non-file parts only, so an anonymous chunked upload was bounded by nothing. In the
shipped self-host compose that lands in a RAM-backed tmpfs.

The exposure predates SEC-20 (no route was capped before it); what this branch
added was the claim. So the list becomes a mapping of per-route CEILINGS instead
of a set of exemptions: every request now gets a finite number, the upload route's
being the same one the route itself enforces. `sysconfig`'s knob is bounded by it
so an admin cannot raise the handler's cap past the edge and start collecting 413s
from a layer the admin page never mentions — held by a test, not by prose.

Two more things the review caught in the same module:

The 413 was not byte-identical to `capped_body`'s after all. `json.dumps` puts
spaces where JSONResponse does not, and the test that promised to hold them
together compared PARSED JSON, so it could never have seen it. Built through the
same JSONResponse now, and compared as raw bytes.

And the middleware sat outside CORS and the security headers, so its 413 carried
neither — on the split-origin dev setup the browser drops it and the user gets a
generic network failure instead of the translated too_large, while the identical
refusal from capped_body one layer down renders fine. It moves to just inside
CORS: still ahead of the session, the setup gate and every auth dependency.

Also corrected: the docblock's "largest non-exempt body is 64 KiB, ~16x headroom"
(it is 512 KiB and 2x — the diagnostics sink, named eleven lines further down),
and CLAUDE.md's Request SIZE row, which forbade the one correct use of
capped_body that is actually in the tree.
2026-09-21 23:32:49 +02:00
peter bf518223e5 fix(frontend): the scheduler's two number editors stop swallowing what you typed
Both refused an out-of-range value by discarding it: the entry vanished, the box
snapped back to the server's number, and nothing said whether the save had
happened, been rejected, or been ignored. That is the house rule about honest
states, applied to a write instead of a read. They now keep the draft, mark the
field aria-invalid and name the range (HU+EN), and the Save control is disabled
rather than silently inert.

The interval editor also dropped a keyboard user's focus on every exit — Escape,
Cancel and Save all unmount the focused input — so it hands focus back to the
pencil that opened it, the pattern IssueComposer already uses.

Tests: the four interval-editor cases added in S2 are relabelled CHARACTERIZATION,
because they pass against the pre-sweep code too (the old effect was already
guarded by `editing`). The editor that genuinely lost typing to a poll is the
maintenance batch box, whose old effect had no guard at all, and it had no such
test — it does now, anchored on a value that follows the server so it cannot pass
on a page that simply has not polled yet.
2026-09-21 23:32:32 +02:00
peter 332d24a646 fix(frontend): four state defects the deep-review found in the E27 S2 sweep
Every one of them is the same shape: a value derived correctly, and a WRITE to
the thing it derives from left unattended. Deriving hides a stale value; it does
not consume it.

PlayerModal — the watch toggle inherited the previous video's auto-watch. Moving
the queue-advance sync into render was right for `currentVideoId`, but the same
block also synced `status`, and that one had an ordering constraint of its own:
useYtProgress's cleanup for the OUTGOING item can still call setWatched(true)
near the end of a clip, and cleanups run after the commit. As an effect the sync
landed last and corrected it; from render it lands first and the stray write
wins. Press Next in the final ten seconds and the NEXT video showed as watched —
one click on the toggle then sent onState(next, "new") for an item already new.
Fixed by keying the mirror to the id it describes, so a write for another item is
ignored rather than displayed, and no ordering has to be argued about at all.

ChannelsProvider — a focus-channel intent was masked by the page rather than
consumed, and nothing ever cleared it. The manager is a page chunk: it unmounts
on every navigation, and its seed effect runs again on each mount, so returning
to it re-applied a channel name from earlier in the session over whatever the
user had typed since. The consumed marker now lives in the provider, keyed on the
token — which also closes the case the old clear-on-leave effect never handled,
opening a channel page (that does not change `page` at all).

TagFilterChips + ChannelTags x2 — `open = openRaw && hidden > 0` hides the
overflow popover when the lane stops overflowing, but floating-ui only reports a
close for a real event on a mounted surface, so the raw flag stayed true and the
menu came back by itself on the next re-fit — a window resize or the text-size
slider is enough, with nobody near it. The flag is now reset in render against
the same condition that hides it.

Avatar + downloadRow — remembering a failed image BY URL fixed the recycled-row
frame but never forgot it, so a transient 429 pinned a channel to its initial for
the life of the instance, never re-requesting. The memory is now dropped once
`src` moves off it, which keeps both properties.

And playerModalHooks' last exhaustive-deps suppression claimed "nothing in the
body reads a render value" while reading five. The invariant that is actually
load-bearing is that each of them is a per-item snapshot — written down instead.

All four fixes are mutation-checked: each new test was seen red, against the
fixed line only, naming the real defect.
2026-09-21 23:32:22 +02:00
peter c5e022f056 refactor(frontend): the long tail — and eslint-suppressions.json is gone (E27 S2)
Closes the burn-down. Hook suppressions 113 -> 34, `exhaustive-deps` 45 -> 2 (the target was
<10), and every one of the 34 survivors is an inline directive NAMING the invariant it
protects. The suppressions file is deleted, because a bulk file hides the next violation
instead of answering it — and it cannot hold a warning-level rule at all.

The sprint plan's three verdicts missed the answer that turned out to be the most common one:
adjust the state DURING RENDER. React re-runs the component and discards that pass rather than
committing it, so the stale frame an effect-based reset paints does not exist. Applied to the
Plex player's up-next card, the playlist view reset, the feed's optimistic overrides, the
channel banner's retry count, the download center's tab guard, the Plex search highlight, the
comfort-pin latch, Welcome's per-view outcome and the card pager's page clamp. It is legal only
for a component's own state — PlexBrowse's language commit resets PlexProvider's filters, so
that one stays an effect and says why.

Mirrors of a prop or query became derived values or `draft ?? server`:
- `downloadRow` had ui/Avatar's defect exactly — a boolean "poster failed" that an effect reset,
  in rows recycled down a virtualized list, so one job's missing poster painted over the next
  one's for a frame. Keyed by URL now.
- Scheduler's two number editors held a mirror an effect re-synced, on a page that POLLS — the
  shape a poll can overwrite mid-edit. A draft while editing, the server's value otherwise; six
  tests where there were none, the one behavioural change (the save button stops being dirty at
  commit rather than at the echo) mutation-checked.
- SetupWizard's "no token" and WatchPage's "gone" are facts about the URL, so they are seeded and
  derived rather than pushed into state by an effect that had to commit first.

Elsewhere: PlexInfo's rating chip was a component declared during render — a new component type
every render, so React remounted the subtree and dropped the focus on its filter button. Hoisted.
Feed's undo action and PlexPlayer's session reclaim both re-entered their own useCallback, which
the checker reads as a forward reference; both go through the ref idiom this file already uses.
BootGate's dep list was load-bearing for correctness — a re-run meant `location.reload()` — so
the "already adopted THIS identity" comparison now enforces it in code, with a test that fails
without it (the existing reload tests all install their spy too late to catch it).

One correction to the batch before this: keying ChatThread's and Messages' decrypt passes on
`q.data` instead of `q.dataUpdatedAt` was wrong. react-query's structural sharing makes it safe
in production, but the component must not melt down when something hands back a fresh object per
render — and Messages.keyboard.dom.test does exactly that, so the effect span the suite. Both are
back on the fetch as the trigger with the payload behind a ref, which is honest AND loop-proof;
the replaced comment had warned about precisely this.

CLAUDE.md gains four canonical-patterns rows for the above, and its "React 18" line is now true.
2026-09-21 01:44:21 +02:00
peter 4065ac7e6a refactor(frontend): the two players — one real ordering bug avoided, five suppressions gone (E27 S2)
The interesting part of this batch is a change that was WRONG and the test suite said so.

PlayerModal synced its local watch-status mirror and `currentVideoId` from an effect on
`active.id`. Moving that into render is right — as an effect there was a frame per queue advance
showing the new episode's chrome over the previous item's watch state, with `navigated` still
true so the "you have navigated away" affordance flashed. But the same effect also advanced
`currentIdRef`, and that line must NOT move: useYtProgress's per-item cleanup persists the
OUTGOING item's position and guards it on `currentIdRef.current === id`. React runs every
cleanup before any setup, so in an effect the ref still names the item being left. Advanced to
render time the flip happens first, the checkpoint is skipped, and stepping next→prev back to an
item resumes from the frozen launch queue's stale position. "resumes an item played earlier in
the SAME session" caught it immediately (748 instead of 830); the ref write stays in its own
effect with the ordering written down.

The rest:
- PlayerModal's keyboard handler is a useEffectEvent, so the "bound once, reads live state"
  contract is enforced instead of asserted. `focusModal` is a useCallback now — it reads a ref
  and nothing else, and three hooks take it as a parameter where a fresh identity would spread.
- PlexPlayer's session reclaim called `loadSession` from inside loadSession's own useCallback,
  which the checker reads as a forward reference. Routed through a ref, the file's existing
  answer for exactly this (detachSessionRef, cancelAutoSkipRef).
- PlexPlayer's item-load effect excluded `prefs` by hand so a track pick from the player's own
  menu would not double-load the session. The two language prefs go through a ref instead —
  "latest value, not a trigger", said in code rather than in a comment.
- PlexPlayer closes its up-next card during the render that changes `id`, not a commit later.
- The one PlexPlayer directive that stays is the live session counter, whose cleanup must bump
  the CURRENT value: the rule's advice to copy it into a variable is precisely wrong there, and
  the comment now says why rather than pointing at another comment.

Suppressions 75 -> 70; exhaustive-deps 20 -> 17.
2026-09-21 00:27:15 +02:00
peter ce08fc2aec refactor(frontend): a stable useHistorySubview, and PlexBrowse can stop disabling the check (E27 S2)
useHistorySubview returned a fresh object with a fresh `open` on every render. Correct, but it
made "re-run when the sub-view moves" unwriteable: PlexBrowse had three effects that wanted
exactly that and had to disable the dependency check to get it, and a fourth that listed `sub`
honestly and therefore re-ran on EVERY parent render, saved only by a ref guard.

Fixed at the source, which is where the house rule says a principle belongs: `open` and `back`
are useCallbacks and the returned object is memoised. `url` is a module constant at every call
site (PLEX_SUB_URL, MODE_URL) and the body otherwise touches only module functions, so there is
nothing left to churn on. All three PlexBrowse dep lists are now the honest ones, and the
comment that claimed `open` was "behaviourally stable" is replaced by it actually being stable.

PlexBrowse's remaining four are escape hatches, and two of them are now ONE:
- `infoScrollRef` is written from four places, and the compiler objects to a ref being modified
  outside the effect that reads it. That objection is a conservative false positive — a scroll
  memory needs exactly that arrangement — but four bare mutations were worth concentrating
  anyway. One `rememberInfoScroll` writer now holds the "stash on the way out, reset on a fresh
  page" rule, and answers the checker once instead of three times.
- The language-commit effect cannot move into render: it resets PlexProvider's filters, and a
  render-phase setState may only touch the component's own state. Said so at the site.

Suppressions 78 -> 75; exhaustive-deps 23 -> 20.
2026-09-21 00:20:05 +02:00
peter ea6d829c29 refactor(frontend): the context providers adjust state in render, not after it (E27 S2)
Eleven suppressions across the five providers. The recurring shape here was "an effect that
undoes something when you navigate away", and every one of them ran a commit LATE — the new
page painted once while the old module's state was still live.

Adjusted during render instead (React discards that pass rather than committing it, so there
is no in-between frame):
- SavedViewsProvider reset the sticky view and closed its modal from an effect on `source`, so
  the first Plex render carried the feed's sticky id — long enough for a stale view chip. It
  also followed the active view into stickyId from a second effect; that one converges by
  construction, since preferSticky returns the sticky match when there is one.
- NavigationProvider ended a live YouTube search from an effect on `page`. Keyed on the page
  CHANGE rather than derived from it, because "ended" has to be permanent — deriving would have
  quietly brought the search back on return to the feed.
- ChannelsProvider's manager focus is now simply DERIVED from the page: it means nothing off
  the Channels page, and focusChannel sets it and navigates in one handler.

Honest deps by freezing what is genuinely mount-time:
- FeedFiltersProvider persisted a share link's filters with an empty dep list; listing `filters`
  honestly would have re-persisted on every edit and spun. The mount value is frozen into its
  own state, so all three deps are stable and the effect says what it does.
- ChannelsProvider's URL-seed layout effect and NavigationProvider's initial history stamp get
  the same treatment. The three slice setters they call were already stable useCallbacks
  (useFilterState / useSortState / useAccountPersistedState all return one).
- SavedViewsProvider reads `cfg` through a ref: it re-memoizes on every filter edit, and the
  per-source seed guard already enforces once. That also made the set-state finding go away —
  with the render value gone, the checker can see the seed for what it is.

WizardProvider keeps its one escape hatch, now inline with the invariant: auto-opening is a
one-way decision taken from server data as it arrives, and deriving it would reopen a wizard
the user had just closed.

Suppressions 89 -> 78; exhaustive-deps 27 -> 23.
2026-09-21 00:12:44 +02:00
peter 6a7ff330f6 refactor(frontend): the shared hooks in lib/ stop lying about their deps (E27 S2)
Ten more suppressions, and React 19 turns out to supply the missing primitive.

useEffectEvent (stable since React 19.2, which S1 brought in) is the canonical answer to the
"bind the listener once, but read the latest values" pattern these hooks were disabling the
checker for. Three of playerModalHooks' four escape hatches become honest dep lists: the
wheel-volume handler, the native-controls handover, and the fullscreen-change sync. The
contract is no longer prose — the compiler enforces it.

The fourth stays, deliberately, and now says why at the site: the YouTube player's lifecycle
effect must run once per video (re-running destroys and rebuilds the iframe), and YT.Player
takes its callbacks as an object literal, which an effect event may not be passed into.

Fixed, not suppressed:
- useDismiss captured onClose when the popover opened, so a caller that re-rendered with a
  different handler while it stayed open kept getting the OLD one called. The disabled dep list
  asserted the opposite ("read fresh from this run's closure"). Both the handler and the ref
  list are read through refs now; three tests, the two that matter mutation-checked.
- usePref wrapped its seed in a useCallback whose dep list had to be excused. useState calls a
  lazy initializer exactly once, so the memo was saving nothing; inlined.
- useCardPager clamped an out-of-range page from an effect. Adjusted during render instead —
  React discards that pass rather than committing it, so no out-of-range page is ever painted.
- plexPlayerHooks' auto-skip countdown reads its marker from the ref that already existed, so
  the dep list names its real inputs. That also made the set-state-in-effect finding go away:
  the progress is genuinely timer-driven, and the checker can now see that.
- useUrlMirror and useChipOverflowFit only needed their stable refs listed.

Two test-harness suppressions (a render-phase probe ref, a module-level setState handle) moved
inline with the reason they exist: both exist to observe a hook from outside without changing
what is being measured.

Suppressions 99 -> 89; exhaustive-deps 35 -> 27.
2026-09-21 00:07:12 +02:00
peter 7e1aeeb9fe refactor(frontend): honest effect deps in the app shell and the ui layer (E27 S2)
Burns the first 14 of the 113 hook suppressions. Three verdicts, applied per site:

A useEffect that should not exist (6):
- Avatar remembered "the image failed" as a boolean an effect reset on src change. The
  component is recycled across items in the virtualized feed, so after one avatar 429s the
  next item's slot committed the fallback initial over a perfectly good image and only the
  post-paint effect cleared it. The failure is now keyed BY URL; derived state cannot be stale.
- Combobox cleared the typed query from an effect on `open`. Moved into setOpenAnd, the single
  funnel every close goes through. That also stops a spurious onQueryChange("") AT MOUNT, which
  told a controlled caller its own initial query had been cleared before the user touched it.
- CommandPalette reset query + picked row from an effect. Closed now means UNMOUNTED, not
  hidden, so a fresh open starts blank because mounting already means that -- and the ranking
  memo over every channel, playlist and saved view stops running while the palette is shut.
- Pager's page box held a copy of the page that an effect re-synced; it now derives it and
  keeps a draft only while being edited.
- App's ?link= / ?oauth= / ?admin= one-shots asserted "once per mount" with an empty dep list.
  The ref guard the codebase already uses in 22 places ENFORCES it, so the honest deps can
  include `t` -- which a language switch legitimately changes, and which would otherwise have
  re-fired the toast.

Honest deps that also fix something (5):
- App's module preload was keyed on the account id, so a permission change that alters the
  module set without moving the id never re-preloaded. `preload()` is idempotent, so depending
  on `me` costs nothing.
- App's access-request nudge now re-issues on a language switch; it used to stay in the old
  language until the pending count changed.
- App's diagnostics effect: applyDiagnostics already no-ops on an unchanged effective level,
  so picking four fields out by hand was guarding nothing and missed any field added later.
- DataTable's two filter effects: a ref holds the latest onFiltersChange so commitFilters can
  be stable, which is what lets the effects name it instead of switching the check off.

Genuine escape hatches (7) moved OUT of eslint-suppressions.json into inline directives that
name the invariant at the site -- the toast handover's render-phase global, the live region's
deliberate arming tick, the docked one-slot component, the popover engine's floating-ui list
state, dnd-kit's per-drag `moved` flag, BackToTop's DOM read and the palette's debounced
announcement. A reader of the code now sees why; a JSON file no one opens does not.

Tests: Avatar (3, new file) and Pager (3) pin the derived-state frames with a layout-phase
probe -- layout effects run before any passive effect, so they read the frame the browser would
paint. Combobox (2) pins the close-clears-query contract and the no-callback-at-mount rule.
All three sets mutation-checked against the effect-based originals.

Suppressions 113 -> 99; exhaustive-deps 45 -> 35.
2026-09-20 23:57:08 +02:00
peter 7d682c90d5 build(frontend): React 18 -> 19 with the React Compiler (E27 S1)
The compiler auto-memoises, which is the point: it infers what the app's
112 `useCallback` and 48 `useMemo` call sites were hand-written for and
applies the equivalent itself. It is on repo-wide with no opt-out list —
it bails on anything it cannot prove safe, and says so through the lint
rules. Verified in the built bundle rather than assumed: 369
`react.memo_cache_sentinel` markers across 35 of the 76 chunks.

ONE transform, shared by vite.config.ts and vitest.config.ts through
reactPlugin.ts. Not two `react()` calls: if the compiler ran only in the
production build, every vitest run would exercise code the user never
receives and a compiler-induced change would ship green — the same
gate-vs-image split 0.93.2 was released to fix. The plugin is imported
rather than named by the usual string, because a string is invisible to
knip and, worse, a typo in it is a silent no-op that ships an uncompiled
bundle, while a bad import is a tsc error.

eslint-plugin-react-hooks 5 -> 7 brings the whole compiler rule set: 283
findings, triaged rather than bulk-suppressed, because the two halves
needed different answers.

  react-hooks/refs (217) is OFF. 153 of those 217 are this app's popover
  architecture, not its history: 79 are @floating-ui/react's documented
  `pop.refs.setReference` / `getReferenceProps` callback-ref setters, and
  74 more are passing the popover handle down as a prop, which is what
  components/ui's popover engine is for. Suppressing them would not help
  the next popover anyone writes. The remaining ~64 are genuine
  render-phase `ref.current = x` writes and are real, but the rule cannot
  tell them apart here; they are E27 S2's work.

  The other 66 go into eslint-suppressions.json as a ratchet — new
  violations fail the gate, and `--prune-suppressions` makes a fixed one
  fail too until it is removed. 45 of them are `set-state-in-effect`,
  which IS E27 S2's subject, so that sprint now has a measured worklist.

react-i18next deliberately stays on 14. It never declared a React-19
peer — every version from 14 to 17 asks only for `react >= 16.8.0` — and
what the newer lines actually require is i18next 25/26 against our 23.
That is an i18n runtime migration wearing a React hat, and it buys this
sprint nothing.

Two React 19 type changes: `useRef<T>(null)` is now `RefObject<T | null>`
(the honest shape — the element is null until it mounts), so
PlexPlayerSettings' prop follows; and reactPlugin.ts joins
tsconfig.node.json's include.

The compiler skips memoising VirtualGrid and reports it: TanStack
Virtual's `useVirtualizer()` returns functions that cannot be memoised
without going stale. That is the right trade for a component whose cost
is its row window, and it is the one such site in the app, so it carries
a targeted disable with the reason — a second incompatible library
anywhere else must still stop the gate.

Nine lanes green, vitest 1558. Browser UAT on the dev server against the
still-running React 18 build as a control: boot, feed (virtualized grid,
popover menu, a filter change that moved the list 195,187 -> 64 items
with the URL mirrored, so no stale memoised UI), command palette, player
modal — no console errors on any of it.
2026-09-20 23:21:08 +02:00
peter cc9fc3b544 build: pin one Python version across the app, the images and the gate
0.93.2 shipped a week's worth of confusion about which Node this repo
runs on; the Python side had the identical shape and nobody had looked.
Prod (Dockerfile), the test image (Dockerfile.test) and CI (check.yml)
each spelled 3.12 as an independent literal, there was no
`.python-version`, no `requires-python`, and ruff had no target-version
so it judged our syntax against some other interpreter. The dev machine
had no 3.12 installed at all — only 3.13 and 3.14 — so every ad-hoc host
script ran two majors ahead of production.

`.python-version` is now the single source of truth, and the contract is
held by tests rather than by prose. It is split in two because no single
runner can see every link: the pytest suite runs inside the test image,
which mounts backend/ alone and cannot read the repo-root Dockerfile or
the workflow.

  test_python_engine_contract.py : running interpreter == pyrightconfig
                                   == ruff.toml
  pythonEngine.contract.test.ts  : .python-version == pyrightconfig ==
                                   ruff.toml == Dockerfile ==
                                   Dockerfile.test == check.yml

Together they close the loop, and the backend half carries the assertion
that pays — that the interpreter running the suite is the pinned one. A
green suite on another minor does not mean the shipped image is green;
it means they have drifted and nobody has noticed yet.
2026-09-20 23:02:11 +02:00
peter c9101e3450 ci: type-check the backend — the gate's ninth lane (E27 S3)
Three tsc projects held the frontend and NOTHING held the backend: ~36k
lines of `app/`, annotated in 113 of 128 files, that no lane had ever
read. Asymmetric by accident, not by decision.

basedpyright, in basic mode. Measured against mypy on this tree with the
dependencies actually installed — without them basedpyright errors on
every import and mypy's --ignore-missing-imports quietly turns them into
Any and UNDER-reports. Both land ~190-200 errors with the same
distribution, so the choice fell to what differs: basedpyright is ~2x
faster (4.4s vs 7.4s, and this runs on every `siftlode lint`), resolves
our `app.*` imports from its own config with no MYPYPATH, and has a
first-class `--writebaseline`. That last one is the ratchet itself;
mypy would have needed a third-party tool for it.

198 pre-existing errors are baselined, so the lane fails on NEW errors
only and the list burns down epic by epic. A green lane means no new
damage, not a clean codebase.

It runs in the pytest test image rather than on the host, and that is
forced rather than chosen: `requirements.lock` cannot be installed on
Windows at all — uvloop refuses to build there — so a host venv could
only ever be the lock minus a package, and a type checker without the
real third-party packages reports nonsense rather than nothing. The
image installs the pin from requirements-dev.txt, so the lane's version
IS the pin and there is no host copy to drift. The cost is that
`siftlode lint` now needs Docker, which it did not before.
2026-09-20 23:02:02 +02:00
peter 51daf3e388 feat(security): cap every request body before auth runs (SEC-20)
FastAPI reads a declared body IN FULL before the first dependency, so
`POST /auth/password-login` drained an arbitrary upload from an anonymous
caller and only then answered 401. 70 of the 72 body-declaring routes
behaved the same way: one unauthenticated client could spend the
process's memory and time at will.

A raw ASGI middleware, not `@app.middleware("http")` — BaseHTTPMiddleware
cannot wrap `receive`, so it could only trust `Content-Length`, which a
client may omit (chunked) or understate. This refuses an oversized
declaration outright and counts the bytes of everything else as they
arrive, the same rule `capped_body` already applies per-route. Registered
directly inside the access log: a refusal is still logged with its
request id, but the session, CORS, the setup gate and every auth
dependency never run for a body that is already too large.

Two routes opt out, under one stated rule: an exemption is ONLY for a
route whose own cap is larger than the global one. That is why
`/api/diagnostics/events` is absent — it caps itself at 512 KiB, below
the ceiling, so exempting it could only widen what it accepts. The list
names route templates and a test resolves each against the live route
table, so a rename fails the gate instead of silently dropping an
exemption in prod.

The 413 is byte-identical to `capped_body`'s and the test compares the
two against each other rather than against a literal — two shapes for one
fact is how the `cancelled`/`canceled` bug happened.
2026-09-20 22:51:32 +02:00
peter 09d351a0e7 docs(ci): the two gate jobs queue, they do not run in parallel
Declared side by side, so they read as parallel; the lab runs a single
act_runner with one slot, so the backend job starts at the exact second the
frontend one ends. Measured across runs #86-#91: ~4 + ~6 = ~10 minutes from the
push, not ~6.

Written down because believing the comment cost something real — the ops
release read-back budgeted the max instead of the sum and timed out on its
first live run, reporting a still-running gate as a failing one.
2026-09-20 22:05:56 +02:00
peter b5078003ce release: 0.93.2 (bug/ci-node-version-split)
gate / frontend (push) Successful in 3m51s
gate / backend (push) Successful in 6m7s
2026-09-20 21:49:56 +02:00
peter f2c32cfb8e chore(release): 0.93.2 — a Plex amount reads the same in every browser 2026-09-20 21:49:36 +02:00
peter 1453fa4a33 ci: run the gate on branch pushes only, not on release tags
A release pushes main and then the v<ver> tag, and a bare `on: [push]` treated
the tag as a second push of the identical commit — every release ran the whole
gate twice to the same verdict (runs #81-#90 are five such pairs).

It has to be `branches`, not the `tags-ignore` this obviously wants to be:
Gitea's matchPushEvent requires every named filter to match and each one bails
on the wrong ref kind, so `tags-ignore` rejects every BRANCH push and silently
disables the gate. Measured in go-gitea v1.27.3, where it differs from GitHub.

A trigger filter that stops firing is invisible, so the ops `release` now reads
the verdict back for the sha it pushed and says so when there is no run for it.
2026-09-20 19:47:19 +02:00
peter efe106dfda build: pin one Node major across the dev, CI and build machines
The server gate ran node:22 while the dev gate ran node:24, and the two disagree about
Intl.NumberFormat's compact currency padding and about which realm's Blob a Response hands
back — so the frontend job had been red for 73 of its last 90 runs while the local gate was
green, and nothing said so.

Align everything on 24, with `.nvmrc` as the single source of truth: the CI container, the
prod build stage, `engines`, and `@types/node` (which was two majors ahead of the runtime, so
tsc was green over APIs the shipped engine does not have). The backend job's node only runs
the checkout action, so it tracks the pin as hygiene — node 20 is EOL — but deliberately
stays outside the contract.

The pin is held by a contract test that reads every one of those places AND the engine
running it, so the next drift is a red lane on its first run instead of a CI log nobody reads.

Verified on node:24-bookworm (the CI lane: 228 files / 1552 tests, typecheck, lint,
format:check, knip all exit 0) and on node:24-alpine (the prod build stage, v24.21.0, SPA
built). Each assertion of the new test was mutation-checked red.
2026-09-20 18:06:24 +02:00
peter ec63056bd7 test(api): assert the export body's brand, not which realm built its Blob
`expect(x).toBeInstanceOf(Blob)` asks which realm constructed the body, not
whether there is one. Under jsdom the ambient `Blob` is jsdom's class, while
`Response.blob()` hands back node's own on Node 22 and jsdom's on Node 24 — so
both assertions passed locally and failed on the server gate, for a reason that
says nothing about the code under test. `.text()` is the mirror image: jsdom's
Blob does not implement it, so reading the bytes back is green on 22 and red on
24.

Both now assert the brand (`[object Blob]`, agreed by both realms) plus size and
type. Mutation-checked in node:22-bookworm: dropping `{ raw: true }` from
`debugExport` still turns the test red, so the repaired assertion still catches
the defect it was written for.

core.test.ts also stops building its Response from a jsdom `Blob`: on Node 22
undici does not recognise it as a body and stringifies it, so the fixture's body
was the 13 bytes of "[object Blob]" rather than the zip header. A string body is
what the sibling test already used.
2026-09-20 16:33:26 +02:00
peter 09becb3d00 fix(plex): pin the compact currency format instead of inheriting the engine's
`{ notation: "compact", maximumFractionDigits: 1 }` leaves the MINIMUM to the
engine, and engines disagree: the same options render "$160.0M" on Node 22 and
"$160M" on Node 24 (measured 2026-09-20 in node:22-bookworm, the image the
server gate runs, against the local Node 24.17). Browsers split along the same
line, so a viewer on an older engine saw the padded form. `minimumFractionDigits:
0` makes both render "$160M"; the plain `compact` helper (vote counts) was
measured too and already agrees on both.

This is why the server gate's frontend job was red on PlexInfo since 09-17 while
the local gate was green: the two run different Node versions.
2026-09-20 16:33:26 +02:00
peter 8d2ec5e867 release: 0.93.1 (bug/preview-crashes-the-tracker)
gate / frontend (push) Failing after 4m10s
gate / backend (push) Successful in 5m50s
2026-09-20 07:55:37 +02:00
peter 0e9839e73d chore(release): 0.93.1 — the report preview works again 2026-09-20 07:51:16 +02:00
peter 05ab963951 fix(issues): the preview must not ask the tracker for a repository context
The composer's preview did not fail a call — it killed the Gitea process, and
with it the git server and the container registry. `POST /api/v1/markdown` with
`{"Mode": "gfm", "Context": "<repo>"}` makes Gitea resolve every hex-shaped word
as a commit link, and on the API's render path the commit checker it asks is nil:

  panic: runtime error: invalid memory address or nil pointer dereference
  renderhelper.(*commitChecker).IsCommitIDExisting(0x0)
   <- markup.hashCurrentPatternProcessor (html_commit.go:217)

Every report carries hex — a tab id, a request id — so this was not an edge case
but the only case: the preview answered "the tracker isn't answering" every time,
because the tracker had just died.

Measured on throwaway instances, not argued. Same body, same call:
  gfm + Context + "5645b1c0ae01"  -> 1.26.4 CRASH, 1.27.1 CRASH, 1.27.3 200
  gfm + Context, no hex           -> 200
  gfm, no Context                 -> 200, and the HTML keeps its tables, task
                                     lists and chroma-highlighted code blocks
Upstream fixed it in 1.27.2 ("fix: avoid markup render panic", #38698), but a
self-hoster runs whatever version they run, so the payload is what gets fixed.

Sending was never affected: the created issue's own page renders the same body
fine (measured) — a real repo context has a real commit checker. The only loss is
that a hash or a `#123` is not yet a link in the preview; the issue page links it.

The test pins the payload dict, because the defect is in what we send.
2026-09-20 07:51:08 +02:00
peter 8142484347 release: 0.93.0 (feature/e16-debug-issue)
gate / frontend (push) Failing after 4m11s
gate / backend (push) Successful in 5m58s
2026-09-20 05:54:26 +02:00
peter 0d91b82f86 chore(release): 0.93.0 — report a fault from inside the app
E16: a recorded debug session becomes a drafted issue with its own logs attached, sent to the private Gitea tracker or downloaded as a zip. Includes the four epic-end review rounds and the Plex player's focus trap.
2026-09-20 05:54:03 +02:00
peter 90fc86d437 fix(diagnostics): deep-review round 4 — a claim only its owner may write, and a player that keeps the Tab
Round 4 was targeted at the surfaces rounds 2 and 3 changed. 29 findings raised,
26 survived deduplication, 12 survived three adversarial readings each.

`set_delivery` renewed the claim but never checked it was still ours, while its
sibling `release_send` always did. `expire_on_commit` is off, so the token it
re-stamped came from the caller's own memory, not the row: a send that outlived
its 600 s lease came back, overwrote the take-over send's committed links with a
full-column write, and put its own dead token back. It now takes the token, reads
the row's claim with a targeted scalar (not a refresh, which would discard the
`issue_url` assigned one line earlier) and raises `SendClaimLost` rather than
writing. An issue created by a send that had already lost its claim is logged with
its number — that record is the only one that will exist.

The tracker's settings are read above the pre-call checkpoint instead of inside
`_client`. Reading them there was a SELECT one line after the commit that was
meant to end the transaction, and held it — idle, snapshot pinned — across the
repository check and the create. The comment and the module docstring said
otherwise; measuring it in pg_stat_activity is what settled it. Two more comments
were saying things the code had stopped doing: the `**state` spread does not carry
the claim (`set_delivery` strips and re-stamps it), and react-query does not
debounce the preview — the editor being unmounted is what paces it.

The composer stopped stranding the focus on `<body>`: a send that creates the
issue and then fails on an attachment unmounts the button that was just pressed,
and only the success path handed the focus on. Export no longer saves into a
refusal it does not need — the zip is the way out of a stuck send, and its
pre-save was the one step that could not survive one. The delivery banner
announces through a role instead of an `aria-live` created with its own message.

`historyState` returns a union, so `chipTone` matching on the same keys is checked
by tsc rather than by hope.

PlexPlayer was the only full-screen overlay in the tree with no focus containment
at all — Tab walked out of an opaque player into a page nobody can see. It now
carries the shared trap. Closing that hole exposed the one underneath it:
`useDismiss` registered as a layer that traps the Tab while installing no trap,
so the player stood its own down for its own menus. Measured in a real browser:
30 Tabs with the settings panel open, 9 escapes into the nav rail, and 0 after.
Escape ordering is unchanged — the panel first, then the player.

Three tests that did not hold their line: the advisory lock's call site (deleting
it left the suite green), the trail assertion whose `all()` was vacuous over an
empty filter, and a `layerStack` assertion the stand-in dialog already forced.
All eleven executable fixes mutation-checked red for the right reason; the two
comment corrections have nothing to pin.
2026-09-20 04:30:42 +02:00
peter 2fe58073e4 fix(diagnostics): deep-review round 3 — a keyboard owner that is not a Tab trap
Round 2 made the step prompt a blocking layer so that `j` and the `g` chords
would not reach the page behind it. The dialog it sits in gave up its Tab trap
because of that, while the prompt's own replacement trap covered only its own
two controls — so Tab from anywhere else in the dialog (the bar's Screenshot or
Stop, the dialog's own buttons) walked out behind the scrim. `layerStack` now
separates the two questions it had folded into one flag: a layer can own the
keyboard and still leave the Tab trap to the dialog below it. The mirror case,
a dialog opening over the prompt, falls out of the same split.

A send's claim is now a lease with an owner. Every progress report re-stamps
it, so a send slower than the lease cannot lose it while it is still running,
and only the send that took a claim can give it back. The issue_create audit
row is staged inside the send and rides the same checkpoint as `issue_url`, so
the trail cannot be missing for an issue that exists. The attachment-name
prefix has one definition, shared by the producer and both readers.

A running send is reported to the UI. A reload or a second tab used to render
it exactly like a send that STOPPED, and offered a retry the server was certain
to refuse; the composer now says the report is being sent, holds the draft, and
puts both the retry and Create issue out. The past-sessions list says the same
for the same reason — it is the list a user opens to check on a send, and both
"draft" and "half sent" describe one that died. `/preview` and the retry path
got a pace of their own, and the per-user byte cap took the user-scoped lock
that makes it a cap instead of a guess.

Verified in a real browser on localdev: Tab cycles the whole dialog with the
prompt open, the prompt still swallows the page's keys, and the issue link is
focusable where the composer puts it. All 24 regression tests mutation-checked.
2026-09-20 02:43:56 +02:00
peter bc03e82687 fix(issues): a send keeps its claim through every step, not just up to the issue
Creating the issue rebuilt the delivery state from scratch and dropped the
claim with it, so for a new issue the claim covered only the creation — a
draft, a screenshot or a second send could change the report under the uploads
and the body edit that follow, which is exactly what the claim is for. The
state is now built on what was read, and a test walks the send's checkpoints
to see the claim still held at each one.

Found by mutation-checking the round-2 tests: removing the claim's release
left them green, because for a new issue there was nothing left to release.
2026-09-20 01:03:23 +02:00
peter 3f983e6d76 fix(diagnostics): deep-review round 2 — a send that claims, not one that blocks
The send no longer holds the session's row lock while it talks to the tracker:
it claims the session, gives the lock back, and checkpoints each step. A Stop
is never held behind a send, while a draft, a screenshot or a second send are
refused at once instead of queueing. The claim ages out, so a request that
died mid-send does not strand the session.

What the composer says is what happened: an attachment that went is no longer
named as one that did not (the missing body edit is its own line), and the
issue's screenshot count comes from the server when the list cannot be read,
so a failed read no longer blocks saving the draft. The zip is fetched as a
request, so its pace is a message and not a saved error page. Screenshots of a
finished report can be removed again; only one still being sent is held.

A recording's log elevation can no longer outlive it. The step prompt keeps
the keyboard (the page's keys stay quiet) and traps Tab itself. An image
pasted into a dialog opened over the composer stays that dialog's.
2026-09-20 00:51:12 +02:00
peter 134ec9a68e fix(diagnostics): deep-review round 1 — a send that cannot race, lose or repeat itself
The send, the draft and the screenshots take the session's row lock, so an
autosave or a paste during a send waits for it and then sees it, and two sends
cannot open two issues. A save keeps the delivery's progress; a failed edit
that shows the screenshots is finished by the retry; an answer that is not the
tracker's API (a redirect, a proxy page) is refused, not a crash that loses the
issue's record. Only an attempt that opened an issue counts toward the pace.

Screenshots: deleting one after the send is refused, a user's storage is
capped across sessions, new sessions and zip exports are paced, images are
stored uncompressed in the zip. One liveness rule, one session view, one image
type table.

Logging: a session's elevation is tracked apart from the user's own, so the
browser's own requests (no header) no longer lower the floor a session needs;
Stop ends it at once.

Composer: an image pasted anywhere is taken; what is still unsaved is saved on
close, and "saved" is shown only when it is; the draft is held while it is
sent; a failed screenshot read says so instead of reading as none; focus moves
to what stays; the preview follows the text size. Pill: the step prompt and a
running Stop survive the pill moving between hosts, the prompt keeps the
dialog's Tab trap, the toast region lets clicks through, other tabs hear of a
Start or Stop, the last seconds are sent before the cap, a capture with no
frame is an error.
2026-09-20 00:09:52 +02:00
peter 9cf177b198 feat(diagnostics): send a bug report to a private Gitea issue tracker (E16 S4)
- admin config group "Bug reports": Gitea address, repository, token (secret),
  and "Check the connection" — the repository must answer, take issues and
  be PRIVATE (a report carries its reporter's log); checked on every send too
- POST /api/debug-sessions/{id}/issue sends the SAVED draft: create the issue,
  upload each file on its own request (the measured API), then show the
  screenshots inline; progress is kept in the draft, so a send that stops
  half-way keeps its issue and the retry sends only what is missing; 5 new
  issues an hour per user; audited as issue_create
- the composer: Create issue (only where a tracker is set up, naming the
  repository), a partial-send banner with retry, and a Preview — the tracker's
  own rendering in a sandboxed frame, or the zip's markdown without one
- the recorder's refusals answer in the localized family (debug_session_*,
  screenshot_*, issue(s)_*), not bare codes the UI could only call generic
- a tracker refusal logs the status only: its message can quote the title back
2026-09-19 22:48:20 +02:00
peter a5fae489e0 test: a request that raises is rolled back, as production rolls it back
The shared test session kept whatever a failing handler flushed and never
committed, so a later read in the test saw a write production loses when
get_db closes the session — and a handler whose failure path must commit
(E16: the tracker's issue link) looked right without the commit.
2026-09-19 22:48:20 +02:00
peter 8d0083d4e4 feat(diagnostics): the issue composer — draft, review, screenshots, zip export (E16 S3)
After Stop (or at the cap, or from Settings → Diagnosztika's past sessions)
one dialog drafts the issue: title, description, the steps drafted from the
session's markers and page changes (a marker without a label is named by its
time), the environment (version, browser, system, window, language). Beside
it, what goes with it: the browser and server logs — each reviewable line by
line, an unticked line is not sent — and the screenshots, removable, with
paste / drop / pick to add more.

- it saves as it goes (debug_sessions.draft, migration 0090): the rendered
  markdown in the user's language, the excluded rows, and its own fields to
  reopen as it was; a sent session is read-only
- the zip export (issue.md, client.log, server.log, screenshots) is built
  from the saved draft by app/issues/bundle.py — the one definition the Gitea
  delivery (S4) will share; .log, not .jsonl, which Gitea refuses
- the log line's level tags and clock move to components/logLineParts so the
  review list draws a level the way both log viewers do
2026-09-19 22:26:49 +02:00
peter 9f177557d2 docs(ui): the toast dock in the primitives catalog 2026-09-19 22:08:21 +02:00
peter 46fce4a767 feat(diagnostics): the debug recorder — start, the pill, steps, screenshots, stop (E16 S2)
- the start dialog is the consent: level (Részletes/Teljes), what is recorded,
  retention and the cap (now served as diagnostics.session_minutes)
- the pill docks under the toasts: elapsed time, Step (an optional label,
  its own Escape layer), Screenshot (tab capture where the browser has it),
  Stop; compact on a phone
- while recording, lib/log tags every event and every request path (req,
  beacon, uploadReq via one identityHeaders) names the session; applyDiagnostics
  raises the level for the session's run and never lowers a raised one, and
  lapses at whichever ends first
- Stop sends what is waiting before the server closes the session; the cap
  says so and opens the composer
- Settings → Diagnosztika and the command palette start/step/stop it
- the evidence leaves the log upload's own request lines out
- the step shortcut is deliberately palette-only: the keymap's D1 rule bars a
  new modifier chord, and a bare letter collides with the players'
2026-09-19 22:08:10 +02:00
peter 2d9595b0e5 feat(ui): a control can dock under the toasts and ride every toast host
The toast stack already has a host in the page and in every overlay (dialogs,
both players, the fullscreen player), the topmost one shows it, and inside an
overlay it sits above the player's control band. lib/toastDock lets one control
use exactly that placement; it renders outside the live region, so a ticking
control is not read out every second.
2026-09-19 22:08:10 +02:00
peter a7d7456279 feat(diagnostics): debug sessions — start, stop, the tag on every line, screenshots (E16 S1)
A session is one recording, capped at 60 minutes by the server. While it is
open, a request naming the caller's own session in X-Debug-Session is logged
at the session's level with every line tagged; a header naming anything else
is ignored. The browser's batches count a session as consent to its level and
keep the tag only on its own events. Screenshots are judged by their bytes,
capped in size and count, and served back to their owner only.

- debug_sessions / debug_screenshots tables, one open session per user
  (partial unique index), log_events.debug_session_id indexed
- /api/debug-sessions: start (joins a running one), stop, list, get, events,
  screenshots; /api/me reports the live session
- the hourly log job closes expired sessions and drops them with the logs
- logsink.serialize and routes._common.capped_body shared by their readers
2026-09-19 21:38:53 +02:00
peter 44a792c34f test(i18n): a 404 target the scan cannot read fails the backend-id contract
what=SOME_CONSTANT reached no locale check at all; only _common.py's own
pass-through may name the target by variable.
2026-09-19 21:38:53 +02:00
peter 3987cd4260 release: 0.92.0 (improvement/debt-burn)
gate / frontend (push) Failing after 4m30s
gate / backend (push) Successful in 5m25s
2026-09-19 19:53:37 +02:00
peter 95d9ed4aa6 chore(release): 0.92.0 2026-09-19 19:53:32 +02:00
peter 8d0063d514 fix(ui): the reader's own use of a stand-in stops the restore; the pill's name carries its cap
Round 2 of the deep review:
- A click on, or a key typed into, the focus keeper's stand-in fires no focusin, so a restore still
  polling for its list could take the focus from under the reader. A pointerdown or keydown now
  makes it the reader's own focus.
- The Jump-to pill hid its "Ctrl K" from its name (the voice command "click Ctrl K" still matched
  nothing); the key cap's tooltip now equals its name, so it is not read twice as a description.
- listNav's comment still cited DataTable's two layouts as the reason for its visibility filter.
2026-09-19 19:30:27 +02:00
peter 0ca0cac534 fix(api): a waiting reorder deals from the fresh order; the commit check sees staged writes
Round 2 of the deep review, on round 1's own fixes:
- `ORDER BY position … FOR UPDATE` sorts before it locks, so the reorder that waited got the
  fresh rows in its STALE order and dealt the pins it left out (a tab with Plex off) back to their
  old slots. It locks in id order — one global order, so two reorders cannot deadlock — and sorts
  the fresh rows itself. The race test now sends that partial list and fails unless tab B really
  blocked.
- The request check saw only writes the handler flushed: a `db.delete` left staged, or a flush
  followed by a released savepoint, passed. Staged changes count at the end of the request, and
  only the ROOT transaction's end clears it (a savepoint test drives the dependency directly).
- The `ok` test exempted every call as a serializer, hiding four domain REPORTS (YouTube sync,
  revert, push; Plex sync) without `ok`. They are spread into `{"ok": True, …}`; the exemption is a
  named list of real serializers.
2026-09-19 19:30:27 +02:00
peter cf0c55ba80 fix(a11y): names that carry what they show; the unused-key check stops covering whole files
- The palette launcher's name replaced its visible "Jump to…" (and the key cap's its "Ctrl K"):
  a voice-control "click Jump to" matched nothing (WCAG 2.5.3). Each name contains its text now.
- SegmentedToggle's `title` was only a tooltip on a non-focusable box — the live search's "how
  many per batch" hint reached no keyboard or screen-reader user. It is the group's accessible
  description, standalone and captioned alike.
- The unused-key contract let a template like `card.${status}` cover its whole namespace, which hid
  five dead keys (deleted). Dynamic keys are matched by shape (`editor.*.hint`); a bare namespace
  shape covers nothing, and the audit/quota namespaces are now held both ways by backendIds.
- CLAUDE.md's "FE control" row matches the lint rule it cites; each no-restricted-syntax entry has
  its reason directly above it.
2026-09-19 19:19:43 +02:00
peter fa4f823fc8 fix(ui): Back finds the item again past the focus keeper; nested drags stay in their list
Deep review of the debt burn, each measured in Chrome:
- Back from a sub-view whose focused control the swap removed: the focus keeper put the focus on a
  stand-in (the returning list's first item) and `restoreNavPosition`, finding the focus "already
  somewhere", gave up — Plex show → Back landed on tile 0, not the tile it was opened from. A
  stand-in the keeper placed now counts as untouched, like <body>. The keeper's walk up also stops
  at its container: a slot left with nothing focusable no longer sends the focus to the first
  control of the whole document.
- DataTable enabled both layouts' keyboard lists with the same flag, and `useListNav` reads its ref
  only when the flag changes: after a resize across `md`, the layout that mounted was never
  registered (no j/k) and the one that left stayed registered. Each is enabled with its layout.
- A Plex playlist nests blocks and each show's episodes in one DndContext; closestCenter and the
  keyboard coordinates weighed every droppable, so a movie landed "over" an episode — announced as
  a place in the other list, dropped somewhere else or nowhere. `sameListCollision` and
  `sameListKeyboardCoordinates` keep a drag among its siblings; onDragEnd's episode → block
  resolution is gone with the case.
- dnd-kit reports the item over itself in the pick-up's own tick; announced, "moved to position 1"
  replaced "picked up … use the arrow keys" before a screen reader read it.
2026-09-19 19:19:43 +02:00
peter 884060891e test: a handler that flushes a write and never commits it fails its test
Helpers flush and handlers commit — but every test request shared the test's session, so a
follow-up GET or an assertion read the flushed write and the suite stayed green over a handler
whose commit was missing (production's get_db closes the session: the write is rolled back behind
a 200). The db fixture now hands out `db.info["request"]`: the same session, recording each request
that ends normally with a flushed, uncommitted write, and failing the test on it at teardown. All
28 get_db overrides use it (a convention test keeps it so); the full suite passes under it, and
dropping the commit from add_item turns a playlist test red.
2026-09-19 19:19:30 +02:00
peter 955c8c3445 fix(api): a token hiccup is not "reconnect"; reorders and approvals are one unit each
Deep review of the debt burn:
- `refusal_reason` told the user to reconnect on ANY token-endpoint failure — a 5xx, a 429, a
  misconfigured client — while the grant stayed and the account still read as connected, with no
  connect button. Only `invalid_grant` (the one that drops the grant) means reconnect now.
- `pins.reorder` writes only the rows that change, so two overlapping reorders each skipped rows
  that matched in their own stale snapshot and interleaved into two pins on one slot. It locks the
  user's pins first; the second waits and deals from the first's result.
- Adding a whitelist entry committed the invite and the activation it implies separately, and its
  audit row was written before a new invite had an id (target NULL). One transaction, flushed for
  the id.
- The mutation convention test skipped async routes and dicts returned by name: request-access,
  purge-discovery and the subscription sync answered without `ok`. They do now, and the test
  covers both. The helper-commit check looked only at `_`-names; it reads every non-route
  function, which listed two deliberate committers (upsert_pending_invite, ensure_welcome) — now
  allowlisted with their reasons.
2026-09-19 19:19:29 +02:00
peter 99df44d7fb fix(api): mutations answer ok; route helpers flush and leave the commit to the handler
Two rows of the canonical-patterns table had drifted, found while closing the roadmap audit:

- 63 mutation responses carried no `ok` — `{"unsubscribed": true}`, `{"status": "ok"}`,
  `{"added": 3, …}` and so on. Each now leads with `"ok": true` (additive: every field a client
  reads is still there).
- Twelve route helpers committed on their own. Six were just the end of their handler's work and now
  flush: the playlist item add/remove and Watch later get-or-create, the Plex series/season bulk
  mark, and the invite decision + account activation — which an approval used to land in separate
  transactions. The other six commit on purpose (a token durable before its link is mailed, a
  single-use token burned as its own unit, the Google callback's tail, a background registration
  with its own session, the staged search ingest around YouTube calls), each listed with its reason.

`tests/test_route_conventions.py` holds both rules in code; CLAUDE.md's two rows point at it.
2026-09-19 18:21:06 +02:00
peter 30fbdc52b2 fix(ui): a focused, disabled FocusSafeButton no longer lights up on hover
A FocusSafeButton that holds the focus is disabled by aria-disabled, not the native attribute, so
every hover gated on `:disabled` / `enabled:` alone still fired on it — the glass lift, the primary /
danger / warn dim, and eight `enabled:hover:` chains (saved views, the player's list buttons, the
undo toolbar). Measured on localdev: the Pager's Next, focused at the last page, now keeps its resting
border and shadow under the pointer. The CSS selectors and the chains also exclude
`[aria-disabled="true"]`, and a lint rule refuses a bare `enabled:hover:` from now on.
2026-09-19 18:14:18 +02:00
peter 046ff735cf test: close the last coverage debt the roadmap still listed
- `overflow-clip` gets a safelist anchor in index.css (its only occurrence is a `.ts` constant),
  and the chip-lane contract keeps it there.
- PlexPlayer's Backspace ownership rule is pinned on all three branches: nothing above, a foreign
  layer above (it defers), and its own panel open under a layer (it still closes the panel).
- Removing a playlist item bumps the list's `updated_at`, like the three other item-only writes.
- Seen in a real browser on localdev: an exhausted comfort-shelf tile (an empty pinned playlist) is
  focusable, aria-disabled, and its Shift+F10 menu offers Play off and Remove from shelf.
2026-09-19 18:14:18 +02:00
peter 7afaba0547 feat(playlists): say which lists YouTube's API cannot give us
The module shows fewer lists than youtube.com, and without a word about it that read as a Siftlode
bug (the owner's ask, 2026-09-03). Owner's pick from the S7 mockups (2026-09-19): one quiet line at
the foot of the rail, "Missing a list? Why?", whose Why? opens three facts — YouTube's own Watch
later and History are closed to its API (the Watch later list here is Siftlode's own), only lists
you own sync, and an account with no YouTube channel has no playlists at all.
2026-09-19 17:58:40 +02:00
peter db8ed5a004 feat(ui): a visible way into the command palette
Ctrl/Cmd+K and the shortcut sheet had no entry point but a palette row, behind the other
undiscoverable chord. Owner's pick from the S7 mockups (2026-09-19): a key cap at the end of the
header's search field opens the palette, and a page with no search field shows a small "Jump to…"
pill with the same key cap in its place, so the entry is on every page. The cap reads the chord the
palette row binds from the keymap table (Ctrl K, or ⌘ K on a Mac) and stays out of a phone's
header, where there is no keyboard to press it on.
2026-09-19 17:58:39 +02:00
peter 0ef25a8037 feat(ui): the six native selects become the house controls
Owner's pick from the S7 mockups (2026-09-19): a short fixed set is a SegmentedToggle — the live
search's result count, the table's rows per page, a download profile's mode (on its own row, so the
three worded choices fit a phone) and audio format — and a longer set or the user's own list is a
FilterSelect: the profile's quality (seven heights) and the download dialog's profile.

A lint rule refuses a native <select> from now on (the canonical "FE control" row). The px-frozen
size rules also caught up with a fraction: `text-[9.5px]` had slipped past the integer-only pattern
in the Glass tuner, which is rem now.
2026-09-19 17:53:22 +02:00
peter 17a26894b0 fix(a11y): every sortable list speaks the user's language to a screen reader
dnd-kit's screen-reader text is hard-coded English and names an item by its id ("Draggable item
10357 was dropped", a role of "sortable"). Only the YouTube playlist replaced it; the Plex playlist,
the saved-views list and the side-panel groups read it out under a Hungarian UI — found while
checking the Plex playlist's keyboard reorder on localdev.

`ui/sortableA11y` is now the one place: `useSortableA11y` gives a DndContext its instructions and
announcements (HU/EN, `common.dnd.*`), `useSortableItem` announces the role. The YouTube playlist's
own announcements move onto it, which also fixes them: its lists reorder on drop, so reading the
dragged row's own index said "moved to position 1" at every step and "dropped at position 1"
wherever the row went. The place announced is now the slot the row would land in.

A `local/sortable-a11y` lint rule (with its contract test) refuses a DndContext without
`accessibility` and a direct `useSortable`.
2026-09-19 17:37:21 +02:00
peter 946c8f26db test(plex): drive the playlist's reorder and rollback through a hook
The Plex playlist's reorder is a dnd-kit drag, and neither of its sensors activates under jsdom,
so `commit()`'s failure path could not be reached from a test. The query, the optimistic copy and
the two writes (reorder, remove) move into `usePlexPlaylistEdits`, unchanged, and are driven
directly: a failed reorder is said and the rows go back; an accepted order lands in the detail
cache; a failed removal comes back.

It also settles the `putBack` question that could not be driven before: with the refreshed list
taken in BEFORE the reorder fails, a rollback bound to the render that started the reorder puts a
removed row back, and the deep-equal refetch never clears it. The live cache read the code has now
is what keeps it out; the test fails if it goes back to the closure's `q.data`.
2026-09-19 17:30:31 +02:00
peter 8759b5860d test(be): every dependency override goes through override_dep
22 test files set `app.dependency_overrides[...]` by hand and tore down with a blanket `.clear()`
in a `finally`, which wipes every override rather than the test's own and stops fixtures composing.
They now take the `override_dep` fixture (helpers get it passed in), whose teardown restores each
key's prior value; no clear() carried meaning between two phases of a test. A convention test fails
any test file that writes `dependency_overrides` directly.
2026-09-19 17:26:07 +02:00
peter dc902d2c0e test(fe): cover the palette's content, AltGr, the banner's name, share colours, playlist headers
- AppCommandPalette had no test: modules gated by the account, hidden channels left out, the
  playlist row selecting before it navigates, and the surprise riding setPage's `then`.
- keymap: an Alt/AltGr-modified key (Ctrl+Alt on Windows, a constant on the HU layout) fires no
  bare-key or Mod row.
- Banner: the dismiss button falls back to "Close" rather than going unnamed.
- ShareDialog: a protected link's lit lock and Revoke's danger colour stay on the icon.
- Both playlist headers are rendered now: the YouTube one needed the toolbar slot it portals into.
  Meta line sums known runtimes and treats an ABSENT duration as nothing (null never exercised the
  guard: `3000 + null` is 3000); a one-row list offers no view controls.
2026-09-19 17:23:55 +02:00
peter cedcbb1434 test(i18n): every locale key must be read by the app; 75 dead keys removed
A contract test scans the source (tests excluded) for each key as a whole literal, or under a
dynamic prefix like `plex.sort.${k}`, with plural forms counted through their base key. It found 75
keys that had outlived the code showing them (the old sidebar sort vocabulary, the old login page, a
pager the DataTable replaced, …); they are gone from EN and HU. A key read in a way the scan cannot
see goes in an explicit ALLOWED map with its reason.
2026-09-19 17:15:18 +02:00
peter 4f72b193aa fix(i18n): every label a screen reader or tooltip says is translated
- The Plex playlist rows' "reorder"/"remove", the Plex filter chip's "remove" and the player
  slider's "reset" were English literals; each now names its row or setting in HU/EN.
- The playlist show header's chevron had no accessible name; its decorative thumbnail button left
  the tab order.
- The opt-in Glass tuner panel was English only: every word is translated, and each slider's label
  now names its number box and range.
- New `local/literal-a11y-text` lint rule (with its contract test): a literal word in aria-label,
  title, placeholder, alt and the other spoken/shown attributes fails lint.
2026-09-19 17:12:47 +02:00
peter 7b94e9a651 fix(home): shelf honours the Plex gate; hero unpins its origin; palette asks for a surprise
- A13: with Plex off, the Home shelf and the hero's comfort tier go YouTube-only like the rails
  beside them (owner's call: hide the tiles). The pins are kept; GET /pins still lists them.
- Reorder deals the listed pins into the slots they already hold, so a partial list (hidden Plex
  pins, a stale tab) no longer puts two pins on one position; repeats are ignored.
- A21: a comfort hero carries `pinned_from`, the shelf entry it was resolved from; its menu acts on
  that pin, so a pinned show's next episode offers "Unpin" instead of creating a second pin.
- B12: the palette's "Meglepetés" asks for the surprise tier (the next press leading with it); the
  rotation carries on from there.
- B13: the surprise token names the history entry it was requested for (minted by setPage's new
  `then`, after the push) and is dropped on any other entry, so a navigation that never mounted
  Home cannot replay it later.
2026-09-19 17:07:11 +02:00
peter c6bf84a578 fix(home): a shelf move acts on the order shown, one write at a time
- During a budget switch the rail shows the previous budget's payload
  (keepPreviousData) while the new key is empty; the move read that empty
  cache, found no tile and silently did nothing. It now acts on the order
  the rail rendered.
- A refused move was undone only by a re-read; when that read failed too,
  the shelf kept showing the order the user had just been told was not
  saved. Moves are now written one at a time (the move rows wait, with a
  "saving" note), which makes restoring the shown order correct — the
  overlapping-moves problem that ruled a rollback out cannot occur.
2026-09-19 08:13:07 +02:00
peter 444f9b8166 fix(ui): a control removed by its own action leaves the focus in its slot
The second half of the pending-button sweep: where the pressed control does
not merely disable but LEAVES the page — an invite row gone after Approve,
the inbox's Clear all gone with the last notification, a section gone once
its last row is handled — the browser dropped the focus to <body> and the
next Tab started over at the top.

useFocusKeeper, on the page scroller: when the element holding the focus is
removed and the focus fell to <body>, the first focusable thing in its slot
takes it (what followed, else what preceded, else the nearest ancestor still
on the page). Portaled layers (menus, dialogs) sit outside the scroller and
return focus themselves; listNav's keyboard lists are left alone — their `j`
already goes on from the slot, and a row scrolled out of a virtualized list
must not drag the focus after it.
2026-09-19 08:10:03 +02:00
peter 7a47223a58 fix(ui): no button drops the keyboard focus when its own action disables it
A focused button that turns natively disabled loses the focus to <body> in
Chrome. The app disables the very button that was pressed on ~60 sites — a
mutation pending, a pager at its last page, undo with nothing left, a player
at the end of its queue, a form gone clean — so a keyboard user was thrown
out of the page on every such press.

One primitive instead of a fix per site: ui/FocusSafeButton renders a
focused-and-disabled (or busy) button as aria-disabled with a click guard,
and applies the native attribute again once the focus is elsewhere. It asks
the document at render time as well as listening for focus events, because
a window without focus fires none (measured). ui/Button and ui/form's Switch
are built on it; 57 hand-rolled buttons swapped their tag for it keeping
their classes, and three hand-copies of the rule (the channel page's Block
and Hide, ChannelDiscovery's Subscribe) now pass `busy`. The soft state's
dim look is one rule in index.css.

`local/focus-safe-disabled` refuses a new raw <button disabled>, pinned by
focusSafe.contract.test.ts. lib/plexDetailUi moved to components/: it is
components, and lib may not import the primitive.

Measured on localdev in Chrome: the audit log's Next button keeps the focus
on reaching the last page, reads aria-disabled, and the guarded click does
nothing.
2026-09-19 08:07:25 +02:00
peter bc138e3b37 fix(ui): a disabled menu row says why it is off, under its label
The reason for a disabled menu row lived in a `title` on a natively disabled
button: no pointer events, so no tooltip, and roving focus skips the row —
four translated sentences reached nobody (the Watch later rename/delete
lock, "no other list to move to", the reorder gate). An aria-disabled row was
tried and reverted in E10's review. ActionMenu items now take a `note`,
rendered as a muted line under a disabled row's label: seen, and readable in
a screen reader's browse mode. Only the icon and label dim — row-level
opacity would dim the explanation too. The reorder group states its reason
once, under its first row.
2026-09-19 07:57:20 +02:00
peter df7c3db085 test(ui): the px-box probe does not trip its own rule; tsc lib-safe last()
The pxBox contract test held the literal w-[150px] it probes, which the new
lint rule refuses, and stickyTop's test used Array.at, outside the tsconfig
lib. Both landed because a local lane check piped its exit code into tail.
2026-09-19 07:53:01 +02:00
peter 6fda49369f fix(ui): a sticky table header turns the scroller's top fade off itself; DataTable renders one layout
- "A page with a sticky ColumnHead must declare fadeTop: false" lived in
  prose: a registry flag on three pages plus an App special case for the
  Channels card layout, and the next table page would have re-acquired the
  fade-on-the-header defect with every lane green. ColumnHead now registers
  itself with the enclosing PageScroller (ui/stickyTop) while mounted; the
  registry flag, the App branch and the prop plumbing are gone. Measured on
  localdev: the top mask is 0px under Channels' and Audit's headers, 20px on
  Home and the Feed.
- ui/DataTable mounted its table AND its card list at every width, one
  hidden by CSS: double DOM and render cost, duplicate ids, two keyboard
  copies of every row. It now mounts the one the `md` media query calls for
  (lib/useMediaQuery); the keyboard test asserts one layout at each width.
- The Plex module's content width went px → rem with the rest of the page.
2026-09-19 07:51:33 +02:00
peter 8018cd4685 fix(ui): the platform layer speaks platform words; boxes scale with the text
- PriorityStepper labelled itself with channels.row.* and told every caller
  it orders the Feed; Combobox said plex.loading / plex.filter.moreRefine /
  sidebar.match on the feed sidebar. The generic strings moved to common.*,
  and what a priority MEANS arrives as the stepper's `hint` prop.
  uiI18n.contract.test.ts pins components/ui/ to the platform namespaces —
  the import boundary could never see an i18n key.
- 32 px-frozen box utilities (w-[150px], max-w-[1600px], left-[18px], …)
  became spacing-scale steps (w-37.5, max-w-400, left-4.5): identical at the
  default text size, and now they grow with the Settings slider instead of
  clipping or re-wrapping their own labels. An eslint rule refuses the px
  form beside the older text-[Npx] one, pinned by pxBox.contract.test.ts;
  measured in the browser that every new step is generated at its old px.
- chipLane's import-strip regex ran past a side-effect import into the next
  statement's `from`, hiding a lane between them; it now stops at the
  import's own `;`.
2026-09-19 07:46:33 +02:00
peter 29875b7367 fix(ui): stable shelf-card menu triggers, a named Button by type, true layout comments
- Home's rail handed each card a fresh `() => menu.triggerProps(c)` per
  render, defeating memo(VideoCard) on every tile whenever the menu's own
  state changed. `useItemTriggers` (ui/useContextMenu) keeps one trigger per
  item, index-aligned; tested for identity across renders.
- The Button accessible-name union had one hole: `<Button icon={…} />` with
  no children and no size typechecked into a nameless button. A textful
  button now needs children or a name; pinned with @ts-expect-error.
- comfortPin's header said the pin belongs "on every card and row", which
  the code never meant (an episode, a season, a person are not pin kinds).
  It now says what is pinnable — and the show page's Related strip, whose
  shows ARE pinnable, gets the pin menu it was missing (and a rem width).
- gridColumns / feedView / VirtualGrid claimed layouts 1024–2560 are
  "byte-identical" under the column cap; the small cards are capped at 2560
  with the filter rail collapsed. The comments say so.
2026-09-19 07:40:48 +02:00
peter 7ae5c57b6b fix(i18n): every refusal the API answers is structured and said in the reader's language
~150 HTTPExceptions carried a hand-written English sentence as their detail,
shown verbatim under the Hungarian UI — in the blocking error dialog and on
every inline surface that printed HttpError.detail (sign-in, setup wizard,
change password, the passphrase gate, live search). Four also quoted an
upstream exception's text back to the user.

- app/refusal.py: `refused(status, reason, **params)` → {code: "refused",
  reason, …params}; localizeDetail translates errors.refused.<reason> with the
  params interpolated. "Unknown X" 404s use the existing not_found(what).
- sysconfig.set_value raises InvalidSetting (reason + bounds) instead of an
  English ValueError; the three writers answer it through setting_refused.
- The Plex connection/write/image failures log what the upstream said and
  answer a reason; the search scraper's failure is logged and answered as
  youtube/unavailable (it crashed refusal_reason before).
- HttpError.detail now only ever holds a translated reason: a string detail
  (only the framework's own can remain) is left out, so reading .detail is
  safe everywhere — including destructured, the hole the raw-detail source
  scan could not see. That scan and its five-file exception list are gone.
- Enforced: test_http_details.py fails on any HTTPException with a sentence
  for a detail; the contract test requires every refused reason and every
  not_found "what" in both languages, and no key for one that is gone.
- local/jsx-comment-text: a // inside JSX children renders as text; the rule
  is ten lines in eslint.config.js rather than eslint-plugin-react's ~1800
  lockfile lines.
2026-09-19 07:34:17 +02:00
peter dc4e8b9fcd fix(plex): playback sessions per viewer, bounded in total; a fair share of subtitle passes
HLS sessions were keyed by the item, so a second account opening the same
film killed the first one's session; the displaced player reclaimed at its
playhead and evicted the other in turn. The registry is now keyed by
(viewer, rating_key), segment requests resolve by both, and the session
directory carries the viewer (the orphan sweep still recognises the older
shapes).

Only transcodes counted toward a cap, so remuxes — each a whole segmented
copy of the film on the scratch volume — were bounded by the library size.
Three bounds now apply after every start, each dropping the least recently
used: two sessions per account, the transcode cap, and a new admin knob,
plex_max_sessions (default 8), over every session. A superseded session
request returns at once, since it waits on the killed ffmpeg, so an account
cannot park more request workers than it may hold sessions.

The subtitle extractor's full-pass slots were global and minutes long each;
one account opening subtitles on film after film could hold them all. An
account may now hold three of the six, and still gets windows past that.
2026-09-19 07:16:17 +02:00
peter 6f9ba428d5 fix(playlists): the move's conflict fallback says why it is silent
The house guard requires a silent handler to carry a silent-ok note; the
savepoint fallback added in e74986cc did not, and the gate's pytest lane
caught it.
2026-09-19 07:10:43 +02:00
peter c60359bfed fix(plex): subtitles — a retried full pass resumes, a stalled track is polled slowly
A full pass that ran out of time kept what it reached, but its retry an hour
later started again at 0, so a file too big to read inside the extraction
timeout never completed. The pass now starts where every embedded track's
coverage from the head ends (less the window pre-roll, so a cue across the
join is read whole). The seek needs -avoid_negative_ts on this read as on a
window, measured on a real file: without it the cue running at the start
shifts the whole output.

A partial track whose full pass had failed never reached a settled state, so
the player polled it every 2 s for the rest of the film. The status now says
whether the track is still growing, and a stalled one is polled every 15 s —
still inside the 60 s lookahead that reads the next window before an edge.
2026-09-19 07:07:53 +02:00
peter e6b76e156a refactor(discover): one shelf query per kind, one RAIL_LIMIT, a true rollup comment
The comfort shelf resolved each pinned channel with three queries (the
row, its unwatched count, its newest video) and each playlist with two.
channels_summary gets the count and the newest row per channel as window
functions over one filtered set, playlists_summary reads every pinned list
in one query, and the rows come in one IN() each — so the shelf's cost no
longer grows with its length. A test counts the statements.

RAIL_LIMIT had two definitions held in step by a comment; the Plex Discover
route now imports the domain's. And rollup's "continue the latest-started
episode" was false: it takes the last in-progress one in broadcast order,
which is also what Home's next_episode picks.
2026-09-19 07:03:35 +02:00
peter e74986cc9a fix(playlists): a racing move is not a 500; ownership through owned_or_404 everywhere
move_item checked "already in the target?" and then re-parented without a
lock, so a second tab's add or the scheduler's YouTube merge committing the
same video in between hit uq_playlist_video and answered a raw 500. The
re-parent now runs in a savepoint and a conflict falls back to the
already-present outcome. The target is touched only when the row landed in
it (an untouched list was promoted in the rail's "recently changed" sort),
and _touch stamps the database clock like the mixin's own onupdate.

The canonical-patterns table names owned_or_404 against ad-hoc ownership
helpers; six were left (_own_playlist, _own_playlist_or_404, _own_view,
_own_tag, notifications._owned, downloads._own_job), two of them
hand-rolled with a bare English 404. All call sites now use owned_or_404,
and the download/link/notification 404s are structured like the rest. The
contract test reads every "what is gone" the routes name, and the YouTube
refusal reasons, and requires both languages to name each.
2026-09-19 07:01:18 +02:00
peter 2df45b27d3 fix(plex): a refused finish restores no resume point at the media's own end
The rollback of a refused watched mark guarded against Plex's REPORTED
duration. On a file shorter than its metadata the end sits outside the
finish margin, so playing to the end and having the mark refused stored the
very end as a resume point on an unwatched row: a ~99% bar that reopened in
the last seconds. The media's own ended flag now decides.
2026-09-19 06:58:11 +02:00
peter bacd6aec4f fix(plex): one scrobble per finish, and no resume push lands after it
Two near-end checkpoints arrive together routinely (the 10 s tick and the
debounced seeked flush); both read "not watched" before writing and both
scrobbled, bumping Plex's viewCount twice. The transition is now decided by
the write: an insert that did not conflict, or an update conditional on the
old status.

The finish also popped the push-ordering entry that _resume_push_superseded
reads, so an older resume push still in flight could land at Plex after the
scrobble and revive a resume point on a watched item. It now stamps the
entry instead; the existing eviction bounds the map.
2026-09-19 06:57:20 +02:00
peter fad301e7a4 fix(youtube): a refused YouTube action answers a 422 that says what to do
The two manual syncs (subscriptions import, playlist pull) had no YouTubeError
handler, so a quota, token or transport failure was a raw 500 and the blocking
"Server error" dialog (E9 F1). The other YouTube routes answered either the
upstream's English or a playlist-scoped "youtube_failed".

One helper, routes/_common.youtube_refusal: 422 {code: "youtube", reason} with
the reason sorted by what the user can do — quota (wait for the reset),
reconnect (the token step refused), unavailable (retry). Used by every route
that surfaces a YouTube failure; localizeDetail translates it.

"Reconnect" had no path: a grant Google revoked still read as granted, so
Settings and the wizard offered no connect button while every call failed.
An invalid_grant from the token endpoint now clears the stored grant.
2026-09-19 06:55:40 +02:00
peter 1cfb19bdc7 release: 0.91.0 (bug/prod-uat-2026-09-19)
gate / frontend (push) Failing after 4m0s
gate / backend (push) Successful in 5m23s
2026-09-19 05:56:19 +02:00
peter 90c30d596f chore(release): 0.91.0 2026-09-19 05:56:12 +02:00
peter 7ed6ccf171 fix(feed): the card menu's reset uses the card's own reset string
Two keys for one action had already drifted once; the menu now reads card.resetState like the
card's button.
2026-09-19 05:54:19 +02:00
peter ce30b99327 docs(nav): restoreNavPosition's doc back on restoreNavPosition 2026-09-19 05:54:19 +02:00
peter a7891d03c5 fix(sync): only a refusal about the channel spends its fallback window
A failed connection or token step answered nothing about the channel, yet stamped it: one proxy blip
during an RSS blackout delayed every due channel by a whole window. YouTubeError now names the
endpoint that answered; a failure no endpoint gave, or a spent quota, ends the round and stamps
nothing.
2026-09-19 05:54:19 +02:00
peter 038da5374f test: the boundary stays quiet while reloading for a new build; Back restores Home's whole question 2026-09-19 05:47:17 +02:00
peter 92979fa920 refactor(ui): one rule for resettable, role key and display name; an honest reset label
The card menu's reset read 'Clear watch progress' but also un-hides and un-watches; it now says so.
The resettable rule, the demo-first role key and the display-name fallback each lived in several
components tied by comments; they are shared helpers now, and the menu reuses the card's own
strings instead of duplicate keys.
2026-09-19 05:47:17 +02:00
peter 2d5bd275fa fix(diagnostics): a pivot or Back applies the search text at once
The query read the typing-debounced search, so for 300 ms after a jump it mixed the new view's
filters with the old view's text and fetched rows neither view shows. useDebounced takes a settle key
whose change applies the value immediately.
2026-09-19 05:47:16 +02:00
peter 49fe56b6e7 fix(nav): Back's focus keeps its place among items of the same name
Restoring by name picked the first item of that name, so Back on the log console (repeated
http.request lines) or two same-titled downloads landed on the wrong one. The saved place wins while
it still holds the name; otherwise the nearest item of that name.
2026-09-19 05:47:16 +02:00
peter 1fa6a0bea8 fix(sync): a failed feed-fallback read waits its window out, and spent quota ends the round
A channel whose uploads playlist also failed was never stamped, so a terminated channel was charged
again on every 20-minute poll instead of once per rss_fallback_hours. An upstream quotaExceeded now
stops the round instead of charging every remaining channel the same refusal.
2026-09-19 05:47:16 +02:00
peter 582f20fab6 fix(sync): the stored RSS warning says what the API fallback read
An INFO line outside a request never reaches log_events, so the fallback's
own summary was invisible in the Server log. Its counts ride on the
sync.rss.failed warning, the one line of a poll the console keeps.
Measured on localdev: 375 of 375 dark channels read, 403 new videos.
2026-09-19 05:05:13 +02:00
peter 6cb8231a9f fix(logs): the browser's copy of a polled read is quiet too
The server now logs polled reads at DEBUG, and the browser's copy of each
request line promises the server's level: at INFO the unread-count polls
filled My log within minutes and, at Detailed, the shipped log. The server
marks a line it logged at DEBUG with X-Log-Quiet, so the polled-route list
stays in one place.
2026-09-19 05:03:31 +02:00
peter 6bbf532b70 fix(diagnostics): count the 24 hours from the moment the level was raised
The countdown's clock was the moment Settings mounted, so a level raised
two minutes after opening the page read "24 h 2 m" until the first tick
(measured on prod). The clock is read at render; the interval only
re-renders.
2026-09-19 05:03:31 +02:00
peter 2fa5c8c394 fix(ui): nothing past the right edge of a phone
The Feed's chips + Sort + View row did not fit one line at 375px and pushed
the page 21px wide; the toast stack capped its width by the viewport while
it sits right of the sidebar. The toolbar rows (Feed, Plex browse) wrap,
and the toasts are capped by their host.
2026-09-19 04:54:18 +02:00
peter 373415a2eb feat(account): show the account its own last sign-in
Settings → Account names the latest sign-in on any device (only the
admin's user list did, though the 0.88 notes said otherwise), and the role
in the reader's language instead of the raw `user`.
2026-09-19 04:51:43 +02:00
peter 582c033e1c feat(feed): the card menu carries the card's own actions
Right-click, long-press or Shift+F10 on a feed card offered only the
comfort-shelf pin; watched, saved, hidden and the progress reset were
reachable only by hovering or Tabbing through every card.
2026-09-19 04:51:43 +02:00
peter ef652ee5d8 fix(ui): honest tab titles and a download field that names every site
Signed out, the tab read Home's question over the sign-in form; the legal
pages read just "Siftlode". The download field asked only for a YouTube
link although Facebook and the rest download too.
2026-09-19 04:51:43 +02:00
peter 250e3a9f90 fix(home): Back keeps the question, and the focus finds its card by name
Opening a card left Home, and Back came back to "Doesn't matter" with a
different hero, whose card then dropped out of the Continue rail and moved
the keyboard onto its neighbour. The budget, reroll seed and seen list live
on the history entry now, and a returning list focuses the item by its
name before its place.
2026-09-19 04:51:43 +02:00
peter 6096dd6927 fix(diagnostics): Back from a request's lines returns to the view it came from
Every filter change replaced the history entry, so Back after a request or
job pivot left the console altogether (the 0.90 notes promised otherwise).
A pivot follows a link and is its own entry now; the console reads the
landed entry back on popstate. The other filters still adjust in place.
2026-09-19 04:51:42 +02:00
peter 51e98bfc38 fix(app): reload onto a new build instead of crashing on a missing chunk
A tab left open across a release asked for a lazy chunk the server no
longer has (prod: DownloadDialog 404 an hour after 0.89) and fell to the
crash screen. `vite:preloadError` now reloads the tab once a minute at
most; a second failure is let through to the error boundary, which now
wraps every entry — the public /watch, /privacy and /terms pages crashed
to a blank page.
2026-09-19 04:51:31 +02:00
peter 3ee00fb051 fix(logs): keep polling reads and a channel-less account out of the warnings
The reads the frontend polls (unread counts, /api/me, the download index,
sync status, subtitle progress) are logged at DEBUG: at INFO they were ~95%
of the stored log on prod. A scheduler job failure and a playlist sync
failure carry event codes, and a Google account without a YouTube channel
has an empty playlist list instead of a warning every six hours.
2026-09-19 04:51:31 +02:00
peter 29094f5f51 fix(sync): read a dark RSS feed through the API instead
YouTube's RSS goes dark per channel for hours (prod 2026-09-18/19: 393 of
396 feeds answered 404 for four hours, sticky per channel, so a retry does
not help), and nothing else read a followed channel's new uploads. A
channel whose feed has not answered for `rss_fallback_hours` (admin knob,
default 2, 0 = off) now gets one page of its uploads playlist through the
Data API (1 unit), at most once per window, yielding to the backfill
reserve. Spend is attributed to its own quota action.
2026-09-19 04:51:31 +02:00
peter 16a44dec79 release: 0.90.0 (feature/e22_6-diagnostics-console)
gate / frontend (push) Failing after 4m16s
gate / backend (push) Successful in 5m19s
2026-09-19 03:57:44 +02:00
peter 4e3e68d6e6 chore(release): 0.90.0 2026-09-19 03:57:29 +02:00
peter eb29f22eb7 fix(diagnostics): say which read failed, and let Live restart
- "Load older" is not offered off a loading filter's placeholder rows,
  whose cursor belongs to the previous filter.
- A failed older page, stale rows and missing counts each get their own
  message (HU/EN).
- Live shows as off once a failed read stopped the polling; pressing it
  again reads now.
2026-09-19 03:48:01 +02:00
peter 8f6f89832f fix(diagnostics): honest live console after review
- Live stops polling on a failed read and marks the list as possibly
  stale (with Retry), instead of re-raising the error dialog every tick.
- While live, a refetch reads the newest page only, whatever the cache
  holds (a late "Load older", a revisited filter set).
- Copy is disabled and the list dimmed while a filter change loads.
- Dates follow the UI language; the pivot chip uses the shared .chip.
- Tests: query-string wire test, Copy, every facet's skip key.
2026-09-19 03:42:26 +02:00
peter 1c49870a7a fix(sync): don't try a timed-out feed again
A request that hung for the full timeout says the way out is hanging, not
that the endpoint is flaky; three of them per channel stretched a poll over
a dead egress to about three times its interval.
2026-09-19 03:42:26 +02:00
peter 142593083f feat(diagnostics): walk the log lines from the keyboard; name the console "Server log"
Both log viewers put their lines in a NavGrid: j/k (and the arrows) step
from line to line, and Enter opens one exactly as a click does (the line's
button is its `data-nav-open`). A focused line is named by its code and
message.

The console's nav entry is "Szervernapló" / "Server log", its page title:
"Diagnosztika" was already the Settings tab that holds the user's own log,
and two places with one name read as one place.
2026-09-19 02:02:45 +02:00
peter 8cfab498cf fix(sync): try a channel's feed again before counting it as failed
YouTube's feed endpoint answers the same channel with 404, 500 and 200 in
turn: on prod 182, then 296 of 396 channels failed a poll, and a probe of
24 channels saw 15 fail a first request while 16 had succeeded by the
fifth. A channel that failed once waited for the next poll, 20 minutes
later, so new uploads showed up late.

A feed is now tried up to three times (after 1 s and 2 s) on a transport
error, a 404, a 429 or a 5xx; a refusal such as 403 is not retried. The
last failure is what gets counted. A feed that is really gone costs two
more requests per poll.
2026-09-19 01:58:40 +02:00
peter b5996a0e56 feat(diagnostics): the admin log console
Admin → Diagnostics reads the server's stored log: one list, newest first,
where a line opens in place to its full record (time, source and logger,
account, request and job ids, version, message, attributes, stack). A
request or job id narrows the list to that request or job with every other
filter off, shown as a removable chip. Filters: level group, source,
account, event, period and a free-text search; all of them live in the URL
(`l*` keys, which setPage clears like the Plex and Channels keys), so a
filtered view is a link and Back restores it. "Live" re-reads the newest
page every few seconds while the tab is visible; paging into older rows
turns it off. Copy puts the loaded lines on the clipboard as JSON lines.

The facet counts now honour every filter but their own, as the feed's do:
a request pivot narrows them, and a picked level does not zero its
siblings. A value in use stays offered even when the counts drop it.

"Saját napló" and the console draw a line with one shared component,
`LogLine`, whose phone layout gives the message its own full-width line
(sharing the code's line left it a few characters). The level groups
moved to lib/log.ts for both.

The console's own reads are logged at DEBUG on success, like the
healthcheck: at INFO its live tail filled the log with itself.
2026-09-19 01:54:48 +02:00
peter 5e208353e3 feat(diagnostics): admin query API over the stored log
GET /api/admin/logs pages `log_events` newest first on a (ts, id) keyset:
a browser row's ts is clamped into its consent window, so ts alone is not
unique. Filters AND together: level, source, account (or `none`), event
code (or `none`), request id, job id, a free-text search over the
message, code, request id, stack and attributes (typed text is a literal),
and a time window. Each page names the accounts it mentions.

GET /api/admin/logs/facets counts what the filter chips offer over the
window alone, so a chip row does not shrink as filters are picked.

Admin-only and read-only; reading is not audited (the console's live tail
polls). There is deliberately no "ids after N" tail: several processes
write this table and a batch takes its ids at insert, not at commit, so a
tail keyed on ids could pass a row that commits late.
2026-09-19 01:40:46 +02:00
peter 66dbd0af81 fix(search): typed text is a literal in every LIKE pattern
The feed's channel-name match and the live search's term fold built their
LIKE pattern from the typed text as it was, so `_` and `%` acted as
wildcards: `build_log` also matched `buildXlog`. The term fold also tested
a substring, so a term already present inside a longer word (`cat` in
`category`) was never folded into search_terms. It now compares whole
terms, escaped.

`like_escape` moves from the Plex query module to `app.utils`, beside
`to_tsquery_str`, and every pattern built from typed text goes through it.
2026-09-19 01:33:32 +02:00
peter 59fd068711 release: 0.89.0 (improvement/e22_5-backend-logging)
gate / frontend (push) Failing after 3m28s
gate / backend (push) Successful in 5m1s
2026-09-19 00:05:53 +02:00
peter 8e12eee996 chore(release): 0.89.0 2026-09-19 00:05:47 +02:00
peter 89c77ae6b5 fix(logs): deep-review round 3 — a failure nobody could record is one scrubbed line
- The download and edit failure lines go back after their DB write: a job the
  write could not fail is not a failed job, and the worker's child_exit line
  already says it failed.
- The download child's runner is the last stop before Python's hook: an
  escaping exception is one download.child.failed line with its chain scrubbed,
  then exit 1, instead of a raw stderr traceback carrying the URL.
- A crash the database will not record carries its stack once, and the
  fail-write line has an event code.
- The job's own generic-extractor name is taken out as a literal (formats.hiding),
  so brackets and colons in a direct link's file name no longer leak; the
  pattern stays as the fallback. `Downloading just the video <id>` is covered.
- Tests: scrubbed_stack's cause, suppress, unprintable and text-format paths,
  the image steps' failure kinds, the watchdog error through the real
  _fail_running, exact poster/storyboard messages, ffmpeg skip marker.
2026-09-18 23:44:42 +02:00
peter 0b672ed6e7 fix(logs): deep-review round 2 — a stack names code, not data, and survives a failed write
- A failure keeps its traceback unless it is yt-dlp's or our own StatedFailure (a
  message that is the whole story). A plain RuntimeError, RecursionError or OSError
  is also what a bug of ours raises, and its stack is the only lead.
- The stack is scrubbed link by link (logs.scrubbed_stack): a chained DownloadError
  under a DB error no longer prints yt-dlp's text unscrubbed.
- Engine hide_parameters: a DB error's message no longer carries the title or path
  it was writing.
- The download and edit failure lines go out even when their DB write fails.
- The worker's child_exit/crash/supervisor lines say download.job.failed only when
  the job failed there; a crash the DB will not record no longer ends the thread.
  Stopping a paused child is download.child.stopped, not a second abort.
- log_safe also covers generic ids with spaces, playlist titles and bare ids.
- The ffmpeg image steps log the failure kind, not the argv with its path.
- The consent contract accepts only the body of a plain consented_trace() check,
  and any level that may be TRACE.
- Tests for each; every one mutation-checked.
2026-09-18 22:50:37 +02:00
peter 54d19cc386 fix(logs): deep-review round 1 — say each failure once, and scrub it everywhere
- yt-dlp's text is scrubbed wherever it is logged, not only in the adapter:
  `formats.log_safe` takes out the URL, the `[extractor] <id>:` prefix and
  file paths, and the job-failed, refused, edit-failed and backfill lines use
  it. Done lines log the asset id, not the title-built path. The PII contract
  now bans `rel`/`rel_path`/`title`.
- A traceback goes with a download/edit failure only when it is our own code
  failing; for yt-dlp's DownloadError, our RuntimeErrors and OSErrors its foot
  is the unscrubbed message.
- yt-dlp's `error` maps to WARNING: it fires on attempts the fallback client
  set recovers from; the job's verdict is `download.job.failed`.
- The sink groups a bare `%s` pass-through by its text (numbers folded), and
  a process's last batch closes the repeat window, so a short-lived download
  child still reports what it dropped.
- Worker-side failures (stall, wall-clock, no asset, cached asset error) log
  `download.job.failed` with a cause; they left no line at all before. The
  crash, supervisor-error and non-zero-exit lines carry the code too.
- An edit failure is one coded line in `_fail_edit`, for every path.
- `logs.get_logger("__main__")` resolves to the `-m` module, so the worker
  and the download child keep their names.
- TRACE lines carrying user content are gated on `logs.consented_trace()`,
  not on the process level; a contract test pins it.
- The OAuth token refresh goes through `_upstream_failure`; transport errors
  carry reason `transport`. Subscribe recognises a duplicate by reason code
  only (the prose fallback was dead).
- Once-only guards for a stale sysconfig override and a refusing HLS session
  directory; item_detail splits Plex failures from our own parse bugs.
- The playlist breaker line names the playlist and the action; comments that
  were false (runner noqa, `_merge_vtt`, sink grouping) are corrected.
2026-09-18 03:16:15 +02:00
peter 04b58e5c6f feat(logs): give the log the events that no request line already carries
The gap this sprint was scoped against turned out to be smaller than the survey
said, and in a way worth writing down: `log_requests` already records every HTTP
call with its route template, status, duration, request id and user id, and the
audit log already records every admin act on someone else's account. Putting a
line in each of routes/downloads.py's twenty mutating endpoints would have added
a second copy of the first and nothing else. So this instruments what neither of
them can see.

yt-dlp now explains itself to our log. `quiet: True` was doing more than quieting:
with no `logger` its whole account of a download went to the child's stdout, so
"Requested format is not available", a signature-extraction failure or a POT
refusal reached us only as whatever text ended up in the raised exception — the
real reason a download failed was routinely one line the app never kept. Measured
against the pinned yt-dlp 2026.07.04 before wiring it: with `logger` set it calls
`debug` for extraction and progress lines and `warning`/`error` for the rest, and
neither `quiet` nor `no_warnings` gates those calls. `debug` lands at TRACE, being
several lines a second, so the noisy half is the half the user's own consent
gates. The full source URL is trimmed to its host on the way in — it is a row of
someone's download history, and `logs.set_job` already puts the job id on every
one of these lines. The option is set AFTER the adapter merge, like `cookiefile`
and the size cap: a source adapter must not be able to take away the one account
of what went wrong that a failed job leaves behind.

An HLS session now says which pipeline it got (transcode/remux, hw, tonemap,
variants, offset) — everything downstream is about that choice, and it was only
inferable from an ffmpeg command line nobody keeps. A session culled by the
concurrent-transcode cap says so: somebody's playback stops there, the route turns
it into a retryable 404, and "my film keeps rebuffering" had no other explanation
anywhere. The HW→SW fallback keeps its line and gains an event code.

A live search refused by a quota cap says WHICH cap: the 429 is in the request
line either way, but the per-user daily cap is answered by tomorrow and the shared
budget by the admin.

The download and edit job lifecycle lines gain event codes, which is what E22.6's
console filters on — the job id was already a structured field.

tests/test_ytdlp_logging.py; three of its four cases mutation-checked. A fourth
(a `%` in a progress line) survived its mutation — stdlib only applies `%`
formatting to a record that has args — so it is gone rather than kept as a case
nothing can break; the reasoning moved into the comment on `_emit`.
2026-09-18 01:28:52 +02:00
peter 6b0401b922 fix(logs): rule on every swallowed exception in the backend
114 exception handlers neither logged nor re-raised. The defect was never that
they are silent — plenty of them should be — it is that their silence said
nothing, so nobody could tell a decision from an oversight. Each one has now been
read. 44 got a line; the other 70 carry `# silent-ok: <why>`.

What the silent ones had been hiding, in the cases that got a line:
- A failed ffprobe sent every HDR file down the 8-bit pipeline, so playback
  degraded to a software transcode with nothing saying why; a failed audio probe
  collapsed a multi-audio file to one track and the player showed no language
  picker; a failed first-PTS probe left subtitles seconds out of sync.
- A subtitle sidecar that existed but would not parse made a PARTIAL track read as
  complete, permanently — the viewer loses every cue past the cached window.
- An unreadable crop/trim/cut-list in an edit spec was dropped, so the clip came
  out uncropped, untrimmed, or shorter than the one the user built.
- A dropped `max_height` downloaded at best quality against an explicit cap.
- A failed videos.list enrich made every candidate undroppable-from-judgement and
  therefore dropped: a working search rendered as "no results".
- An unusable password hash answered as "wrong password", locking the owner out
  and telling them they mistyped; a rotated TOKEN_ENCRYPTION_KEY turned every
  stored token into "not connected".
- A stored admin setting that would not coerce fell back to the env default while
  the settings page went on showing the saved value.
- A one-off InnerTube config failure pinned the whole process to the hard-coded
  fallback key, and searches return nothing when that goes stale.
- The HLS sweep reported 0 swept — like a clean run — when it could not list its
  own root, leaving every crashed session's segments on disk forever.

Four of them are counted rather than logged per item (dropped cues, dropped cut
list segments, unreclaimed GC files) and reported once per call, because the
alternative is one line per cue over a whole subtitle track.

ruff gains S110 and BLE001. Every blind `except Exception` that remains carries
`# noqa: BLE001 — <why>`, which is the point: catching everything is sometimes
right, and always worth a sentence. plex/sync.py's three best-effort backfill
catches now keep the traceback — a blind catch over a batch that fetches AND
writes could be Plex, the database or our own bug, and `%s` of the exception told
them apart badly. main.py's boot-time log-level read likewise.

logs.event's `code` is positional-only now: `event("x", code=...)` was a TypeError,
and `code` is exactly the word a caller reaches for when the thing that failed has
one of its own.

tests/test_swallowed_exceptions.py is the wider rule ruff cannot see: a handler
that neither logs nor re-raises must say why. Both cases mutation-checked — a
marker removed and a log line removed each fail it, naming the handler.
2026-09-18 01:20:42 +02:00
peter 0cff964ff2 refactor(logs): name every logger after its module, and hold it there
25 literal names covered 34 modules. plex/client.py, plex/stream.py,
plex/subtitles.py, plex/sync.py and two Plex route modules all called themselves
`siftlode.plex`; downloads/pipeline.py called itself `siftlode.worker`;
routes/channels.py called itself `siftlode.api`. The `logger` column is in every
stored row so E22.6's console can filter on it, and so an operator can pin a level
to one part of the system — neither works when six modules share a name.

`logs.get_logger(__name__)` derives it: app/plex/sync.py -> siftlode.plex.sync.
One name per module, and it cannot drift from the file it sits in.
tests/test_logger_names.py pins the derivation, that everything stays under the
one `siftlode` tree the handler and the tests attach to, and that no module
hand-rolls a name again unless the line says why (`# logger-name-ok: <why>`).
Two do: get_logger's own call, and `siftlode.http`, which is named for the traffic
it carries rather than the module it lives in — it replaced uvicorn's access log
and is what an operator pins when they want request lines and nothing else.

app/scheduler.py called its logger `logger`; every other module calls it `log`.

ruff gains LOG, G, T20, TRY400, TRY401 and S112. Four findings cleared with them:
worker.py logged `str(exc)[:120]` where the traceback was the whole question, and
three prints/f-strings that are deliberate now say so on the line. S110 and BLE001
stay off until S3 — 23 handlers are outstanding and each is a judgement about one
of them; enabling the rules first would just mean 23 blanket noqas.
2026-09-18 00:57:57 +02:00
peter dc221338be fix(logs): keep what a person typed out of the backend log lines
The E22 redactor is a safety net over addresses and credential params, and every
one of these call sites was leaning on it to do the whole job. What it does not
cover went in whole, and what it does cover came out as a constant.

Not covered, so a real leak:
- routes/search.py logged the raw search term at ERROR. The privacy policy shipped
  in 0.88.0 says a search string is kept only at Teljes/TRACE, with the user's own
  consent. The ERROR line now carries the shape of the failure (source, candidate
  and collected counts) and the term follows on a TRACE line, which nothing emits
  unless that consent is in force.
- downloads/backfill.py logged the full source URL at INFO — a row of the user's
  download history. Now the asset id and the host.
- youtube/client.py logged the upstream error body, which is prose that quotes the
  request back. Every failing YouTube call now goes through one _upstream_failure()
  that logs the API's `reason` code, keeps the body for the same consented TRACE
  line, and raises a YouTubeError carrying `reason` as a field — callers put that
  error in log lines of their own (routes/channels.py) and in a 422 answered to the
  user, so the body must not travel in it either.

Covered, so merely useless: routes/auth.py named the account by an address that
reached both sinks as `[email]`, and app/email.py printed recipient lists that read
as `['[email]']`. Both now carry the uid (or a count) and a reason code, which is
also the first half of E22.5 S4's failed-sign-in coverage.

routes/channels.py branched on the text of that error message to recognise an
already-existing subscription. It now branches on `exc.reason`, with the substring
check kept beside it for a body that carried no reason at all.

tests/test_log_pii.py pins the call sites and the family: a source-level contract
test walks every log call in app/ and fails on an argument naming user-typed
content unless the line says why it is safe (`# log-pii-ok: <why>`). Eight sites
carry that marker today, each with its reason. Gate 8/8 green; all five cases
mutation-checked.
2026-09-18 00:49:43 +02:00
peter 8c10e57208 release: 0.88.0 (feature/e22-activity-trail)
gate / frontend (push) Failing after 3m31s
gate / backend (push) Successful in 4m44s
2026-09-18 00:05:50 +02:00
peter dd040adc96 chore: 0.88.0 release notes
E22 (activity trail): a consented, self-expiring diagnostics level with the user's own log visible
before anything is sent, last sign-in, bilingual privacy policy and terms, an admin diagnostics
section, and pickers for every closed-set setting. Plus what the epic's measurements turned up on
the way: doubled chat delivery, Settings on a phone, locale-ignoring dates and numbers, and the
feed poll narrowed to followed channels.
2026-09-18 00:05:46 +02:00
peter 4ab128ccab Merge branch 'main' into feature/e22-activity-trail 2026-09-18 00:02:12 +02:00
peter a28193a00c test: pin the focus restore, the choice labels and the phone layout to their fix
Three round-1 fixes whose tests passed against the broken version too.

The focus restore keeps watching until the element that closed the box is GONE, rather than spending
itself on the first render after the change — the distinction Chrome exposed and jsdom did not,
because jsdom commits the save in one render where Chrome takes two. Driven now through the general
case instead: the user moved the focus elsewhere while the request was out, renders keep coming
while that element is still there, and the restore happens only once it disappears.

The choice test's i18n mock returned an option's bare value, which is exactly what dropping the
`config.choices.*` lookup renders — a Hungarian admin would have seen `on`/`off` instead of
`GPU`/`Szoftver` with the gate green. Marked labels now, still short enough not to move the
segmented-bar budget the same file measures.

The admin config row's stacking had no test at all, only its sibling fix did — and stacking is the
only fix for a text or number key, whose fixed-width input is the wide control. Class contract, as
SearchBar.dom.test.tsx already does for the narrow-header rules, since jsdom does no layout.

Also: the time-zone cleanup deleted TZ, and Node re-reads it on assignment only, so the process kept
the last zone the file set. Harmless until someone appends a test below it.
2026-09-17 23:45:41 +02:00
peter 986bc8d01e fix: a batch the user themselves cancelled is not called a lapse
Picking "Alap" revokes the consent, and a batch still in flight is then answered by the server with
the same 409 a real expiry gets. The 409 arm only checked that the consent session had not changed,
not that this tab was still streaming — so the user's own log gained a "diagnostics level lapsed"
line for a level they had just turned off themselves, plus a pointless `me` refetch. The same guard
also stops a lapse this tab has already announced being announced a second time.

Measured in Chrome by holding the POST at the network layer, switching to Alap and only then
answering 409: the line appears without the guard and not with it.

Three tests that named the session counter never reached it — they settled the refusal while the
stream was already stopped, where `unsubscribe` does the work, so the counter could be deleted from
both arms with the suite green. The refusal now settles after the re-raise, which is the ordering it
exists for. Same for the queue trim on the 429 path: the old test unshifted a batch into an empty
queue, so the trim was a no-op and only the subscribe path was ever measured.
2026-09-17 23:45:29 +02:00
peter 07e3ea3737 docs: the deep-backfill card counts the backlog, not the next pass
Two reviewers in a row read the admin card's selection against `run_deep_backfill`'s, saw it was
missing the job's `recent_synced_at` filter, and called it a card that counts channels the job will
never touch. It is the opposite: the card is the same backlog figure /api/sync/my-status shows the
user, and a channel still missing its recent pull is one the same job tick fetches first. Narrowing
it would hide genuinely queued work. Written down where the next reader will look.
2026-09-17 23:45:29 +02:00
peter b2eaa9e00b fix: a credential, an erased account and a stale level stay out of the log
Round 2 of the E22 review, on what round 1 changed.

`redact_query` decodes a parameter before the text rules see it, which is what lets `%40` be
recognised as an address — but `_SECRET_PARAM_RE` was written for the raw, encoded form, where a
value holds no space, `&` or quote. On decoded text its value class stops at the first one, so
`?state=a%20b` kept its tail and `?code=%20abc` was not matched at all. The parameter-NAME rule now
decides, on the key, before any text rule runs.

`forget_user` dropped the account's level elevation without recomputing the root level, so a process
whose only elevated account had just deleted itself kept building TRACE records to throw away, until
some later request happened to recompute it. Every other path that drops an elevation recomputes.

The sink's writer took the erasure lock and THEN waited for a pooled connection, while every request
path already holds a connection when it reaches that lock — so under pool pressure the writer held a
process-global lock against the very requests that could hand a connection back. Same order on both
sides now; the lock still spans the check, the insert and the commit.

The storage-budget notice said the rest of the hour was not kept, and then smaller batches kept
landing underneath it: the window slides and a refused batch is not charged to it. Says what it does.

Tests: the route's half of the erasure rule had no test — only the sink's did — and a source-level
contract test now covers the third insert nobody has written yet. `logs._erased` is process-global
with an hour's memory while the DB fixture hands serial ids back out from 1, so it is swept before
each test like the rate limiters are.
2026-09-17 23:45:16 +02:00
peter 791ba5f507 fix: a calendar date is the same day everywhere; legal pages use the shared language picker
formatDate read a date-only value as UTC midnight and showed it in the viewer's zone, so the legal pages' last update read a day early west of UTC. Date-only values are shown in UTC. The legal pages' hand-rolled language buttons are the landing page's LanguageSwitcher. formatDateTime and formatDate get the two-language test formatNumber already had.
2026-09-17 03:53:40 +02:00
peter 7789db8240 fix: admin config pickers fit a phone
A segmented bar cannot shrink or wrap, and the five log levels (~300 px) are wider than a phone leaves beside the navigation rail. A choice is a bar only when its labels are short; the rows stack below sm.
2026-09-17 03:53:40 +02:00
peter f92cd9017f fix: the diagnostics tab keeps its focus, its level and its consent
Deep-review round 1 of E22.

- After a 429 the stream sent one POST per new event while a full batch
  waited; it now stays quiet for 15 s. Past the queue cap the oldest
  events go, as the comment always said, a refused batch included.
- A batch refused after the level was lowered came back into the queue
  of a later consent in the same tab; each consent is its own session.
- A `me` fetch that left before a level or preference change could put the
  old value back for up to a minute. `updateMeCache` writes the cache and
  cancels that fetch; usePref goes through it too.
- The consent box's buttons and "revert now" took the focus with them to
  <body>; it returns to the level picker. Watched until the button is
  gone — in Chrome the saved level arrives in two renders, and a one-shot
  check spent itself on the first (jsdom never showed it).
- The countdown line is no longer a live region re-read every minute.
- "Saját napló" rows were keyed by index on a newest-first list, so every
  new event remounted every row and dropped the focus off a row button.
2026-09-17 03:53:40 +02:00
peter 4bfba4aa86 refactor: one definition of a followed channel and of a choice spelling
The explore purge rebuilt followed_channel's two EXISTS clauses inline, so what is polled and what may be deleted could drift; it now uses the predicate, which moved to sync/explore.py. The env validator and sysconfig each folded a choice value on their own; both call config.canonical_choice. The deep-backfill comment no longer calls recent backfill unconditional.
2026-09-17 03:53:29 +02:00
peter 86bc008a81 fix: a crashed job's crash line carries its job id
The bind sat inside the try, so it was already reset when the crash line and the cleanup were logged — the one line most worth finding by job id had none.
2026-09-17 03:53:29 +02:00
peter fac5c4465a fix: stored logs keep bad rows, erased accounts and unconsented queries out
Deep-review round 1 of E22.

- A NUL character (`GET /assets/%00` logs the decoded path) made Postgres
  refuse the row and with it the sink's whole batch, every user's rows
  included. `to_row` replaces NUL in every text column and in attrs.
- An erased account's lines kept arriving after purge_user's DELETE: the
  erasing request's own access line, a queued row, a background task.
  `logs.forget_user` runs first; later rows are stored without the account,
  and the check and the insert share a lock with the erasure.
- An admin setting the whole process to TRACE logged every user's query
  string. The query is now logged only under the user's own Teljes consent,
  and decoded before redaction, so `%40` no longer hides an address.
- The browser log ingest has an hourly storage budget per account (5 MiB,
  one notice when spent) and reads only a JSON Content-Type, so a
  same-site text/plain form post cannot write into a user's log.
  RateLimiter takes a per-attempt cost for the byte budget.
- The root level is applied under the elevation lock: two concurrent
  changes could leave it at the base while an elevation was active.
- Tests for the running sink (level gate, drain on stop) and for the
  admin log level applied at runtime and at boot.
2026-09-17 03:53:29 +02:00
peter 9c9b1cf6d5 test: the gate's output carries no noise to hide a real warning in
- Home tests mocked the api without downloadIndex, so every card's download button logged a
  missing-queryFn error (64 stderr blocks per run).
- QueryState rendered without an i18n instance; Modal.fullscreen awaited a synchronous act,
  which React reports as an act environment it does not support.
- pytest: Starlette 1.x's TestClient is built on httpx2 and warned on every run without it
  (pinned in requirements-dev); alembic.ini declares its path separator.
2026-09-17 03:03:37 +02:00
peter c3d88d9b20 refactor: component files export only components; lint fails on a warning
The 43 react-refresh/only-export-components warnings the gate let through meant an edit to a
provider (or AnchoredPopover, Toaster, plexDetailUi, …) could not be hot-swapped: Fast Refresh
reloads a module that also exports hooks or helpers.

- Each provider's contexts, the types its hooks return and the hooks themselves live in a sibling
  context module (navigationContext.ts, channelsContext.ts, …); the provider file keeps the
  component. loadAccountFilters moved next to the feed filters' defaults (feedFiltersInit.ts),
  following navigationInit/playlistsInit.
- Helpers that sat in component files have their own modules: welcomeMode, downloadTitle,
  plexPlayable, channelSync, ui/anchoredPopoverHooks, ui/pageToolbarSlot, ui/pagerContent,
  ui/toastLevels, ui/overlayOrder, ui/tableAlign, lib/plexDetail; the cells of the channel columns
  and the registry's page wrappers are components files of their own (channelCells, modulePages).
- usePersistedTab, an alias of useAccountPersistedState, is gone; its callers use the original.

npm run lint is now eslint --max-warnings 0, so a warning fails the gate like an error does.
2026-09-17 02:59:09 +02:00
peter 569190480d refactor: hook dependencies name everything the hooks read
The 12 react-hooks/exhaustive-deps warnings the gate let through:
- ShareDialog and YearFilter built a fresh [] per render while their query loaded, so their
  memos recomputed on every render; they share one empty list now.
- VideoEditor's drag effect called a seek/moveBoundary recreated each render; both are stable
  callbacks in its deps.
- WizardProvider re-checks onboarding on exactly the fields it reads (not the me object, whose
  60 s refetch would reopen a closed wizard).
- Channels, Feed and PlexPlayer list the stable values they already used.
- The player's keydown effect is bound once on purpose (its helpers read refs): said so, with a
  disable, as the codebase's other deliberate sites do.
- dialogFocus and the YouTube player's cleanup read refs React may already have let go of; they
  use what they captured when the effect ran.
2026-09-17 02:45:19 +02:00
peter 5883ea509d feat: Settings → Diagnostics, a consented log level and the tab's own log
A user can raise their diagnostics level to Részletes (DEBUG) or Teljes (TRACE) for 24 hours from a
new Settings tab, after a consent box that says what is recorded, that the server keeps it and for
how long (the instance's own retention), and that it lapses. Alap is the base and needs no consent.

- users.diagnostics_level/_until (migration 0087), written only by PUT /api/me/diagnostics, which
  computes the expiry itself; the demo account cannot raise it. /api/me reports the level in force.
- The server logs an elevated user's requests at that level and stores every line of them
  (logs.LevelGate on the stdout handler and the sink; the root is lowered only while an elevation
  is active). Teljes adds the redacted query string to the request line.
- POST /api/diagnostics/events stores the browser's log as source=web rows: account from the
  session, body capped while streaming, events/attrs capped, re-redacted, events under the
  consented level dropped, timestamps clamped, 30 batches a minute, 409 once the level lapsed.
- lib/diagnostics.ts applies the level to the tab and streams only what happens after consent, in
  batches, never its own request lines, with a keepalive flush on pagehide; stops on lapse (its
  own timer), on withdrawal and on 409.
- Detail hooks at DEBUG: which control was clicked (test id, icon, app copy in English — never user
  text, input values or a link's query), visibility/bfcache/resize, filter changes
  (lib/filterLog.ts; typed search only at TRACE), player events (lib/mediaLog.ts; media errors are
  warnings at every level). Offline/online at the base level.
- "Saját napló" viewer: live, newest first, level groups with counts, search, row details, copy
  the filtered events as JSON lines, clear.
- Privacy policy and Terms exist in Hungarian and English (a contract test holds both languages to
  the same sections); the policy now describes the server logs and diagnostic logging, self-service
  deletion and password sign-in.

Fixed on the way:
- A sync auth dependency ran logs.set_user on a copied context, so the account never reached a
  handler's own log lines (measured); a per-request mutable scope carries it now.
- The Settings tab rail left a phone no room (clipped labels, buttons over text): below `sm` it is
  a scrolling row above the content.
- PUT /api/me/preferences took a bare dict body; it is a Pydantic model now.
2026-09-17 02:42:12 +02:00
peter a210992bfc feat: client log with request ids, joined to the server's
The browser now keeps its half of the one event shape logs.py defines:
lib/log.ts is a levelled, redacted, bounded (2000 events) in-memory ring
with the server's level names and severity numbers. Nothing leaves the
browser yet; sending is the consented diagnostics level (S5).

- lib/http.ts `send` is the one fetch: every request carries an
  X-Request-Id and becomes one http.request event. The server echoes the
  matched route template in X-Route (only for a matched route, so a raw
  path with a value in it is never sent back); without one the client
  templates the path itself. req, beacon, the upload XHR, the public
  watch page and the legal pages all go through it.
- Hooks: app start snapshot, uncaught errors and rejections, History
  push/replace/popstate (a scroll-only replace is not an event; parameter
  values only at DEBUG, typed search only at TRACE), notifications, the
  error dialog, the crash boundary, failed queries and mutations (key
  head only), and the messages socket (lifecycle, frame types at DEBUG).
- Redaction is one contract: backend/tests/fixtures/log_redaction_cases.json
  runs against both redact() implementations.
- Lint: fetch, XMLHttpRequest and WebSocket each belong to one logging
  module, console to none; a contract test proves the rules fire.

Fixes a socket takeover: the close of a socket released by the last
subscriber cleared the newer socket a quick re-subscribe had opened and
scheduled a third, leaving two live sockets that delivered every message
twice (measured in Chromium: 3 created / 2 live before, 2 / 1 after).
2026-09-17 01:44:29 +02:00
peter 0517de1f59 fix: poll and backfill only channels somebody follows
The scheduled RSS poll and recent backfill covered every channel row,
but rows also come from search results, playlist sync and dropped
subscriptions, and nothing deletes them. On prod 582 of 979 polled
channels had no follower, three of them dead feeds answering 404 on
every poll. Both jobs now take subscribed or explored channels only,
and the scheduler card counts the same set. As a deliberate trade-off,
the shared Library no longer receives new uploads from channels nobody
follows.
2026-09-17 00:38:46 +02:00
peter b59fd38a9c feat: store warnings and request lines in log_events
The log now has a stored, queryable copy for the diagnostics console:
every WARNING and above plus the request line of every request, kept
for log_retention_days (default 7, no "forever") and pruned hourly by
the new log_gc job.

Rows are built on the logging thread from the same redacted fields as
the stdout line, queued without blocking and written in batches by one
background thread on its own connection. A full queue or a failed write
drops and reports instead of retrying. Flood control keeps 30 of any
one message per minute plus a count of the rest; anonymous request
lines are capped at 60 a minute per client, and the client address is
never stored. Deleting an account deletes its stored events.

Measured on prod first: a YouTube RSS outage wrote 11k warnings in four
hours, one per channel per poll. The poll now writes one warning with
per-reason counts and a few examples, and a failure on our side gets
one traceback per poll instead of one per channel.
2026-09-17 00:20:14 +02:00
peter 88a166e4aa fix: call the audit log "Auditnapló" in Hungarian
"Naplózás" (logging) named the audit log in the nav, its page and its
Configuration tab, right beside the new "Naplózási szint" (log level)
under Diagnosztika — two different things under one word. The scheduler
hint also pointed at a "Beállítások → Naplózás" path that does not
exist; it is Konfiguráció → Auditnapló, as the English already says.
2026-09-17 00:04:01 +02:00
peter 07766c43e8 feat: closed-set admin knobs are pickers, not free text
log_level, search_source, download_layout and the transcode hw/preset
were typed by hand, and every consumer carried its own guess at what a
typo meant. The value set is now a Literal in config.py: pydantic
rejects a bad env value at boot (any letter case accepted), sysconfig
refuses a bad override and derives the admin picker's options from the
same Literal, and the consumers read the value as-is.

The Configuration page renders a choice key as a segmented bar, or a
dropdown past six options, with translated option names. Its tabs now
follow the registry's order instead of a hand-kept copy that had
already dropped the explore group.

A single-choice FilterSelect now opens on the option in use: in the
browser it could land on the first row instead.
2026-09-17 00:03:56 +02:00
peter bb3c6dec45 release: 0.87.1 (bug/youtube-cookie-host)
gate / frontend (push) Failing after 3m13s
gate / backend (push) Successful in 4m22s
2026-09-16 04:41:40 +02:00
peter 3cf8f04d73 chore: 0.87.1 release notes 2026-09-16 04:41:10 +02:00
peter 2b56c5c2eb fix: name the error yt-dlp actually raises for a cookie-emptied client set
A review round read yt-dlp's source and found the comment behind the previous commit's safety-net
marker did not match it. Measured in the worker with a real account cookie and
`player_client=android_vr`: yt-dlp logs `Skipping client "android_vr" since it does not support
cookies`, drops it, and the request left with no client fails with "Requested format is not
available" — not "No player clients have been requested". That raise sits EARLIER in yt-dlp, before
the cookie-capability removal, so the cookie path can never reach it; and an unknown or renamed
client is skipped with a warning and replaced by yt-dlp's defaults, so the other scenario the
comment named cannot reach it either.

So the marker bought nothing: the symptom that does occur was already retriable here before this
branch. It is dropped, the comment now states what was measured, and the test asserts the real
symptom instead of a message the code cannot produce. The cookie-capability filter itself is
vindicated by the same measurement — without it every signed-in download would burn its first
attempt on a client yt-dlp discards.

Also from the round: an enqueue test for a `source_kind="youtube"` job, the one join nothing else
made (a YouTube job stores a bare 11-char ref and the URL the cookie gate keys on is rebuilt by the
adapter — both ends were pinned apart, the join was not, and a break there would stop every YouTube
cookie silently while signed-out downloads kept working). And copy corrections: the group's only
unhedged claim is hedged, "described the video" loses the pipeline vocabulary, and four Hungarian
phrasings that read as translated English are rewritten.
2026-09-16 04:29:13 +02:00
peter c06a456a9a fix: a YouTube stream refused after extraction is the gate, not a network fault
Measured on the isolated test stack the same night, on the very same video ~90 minutes after it was
answering with the bot message: yt-dlp got the player response and picked its formats, and the CDN
then refused the stream itself — "unable to download video data: HTTP Error 403: Forbidden".
Signed out it did that on every attempt; with the owner's cookie attached the video downloaded in
full. So the wall moves between the player response and the stream, and the text it wears there was
costing the download twice.

retriable: the format URLs belong to the client that minted them, so the next client set is a real
second chance — but this text classifies as `network` and nothing in the YouTube marker list matched
it, so the ladder ended the job on attempt 1 and never tried the fallback.

catalog: `youtube.media_forbidden` for the cookie-less case, so the UI stops calling it a network
error when nothing was wrong with the network, and names what actually gets past it. Scoped exactly
as narrowly as the measurement: another host keeps the generic `network` category, and so does a
download that already carried a cookie — telling a signed-in user to sign in is the failure the
catalog exists to avoid. EN + HU copy.
2026-09-16 03:43:29 +02:00
peter 864f673ef5 fix: a YouTube video behind the bot wall is downloadable with cookies
A YouTube link can fail with "Sign in to confirm you're not a bot" while every
other YouTube download works — YouTube gates INDIVIDUAL videos behind a
signed-in session. Measured on prod job 273: all twelve of yt-dlp's player
clients fail on that one video while its neighbours download fine, the bgutil
PO-token sidecar is live and answering, and the newest yt-dlp fails identically.
So the download is not broken by our stack or a stale yt-dlp, and no retry,
client switch or upgrade gets past it. A cookie is the only way in.

The failure copy already sends the user to Settings -> Account -> Download
authentication, but YouTube was not in the cookie host registry, so that
dropdown could not help them: the app asked for something it did not offer.

credentials: YouTube joins SUPPORTED_HOSTS (youtube.com, youtu.be,
youtube-nocookie.com). The Settings dropdown, the SEC-12 host scope and the
private-asset cache salt all derive from the row, so that is the whole wiring.

adapters: a signed-in YouTube download runs only cookie-capable player clients.
Measured against yt-dlp's own INNERTUBE_CLIENTS table: with account cookies
attached yt-dlp REMOVES every client whose SUPPORTS_COOKIES is False, and a set
it empties that way raises "No player clients have been requested" — which is
not a per-client flake it retries past. Our primary set is android_vr, one of
those, so without this the cookie would have failed the download on attempt 1
and never reached the POT-backed web fallback it exists for. A test pins the
client list against the installed yt-dlp so a version bump turns the gate red
instead of silently un-authing every YouTube download, and the empty-set message
joins the retriable markers as the second line of defence.

errors: catalog entries for the wall with and without a cookie, so a user who
has already added cookies is told they were rejected (YouTube rotates exported
ones) rather than to add some. The marker is quote-agnostic: YouTube writes the
apostrophe as U+2019, so "you're not a bot" with a straight quote never matches
the real text. EN + HU copy.
2026-09-16 02:03:00 +02:00
peter 6c63df6704 feat: one structured log for every process, with a request id
A deployment printed four different log formats and none of them could be
joined up: uvicorn's log_config.json for the API, a hand-rolled StreamHandler in
main.py for anything not started by uvicorn, logging.basicConfig() in the worker
and in the download child (no timestamp at all), and alembic's fileConfig.
app/logs.py replaces all four with one JSON line per event (OpenTelemetry's log
data model, no SDK): ts, level, severity, source, logger, event code, message,
attributes, plus the correlation the context carries — request id, account,
download job.

Redaction is applied at the HANDLER, not at the call sites: every message, every
attribute and every traceback is scrubbed on the way out, so a secret cannot
leak by being forgotten at one of ~125 call sites. Passwords, tokens, cookies,
secret config values and email addresses never reach a line.

The access log is ours now (`http.request`): the route template instead of the
concrete path, the status, the duration and the account — and never the query
string, which uvicorn's access log printed on every single request (`?account=`,
`?source=`, the `?token=` of a share link). A well-formed client request id is
accepted and echoed in X-Request-Id, so a browser line and a server line join up.
The account comes from `request.state`, filled by the auth dependency: the
middleware is the outermost one, so it runs outside SessionMiddleware and in a
different task context.

`log_level` becomes an admin knob (sysconfig), applied without a restart and
re-applied at startup — best-effort on purpose, since a logging knob must never
be what stops the app from booting. Every compose service gets bounded json-file
logs (3×50 MB): Docker keeps every line forever by default, and the API now
writes one line per request.

Verified on the localdev stack: JSON everywhere including alembic and the
worker, the query strings gone, a supplied X-Request-Id echoed and logged, and
three /healthz hits of which only the one made while the level was DEBUG
produced a line.
2026-09-16 01:37:02 +02:00
peter a5bcd9fca2 fix: numbers follow the UI language, not the browser's
`n.toLocaleString()` groups in the BROWSER's locale, so an English UI in a
Hungarian browser printed "1 234 567" while a Hungarian UI in an English
browser printed "1,234,567" — the number half of the date defect fixed in
618ef812. One `formatNumber` in lib/format.ts (a per-language cached
Intl.NumberFormat, like the relative-time formatters beside it) now serves
every grouped figure: Stats, Scheduler, the channel columns and the stats
popover, VideoCard's view tooltip, the ItemCount pill, and formatTotalHours.

ReleaseNotes' `toLocaleDateString(undefined, …)` was the one DATE site the
previous sweep missed; it moves onto formatDate.

The new cases assert both languages and that the two differ, instead of
comparing against the runtime's own default: that default is Hungarian on the
dev box and English on a CI image, so a one-language assertion would pass
under the old bug on half the machines. Verified in Chromium on :5173 with the
browser forced to hu-HU and the UI in English — Stats, Scheduler and the
channel table all keep comma grouping.
2026-09-16 01:05:53 +02:00
peter 618ef812e6 fix: dates follow the UI language, not the browser's
Seven sites formatted with a bare toLocaleString()/toLocaleDateString(), so an
English UI on a Hungarian machine printed Hungarian-shaped dates (and the reverse).
They now go through formatDateTime/formatDate, which take the i18n language.
2026-09-16 00:49:23 +02:00
peter e299c66121 feat: show each account's last sign-in in Users
A right-aligned column at the end of every Users & roles row says how long ago the
account signed in, with the exact moment as the hint and as a line in the account
panel; a dash where nothing is recorded, and the demo account says it is not recorded.
formatDateTime formats that moment in the UI language rather than the browser's.
2026-09-16 00:45:03 +02:00
peter 11aef4ab7a feat: record when an account last signed in
users.last_login_at (migration 0085, no backfill) is stamped by _establish_session,
which the Google callback now shares with password login instead of repeating its
session setup inline. The demo entry does not stamp its shared account. The admin
user list carries the field.
2026-09-16 00:34:50 +02:00
peter 4e52bdd4ca fix: a signed-out tab no longer asks for saved views
The providers mount for the public landing too, and the saved-views query had no
enabled gate, so every signed-out load logged a 401 for /api/saved-views. The query
now waits for a signed-in account in BootGate's me cache, like WizardProvider.
2026-09-15 23:45:13 +02:00
peter 595490cb98 release: 0.87.0 (improvement/watch-background-audio)
gate / frontend (push) Failing after 3m19s
gate / backend (push) Successful in 4m25s
2026-09-15 03:02:19 +02:00
peter 7537882235 chore(release): 0.87.0 2026-09-15 03:01:59 +02:00
peter 47d968eeea fix: review round 4 — an account switch unloads as the old account, a share link's lockout costs five networks
Deep-review round 4, targeted at round 3's changes (f569b19c..e7f38396).

Accounts
- Switching, adding or signing out an account moved the tab's pin before
  the page unloaded, so the players' resume beacons fired on the way out
  under the NEW account: one account's playback position saved into the
  other. The pin is now handed to the next page (pinOnNextPage) and taken
  over at import; clearActiveAccount is gone.
- Switching to a suspended account was refused but left it in the
  switcher, refused on every click; a deactivated one was not refused at
  all, and made the default it signed every account in the browser out on
  the next request. Every account _session_user would refuse — gone,
  deactivated, suspended, revoked — is refused and dropped from the list.

Share page (/watch)
- An IPv6 client is limited by its /64 (client_ip becomes client_key, for
  every limiter): keyed on the full address, one subscriber could rotate
  through endless identities. The comments now state what the per-link
  ceiling costs: guessing from five networks can keep a link shut for its
  window, and guesses already running when it fills are still checked.
- The link is looked up before the limiter records the attempt, so made-up
  tokens no longer grow it by an entry each.

Forms
- Failing the same way twice is announced again: the sign-in, passphrase
  and setup errors are keyed on the attempt (a clear and a set in one
  handler left the alert node untouched).
- The install wizard's own validation showed the generic error — it threw
  plain objects its catch did not read. It sets the step's message.

Tests: a re-pin leaving the previous account's notices behind; the pin
hand-over; switching to an ended account; the /64 viewer; made-up tokens;
the repeated alerts. Each mutation-checked RED for its own reason.
2026-09-15 02:53:22 +02:00
peter e7f38396a2 fix: review round 3 — a new tab keeps its account's notices, a revoked tab account lands on the default
Deep-review round 3 over main..improvement/watch-background-audio.

Account boot
- The notification history and settings, and the hints toggle, are read
  when their modules load, which in a new tab (bookmark, typed URL) is
  before BootGate pins the account: they read as nobody's, and the tab's
  first notice replaced the account's stored history (up to 100 entries)
  with a one-item list. They now read again when the tab is pinned
  (onAccountPinned, run by setActiveAccount); a notice raised before the
  pin stays, after the history.
- A tab pinned to an account still in this browser's wallet but whose
  session ended (suspended, deleted, or revoked by "log out everywhere")
  got "signed out" from /api/me while the default account was still
  signed in, and signing in again as anyone else left it there. A refused
  per-tab account now leaves the wallet, and the probe answers as the
  default, so the tab reloads onto it.

Share page (/watch)
- The password unlock is limited per viewer (link + client IP), under a
  per-link ceiling that counts only wrong passwords. Keyed on the link
  alone, ten wrong guesses every five minutes locked out every viewer,
  the ones with the password included. client_ip moves to app.ratelimit
  so the public routes can key on it.
- Its failure messages and the load-failure screen are announced
  (role=alert, the password input described by its error). The sign-in,
  passphrase, password-change and setup form errors get role=alert too.

Comments that still described removed machinery are corrected: App's
collapse flags, SettingsPanel's Save bar, ConfigPanel, authed_format_sig.

Tests: the account's history and hints across the pin; a revoked and a
suspended pinned account; the per-viewer limit and the ceiling; the
limiter's exhausted(); the share page's announced failures; a share
link's feed filters persisted. Each mutation-checked RED for its own
reason.
2026-09-15 02:25:00 +02:00
peter f569b19c49 fix: review round 2 — a signed-out tab acts as nobody, honest share-page failures
Deep-review round 2 over main..improvement/watch-background-audio.

Account resolution (server)
- A request naming an account that is no longer signed in to this browser
  (X-Siftlode-Account or ?account=) now acts as nobody instead of the
  session's default account. A tab still pinned to an account signed out in
  another tab went on acting as the default: its player checkpoints and
  preference saves landed in that account, the messages socket and Plex
  playback ran as it, and its Sign out signed THAT account out. A hidden tab
  never polls /api/me, so a tab playing in the background kept doing it.
  The /api/me bootstrap probe alone keeps the fallback, which is how such a
  tab learns who it now is: the app's 401 handler reloads, and BootGate
  re-pins the tab.

BootGate
- On an identity change the tab reloads without moving its pin first. The
  tree being torn down still runs its unmount saves before the page goes,
  and with the pin already moved they wrote into the new account's local
  keys (a player volume still pending). The reloaded page corrects the pin.

Share page (/watch)
- Only a 404 says the link is gone. A 5xx, a dropped connection, and the
  unlock's own rate limit all rendered "no longer available ... expired or
  revoked" to a viewer with a valid link, with no way to try again. A load
  failure now offers Retry; too many password tries says so and keeps the
  form; an unlock that gets no answer says so instead of doing nothing.
  Strings in en/hu/de.

Downloads
- A cookie-authed asset is recognised under the current signature version
  and the one before it, so a job queued before a deploy that bumps the
  version still downloads with its cookie instead of failing on the login
  wall with an error that says no cookie was used.

The app chunk is preloaded through lazyWithPreload, so a failed fetch no
longer leaves an unhandled rejection beside the ErrorBoundary. The panel
layout seed is one helper (seedLayout).

Tests: the stale-tab refusal, its harmless Sign out and the probe fallback;
the pre-bump authed asset; BootGate keeping the pin on reload and staying
mounted through a same-account change; the reveal clock's later-commit
hand-off; the prefs, panel layout and demo-scope seeds from the me cache;
the share page's failure states. Each mutation-checked RED for its own
reason.
2026-09-15 01:35:25 +02:00
peter e795d191c5 fix(boot): review round 1 — reconcile the tab pin, one loading clock, honest probe failures
Deep-review round 1 over main..improvement/watch-background-audio.

BootGate
- The tab's account pin is corrected to the account the server answered as.
  The server honours the pin only for an account still signed in to this
  browser and otherwise answers as the default, so a tab pinned to an account
  signed out in another tab ran as the default on the server while reading
  and writing the gone account's local state.
- A mounted tab whose identity changes (its session ended, or the case above)
  reloads instead of rebuilding the tree in place, like every account switch;
  that also drops the previous account's query cache. The account key on the
  tree goes: the readyFor gate already unmounted it on every change.
- An unanswered setup probe shows the error with Retry, like an unanswered
  account probe, instead of the signed-out landing to a signed-in user.
- Comments: /api/me never 401s; the ended-session path is identity → nobody.

Loading states
- LoadingState's reveal delay is one clock across loading states that hand
  over to each other (boot screen → page-chunk fallback → the page's own
  fetching state), instead of a CSS delay restarted by each, which blanked a
  visible spinner for another 300 ms at every hand-off. BootScreen is now the
  shared loading visual, full screen, counting from navigation start.

Plex shared link
- Persists only the dimensions the link carries again. Writing the default
  filters for an omitted `pf` created the stored-filters key, which is what
  tells SavedViewsProvider the account picked filters, so the account's
  default Plex saved view was never applied again.

Watch page
- The Repeat toggle uses the shared pressed-pill paint (a solid fill when on)
  instead of Button variants that differ only in colour, and its flag goes
  through storage.ts readJSON/writeJSON.
- The actions wrap under the title when the row is too narrow, instead of
  squeezing the title into a column a word wide on a phone; the title breaks
  long words.

Comments that still justified code by the removed adopt effects (usePref,
PrefsProvider, FeedViewProvider) are rewritten against the seed-once model.

Tests: BootGate error/retry for both probes, setup mode, a failed background
refetch keeping the app, the pin correction and the reload on an identity
change; the shared reveal clock; the Plex link persist; the audio signature
actually moved off version 4; a video download (av and v) still remuxed to
an mp4. Each mutation-checked RED for its own reason.
2026-09-15 00:53:15 +02:00
peter 4ad7183102 feat(players): keep a video playing in the background on Brave iOS; repeat on the watch page
On iPad Brave, a shared video stopped the moment the browser left the
foreground, even with Brave's "Enable Background Audio" on, while an mp3
kept playing. Brave iOS keeps a web <video> going by re-playing one the OS
paused (brave-core ios/browser/web/media/resources/media_backgrounding.ts),
but it only hooks a <video> present at document start or added as a
MutationObserver addedNode that IS a video. React inserts a player's
<video> inside its freshly mounted wrapper in one go, so Brave never saw it.

lib/browserBackgroundPlay re-inserts a mounted <video> at its own position,
once per element: it becomes an added node, the DOM React manages does not
change, and playback is not interrupted. The watch page, the downloads
library player and the Plex player hand their <video> to it. Measured on the
owner's iPad (Brave) with a prototype of the same idea: the video kept
playing in another app and with the screen locked. The YouTube modal is not
ours to change: its <video> lives in YouTube's cross-origin iframe, and
nothing of ours pauses it on backgrounding.

The prototype's two audio-element variants (a muted video following an
<audio>, and an <audio> taking over on hide) went silent in the background on
the iPad and are removed.

The watch page gets a Repeat toggle, on by default and remembered per
browser (LS.watchRepeat; the page has no account): it sets `loop` on the
<video> or the audio-only <audio>.
2026-09-14 23:58:30 +02:00
peter 2c3aaec62d feat(watch): prototype background sound for a shared video on iPad (dev only)
On iPad Brave a shared mp3 keeps playing when the browser leaves the
foreground, but a shared h264 video stops at once, even with Brave's own
"Enable Background Audio" on. Three variants behind a `?bg=` param on the
/watch page, for a real-iPad comparison; without the param the page is
unchanged.

  a: the sound comes from a hidden <audio> on the same file, the <video> runs
     muted and follows it (drift correction, user seeks and mute moved onto
     the audio; our own catch-up seeks are not copied back, which re-seeked
     the sound on every correction and sent both back seconds in WebKit).
  b: the <video> plays with sound; an <audio> unlocked by the start tap takes
     over while the page is hidden and hands back on return.
  c: no audio trick: the <video> is inserted as a node of its own into an
     already-mounted wrapper. Brave iOS's background play only hooks a video
     present at document start or added as a MutationObserver addedNode that
     IS a video (brave-core ios/browser/web/media/resources/
     media_backgrounding.ts); the page inserted the video inside a new wrapper
     in one go, so Brave never saw it. A re-implementation of that detection
     rule reports the default page's video unhooked and mode c's hooked.

Modes a and b were driven in Chromium and WebKit with a simulated backgrounding
(document.hidden forced plus the OS's pause): sound continues, the picture
catches up on return, a user pause stops both. iPadOS's own media policy can
only be judged on the iPad.
2026-09-14 23:40:51 +02:00
peter 70598d47a3 fix(dev): proxy the messages WebSocket through Vite, which stops a reset killing the dev server
The Vite dev proxy had no `ws: true` on /api, so the in-app messages socket
(/api/messages/ws) was an upgrade nobody handled: it never connected on :5173,
and the first reset on it (a closed tab, a dropped connection) threw an
unhandled ECONNRESET that took the whole dev server down. Reproduced with a raw
upgrade + RST (dev server dead); with `ws: true` the backend answers the
upgrade, the dev server survives the reset and a closed tab, and the signed-in
socket is accepted through the proxy.
2026-09-14 23:08:41 +02:00
peter 5858e7b513 test(e2e): run the net on WebKit and an iPad profile; opt the account out of Home
The Playwright net now runs every flow in three projects: chromium (Desktop
Chrome), webkit (Desktop Safari) and ipad (iPad gen 7 landscape: touch, mobile
viewport). WebKit catches the engine-level differences every iOS browser
inherits; the config and the README say plainly what it does NOT catch: iPad
media refusals. Playwright's WebKit decodes through the host OS, and on Windows
it played an mp3-in-mp4 and an AV1 mp4 that a real iPad refuses, so media
formats are checked on a real iPad against the dev stack over the LAN.

global.setup opts the e2e account out of starting on Home (the Startup setting),
which the feed, channel and player specs need since E24. It only takes effect
with the new-tab boot fix: before it, all twelve of those specs landed on Home
in every project. 49/49 green on :8080 across the three projects.
2026-09-14 22:29:33 +02:00
peter 5346efcd10 fix(boot): a new tab knows its account before anything renders
With Settings -> Appearance -> Startup turned off, every NEW tab or window still
opened on Home; only a same-tab reload honoured it. The provider tree mounted
before /api/me answered, so NavigationProvider's initializer could not read the
per-account opt-out, landed on Home and mirrored ?page=home into the address bar.
When App pinned the account a moment later, the re-hydrate re-read that mirror
and stayed on Home. The same mount-then-patch shape made a new tab start on
default filters, the default theme and the browser language, and correct each
after the first paint.

BootGate (main.tsx, above the providers) now owns the setup-status and /api/me
queries and mounts the tree only once the identity is settled: it pins the tab's
account and adopts the account's language in a layout effect, so the first frame
of the app is already right. The tree is keyed on the account id, which makes
"per-account state is read once, for this account" a structural guarantee.

With that, the whole re-hydrate machinery is gone: useAccountRehydrate, the
reactive active-account store, the markDirty guards, the pin-deferred saved-view
seed and the present-dimension bookkeeping of the shared Channels/Plex links.
PrefsProvider, FeedFiltersProvider, usePref and App's panel layouts seed from the
me cache instead of adopting it in an effect; the theme is applied in layout
effects. The app chunk is fetched in parallel with the probe, and the boot screen
only fades its label in after 400 ms, on one clock across the gate and the
Suspense fallback.

Module switches no longer flash a placeholder either. Measured frame by frame
in Chromium and WebKit, every first visit to a module showed "Loading..." for
17-70 ms: the lazy chunk's Suspense fallback, then often the first fetch's
LoadingState. Pages are now lazyWithPreload components, and App fetches the
chunks of every module the account can reach (plus the channel page) once the
first page is idle; a preloaded page renders synchronously and never suspends.
LoadingState, the one shared loading visual, fades in only after 300 ms, so a
fast load shows nothing in between and a slow one still gets its spinner.
2026-09-14 22:29:33 +02:00
peter 6169e32ca5 fix(downloads): a signed-in Facebook reel keeps its title, and an mp3 stays an mp3
Since a Facebook cookie was added on prod, every reel came down titled
"Video": measured on dev, a signed-in read gives yt-dlp the placeholder
"Video" (an 18+ reel read as Chrome gives "Facebook") while the post's text
is still in the description. A new SourceAdapter.title hook takes the whole
first line of the description in that case, and both title writes (the early
fill and the finished download) go through it, so the row, the file name and
the share preview carry the post's text again. The line is not cut with an
ellipsis: normalize_title's NFKC turns "…" into "..." and collapses it to
".", which read as a finished sentence (seen in dev UAT).

The "Audio (MP3)" profile's extracted .mp3 was remuxed into an .mp4 by the
browser-safe step (the audio spec is clamped to container mp4 too) and served
as video/mp4, which iPad's WebKit will not play (prod job 268). Audio mode now
skips that step, so an .mp3 is served as audio/mpeg and an .m4a as audio/mp4.
Audio downloads version their cache identity on their own (v5), so a cached
remuxed audio re-derives without re-downloading every cached video.
2026-09-14 17:46:46 +02:00
peter 90523284ab release: 0.86.0 (feature/e11.5-download-failures)
gate / frontend (push) Failing after 3m17s
gate / backend (push) Successful in 4m17s
2026-09-14 01:53:06 +02:00
peter 17bfea8754 release: 0.86.0 notes 2026-09-14 01:52:34 +02:00
peter 2a9ddb78ac fix(downloads): E11.5 epic-end review round 2
Pause now re-reads the job under the same asset-then-job row lock as resume,
cancel, delete and redownload. Its unlocked write waited for a concurrent
cancel's lock and then set the released job back to paused, a holding status,
so a later delete released the same hold again and could free a file another
user's job still held (measured: the overlapping-request test ends 'paused'
instead of 'canceled' with the old read). The lock test now calls the route
handlers, so it also pins which helper each route uses.

Comments follow the code: resolve_url's contract names both verdicts and that a
failed page read is raised as itself; the inherited-failure diagnosis asks the
cookie gate as it stands when the job is claimed, so its advice fits the cookie
that job's Retry would use; the undecryptable-cookie warning no longer claims a
download goes ahead, since the worker parent reaches it only to diagnose.
2026-09-14 01:42:57 +02:00
peter e887233210 fix(downloads): E11.5 epic-end review round 1
A refused Facebook cookie on a group post is no longer "no video in this post":
measured on dev, Facebook answers with its login page (no post id at all), so the
adapter now raises its own POST_NOT_SHOWN verdict and the catalog explains it as
the refused cookie. The adapter owns both post verdicts and the "cannot parse
data" wall wording, so the Chrome retry and the catalog read one constant. The
group-post "no formats" entry is cookie-scoped (with a cookie yt-dlp only gets the
post's video link) and uses the adapter's own group-post test instead of a
"/groups/" substring.

Job transitions that move an asset's ref count (resume, cancel, delete,
redownload) re-read the job under a row lock, asset first, so two overlapping
requests can no longer release the same hold. get_or_create_asset loses an insert
race inside a SAVEPOINT instead of rolling back the caller's pending writes (Retry
lost its cleared error). Every worker-side job failure goes through
finalize_if_running and writes error_detail, so a cancel that landed first is kept
and a failure never shows an earlier one's copy. The worker asks the download
child's own cookie gate (credentials.usable_cookie) when a job inherits a failed
asset.

UI: the Facebook copy no longer points at a Retry button the failure window does
not have, says "paste" like Settings, and the HU wording is fixed
("Újraletöltés" on the re-download confirm). A category this client has no copy
for falls back to the generic sentence instead of a raw key.

Tests: route serialization of error_detail, the probed duration through a real
_download, the page-read cap and a failed read, a Retry off a pending asset, the
confirm-button labels, the lock and savepoint races.
2026-09-14 00:50:05 +02:00
peter b17d92d7c2 fix(downloads): confirm dialogs name the action on their confirm button
The download center's confirm button repeated the dialog's question
("Re-download this file?"). It now reads the action: Stop download,
Re-download, Remove, Remove from my list.
2026-09-13 23:38:11 +02:00
peter 90ee34a1b4 feat(downloads): a Facebook group post downloads its own video, or says it has none
Given a group post's link, yt-dlp either found no formats or took another
video from the page's feed data, so a "successful" retry of such a post was a
wrong download. With the user's cookie the Facebook adapter now reads the post
page, takes the single video node in the script blocks that carry the post id,
and downloads that video's watch link (the Chrome retry does the rest). No
single video in the post fails with its own message and the group-post copy,
reworded to say the post has no video; the post link is never handed to yt-dlp
as a fallback. Without a cookie nothing changes.
2026-09-13 23:38:11 +02:00
peter 118fc70e08 fix(downloads): take the duration from the file when yt-dlp gives none
A Facebook or Instagram reel extracts with no duration, so the asset kept a
NULL duration_s although the file plays for minutes, and the Library row had
no trim action. The finalize probe now returns the duration with the codecs
(the file wins, info fills a gap), and runs before the sidecars so the .nfo
runtime gets it too.
2026-09-13 23:12:16 +02:00
peter 636372511b chore: remove the unreferenced backend/Dockerfile
Nothing builds from it: both compose files build the root Dockerfile, and the
test image has its own Dockerfile.test. It installed the unpinned
requirements.txt, contradicting the lock header that every image installs
requirements.lock.
2026-09-13 23:12:16 +02:00
peter 1354f5b503 feat(downloads): a Facebook cookie download stuck at "Cannot parse data" is retried as Chrome
Measured on dev: with the user's cookie some reels still answer "Cannot parse
data" and only extract when the request impersonates Chrome. Impersonation is
a second attempt behind that one error, only when a cookie is attached — links
that already download with the cookie keep their plain path, and without a
cookie the same error is the login wall.

curl_cffi is pinned inside yt-dlp's supported range (0.10-0.15; outside it
every target silently reports "unavailable"); the lock only gains the new
packages. A test fails the gate if the locked pair can no longer impersonate.
2026-09-13 22:48:21 +02:00
peter 98ec1ffcab fix(downloads): Retry and Re-download take a cookie added after the failure
The login-wall copy tells the user to add their cookies and press Retry, but
Retry kept the job on the shared asset the worker never attaches a cookie to,
so it failed on the same wall. Resume now moves a pending/errored job onto the
asset enqueue would pick with the cookie (never off a file or a download in
flight).

Re-download salted its fresh asset with the job's previous asset id, which the
cookie gate cannot recompute, so even a cookie download rebuilt without the
cookie. The rebuild salt is now the new asset's own id (plus the user's cookie
salt when one applies), and is_authed_asset recognises it.
2026-09-13 22:31:49 +02:00
peter 18c11ca437 feat(downloads): a per-site failure catalog — Facebook's walls say why and what to do
A failed Facebook download read "Download failed": yt-dlp words the login wall
(18+ reel, closed group) as "Cannot parse data", which no category matched.
Measured on dev with the owner's cookie before writing a rule — the same text is
the login wall without a cookie and a refused cookie with one.

- errors.diagnose(msg, url, cookie_attached) tries a host-scoped catalog first
  (all-of markers, cookie state, link fragment) and falls back to the category.
  First entries: facebook.login_wall, facebook.cookie_rejected,
  facebook.group_post_video. The module doc is the recipe for the next entry.
- download_jobs.error_detail (0084) + backfill of the stored Facebook parse
  failures that were unknown.
- The pipeline's failure branch passes the link and whether the cookie went
  with it; resume and redownload clear code and detail (redownload never
  cleared error_code).
- A job meeting an already-failed asset got the text but no code, so it read
  "Download failed" where the first job said why. It is diagnosed again, with
  the cookie state from credentials.is_authed_asset (now shared with the
  cookie gate); the redacted network text classifies back to network.
- FE: failCopyKeys gives the row and the failure window one lookup — catalog
  copy, else the category. Copy in EN and HU; a contract test pins every
  backend entry to both locales.
2026-09-13 21:53:25 +02:00
1600 changed files with 178716 additions and 46025 deletions
+1
View File
@@ -4,6 +4,7 @@
*.env.local
**/__pycache__/
**/*.pyc
**/.coverage
.venv/
venv/
# Docker patterns are anchored at the context root with NO implicit `**/` (unlike .gitignore, where
+18 -8
View File
@@ -28,6 +28,14 @@ ALLOWED_EMAILS=
# Admin emails (subset of the above) get the admin role.
ADMIN_EMAILS=
# Logging. LOG_LEVEL is the BOOTSTRAP level — the level a process starts at and the only one
# the worker/download child read; the API's running level is the admin Configuration page's
# `log_level` knob, which takes effect without a restart. TRACE | DEBUG | INFO | WARNING | ERROR, in
# any letter case; any other value stops the boot with an error. LOG_FORMAT is "json" (one object per
# line — what a log collector ingests) or "text" (a readable line for a local terminal).
LOG_LEVEL=INFO
LOG_FORMAT=json
# Optional: origin of a separately-served frontend dev server (enables CORS). Leave empty in production.
FRONTEND_ORIGIN=
@@ -60,18 +68,20 @@ SMTP_FROM=
# --- Scheduler ---
# The background scheduler (subscription sync, backfill, enrichment) runs inside the app.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. If you ever run
# more than one instance against the same database, keep SCHEDULER_ENABLED=true on exactly one
# of them and false on the rest, to avoid double quota use and write races.
# DATABASE_URL is set for you by docker-compose to the bundled `db` service. Run ONE app instance
# per database: a starting instance closes every run still open as interrupted (it assumes the
# previous process is gone), and "is this job already running" is held in the process. A second
# instance would cut the first one's runs short and could run the same job twice.
# SCHEDULER_ENABLED=false keeps the recurring jobs off; "run now" still works.
SCHEDULER_ENABLED=true
# --- Download center ---
# The Download Center adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# --- Files library & downloads ---
# Downloading adds a `worker` container (runs the yt-dlp/ffmpeg job loop) and a small
# `bgutil-pot` sidecar (mints YouTube tokens) — both come up automatically with docker compose.
# Downloaded media defaults to a Docker-managed named volume. Set DOWNLOAD_HOST_PATH to a host
# The Files library defaults to a Docker-managed named volume. Set FILES_HOST_PATH to a host
# directory instead — e.g. one your Plex server can read — to keep the Plex-style tree there.
# The path must be writable by the container user (uid of `appuser`, 1000): chown 1000:1000 <dir>.
# DOWNLOAD_HOST_PATH=/mnt/media/youtube
# FILES_HOST_PATH=/mnt/media/youtube
# --- Plex integration (optional) ---
# The optional Plex module browses/plays your Plex library, playing the LOCAL physical file
@@ -81,6 +91,6 @@ SCHEDULER_ENABLED=true
# to /plex-media inside the container (the local side of the admin "path map", whose default is
# /data=/plex-media). Leave unset if you don't use Plex.
# PLEX_MEDIA_HOST_PATH=/mnt/media
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to DOWNLOAD_ROOT; point it at
# Optional: scratch dir for on-the-fly HLS remux segments. Defaults to FILES_ROOT; point it at
# fast local storage if your download volume is slow.
# PLEX_HLS_DIR=/var/tmp/plex-hls
+4 -1
View File
@@ -2,8 +2,11 @@
# whole-repo reformatting does not bury the real authorship of each line.
# Configure once per clone: git config blame.ignoreRevsFile .git-blame-ignore-revs
#
# Prettier one-time repo-wide format pass (R2 S1b, 2026-07-21)
# Prettier one-time repo-wide format pass (2026-07-21)
51c36c2071728d2cd7ae580b7bfdf21f6867a347
# ruff format one-time backend + scripts pass (line-length 100, 2026-08-23)
b1832afd180866b922521148603db447020c57a4
# Plan-code purge from comments, one-time (2026-10-04)
4aaaaa786017278460e0b4f94cc0d3374e0e37e2
-71
View File
@@ -1,71 +0,0 @@
# Server-side mirror of the local `siftlode check` gate (E1 S3 / DX-1), so `main` is always green
# independent of any one machine. Two parallel jobs match the two local lanes:
# frontend — tsc (3 projects) + eslint + prettier --check + knip + vitest
# backend — ruff + the full pytest suite (pure-logic + the DB-backed lane) against a real Postgres
#
# Runner: the lab act_runner (LXC 115) mounts the Docker socket, so service containers work. The backend
# runs pytest NATIVELY against a `postgres` service (not the local docker-compose.test.yml, whose
# bind-mount is unreliable under a mounted-socket runner); fidelity is kept by installing the SAME pinned
# requirements, Postgres 16, and ffmpeg, and by setting SIFTLODE_TEST_REQUIRE_DB=1 so a DB/ffmpeg that
# failed to come up is a hard error, never a silent skip. node 22 matches the prod build image
# (Dockerfile `node:22-alpine`).
name: gate
on: [push, pull_request]
jobs:
frontend:
runs-on: ubuntu-latest
container: node:22-bookworm
defaults:
run:
working-directory: frontend
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run typecheck
- run: npm run lint
- run: npm run format:check
- run: npm run knip
- run: npm run test
backend:
runs-on: ubuntu-latest
# A combined python+node image: python 3.12 (matches the prod Dockerfile) for the tests, and node
# so the JS actions (checkout) run. actions/setup-python is NOT usable here — the runner can't reach
# GitHub's python-versions download, so it must be baked into the image.
container: nikolaik/python-nodejs:python3.12-nodejs20
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: siftlode
POSTGRES_PASSWORD: siftlode
POSTGRES_DB: siftlode_test
options: >-
--health-cmd "pg_isready -U siftlode -d siftlode_test"
--health-interval 2s
--health-timeout 3s
--health-retries 30
env:
DATABASE_URL: postgresql+psycopg://siftlode:siftlode@postgres:5432/siftlode_test
SIFTLODE_TEST_REQUIRE_DB: "1"
steps:
- uses: actions/checkout@v4
# ffmpeg for the poster/thumbnail tests; postgresql-client for the readiness wait below.
- run: apt-get update && apt-get install -y --no-install-recommends ffmpeg postgresql-client
- name: Wait for Postgres
run: until pg_isready -h postgres -U siftlode -d siftlode_test; do sleep 1; done
- run: pip install --no-cache-dir -r backend/requirements.lock -r backend/requirements-dev.txt
- name: ruff
run: python -m ruff check --config backend/ruff.toml backend scripts
- name: ruff format --check
run: python -m ruff format --check --config backend/ruff.toml backend scripts
- name: pytest
run: python -m pytest
working-directory: backend
# Dependency CVE audit over the pinned lock. Non-blocking for now (informational) — a real CVE
# shows up in the run without failing the gate; tighten to a hard fail once the noise is known.
- name: pip-audit (advisory)
run: pip install --no-cache-dir pip-audit && pip-audit -r backend/requirements.lock
continue-on-error: true
+8 -2
View File
@@ -9,6 +9,7 @@ __pycache__/
venv/
.mypy_cache/
.pytest_cache/
.coverage
.ruff_cache/
# Node / frontend
@@ -28,12 +29,17 @@ frontend/e2e/.auth/
*.sql.gz
backups/
# Download center: locally downloaded media (the DOWNLOAD_ROOT bind mount at the repo root).
# Download center: locally downloaded media (the FILES_ROOT bind mount at the repo root).
# Anchored with a leading slash so it does NOT also ignore the backend/app/downloads package.
/downloads/
/files/
# OS / editor
.DS_Store
Thumbs.db
.idea/
.vscode/
# TypeScript incremental build artifacts — machine-local, absolute-path-bearing, and no gate lane
# reads or regenerates them, so a committed one is stale from the commit that adds it.
*.tsbuildinfo
frontend/e2e/.speed/
+8
View File
@@ -0,0 +1,8 @@
# gitleaks false positives, one fingerprint per finding (commit:file:rule:line). Each says why it is no secret.
#
# YouTube's public WEB InnerTube key: embedded in every youtube.com results page, read live and cached;
# this constant is only the fallback (backend/app/youtube/innertube.py).
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/app/youtube/innertube.py:gcp-api-key:23
# A recorded watch-next page's continuation cursors (continuationCommand.token): page pointers, not credentials.
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:768
8b37c470ba5d11c45757f0b2e62d660b782e816d:backend/tests/fixtures/innertube_next_first.json:generic-api-key:792
+1
View File
@@ -0,0 +1 @@
24
+1
View File
@@ -0,0 +1 @@
3.12
+15
View File
@@ -0,0 +1,15 @@
# The CVE gate's allowlist. The publish gate scans the built image and the committed tree with
# Trivy and fails on any HIGH or CRITICAL finding that has a fix available. A finding that cannot be
# fixed yet is let through ONLY by an entry here, and every entry carries both:
# statement — why it is acceptable for now (what is exposed, what is not)
# expired_at — the date it stops being accepted; after it, the gate fails again on its own
# A contract test (frontend/src/lib/gates.contract.test.ts) refuses an entry without either, and
# docs/gates.md lists them.
#
# Example:
# vulnerabilities:
# - id: CVE-2026-00000
# statement: in libfoo's XML path, which nothing in the image calls; no fixed Debian build yet
# expired_at: 2026-12-31
vulnerabilities: []
secrets: []
+1184 -1205
View File
File diff suppressed because it is too large Load Diff
+87 -44
View File
@@ -5,19 +5,23 @@ be developed that way. This file is the rules of the road: read it before you to
loaded into every agent session, so keep it **short and true**. A rule here that has gone stale is
worse than no rule — if you change how the code works, change this file in the same commit.
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a download center). It is a
Siftlode is a self-hosted YouTube reader (+ a local Plex browser/player and a files library with downloads). It is a
**modular monolith**: one Postgres, one FastAPI process (uvicorn + APScheduler), one worker process
(yt-dlp/ffmpeg), one PO-token sidecar. React 18 + react-query frontend with a router-free navigation
provider.
(yt-dlp/ffmpeg), one PO-token sidecar. React 19 (with the React Compiler) + react-query frontend, with a
router-free navigation provider.
## The gate — never merge or ship red
The local gate is **ruff (`check` + `format --check`) + typecheck (3 tsc projects) + eslint + prettier
--check + knip + vitest + pytest**. It runs on every `publish`, which **refuses to build on failure — there is deliberately no
`--skip` flag.** Run it and get it green **before** any merge, release, or handing work to review.
The local gate is **ruff (`check` + `format --check`) + shellcheck + basedpyright (backend) + the lock check + typecheck (3 tsc projects) +
eslint + prettier --check + knip + vitest + vite build + pytest + pip-audit + npm audit**. It runs
before every `release` merges and on every `publish`, which **refuses to build on failure — there is deliberately no `--skip`
flag**; a release also requires the Playwright suite, and the image must pass a Trivy scan before it is pushed. Run it and get it
green **before** any merge, release, or handing work to review.
- Full gate (agents with the private ops dispatcher): `siftlode check`.
- Directly, without the dispatcher: `cd frontend && npm run typecheck && npm run lint && npm run knip && npm run test`, plus backend `ruff check backend` + `ruff format --check backend` and `pytest` (the DB-lane needs Postgres).
- Full gate: `bash scripts/check.sh` (`lint` for the static lanes only); the private ops dispatcher's `siftlode check` runs the same script. Single lanes: [`docs/development.md`](docs/development.md) § The gate.
- **There is no CI** (owner, 2026-10-04): this gate, run here, is the only one — green before every hand-over, and `release` refuses to merge without it.
- **A gate with any real hit is never switched off or narrowed** — not a lint rule, a compiler flag, a lane or a release step. Fix the sites, or suppress each one inline naming its invariant (`-- why`, lint-enforced). What stays narrowed is in [`docs/gates.md`](docs/gates.md), and a contract test fails when a narrowing is missing from it. The effective ESLint config of every scope and the gate/lane lines are pinned in `frontend/src/lib/gates.snapshot.json`: any gate change fails until you regenerate it (`GATE_SNAPSHOT=update`, see gates.md) — then read that diff as a gate change. A comment saying "X is enforced by Y" is checked by running Y (E32: one such comment hid 95 functions the React Compiler skipped).
- **The backend type lane runs in the test image, not on the host** — `requirements.lock` cannot be installed on Windows (uvloop), and a type checker without the real packages reports nonsense. It fails on NEW errors only; the pre-existing ones sit in `backend/.basedpyright/baseline.json` (no burn-down mandate, but none was ever triaged: a `reportOptional*` entry in code you touch may be a real bug). A run that fixed some rewrites it — commit that.
- New DB-backed test? It must actually run, not silently skip — assume `SIFTLODE_TEST_REQUIRE_DB=1`.
## Workflow
@@ -45,56 +49,94 @@ how it gets built. Concretely:
behind each option and the weight of the decision. These decision points are where the product owner
engages with the build and learns from it, so make them understandable — a reasoned menu, not a bare one.
- **Mockups for design/UX decisions.** For any visual or interaction choice, show mockup options to
choose from before building.
choose from before building. How the UI looks and behaves is decided in
[`docs/DESIGN.md`](docs/DESIGN.md) (its variants render in the dev-only `/design-lab`): build to it,
and change it only through its decision log.
- **Challenge unsound requests.** If a request is technically ill-advised and you can back it with real
arguments, push back or propose an alternative — don't comply blindly.
- **Verify UI in the browser before calling it done.** tsc/unit/DOM tests miss visual and interaction
bugs; do a real visual check of any UI change.
bugs; do a real visual check of any UI change, and run DESIGN.md's Conformance check (a copy-only
change runs its step 4 alone: every state the new text renders in, in both languages).
- **Be terse everywhere else.** Outside those decision points, lead with the answer or decision. During
implementation and debugging especially: no restating the same point reworded, no filler.
## Architecture shape (don't fight it)
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `youtube/` (shared by routes + scheduler + worker) → flat core (`auth`, `quota`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `realtime`, `ratelimit`, `progress`,
`security`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
Frontend: `main.tsx` provider tree → `App.tsx` shell → `moduleRegistry` route-chunks → module
components → the **platform layer `components/ui/`** (domain-free primitives: the popover engine and
its menus, `Modal`/`Overlay`, `Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`, …). That
boundary is one-way and **lint-enforced**, not just described: a platform component may not import a
feature component or a domain API slice, and one layer down `lib/` may not import from `components/` at
all, not even a type (`no-restricted-imports` in `eslint.config.js`, with its own test). A new generic
component goes in `ui/`; see [`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
Data via react-query (`qk` factory), per-account `localStorage`, and server prefs; API through
`lib/api/*` slices over `core.req<T>`.
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
only along a declared edge and writes them only through that unit's service; core reaches a unit's tables through its
`queries`. **Fence-held, with no exemptions** (`backend/tests/test_unit_fences.py`) — as far as an AST sees: what it
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
[`backend/app/README.md`](backend/app/README.md) with the conventions.
Frontend, by folder under `src/`: `main.tsx` provider tree → `shell/` (`App`, header, nav rail, palette,
boot; a shared `/watch` link boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
app-free by a contract test) → **`modules/<page>/`** (one folder per page: the page, its rail, its store, its tests,
and its `module.ts` definition — label, icon, gate, the lazy page and rail, its store — collected in nav order by
`features/navigation/moduleRegistry`, which also mounts every store at the root; a new page is its folder, its id in
`lib/urlState` and its entry there, plus its panel in `lib/panelLayout` if it has a rail) →
**`features/<name>/`** (what several pages share: video cards, filters, downloads, the player, navigation) → the
**platform layer `components/ui/`** (domain-free primitives: the popover engine and its menus, `Modal`/`Overlay`,
`Button`/`form`, `DataTable`/`VirtualGrid`, `QueryState`,
…) → `lib/`. The direction is one-way and **lint-enforced**, not just described (`local/layers`, with
its own test): an import only goes down; another module is reached only through its **`index.ts`** (its
public face — the stores, mounted at the root and read app-wide, which `features/` may read too) or
lazily by `import()`; a platform component may not import a domain API slice, and `lib/` nothing above
it, not even a type. A lazily imported page, rail or dialog that a static import drags onto the boot
path fails the build (`chunkGuard.ts`). A new generic component goes in `ui/`; see
[`frontend/src/components/ui/README.md`](frontend/src/components/ui/README.md).
Data via react-query (`qk` factory), the signed-in account's `localStorage` (`storage.ts`), and server
prefs; API through `lib/api/*` slices over `core.req<T>`.
**Not wanted** (reviewed and rejected as over-engineering for this size): microservices, event bus,
CQRS, a repository/DI framework, a router library, a generic CRUD-router factory or base serializer.
## Canonical patterns — when you have a choice, pick THIS
| Area | Use | Not |
| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------ |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Ownership check | `owned_or_404` / `get_or_404` | ad-hoc `_own_playlist`/`_owned`/`_own_job` |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`) | ad-hoc `{"unsubscribed"}`/`{"cleared": n}`/echo |
| Transaction commit | the handler/job-wrapper commits; helpers `flush` | helpers that commit (only `quota` owns its session) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / `scheduler_settings` scatter |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE per-account state | `usePref` | raw `readAccount`/`writeAccount` or bespoke hooks |
| FE action button | `ui/Button` — pick a `SIZES` rung | a hand-rolled `<button className="px-3 py-2 …">` |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE control | the chip/segmented primitives (`Select` only for >6 options) | a native `<select>` |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)` | a bare `catch {}` or `notify()` with no error attached |
| Area | Use | Not |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ORM | SQLAlchemy 2.0 `select()` | `db.query(...)` |
| Request body | a Pydantic model | `payload: dict` |
| Request SIZE | nothing — `bodylimit` caps every route (SEC-20); need more? add a ceiling to `ROUTE_MAX_BODY_BYTES` | `capped_body` on a route that does not read raw bytes itself |
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
| Job status word | the `downloads/status.py` StrEnum | loose `"canceled"`/`"cancelled"` literals |
| Current time | inline `datetime.now(timezone.utc)` | new `now_utc`/`_now` helpers |
| FE browser storage | a key in `storage.ts` with its kind (contract-held): a `pref` through `usePref` (the server's, follows the account); `device` / `view` / `cache` this browser's; `public` kept for the /watch page | a `"siftlode.x"` literal; a preference kept only in `localStorage` (it stays on one device) |
| FE keyboard shortcut | a row in `lib/keymap.ts` + `dispatch()` | a private `keydown` listener with its own key literals |
| FE list/grid keys | `NavGrid`/`keyboardNav` + `navItemAttrs` + `data-nav-open` | a private arrow-key / roving-tabindex handler |
| FE view push (history) | `rememberNavPosition()` right before `history.pushState` | a push that forgets: Back strands focus on `<body>` |
| FE view state | the module's URL codec: its keys in `urlKeys` (`module.ts`), read and written by `lib/<id>UrlState.ts` through `useUrlMirror` (contract-held, `e2e/bookmark.spec.ts`) | `useState` / sessionStorage alone: a reload, a bookmark or a shared link loses the view |
| FE a page's header content | the page renders it into the floating header itself (`PageHeader`, `ui/PageShell`) | a `page === …` branch in the shell's header, or a bridge carrying a page's state up to it |
| FE module store | its definition's `Store` (`module.ts`), mounted at the root by `ModuleStores`; it reads no other module's store (contract-held) | a provider hand-nested in `main.tsx` |
| FE saved views | the module's own store from `features/filters` (`createSavedViewsStore` + `useSavedViewsStore`) | one store switched by the active page — every visit elsewhere reset it |
| FE static label / button-like link | `Badge` (flat, no edge) · `LinkButton` / `IconLink` | a glass pill that reads as a chip; an `<a>` hand-painted like a button |
| FE loading/empty/error | `QueryState` / `StateMessage` | ad-hoc conditionals |
| FE failed write | `meta.failTitle`, else `notifyActionError(err, title)`; a YouTube write action (subscribe, sync, …): `notifyYouTubeActionError` — a 403 offers Connect | a bare `catch {}` or `notify()` with no error attached |
| FE floating or full-screen layer | `ui/Overlay` (portals to `<body>`) or the popover engine (`AnchoredPopover` portals itself); stacking by the z-index scale (`z-menu` … `z-popover`, docs/DESIGN.md "Layers") | one rendered in place inside a virtualized row or a masked/transformed scroller — clipped or painted over; a magic `z-[n]` |
| FE player on a native media element | `lib/mediaSession` (`useMediaSession`) + `lib/mediaKeys` + `lib/resume` (`useResume`), the element held as STATE | a private `navigator.mediaSession` / `keydown` / position-saving effect per player |
| FE diagnostics | `lib/log.ts` (`log.info("event.code", msg, attrs)`); a filter change → `lib/filterLog.ts`, a player → `lib/mediaLog.ts` | `console.*` (lint-enforced); a value only a higher level may carry, logged unguarded |
| FE state that MIRRORS a prop/query | derive it, or hold a draft (`draft ?? server`) | a copy an effect re-syncs — that paints the stale value first |
| FE state reset on a key change | adjust it DURING render against a remembered key | `useEffect(() => setX(init), [key])` |
| FE effect that must not re-subscribe | `useEffectEvent` (React 19.2+); a handler or timer: `useLatest` / `useStableCallback` | a render-phase `ref.current = x` — the React Compiler skips the component |
| FE hook suppression | an inline `eslint-disable-next-line` NAMING the invariant | `eslint-suppressions.json` (deliberately absent — E27 S2) |
| FE popover / scroll-fade ref | destructure it: `const { setReference } = pop.refs`, then `ref={setReference}` | `ref={pop.refs.setReference}` / `ref={fade.ref}` — the compiler then treats the whole object as a ref and skips |
| FE async handler with cleanup | `try { … } catch { … }`, then the cleanup after it (steps that `return` early go in their own function); a conditional, loop or `?.` the `try` needs goes before it or into a module-level helper | `try/finally`, or a `?:` / `??` / loop / `?.` inside `try` — the React Compiler skips the whole component (and `react-hooks/todo` misses most of these) |
Not repeated here, because a lint rule refuses the wrong one and its message names the right one:
buttons (`ui/Button`, `IconButton`, `Pressable`, `Chip`, `DragHandle`), text fields and native
controls (`ui/Input`, `Checkbox`, `Slider`, …), glass surfaces (`SURFACE.*`), a native `<select>`
(`SegmentedToggle` / `FilterSelect`), a picture (`ui/RemoteImage`, never a bare `<img>`), a box or font size (the spacing scale or rem,
never a px-frozen `w-[Npx]` / `text-[Npx]`: the text-size slider scales the root font), and network calls (`req` / `uploadReq` / `beacon`, `lib/http.ts`).
## House rules that bite (learned from real bugs)
@@ -107,7 +149,8 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
"no results". Reads degrade honestly; writes are conditional (`UPDATE … WHERE status='running'`).
- **i18n parity.** HU/EN, zero key drift. Every user-facing string is translated.
- **Comment the "why", keep it current.** This codebase uses comments as its architecture doc; a stale
"why" is a liability, so update it when the behavior changes.
"why" is a liability, so update it when the behavior changes. Say the why itself, never the plan
step that produced it: no epic, sprint or review-round ids in the code (`planCodes.contract.test.ts`).
---
+23 -9
View File
@@ -10,7 +10,9 @@ ARG GIT_SHA=unknown
ARG BUILD_DATE=
# Stage 1: build the frontend SPA
FROM node:22-alpine AS frontend
# The major is pinned to `.nvmrc` and held there by frontend/src/lib/nodeEngine.contract.test.ts:
# this image and the dev machine the gate runs on must be ONE engine (see that test for why).
FROM node:24-alpine AS frontend
WORKDIR /fe
# Deps first, keyed on the lockfile, so editing app source never re-resolves node_modules.
COPY frontend/package.json frontend/package-lock.json ./
@@ -37,22 +39,31 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
# Create the app user BEFORE the copies so files land already-owned via COPY --chown (a trailing
# `chown -R /app` would rewrite every file into a duplicate layer). /downloads must be appuser-owned
# `chown -R /app` would rewrite every file into a duplicate layer). /files must be appuser-owned
# so a fresh named volume mounted there inherits that ownership (Docker copies the image dir's
# ownership into a new empty volume) — the worker/API can then write without a manual chown. A bind
# mount to a host path instead needs that path writable by this uid (see docs/self-hosting.md).
RUN adduser --disabled-password --gecos "" appuser \
&& mkdir -p /downloads \
&& chown appuser /app /downloads
&& mkdir -p /files \
&& chown appuser /app /files
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode (E6 S1).
# ffmpeg: merge separate video+audio streams + postprocessing, and the Plex VAAPI transcode.
# intel-media-va-driver (iHD): the VA driver ffmpeg's h264_vaapi needs — libva2 ships in the base but
# no driver, so without this `vainfo`/`h264_vaapi` fail and transcode silently falls back to libx264.
# Harmless on hosts with no /dev/dri (the driver just goes unused). Deno: the JavaScript runtime yt-dlp
# uses to solve YouTube's "n" signature challenge for the web player clients (required alongside the PO
# token, else only storyboards are offered). yt-dlp auto-detects deno on PATH.
# `apt-get upgrade`: the base image is a snapshot, and the Debian security fixes released since it
# was cut are only in the archive — the first Trivy scan found one (pcre2) fixed upstream but not in it.
# APT_REFRESH: the upgrade only runs when this layer is rebuilt, and a build cache keeps it for as long
# as nothing above it changes — a publish shipped an image whose curl and librsvg were a release behind
# fixes already in the archive. The release build passes the day here, so the layer is rebuilt at
# least daily; empty (the default, a self-hosted build) leaves the cache alone.
ARG DENO_VERSION=v2.9.1
RUN apt-get update \
ARG APT_REFRESH=
RUN echo "apt refresh: ${APT_REFRESH:-cached}" \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends ffmpeg intel-media-va-driver unzip curl ca-certificates \
&& curl -fsSL "https://github.com/denoland/deno/releases/download/${DENO_VERSION}/deno-x86_64-unknown-linux-gnu.zip" -o /tmp/deno.zip \
&& unzip -o /tmp/deno.zip -d /usr/local/bin \
@@ -74,11 +85,14 @@ RUN printf 'Types: deb\nURIs: http://deb.debian.org/debian\nSuites: bookworm\nCo
&& rm -rf /var/lib/apt/lists/*
# Install from the PINNED lock (requirements.lock, generated from requirements.txt) so the image is
# byte-reproducible and the exact versions are the ones pip-audit scanned in CI — not whatever floats
# up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# byte-reproducible and the exact versions are the ones the gate's pip-audit scanned — not whatever
# floats up from the floor ranges on build day. Regenerate the lock when requirements.txt changes (see its header).
# pip itself is removed once it has installed the lock: nothing runs it at runtime, and it carries
# its own VENDORED copies of urllib3, msgpack and setuptools, which Trivy reports as the image's own —
# an attack surface and a CVE stream for a tool the running app never calls.
COPY backend/requirements.lock .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && pip install -r requirements.lock
pip install --upgrade pip && pip install -r requirements.lock && pip uninstall -y pip
COPY --chown=appuser backend/ .
# Read at runtime by config.py as the app_version (a committed single source of truth).
+5 -3
View File
@@ -22,9 +22,11 @@ blocker and SponsorBlock keep working.
tags to slice the feed by.
- **Playlists with two-way YouTube sync** — build them locally, keep them in sync in both directions.
- **In-app player** with resume, plus keyboard/scroll controls.
- **Download Center** — save videos to the server with yt-dlp in a Plex-friendly layout (format
presets, per-user storage quota), trim / crop / split & join them in a built-in editor, then save
to your device, share with another user, or hand out a public watch link.
- **Files** — one library for everything you keep: paste a link anywhere in the app to download it
with yt-dlp (format presets, per-user storage quota, the running downloads in a tray), upload your
own files, sort them into folders, trim / crop / split & join videos in a built-in editor, play them
where you left off, then save to your device, share with another user, or hand out a public watch
link that plays on any phone.
- **Plex library** *(optional)* — point Siftlode at your own Plex server to browse and play its
movies and shows alongside your feed, with watch state kept in sync.
- **Multi-user** with per-user private state, a shared catalog, and a fair daily API-quota guard.
+1 -1
View File
@@ -1 +1 @@
0.85.0
0.114.0
+918
View File
@@ -0,0 +1,918 @@
{
"files": {
"./app/audit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/debugsession.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/diagnostics.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 40,
"endColumn": 49,
"lineCount": 1
}
}
],
"./app/discover/pins.py": [
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 8,
"endColumn": 9,
"lineCount": 5
}
}
],
"./app/discover/queries.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 33,
"endColumn": 44,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 39,
"lineCount": 1
}
}
],
"./app/discover/rails.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 4,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 18,
"endColumn": 41,
"lineCount": 1
}
}
],
"./app/downloads/adapters.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 38,
"lineCount": 1
}
}
],
"./app/downloads/backfill.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportReturnType",
"range": {
"startColumn": 23,
"endColumn": 27,
"lineCount": 1
}
}
],
"./app/downloads/credentials.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 27,
"endColumn": 30,
"lineCount": 1
}
}
],
"./app/downloads/gc.py": [
{
"code": "reportOperatorIssue",
"range": {
"startColumn": 16,
"endColumn": 28,
"lineCount": 1
}
}
],
"./app/downloads/pipeline.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 22,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportGeneralTypeIssues",
"range": {
"startColumn": 10,
"endColumn": 18,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 31,
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 43,
"lineCount": 1
}
}
],
"./app/downloads/service.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/downloads/urlguard.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 11,
"endColumn": 41,
"lineCount": 1
}
}
],
"./app/feedquery.py": [
{
"code": "reportCallIssue",
"range": {
"startColumn": 37,
"endColumn": 5,
"lineCount": 3
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 37,
"endColumn": 5,
"lineCount": 3
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 8,
"endColumn": 97,
"lineCount": 1
}
}
],
"./app/logsink.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/notifications.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 52,
"endColumn": 58,
"lineCount": 1
}
}
],
"./app/plex/lookups.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 48,
"endColumn": 58,
"lineCount": 1
}
}
],
"./app/plex/people.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 15,
"endColumn": 23,
"lineCount": 1
}
}
],
"./app/plex/sync.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 21,
"endColumn": 69,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 21,
"endColumn": 69,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 70,
"endColumn": 83,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 76,
"endColumn": 89,
"lineCount": 1
}
}
],
"./app/plex/tmdb.py": [
{
"code": "reportAssignmentType",
"range": {
"startColumn": 16,
"endColumn": 9,
"lineCount": 10
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 36,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 29,
"endColumn": 35,
"lineCount": 1
}
}
],
"./app/plex/watch_sync.py": [
{
"code": "reportReturnType",
"range": {
"startColumn": 11,
"endColumn": 5,
"lineCount": 11
}
}
],
"./app/ratelimit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 14,
"endColumn": 25,
"lineCount": 1
}
}
],
"./app/routes/admin.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 9,
"endColumn": 13,
"lineCount": 1
}
}
],
"./app/routes/audit.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 37,
"endColumn": 45,
"lineCount": 1
}
}
],
"./app/routes/discover.py": [
{
"code": "reportInvalidTypeForm",
"range": {
"startColumn": 19,
"endColumn": 33,
"lineCount": 1
}
}
],
"./app/routes/plex/collections.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 20,
"endColumn": 45,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 20,
"endColumn": 46,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 39,
"endColumn": 47,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 56,
"endColumn": 59,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 46,
"endColumn": 49,
"lineCount": 1
}
}
],
"./app/routes/scheduler.py": [
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 25,
"endColumn": 28,
"lineCount": 1
}
},
{
"code": "reportOptionalMemberAccess",
"range": {
"startColumn": 25,
"endColumn": 28,
"lineCount": 1
}
}
],
"./app/sync/explore.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 6,
"endColumn": 14,
"lineCount": 1
}
}
],
"./app/sync/maintenance.py": [
{
"code": "reportOptionalOperand",
"range": {
"startColumn": 19,
"endColumn": 42,
"lineCount": 1
}
}
],
"./app/sync/playlists.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 55,
"endColumn": 72,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 51,
"endColumn": 68,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 39,
"endColumn": 56,
"lineCount": 1
}
}
],
"./app/sync/runner.py": [
{
"code": "reportAssignmentType",
"range": {
"startColumn": 19,
"endColumn": 88,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 16,
"endColumn": 24,
"lineCount": 1
}
},
{
"code": "reportOptionalIterable",
"range": {
"startColumn": 23,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 45,
"endColumn": 72,
"lineCount": 1
}
},
{
"code": "reportAssignmentType",
"range": {
"startColumn": 19,
"endColumn": 9,
"lineCount": 7
}
},
{
"code": "reportOptionalSubscript",
"range": {
"startColumn": 19,
"endColumn": 27,
"lineCount": 1
}
},
{
"code": "reportOptionalIterable",
"range": {
"startColumn": 23,
"endColumn": 31,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 47,
"endColumn": 55,
"lineCount": 1
}
}
],
"./app/sync/videos.py": [
{
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 18,
"endColumn": 26,
"lineCount": 1
}
}
],
"./app/worker.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 40,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 36,
"endColumn": 40,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 35,
"endColumn": 43,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 24,
"endColumn": 32,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 28,
"endColumn": 36,
"lineCount": 1
}
}
]
}
}
-19
View File
@@ -1,19 +0,0 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN adduser --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
EXPOSE 8000
CMD ["sh", "entrypoint.sh"]
+2
View File
@@ -1,6 +1,8 @@
[alembic]
script_location = alembic
prepend_sys_path = .
# Split the path list the way the OS does (alembic's legacy space/comma/colon split is deprecated).
path_separator = os
[loggers]
keys = root,sqlalchemy,alembic
+5 -5
View File
@@ -1,17 +1,17 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.config import settings
from app.db import Base
import app.models # noqa: F401 (import so models register on Base.metadata)
from app.logs import configure_logging
import app.tables # noqa: F401 (import so every table registers on Base.metadata)
config = context.config
config.set_main_option("sqlalchemy.url", settings.database_url)
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# Migrations run at container start (entrypoint.sh), so their output is part of the API's log
# stream — one format with everything else, not alembic.ini's own fileConfig shape.
configure_logging("alembic")
target_metadata = Base.metadata
@@ -6,7 +6,7 @@ Create Date: 2026-07-07
Some sources (a direct media URL like a reddit v.redd.it HLS manifest) carry no thumbnail, so the
download shows a blank image box. The worker now extracts a representative frame with ffmpeg and
saves it as a poster sidecar; `poster_path` (relative to download_root) records it so the card /
saves it as a poster sidecar; `poster_path` (relative to files_root) records it so the card /
watch page / link preview can fall back to it. Nullable — only set when there was no thumbnail.
"""
@@ -0,0 +1,41 @@
"""add download_jobs.error_detail — the domain-specific entry of the failure catalog (E11.5 S1)
Revision ID: 0084_dl_job_error_detail
Revises: 0083_comfort_pins
Create Date: 2026-09-13
`error_code` is the generic category (it decides whether a Retry is offered). `error_detail` names
the host-specific catalog entry `downloads.errors.classify` matched on top of it — `facebook.login_wall`
and its siblings — so the UI can say what went wrong ON THAT SITE and what to do about it, instead of
the category's generic sentence. Nullable: most failures have no host-specific entry.
Backfill: a job's code is written once, at failure time, so a new rule never reaches the rows that
failed before it. The one shape measured on prod (2026-09-13, jobs 9/66/75/145/156/157/182/256/257) is
yt-dlp's `[facebook] <id>: Cannot parse data`, stored as `unknown`. Without a cookie that is Facebook's
login wall. Whether a cookie was attached at the time is not recorded anywhere, so every old row gets
the no-cookie entry — on prod no account had a Facebook cookie, so that is what they all were.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0084_dl_job_error_detail"
down_revision: str | None = "0083_comfort_pins"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("error_detail", sa.String(length=48), nullable=True))
op.execute(
sa.text(
"UPDATE download_jobs SET error_code = 'login_required', "
"error_detail = 'facebook.login_wall' "
"WHERE (error_code IS NULL OR error_code = 'unknown') "
"AND error ILIKE '%[facebook]%cannot parse data%'"
)
)
def downgrade() -> None:
op.drop_column("download_jobs", "error_detail")
@@ -0,0 +1,26 @@
"""add users.last_login_at — when the account last signed in (E22 S1)
Revision ID: 0085_users_last_login_at
Revises: 0084_dl_job_error_detail
Create Date: 2026-09-16
Stamped by the one sign-in path (`routes/auth._establish_session`), which the Google callback and
password login share. No backfill is possible: the signal exists only at sign-in, so existing rows
read NULL until each account signs in again, and the admin UI says so rather than inventing a date.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0085_users_last_login_at"
down_revision: str | None = "0084_dl_job_error_detail"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True))
def downgrade() -> None:
op.drop_column("users", "last_login_at")
@@ -0,0 +1,48 @@
"""add log_events — stored log events for diagnostics (E22 S3)
Revision ID: 0086_log_events
Revises: 0085_users_last_login_at
Create Date: 2026-09-17
Written by app/logsink.py's background writer (WARNING and above, request lines), pruned by the log_gc
job after `log_retention_days`. Sized against prod on 2026-09-16: ~2k rows on a normal day, ~14k on a
YouTube-RSS-outage day before flood control — a few MB over the 7-day window. `user_id` carries no
foreign key on purpose (see models.LogEvent).
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0086_log_events"
down_revision: str | None = "0085_users_last_login_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"log_events",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("ts", sa.DateTime(timezone=True), nullable=False),
sa.Column("level", sa.String(10), nullable=False),
sa.Column("sev", sa.SmallInteger(), nullable=False),
sa.Column("source", sa.String(16), nullable=False),
sa.Column("logger", sa.String(128), nullable=False),
sa.Column("event", sa.String(64), nullable=True),
sa.Column("msg", sa.Text(), nullable=False),
sa.Column("attrs", postgresql.JSONB(), nullable=True),
sa.Column("exc", sa.Text(), nullable=True),
sa.Column("request_id", sa.String(64), nullable=True),
sa.Column("user_id", sa.Integer(), nullable=True),
sa.Column("debug_session_id", sa.String(64), nullable=True),
sa.Column("job_id", sa.Integer(), nullable=True),
sa.Column("version", sa.String(64), nullable=True),
)
op.create_index("ix_log_events_ts", "log_events", ["ts"])
op.create_index("ix_log_events_request_id", "log_events", ["request_id"])
op.create_index("ix_log_events_user_ts", "log_events", ["user_id", "ts"])
def downgrade() -> None:
op.drop_table("log_events")
@@ -0,0 +1,31 @@
"""add users.diagnostics_level + diagnostics_until — a user's consented diagnostics level (E22 S5)
Revision ID: 0087_users_diagnostics_level
Revises: 0086_log_events
Create Date: 2026-09-17
Both NULL = the base level (Alap). A raised level is ALWAYS paired with its expiry, computed by the
server at consent time (`app/diagnostics.py`), so no client can ask for a level that never ends.
Columns rather than `users.preferences`: that JSON is written wholesale by the client, and a consent
with a lifetime must not be something a client can write.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0087_users_diagnostics_level"
down_revision: str | None = "0086_log_events"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("users", sa.Column("diagnostics_level", sa.String(8), nullable=True))
op.add_column(
"users", sa.Column("diagnostics_until", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("users", "diagnostics_until")
op.drop_column("users", "diagnostics_level")
@@ -0,0 +1,30 @@
"""add channels.feed_fallback_at — when a channel's uploads were last read through the API because
its RSS feed would not answer
Revision ID: 0088_channels_feed_fallback_at
Revises: 0087_users_diagnostics_level
Create Date: 2026-09-19
YouTube's RSS feeds go dark for hours at a time, per channel (2026-09-18/19: 393 of 396 answered 404
for four hours; the same channel answers 404 to every retry). The RSS poll then reads the newest
uploads through the Data API instead, at a quota cost, and at most once per `rss_fallback_hours` per
channel: this column is that clock. NULL = never needed it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0088_channels_feed_fallback_at"
down_revision: str | None = "0087_users_diagnostics_level"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"channels", sa.Column("feed_fallback_at", sa.DateTime(timezone=True), nullable=True)
)
def downgrade() -> None:
op.drop_column("channels", "feed_fallback_at")
@@ -0,0 +1,81 @@
"""add debug_sessions + debug_screenshots, and index log_events.debug_session_id (E16)
Revision ID: 0089_debug_sessions
Revises: 0088_channels_feed_fallback_at
Create Date: 2026-09-19
A debug session is one recording: Start → reproduce → Stop (app/debugsession.py). Every log line
of that user's requests while it runs carries its id in `log_events.debug_session_id` — the column
E22 created and left NULL — and the issue it ends in is built from exactly those rows, so the column
gets its index now. Partial: nearly every row has no session, and an index of NULLs would be most
of the table for nothing.
At most one OPEN session per user, held by a partial unique index rather than by the start handler
alone: two tabs pressing Start at once would otherwise both pass a "none open" check.
Screenshots are rows, not files: bounded (size and count per session, app/debugsession.py), gone
with their session by cascade, and a self-host has no file volume to configure for them.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0089_debug_sessions"
down_revision: str | None = "0088_channels_feed_fallback_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"debug_sessions",
sa.Column("id", sa.String(16), primary_key=True),
sa.Column(
"user_id",
sa.Integer,
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("level", sa.String(8), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("until", sa.DateTime(timezone=True), nullable=False),
sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("ended_reason", sa.String(16), nullable=True),
sa.Column("issue_url", sa.String(512), nullable=True),
)
op.create_index(
"uq_debug_sessions_open_per_user",
"debug_sessions",
["user_id"],
unique=True,
postgresql_where=sa.text("ended_at IS NULL"),
)
op.create_table(
"debug_screenshots",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"session_id",
sa.String(16),
sa.ForeignKey("debug_sessions.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("mime", sa.String(16), nullable=False),
sa.Column("size", sa.Integer, nullable=False),
sa.Column("data", sa.LargeBinary, nullable=False),
)
op.create_index(
"ix_log_events_debug_session",
"log_events",
["debug_session_id"],
postgresql_where=sa.text("debug_session_id IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_log_events_debug_session", table_name="log_events")
op.drop_table("debug_screenshots")
op.drop_index("uq_debug_sessions_open_per_user", table_name="debug_sessions")
op.drop_table("debug_sessions")
@@ -0,0 +1,27 @@
"""add debug_sessions.draft — the issue being written from a session (E16 S3)
Revision ID: 0090_debug_session_draft
Revises: 0089_debug_sessions
Create Date: 2026-09-19
The composer saves as it goes: the title, the rendered markdown body that an export or an issue
carries, the log rows the user took out, and the form's own fields so reopening it later (Settings →
Diagnosztika lists past sessions) shows what was written. NULL until the composer first saves.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0090_debug_session_draft"
down_revision: str | None = "0089_debug_sessions"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("debug_sessions", sa.Column("draft", JSONB, nullable=True))
def downgrade() -> None:
op.drop_column("debug_sessions", "draft")
@@ -0,0 +1,27 @@
"""add tagline_hu to plex_items + plex_shows — the HU overlay's tagline (E30 S1)
Revision ID: 0091_plex_tagline_hu
Revises: 0090_debug_session_draft
Create Date: 2026-09-29
The Hungarian overlay section lists a tagline beside the title and summary; the info page showed the
English one to a Hungarian reader. NULL until the next overlay sync fills it (serving falls back to EN).
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0091_plex_tagline_hu"
down_revision: str | None = "0090_debug_session_draft"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("plex_items", sa.Column("tagline_hu", sa.Text(), nullable=True))
op.add_column("plex_shows", sa.Column("tagline_hu", sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column("plex_shows", "tagline_hu")
op.drop_column("plex_items", "tagline_hu")
@@ -0,0 +1,31 @@
"""add biography_hu to plex_people — TMDB's Hungarian biography (E30 S4)
Revision ID: 0092_plex_person_biography_hu
Revises: 0091_plex_tagline_hu
Create Date: 2026-09-29
The person page showed a Hungarian reader the English biography although TMDB has a Hungarian one for
most people. The person fetch now appends `translations` and keeps the Hungarian biography beside the
English one. Already-enriched people are un-stamped, so their next page open refetches them once (the
same lazy path as a first open) and fills it; until then the English biography stands in.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0092_plex_person_biography_hu"
down_revision: str | None = "0091_plex_tagline_hu"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("plex_people", sa.Column("biography_hu", sa.Text(), nullable=True))
op.execute(
"UPDATE plex_people SET tmdb_enriched_at = NULL "
"WHERE tmdb_person_id IS NOT NULL AND tmdb_enriched_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_column("plex_people", "biography_hu")
@@ -0,0 +1,29 @@
"""re-fetch linked people — "Known for" ranked by votes, without guest visits (E30 S6)
Revision ID: 0093_plex_person_known_for_votes
Revises: 0092_plex_person_biography_hu
Create Date: 2026-09-30
A person's "Known for" pool was their credits ranked by TMDB popularity, so talk-show visits as
themselves crowded out the films (Tom Cruise: 14 of 15 were talk shows, none in the library). The pool is
now ranked by vote count, without guest visits, with directed works, and deeper. Already-enriched people
are un-stamped so the people_enrich job fetches them again; their biography stays on the row meanwhile.
"""
from alembic import op
revision: str = "0093_plex_person_known_for_votes"
down_revision: str | None = "0092_plex_person_biography_hu"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute(
"UPDATE plex_people SET tmdb_enriched_at = NULL "
"WHERE tmdb_person_id IS NOT NULL AND tmdb_enriched_at IS NOT NULL"
)
def downgrade() -> None:
pass # the watermark only says "fetched"; a refetch under the old ranking is not worth undoing to
@@ -0,0 +1,45 @@
"""Hungarian genres on every Plex row, not only on overlaid ones (E30 review)
Revision ID: 0094_plex_genres_hu_everywhere
Revises: 0093_plex_person_known_for_votes
Create Date: 2026-09-30
The HU genre facets and filter read `genres_hu`, which only the overlay pass wrote — so a library with
no HU twin, or a guid-less row, offered English genres a Hungarian filter could never match. The sync
now writes `genres_hu` for every row (plex/sync.py `_apply_facet_fields`); this fills the rows it has
not reached yet, from their own English list through the same map.
"""
import json
import sqlalchemy as sa
from alembic import op
from app.plex.genres import hu_genres
revision: str = "0094_plex_genres_hu_everywhere"
down_revision: str | None = "0093_plex_person_known_for_votes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
conn = op.get_bind()
for table in ("plex_items", "plex_shows"):
rows = conn.execute(
sa.text(
f"SELECT id, genres FROM {table} "
"WHERE genres_hu IS NULL AND jsonb_typeof(genres) = 'array'"
)
).fetchall()
updates = [
{"id": rid, "g": json.dumps(hu)} for rid, genres in rows if (hu := hu_genres(genres))
]
if updates:
conn.execute(
sa.text(f"UPDATE {table} SET genres_hu = CAST(:g AS jsonb) WHERE id = :id"), updates
)
def downgrade() -> None:
pass # derived from `genres`; the sync writes the same values again
@@ -0,0 +1,83 @@
"""videos: the per-channel index covers what the channel manager and the feed sidebar read (E31 S4d)
The channel manager (`GET /api/channels`) sums each subscribed channel's videos: count, last upload, total
duration. The feed sidebar (`/feed/summary`, `/feed/years`) counts the reader's videos per channel and per
year under the default filters: available, not via explore, not a short, not live. Every one of them found
its rows through `ix_videos_channel_pub_desc` (channel_id, published_at DESC NULLS LAST, id) and then read
each video's heap page for the rest: ~270k rows at ~2.5 KB each, 681 MB on prod. That stays in cache only
while nothing else needs the memory. Measured on prod (2026-10-02): `/api/channels` 51–61 ms warm, but
2.25 s and 1.96 s on a first open in the browser and 6.4 s under EXPLAIN with 30k blocks read from disk;
`/feed/summary` and `/feed/years` ~140 ms warm, ~800 ms cold.
The same key with those columns INCLUDEd makes them Index Only Scans. On a prod-copy localdev, after a
VACUUM, the manager's sums read 0 heap pages and the summary reads the 25 MB index instead of the heap.
The visibility map is what lets them skip the heap; prod's autovacuum kept it complete (relallvisible ==
relpages). The key columns are unchanged, so every reader of the old index (the per-channel feed, the FK
cascade, the sync upsert, channel detail) rides this one. The old index is dropped.
tests/test_channel_list_index_only.py holds the queries and this column list together.
CONCURRENTLY (autocommit_block) builds it without blocking a live prod. IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent. A re-run after a build that failed part-way rebuilds
the INVALID leftover, and the old index is dropped only once the new one is valid: a re-run that kept the
leftover and dropped the old one left `videos` with no usable channel_id index (E31 S5 review).
"""
from alembic import op
from sqlalchemy import text
revision: str = "0095_videos_channel_cover_index"
down_revision: str | None = "0094_plex_genres_hu_everywhere"
branch_labels: str | None = None
depends_on: str | None = None
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def _swap(build: str, ddl: str, drop: str) -> None:
_build(build, ddl)
if not _valid(build):
raise RuntimeError(f"{build} did not build valid; {drop} is kept")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {drop}")
def upgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_cover",
"ON videos (channel_id, published_at DESC NULLS LAST, id) "
"INCLUDE (duration_seconds, unavailable_since, via_explore, is_short, live_status)",
"ix_videos_channel_pub_desc",
)
def downgrade() -> None:
with op.get_context().autocommit_block():
_swap(
"ix_videos_channel_pub_desc",
"ON videos (channel_id, published_at DESC NULLS LAST, id ASC)",
"ix_videos_channel_pub_cover",
)
@@ -0,0 +1,86 @@
"""videos: an index per feed sort, so a sorted page stops after its rows (E31 S4e)
The feed's views, duration and title sorts had no index. Every page (the first and each next cursor)
read the whole videos heap, filtered it and kept a top-N: on prod ~350 ms a page for views/duration,
550–780 ms for title (the ICU comparison on every row), 2.3 s for page 2 when the copies of one request
queued. With an index in the exact ORDER BY (`feedquery.order_by`: the key, NULLS LAST, then id) the
planner walks it and stops after the page. On a prod-copy localdev every plain sort fell from 180–310 ms
to ~1 ms, deep pages (20) to 3–5 ms, and narrow filters (a tag, a channel, a year, nothing found) stayed
at 0.3–25 ms.
One index per direction: a backward scan of `DESC NULLS LAST` gives `ASC NULLS FIRST`, not the
`ASC NULLS LAST` the asc sorts ask for. The title pair is per reader language (`app.collation.ICU`):
an index only serves an ORDER BY in its own collation.
Cost: ~130 MB (localdev: 10 + 10 + 8 + 8 + 4 × 23). Writes stay cheap: view_count and duration are
written by the enrich, which rewrites indexed columns already (the revalidation job only touches
last_checked_at), so no HOT update is lost. A search combined with one of these sorts can now walk the
index too: a term matching ~2 % of the videos took 130–200 ms instead of 7 (still under the plain sort it
replaces); rarer and broader terms stay fast. Fencing the index off under a search made broad terms
(`m`, `the`) take 2 s, so it is left to the planner.
The title indexes are the schema's first ones on an ICU collation: an ICU upgrade in the Postgres image
reorders them, and `app.collation.heal_drift` rebuilds them at startup when it sees one.
tests/test_feed_sort_indexes.py holds each sort and its index together.
CONCURRENTLY (autocommit_block) builds them without blocking a live prod; IF (NOT) EXISTS keeps the drift
test's downgrade → re-upgrade round trip idempotent.
"""
from alembic import op
from sqlalchemy import text
revision: str = "0096_videos_sort_indexes"
down_revision: str | None = "0095_videos_channel_cover_index"
branch_labels: str | None = None
depends_on: str | None = None
# name → the indexed ORDER BY (feedquery.sort_keys + order_by: key NULLS LAST, then id).
INDEXES = {
"ix_videos_views_desc": "view_count DESC NULLS LAST, id",
"ix_videos_views_asc": "view_count ASC NULLS LAST, id",
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
"ix_videos_title_en": '(lower(title) COLLATE "en-x-icu") ASC NULLS LAST, id',
"ix_videos_title_en_desc": '(lower(title) COLLATE "en-x-icu") DESC NULLS LAST, id',
"ix_videos_title_hu": '(lower(title) COLLATE "hu-x-icu") ASC NULLS LAST, id',
"ix_videos_title_hu_desc": '(lower(title) COLLATE "hu-x-icu") DESC NULLS LAST, id',
}
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, ddl: str) -> None:
"""CREATE INDEX CONCURRENTLY `name` (`ddl` is what follows the name), re-runnable. A build that
failed part-way leaves an INVALID index under the name, and `IF NOT EXISTS` would keep it: the
planner never uses it, and nothing says so (E31 S5 review). Such a leftover is dropped and built
again."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} {ddl}")
def upgrade() -> None:
with op.get_context().autocommit_block():
for name, columns in INDEXES.items():
_build(name, f"ON videos ({columns})")
def downgrade() -> None:
with op.get_context().autocommit_block():
for name in INDEXES:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
@@ -0,0 +1,69 @@
"""messages: drop direct messaging (the owner's decision, 2026-10-04)
Chat — end-to-end encrypted direct messages plus the server's welcome note — was never used: prod held
two welcome notes, no message between people, and one key. Its tables go with its code. The rows are
not migrated anywhere; the nightly backup holds them.
The downgrade restores the schema as it stood (0026 + 0027), empty: a message can't be brought back
from a migration, and the E2EE key blobs were only ever readable by their owner's browser anyway.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0097_drop_messages"
down_revision: str | None = "0096_videos_sort_indexes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_table("message_keys")
op.drop_index("ix_messages_created_at", table_name="messages")
op.drop_index("ix_messages_recipient_id", table_name="messages")
op.drop_index("ix_messages_sender_id", table_name="messages")
op.drop_table("messages")
def downgrade() -> None:
op.create_table(
"messages",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"sender_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column(
"recipient_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("body", sa.Text(), nullable=True),
sa.Column("read_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("kind", sa.String(length=16), nullable=False, server_default="user"),
sa.Column("ciphertext", sa.Text(), nullable=True),
sa.Column("iv", sa.String(length=32), nullable=True),
)
op.create_index("ix_messages_sender_id", "messages", ["sender_id"])
op.create_index("ix_messages_recipient_id", "messages", ["recipient_id"])
op.create_index("ix_messages_created_at", "messages", ["created_at"])
op.create_table(
"message_keys",
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column("public_key", sa.Text(), nullable=False),
sa.Column("wrapped_private_key", sa.Text(), nullable=False),
sa.Column("salt", sa.String(length=64), nullable=False),
sa.Column("wrap_iv", sa.String(length=32), nullable=False),
sa.Column("key_check", sa.Text(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
@@ -0,0 +1,55 @@
"""demo: drop the shared demo account (the owner's decision, 2026-10-04)
The demo account — one shared user that whitelisted emails entered from the login page, reset on a
schedule — goes with its code. Prod held the demo user and 10 whitelist rows.
The upgrade deletes the demo user first (its playlists, states and subscriptions cascade; audit and
quota rows keep their history with the actor nulled, as for any deleted account), then the traces
that only the demo code could render: its audit rows (whitelist edits, manual resets, and every row
about the reset job — its runs and its interval changes), the reset job's interval override and its
completion notices. Nothing is migrated anywhere; the nightly backup holds the rows.
The downgrade restores the schema as it stood (0014), empty: the demo user is created lazily on its
first sign-in, so an empty `demo_whitelist` is the whole of its state.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0098_drop_demo_account"
down_revision: str | None = "0097_drop_messages"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute("DELETE FROM users WHERE is_demo")
op.execute(
"DELETE FROM audit_log WHERE action IN ('demo_add', 'demo_remove', 'demo_reset')"
" OR (target_type = 'job' AND target_id = 'demo_reset')"
)
op.execute("DELETE FROM scheduler_settings WHERE job_id = 'demo_reset'")
op.execute("DELETE FROM notifications WHERE data->>'job_id' = 'demo_reset'")
op.drop_index("ix_demo_whitelist_email", table_name="demo_whitelist")
op.drop_table("demo_whitelist")
op.drop_index("ix_users_is_demo", table_name="users")
op.drop_column("users", "is_demo")
def downgrade() -> None:
op.add_column(
"users",
sa.Column("is_demo", sa.Boolean(), nullable=False, server_default=sa.false()),
)
op.create_index("ix_users_is_demo", "users", ["is_demo"])
op.create_table(
"demo_whitelist",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("email", sa.String(length=320), nullable=False),
sa.Column("note", sa.Text(), nullable=True),
sa.Column("added_by", sa.String(length=320), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index("ix_demo_whitelist_email", "demo_whitelist", ["email"], unique=True)
@@ -0,0 +1,42 @@
"""library: a resume point per user and library item
The library player opened every item at 0:00. `library_states` keeps where a user stopped, per job
(their own download, or one shared with them) — the same shape as `video_states`' position, without a
status: the library has no "watched". It starts empty; the downgrade drops it.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0099_library_states"
down_revision: str | None = "0098_drop_demo_account"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"library_states",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"job_id",
sa.Integer(),
sa.ForeignKey("download_jobs.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("position_seconds", sa.Integer(), nullable=False),
sa.Column("progress_updated_at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),
)
op.create_index("ix_library_states_job_id", "library_states", ["job_id"])
def downgrade() -> None:
op.drop_index("ix_library_states_job_id", table_name="library_states")
op.drop_table("library_states")
@@ -0,0 +1,22 @@
"""plex_items: drop `markers`, a column nothing writes or reads
Intro/credits markers are read live from Plex for the item the player opens (with Plex's `final` flag
on the block that ends it); this cached copy was never filled — every row is NULL. The downgrade
restores the empty column.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0100_drop_plex_item_markers"
down_revision: str | None = "0099_library_states"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.drop_column("plex_items", "markers")
def downgrade() -> None:
op.add_column("plex_items", sa.Column("markers", sa.JSON(), nullable=True))
@@ -0,0 +1,46 @@
"""Plex summaries and taglines as plain text
Revision ID: 0101_plex_text_plain
Revises: 0100_drop_plex_item_markers
Create Date: 2026-10-08
Some Plex agents write a summary as HTML (`<p>…</p>`, `&aacute;`), and the app showed the tags as
text. The sync now stores every summary and tagline through `plain_text` (app/plex/plaintext.py);
this cleans the rows it wrote before, through the same function, so nothing waits for a full resync.
"""
import sqlalchemy as sa
from alembic import op
from app.plex.plaintext import plain_text
revision: str = "0101_plex_text_plain"
down_revision: str | None = "0100_drop_plex_item_markers"
branch_labels: str | None = None
depends_on: str | None = None
_COLUMNS = {
"plex_items": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_shows": ("summary", "summary_hu", "tagline", "tagline_hu"),
"plex_collections": ("summary",),
}
def upgrade() -> None:
conn = op.get_bind()
for table, columns in _COLUMNS.items():
for col in columns:
rows = conn.execute(
sa.text(f"SELECT id, {col} FROM {table} WHERE {col} ~ '[<&]'")
).fetchall()
updates = [
{"id": rid, "v": clean}
for rid, value in rows
if (clean := plain_text(value)) != value
]
if updates:
conn.execute(sa.text(f"UPDATE {table} SET {col} = :v WHERE id = :id"), updates)
def downgrade() -> None:
pass # the markup is not wanted back; the next sync would clean it again
@@ -0,0 +1,193 @@
"""The Files library: folders, and a file's retention on the user's job
Revision ID: 0102_file_folders
Revises: 0101_plex_text_plain
Create Date: 2026-10-11
Every download, clip and upload of a user now sits in a folder of their Files library
(`app.downloads.folders`): a source folder made per origin (YouTube, Plex, uploads, one per other
site) or one of their own. How long a file stays moves from the shared asset to each user's job
(`download_jobs.expires_at`, stamped from the folder's `retention_days`), so one user keeping a file
no longer depends on another's copy; the asset's own `expires_at` / `gc_notified` go.
This places every existing job in its source folder and carries each done job's expiry over from
the asset it holds (an upload keeps none). The source-key rule is copied here, frozen, from
`folders.source_key_for` as it is today — a migration must not follow the code's later changes.
"""
from urllib.parse import urlsplit
import sqlalchemy as sa
from alembic import op
revision: str = "0102_file_folders"
down_revision: str | None = "0101_plex_text_plain"
branch_labels: str | None = None
depends_on: str | None = None
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
_DEFAULT_RETENTION_DAYS = 30 # config.download_retention_days on the day this was written
def _site_key(url: str) -> str:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return "youtube"
return f"site:{host or 'unknown'}"
def _asset_key(kind: str | None, ref: str | None) -> str:
if kind == "youtube":
return "youtube"
if kind == "url" and ref:
return _site_key(ref)
return "upload"
def _key(job, jobs_by_id, assets) -> str:
kind = job.source_kind
if kind == "youtube":
return "youtube"
if kind == "upload":
return "upload"
if kind == "clip":
spec = job.edit_spec or {}
return "plex" if spec.get("source") == "plex" else "youtube"
if kind == "url":
return _site_key(job.source_ref)
if kind == "edit":
parent = jobs_by_id.get(job.source_job_id)
if parent is not None and parent.id != job.id:
return _key(parent, jobs_by_id, assets)
asset = assets.get(job.source_asset_id)
if asset is not None:
return _asset_key(asset.source_kind, asset.source_ref)
return "upload"
def upgrade() -> None:
op.create_table(
"file_folders",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(16), nullable=False),
sa.Column("source_key", sa.String(260)), # "site:" + a host (<= 253)
sa.Column("name", sa.String(255)),
sa.Column("retention_days", sa.Integer()),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
sa.CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
op.create_index("ix_file_folders_user_id", "file_folders", ["user_id"])
op.create_index(
"uq_file_folder_own_name",
"file_folders",
["user_id", sa.text("lower(name)")],
unique=True,
postgresql_where=sa.text("name IS NOT NULL"),
)
op.add_column(
"download_jobs",
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="RESTRICT"),
),
)
op.add_column(
"download_jobs",
sa.Column("kept", sa.Boolean(), server_default="false", nullable=False),
)
op.add_column("download_jobs", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"download_jobs",
sa.Column("expiry_warned", sa.Boolean(), server_default="false", nullable=False),
)
op.create_index("ix_download_jobs_folder_id", "download_jobs", ["folder_id"])
op.create_index("ix_download_jobs_expires_at", "download_jobs", ["expires_at"])
conn = op.get_bind()
raw = conn.execute(
sa.text("SELECT value FROM system_config WHERE key = 'download_retention_days'")
).scalar()
retention = int(raw) if raw not in (None, "") else _DEFAULT_RETENTION_DAYS
jobs = conn.execute(
sa.text(
"SELECT j.id, j.user_id, j.source_kind, j.source_ref, j.edit_spec, j.source_job_id,"
" j.source_asset_id, j.status, j.asset_id, a.expires_at AS asset_expires_at"
" FROM download_jobs j LEFT JOIN media_assets a ON a.id = j.asset_id"
)
).fetchall()
jobs_by_id = {j.id: j for j in jobs}
assets = {
a.id: a
for a in conn.execute(sa.text("SELECT id, source_kind, source_ref FROM media_assets"))
}
folder_ids: dict[tuple[int, str], int] = {}
for job in jobs:
key = _key(job, jobs_by_id, assets)
slot = (job.user_id, key)
if slot not in folder_ids:
folder_ids[slot] = conn.execute(
sa.text(
"INSERT INTO file_folders (user_id, kind, source_key, retention_days)"
" VALUES (:u, 'source', :k, :r) RETURNING id"
),
{"u": job.user_id, "k": key, "r": None if key == "upload" else retention},
).scalar_one()
# A done file keeps the expiry it had (its asset's); an upload had none and keeps none.
expires = job.asset_expires_at if job.status == "done" and key != "upload" else None
conn.execute(
sa.text("UPDATE download_jobs SET folder_id = :f, expires_at = :e WHERE id = :id"),
{"f": folder_ids[slot], "e": expires, "id": job.id},
)
op.drop_column("media_assets", "expires_at")
op.drop_column("media_assets", "gc_notified")
def downgrade() -> None:
op.add_column("media_assets", sa.Column("expires_at", sa.DateTime(timezone=True)))
op.add_column(
"media_assets",
sa.Column("gc_notified", sa.Boolean(), server_default="false", nullable=False),
)
# The asset goes when its last holder would: the latest of its holders' expiries (a holder that
# never expires keeps the asset unexpired).
op.execute(
"UPDATE media_assets a SET expires_at = sub.e FROM ("
" SELECT asset_id, CASE WHEN bool_or(expires_at IS NULL) THEN NULL ELSE max(expires_at) END"
" AS e FROM download_jobs WHERE asset_id IS NOT NULL AND status = 'done' GROUP BY asset_id"
") sub WHERE a.id = sub.asset_id AND a.source_kind <> 'upload'"
)
op.drop_index("ix_download_jobs_expires_at", table_name="download_jobs")
op.drop_index("ix_download_jobs_folder_id", table_name="download_jobs")
op.drop_column("download_jobs", "expiry_warned")
op.drop_column("download_jobs", "expires_at")
op.drop_column("download_jobs", "kept")
op.drop_column("download_jobs", "folder_id")
op.drop_index("uq_file_folder_own_name", table_name="file_folders")
op.drop_index("ix_file_folders_user_id", table_name="file_folders")
op.drop_table("file_folders")
@@ -0,0 +1,32 @@
"""The Files library: a file's SHA-256, and the expired rows retired
Revision ID: 0103_files_sha256
Revises: 0102_file_folders
Create Date: 2026-10-11
`media_assets.sha256` holds the file's hash once it is known (an upload is hashed while it streams
in, any other file on the first ask), so a machine fetching a file can check what it got.
An expired file now leaves its user's library (`downloads.gc`). Before, an expired download stayed
as a fileless "expired · download again" row (a done job with no asset). Those rows hold nothing on
disk and the library no longer shows them, so they are deleted here rather than left behind
unseen; their shares and public links go with them (ON DELETE CASCADE), and those already served
nothing. The downgrade cannot bring them back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0103_files_sha256"
down_revision: str | None = "0102_file_folders"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("media_assets", sa.Column("sha256", sa.String(64)))
op.execute("DELETE FROM download_jobs WHERE status = 'done' AND asset_id IS NULL")
def downgrade() -> None:
op.drop_column("media_assets", "sha256")
@@ -0,0 +1,38 @@
"""A file's "Added" is when it became done
Revision ID: 0104_job_ready_at
Revises: 0103_files_sha256
Create Date: 2026-10-12
`download_jobs.ready_at` is when a job became done, and the Files library sorts and dates by it: a
download that failed for days and then went through used to sit days down the list, under its
request date. Every path to done stamps it, held by `ck_download_jobs_ready_at`.
A done job from before has no record of that moment (its "ready to watch" notice never named the
job, so it cannot be matched back), so it takes its request date, which is where it already sat.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0104_job_ready_at"
down_revision: str | None = "0103_files_sha256"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("ready_at", sa.DateTime(timezone=True)))
op.execute(
"""
UPDATE download_jobs SET ready_at = created_at WHERE status = 'done'
"""
)
op.create_check_constraint(
"ck_download_jobs_ready_at", "download_jobs", "status <> 'done' OR ready_at IS NOT NULL"
)
def downgrade() -> None:
op.drop_constraint("ck_download_jobs_ready_at", "download_jobs", type_="check")
op.drop_column("download_jobs", "ready_at")
@@ -0,0 +1,37 @@
"""A file remembers where it came from
Revision ID: 0105_job_source_key
Revises: 0104_job_ready_at
Create Date: 2026-10-12
`download_jobs.source_key` is the origin a file came from — the source folder it may go back to —
recorded when it is placed. An edit used to work its origin out through its parent each time, and
once the parent was deleted or expired it fell back to "upload": moved into an own folder, it could
no longer go back to its folder, and deleting that folder sent it to Uploads to stay forever.
A file in a source folder takes that folder's key. One already in an own folder is left NULL and
keeps being worked out (`folders.source_key_for`), as before.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0105_job_source_key"
down_revision: str | None = "0104_job_ready_at"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_jobs", sa.Column("source_key", sa.String(260)))
op.execute(
"""
UPDATE download_jobs j SET source_key = f.source_key
FROM file_folders f
WHERE f.id = j.folder_id AND f.kind = 'source'
"""
)
def downgrade() -> None:
op.drop_column("download_jobs", "source_key")
+110
View File
@@ -0,0 +1,110 @@
"""Machine access to the Files library: API tokens bound to folders, and the Hub folder
Revision ID: 0106_api_tokens
Revises: 0105_job_source_key
Create Date: 2026-10-14
`api_tokens` is a user's token for a machine (a script, the hub): only its SHA-256 is kept, plus the
first characters to recognise it by, and whether its user was told it is about to run out. `api_token_grants` says what it may do in which folder — list,
read, write, delete — so a token reaches nothing it was not given, folder by folder.
A third folder kind, `hub`: the one folder a user exchanges files with their hub machine through.
Like a source folder its name is the client's to render; like an own folder a file may be moved in.
One per user.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0106_api_tokens"
down_revision: str | None = "0105_job_source_key"
branch_labels: str | None = None
depends_on: str | None = None
_KIND_OLD = (
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
)
_KIND_NEW = _KIND_OLD + " OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)"
def upgrade() -> None:
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_NEW)
op.create_index(
"uq_file_folder_hub",
"file_folders",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'hub'"),
)
op.create_table(
"api_tokens",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("name", sa.String(80), nullable=False),
sa.Column("prefix", sa.String(16), nullable=False),
sa.Column("token_hash", sa.String(64), nullable=False, unique=True),
sa.Column("expires_at", sa.DateTime(timezone=True)),
sa.Column("last_used_at", sa.DateTime(timezone=True)),
sa.Column("expiry_warned", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_table(
"api_token_grants",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"token_id",
sa.Integer(),
sa.ForeignKey("api_tokens.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"folder_id",
sa.Integer(),
sa.ForeignKey("file_folders.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("can_list", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_read", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_write", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.Column("can_delete", sa.Boolean(), nullable=False, server_default=sa.text("false")),
sa.UniqueConstraint("token_id", "folder_id", name="uq_api_token_grant"),
sa.CheckConstraint(
"can_list OR can_read OR can_write OR can_delete", name="ck_api_token_grant_some"
),
)
def downgrade() -> None:
op.drop_table("api_token_grants")
op.drop_table("api_tokens")
op.drop_index("uq_file_folder_hub", table_name="file_folders")
# A Hub folder may hold files (their foreign key RESTRICTs a delete): it becomes an own folder,
# named "Hub <id>" — or "Hub <id> (2)", … when its user already has a folder of that name,
# which `uq_file_folder_own_name` (per user, case-blind) would refuse.
conn = op.get_bind()
hubs = conn.execute(sa.text("SELECT id, user_id FROM file_folders WHERE kind = 'hub'")).all()
for folder_id, user_id in hubs:
name, n = f"Hub {folder_id}", 1
while conn.execute(
sa.text("SELECT 1 FROM file_folders WHERE user_id = :u AND lower(name) = lower(:n)"),
{"u": user_id, "n": name},
).first():
n += 1
name = f"Hub {folder_id} ({n})"
conn.execute(
sa.text("UPDATE file_folders SET kind = 'own', name = :n WHERE id = :i"),
{"n": name, "i": folder_id},
)
op.drop_constraint("ck_file_folder_kind", "file_folders", type_="check")
op.create_check_constraint("ck_file_folder_kind", "file_folders", _KIND_OLD)
@@ -0,0 +1,66 @@
"""Share-link tokens no longer stored in the clear
Revision ID: 0107_link_token_at_rest
Revises: 0106_api_tokens
Create Date: 2026-10-15
A `/watch` link's token is the credential itself, and it sat in `download_links.token` as written,
so a copy of the database was every shared file. Now a link is found by the token's SHA-256
(`token_hash`), and the token is kept only Fernet-encrypted (`token_enc`, TOKEN_ENCRYPTION_KEY) so its
owner can copy the same link again. Without a key the copy is not kept: the link still opens, its
address just cannot be shown again (`app/downloads/links.py`).
Every existing link is converted here, so the addresses already sent keep working. The downgrade
writes the tokens back from their encrypted copies; a link with none left (no key, or a key since
changed) gets a fresh random token, which is the same as revoking it.
"""
import hashlib
import secrets
import sqlalchemy as sa
from alembic import op
from app import security
revision: str = "0107_link_token_at_rest"
down_revision: str | None = "0106_api_tokens"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("download_links", sa.Column("token_hash", sa.String(64)))
op.add_column("download_links", sa.Column("token_enc", sa.Text()))
conn = op.get_bind()
sealing = security.can_encrypt()
for link_id, token in conn.execute(sa.text("SELECT id, token FROM download_links")).fetchall():
conn.execute(
sa.text("UPDATE download_links SET token_hash = :h, token_enc = :e WHERE id = :id"),
{
"id": link_id,
"h": hashlib.sha256(token.encode()).hexdigest(),
"e": security.encrypt(token) if sealing else None,
},
)
op.alter_column("download_links", "token_hash", nullable=False)
op.create_unique_constraint("uq_download_link_token_hash", "download_links", ["token_hash"])
op.drop_constraint("uq_download_link_token", "download_links", type_="unique")
op.drop_column("download_links", "token")
def downgrade() -> None:
op.add_column("download_links", sa.Column("token", sa.String(64)))
conn = op.get_bind()
rows = conn.execute(sa.text("SELECT id, token_enc FROM download_links")).fetchall()
for link_id, enc in rows:
token = security.decrypt_optional(enc) or secrets.token_urlsafe(24)
conn.execute(
sa.text("UPDATE download_links SET token = :t WHERE id = :id"),
{"id": link_id, "t": token},
)
op.alter_column("download_links", "token", nullable=False)
op.create_unique_constraint("uq_download_link_token", "download_links", ["token"])
op.drop_constraint("uq_download_link_token_hash", "download_links", type_="unique")
op.drop_column("download_links", "token_enc")
op.drop_column("download_links", "token_hash")
@@ -0,0 +1,40 @@
"""A share link may open a folder
Revision ID: 0108_folder_links
Revises: 0107_link_token_at_rest
Create Date: 2026-10-15
A `/watch` link opened one file. Now it may open one of its owner's own folders instead: the link's
page lists what is in the folder when it is opened. `folder_id` sits beside `job_id`, exactly one of
the two set; a folder's links go with the folder.
The downgrade has nowhere to keep a folder link, so it deletes them: they stop opening.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0108_folder_links"
down_revision: str | None = "0107_link_token_at_rest"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"download_links",
sa.Column("folder_id", sa.Integer(), sa.ForeignKey("file_folders.id", ondelete="CASCADE")),
)
op.create_index("ix_download_links_folder_id", "download_links", ["folder_id"])
op.alter_column("download_links", "job_id", nullable=True)
op.create_check_constraint(
"ck_download_link_target", "download_links", "(job_id IS NULL) <> (folder_id IS NULL)"
)
def downgrade() -> None:
op.execute("DELETE FROM download_links WHERE folder_id IS NOT NULL")
op.drop_constraint("ck_download_link_target", "download_links", type_="check")
op.alter_column("download_links", "job_id", nullable=False)
op.drop_index("ix_download_links_folder_id", table_name="download_links")
op.drop_column("download_links", "folder_id")
+160
View File
@@ -0,0 +1,160 @@
"""Background runs get a durable history
Revision ID: 0109_job_runs
Revises: 0108_folder_links
Create Date: 2026-10-17
What a scheduler job last did lived in the API process's memory, gone at every restart; the audit
log kept a row for a run only when it was manual, failed, or changed something. Now every run is a
`job_runs` row, written when it starts and when it ends. A run's quota spend points at it
(`quota_events.job_run_id`), and so do the lines it logged (`log_events.run_id`, no foreign key: the
log writer runs behind the event).
Runs are not admin actions, so the audit log stops carrying them: the existing `job_run` audit rows
MOVE into `job_runs` (start = end = when the row was written, the only time it kept), and leave the
audit log. The downgrade moves every run back into the audit log as such a row (one still running as
interrupted, what the audit row has no column for in its `after`), then drops the table.
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0109_job_runs"
down_revision: str | None = "0108_folder_links"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"job_runs",
sa.Column("id", sa.BigInteger(), primary_key=True),
sa.Column("job", sa.String(40), nullable=False),
sa.Column("trigger", sa.String(16), nullable=False),
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="SET NULL")),
sa.Column("target", sa.String(128)),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True)),
sa.Column("summary", sa.Text()),
sa.Column("result", postgresql.JSONB()),
sa.Column("error", sa.Text()),
sa.Column("failure", postgresql.JSONB()),
sa.Column("quota_units", sa.Integer(), nullable=False, server_default="0"),
sa.CheckConstraint(
"status IN ('running', 'ok', 'error', 'skipped', 'interrupted')",
name="ck_job_runs_status",
),
sa.CheckConstraint(
"trigger IN ('scheduled', 'manual', 'ad_hoc')", name="ck_job_runs_trigger"
),
sa.CheckConstraint(
"(status = 'running') = (finished_at IS NULL)", name="ck_job_runs_finished"
),
)
op.create_index("ix_job_runs_job_started", "job_runs", ["job", "started_at"])
op.create_index("ix_job_runs_actor_id", "job_runs", ["actor_id"])
op.create_index("ix_job_runs_started_at", "job_runs", ["started_at"])
op.add_column(
"quota_events",
sa.Column("job_run_id", sa.BigInteger(), sa.ForeignKey("job_runs.id", ondelete="SET NULL")),
)
op.create_index("ix_quota_events_job_run_id", "quota_events", ["job_run_id"])
op.add_column("log_events", sa.Column("run_id", sa.BigInteger()))
op.create_index(
"ix_log_events_run",
"log_events",
["run_id"],
postgresql_where=sa.text("run_id IS NOT NULL"),
)
# The audit trail's runs move over: the three statuses the scheduler wrote, and `interrupted`,
# which a downgrade writes back (as it does a run's trigger, target, spend and start, which the
# audit row has no column for: its actor alone cannot say "manual" once that admin is deleted),
# so a rollback and a redeploy keep every run as it was. A status the run table does
# not know would fail the CHECK.
op.execute(
"""
INSERT INTO job_runs (job, trigger, actor_id, target, status, started_at, finished_at,
summary, result, error, failure, quota_units)
SELECT target_id,
coalesce(after::jsonb ->> 'trigger',
CASE WHEN actor_id IS NULL THEN 'scheduled' ELSE 'manual' END),
actor_id,
after::jsonb ->> 'target',
after::jsonb ->> 'status',
coalesce((after::jsonb ->> 'started_at')::timestamptz, created_at),
created_at,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' <> 'error' THEN after::jsonb -> 'result' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb ->> 'summary' END,
CASE WHEN after::jsonb ->> 'status' = 'error' THEN after::jsonb -> 'result' END,
coalesce((after::jsonb ->> 'quota_units')::int, 0)
FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
ORDER BY id
"""
)
op.execute(
"""
DELETE FROM audit_log
WHERE action = 'job_run'
AND target_id IS NOT NULL
AND after::jsonb ->> 'status' IN ('ok', 'error', 'skipped', 'interrupted')
"""
)
def downgrade() -> None:
# A run still `running` is one the rollback stopped: it goes back as interrupted. What the audit
# row's own columns cannot say rides in `after` (its trigger, target, spend and start), so the
# upgrade restores them.
op.execute(
"""
INSERT INTO audit_log (actor_id, action, target_type, target_id, summary, after, created_at)
SELECT actor_id,
'job_run',
'job',
job,
left(job || ': ' || st
|| coalesce(' — ' || CASE WHEN st = 'error' THEN error ELSE summary END, ''),
255),
(jsonb_build_object(
'status', st,
'summary', CASE WHEN st = 'error' THEN error ELSE summary END)
|| CASE
WHEN st = 'error' AND failure IS NOT NULL
THEN jsonb_build_object('result', failure)
WHEN st <> 'error' AND result IS NOT NULL
THEN jsonb_build_object('result', result)
ELSE '{}'::jsonb
END
|| jsonb_build_object('trigger', trigger, 'quota_units', units,
'started_at', started_at)
|| CASE WHEN target IS NOT NULL
THEN jsonb_build_object('target', target) ELSE '{}'::jsonb END)::json,
coalesce(finished_at, started_at)
FROM (SELECT *, CASE WHEN status = 'running' THEN 'interrupted' ELSE status END AS st,
-- A run stopped mid-way has not summed its spend yet: sum it as the startup
-- sweep would, before the link to it is dropped.
CASE WHEN status = 'running'
THEN (SELECT coalesce(sum(q.units), 0) FROM quota_events q
WHERE q.job_run_id = job_runs.id)
ELSE quota_units END AS units
FROM job_runs) AS r
ORDER BY id
"""
)
op.drop_index("ix_log_events_run", table_name="log_events")
op.drop_column("log_events", "run_id")
op.drop_index("ix_quota_events_job_run_id", table_name="quota_events")
op.drop_column("quota_events", "job_run_id")
op.drop_index("ix_job_runs_started_at", table_name="job_runs")
op.drop_index("ix_job_runs_actor_id", table_name="job_runs")
op.drop_index("ix_job_runs_job_started", table_name="job_runs")
op.drop_table("job_runs")
@@ -0,0 +1,79 @@
"""Job intervals move into sysconfig; the unread SponsorBlock default goes
Revision ID: 0110_intervals_in_sysconfig
Revises: 0109_job_runs
Create Date: 2026-10-18
An admin's interval override lived in its own table, `scheduler_settings`, beside the one store every
other admin knob uses. Each override becomes the `job_interval_<job>` sysconfig key and the table
goes. Only the jobs that still exist are copied: a row for any other id never did anything.
The `sponsorblock_default` setting was offered on the Configuration page but nothing ever read it, so
its key is gone and its stored value with it — the downgrade cannot bring that value back.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0110_intervals_in_sysconfig"
down_revision: str | None = "0109_job_runs"
branch_labels: str | None = None
depends_on: str | None = None
# The interval jobs as of this revision (a migration does not read the live code's list).
_JOBS = (
"rss_poll",
"enrich",
"backfill",
"autotag",
"shorts",
"subscriptions",
"playlist_sync",
"maintenance",
"explore_cleanup",
"download_gc",
"plex_sync",
"plex_watch_sync",
"plex_watch_reconcile",
"tmdb_enrich",
"people_enrich",
"audit_gc",
"log_gc",
)
_PREFIX = "job_interval_"
def upgrade() -> None:
op.execute(
sa.text(
"INSERT INTO system_config (key, value, encrypted)"
" SELECT :prefix || job_id, interval_minutes::text, false FROM scheduler_settings"
" WHERE interval_minutes IS NOT NULL AND job_id IN :jobs"
" ON CONFLICT (key) DO NOTHING"
).bindparams(sa.bindparam("jobs", expanding=True), prefix=_PREFIX, jobs=list(_JOBS))
)
op.drop_table("scheduler_settings")
op.execute("DELETE FROM system_config WHERE key = 'sponsorblock_default'")
def downgrade() -> None:
op.create_table(
"scheduler_settings",
sa.Column("job_id", sa.String(40), primary_key=True),
sa.Column("interval_minutes", sa.Integer(), nullable=False),
)
op.execute(
sa.text(
"INSERT INTO scheduler_settings (job_id, interval_minutes)"
" SELECT substr(key, :start), value::int FROM system_config WHERE key IN :keys"
).bindparams(
sa.bindparam("keys", expanding=True),
start=len(_PREFIX) + 1,
keys=[_PREFIX + j for j in _JOBS],
)
)
op.execute(
sa.text("DELETE FROM system_config WHERE key IN :keys").bindparams(
sa.bindparam("keys", expanding=True), keys=[_PREFIX + j for j in _JOBS]
)
)
@@ -0,0 +1,28 @@
"""An audit row keeps the request that made it
Revision ID: 0111_audit_request_id
Revises: 0110_intervals_in_sysconfig
Create Date: 2026-10-18
The activity page opens an admin action's log lines from its row: `audit_log.request_id` is the
request the change was made in, the same id `log_events.request_id` carries. Older rows have none —
the request that made them was never recorded.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0111_audit_request_id"
down_revision: str | None = "0110_intervals_in_sysconfig"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column("audit_log", sa.Column("request_id", sa.String(64)))
op.create_index("ix_audit_log_request_id", "audit_log", ["request_id"])
def downgrade() -> None:
op.drop_index("ix_audit_log_request_id", table_name="audit_log")
op.drop_column("audit_log", "request_id")
@@ -0,0 +1,70 @@
"""One Watch later list per user, held by the database
Revision ID: 0112_one_watch_later
Revises: 0111_audit_request_id
Create Date: 2026-10-20
Watch later was made on a user's first save by a read then an insert, so two first saves at once
each made a list, and the reader then picked between them at random. A partial unique index holds
"one per user" now. An instance that already has a second list keeps the oldest: the others' videos
move into it (after its own, in their order; one it already holds is not added twice), a Home pin
of one of them moves to it (unless it is pinned itself; of several, the first on the shelf), then
they go — a pin left on them goes with them, as the kept list already holds one.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0112_one_watch_later"
down_revision: str | None = "0111_audit_request_id"
branch_labels: str | None = None
depends_on: str | None = None
_EXTRA = """
SELECT p.id, k.keep_id
FROM playlists p
JOIN (
SELECT user_id, min(id) AS keep_id FROM playlists WHERE kind = 'watch_later' GROUP BY user_id
) k ON k.user_id = p.user_id
WHERE p.kind = 'watch_later' AND p.id <> k.keep_id
"""
def upgrade() -> None:
op.execute(
f"""
INSERT INTO playlist_items (playlist_id, video_id, position, added_at)
SELECT e.keep_id, i.video_id,
(SELECT coalesce(max(position), -1) FROM playlist_items WHERE playlist_id = e.keep_id)
+ row_number() OVER (PARTITION BY e.keep_id ORDER BY i.playlist_id, i.position, i.id),
i.added_at
FROM playlist_items i
JOIN ({_EXTRA}) e ON e.id = i.playlist_id
ON CONFLICT ON CONSTRAINT uq_playlist_video DO NOTHING
"""
)
op.execute(
f"""
UPDATE comfort_pins c SET playlist_id = m.keep_id
FROM (
SELECT DISTINCT ON (e.keep_id) p.id AS pin_id, e.keep_id
FROM comfort_pins p
JOIN ({_EXTRA}) e ON e.id = p.playlist_id
WHERE NOT EXISTS (SELECT 1 FROM comfort_pins k WHERE k.playlist_id = e.keep_id)
ORDER BY e.keep_id, p.position, p.id
) m
WHERE c.id = m.pin_id
"""
)
op.execute(f"DELETE FROM playlists WHERE id IN (SELECT id FROM ({_EXTRA}) e)")
op.create_index(
"uq_playlist_watch_later",
"playlists",
["user_id"],
unique=True,
postgresql_where=sa.text("kind = 'watch_later'"),
)
def downgrade() -> None:
op.drop_index("uq_playlist_watch_later", table_name="playlists")
@@ -0,0 +1,51 @@
"""A user's Google Takeout export: what they watched on YouTube, and their Watch later list
Revision ID: 0113_takeout_items
Revises: 0112_one_watch_later
Create Date: 2026-10-23
One row per video and kind. A video the catalog does not know yet waits with no outcome, so the id
is not a foreign key; the partial index is what a newly known video looks its waiting rows up by.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0113_takeout_items"
down_revision: str | None = "0112_one_watch_later"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"takeout_items",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("kind", sa.String(12), nullable=False),
sa.Column("video_id", sa.String(16), nullable=False),
sa.Column("channel_id", sa.String(32), nullable=True),
sa.Column("at", sa.DateTime(timezone=True), nullable=False),
sa.Column("outcome", sa.String(12), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "kind", "video_id", name="uq_takeout_item"),
sa.CheckConstraint("kind IN ('watched', 'watch_later')", name="ck_takeout_item_kind"),
)
op.create_index(
"ix_takeout_items_waiting",
"takeout_items",
["video_id"],
postgresql_where=sa.text("outcome IS NULL"),
)
def downgrade() -> None:
op.drop_index("ix_takeout_items_waiting", table_name="takeout_items")
op.drop_table("takeout_items")
@@ -0,0 +1,44 @@
"""When a user last took a video back to "new"
Revision ID: 0114_video_unmarks
Revises: 0113_takeout_items
Create Date: 2026-10-23
Un-marking or resetting a video leaves no `video_states` row, so this is what tells an import of the
user's YouTube history a video they un-marked here from one never touched.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0114_video_unmarks"
down_revision: str | None = "0113_takeout_items"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.create_table(
"video_unmarks",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"video_id",
sa.String(16),
sa.ForeignKey("videos.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("user_id", "video_id", name="uq_video_unmark"),
)
op.create_index("ix_video_unmarks_video_id", "video_unmarks", ["video_id"])
def downgrade() -> None:
op.drop_index("ix_video_unmarks_video_id", table_name="video_unmarks")
op.drop_table("video_unmarks")
@@ -0,0 +1,49 @@
"""A watched Takeout video's length, and the channels a user put out of their list
Revision ID: 0115_takeout_channels
Revises: 0114_video_unmarks
Create Date: 2026-10-24
The "channels you watch but do not follow" list counts a user's watched videos over five minutes
by channel. The length is asked of YouTube for that list and kept on the Takeout row; a channel the
user does not want on the list is kept apart.
"""
import sqlalchemy as sa
from alembic import op
revision: str = "0115_takeout_channels"
down_revision: str | None = "0114_video_unmarks"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.add_column(
"takeout_items", sa.Column("looked_up_at", sa.DateTime(timezone=True), nullable=True)
)
op.add_column("takeout_items", sa.Column("duration_seconds", sa.Integer(), nullable=True))
op.create_table(
"takeout_channel_dismissals",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"user_id",
sa.Integer(),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("channel_id", sa.String(32), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.UniqueConstraint("user_id", "channel_id", name="uq_takeout_channel_dismissal"),
)
def downgrade() -> None:
op.drop_table("takeout_channel_dismissals")
op.drop_column("takeout_items", "duration_seconds")
op.drop_column("takeout_items", "looked_up_at")
@@ -0,0 +1,34 @@
"""An explored channel someone follows is the catalog's own, and so are its videos
Revision ID: 0116_followed_explored_channels
Revises: 0115_takeout_channels
Create Date: 2026-10-24
A follow pulled in from YouTube (the subscription import) left an explored channel flagged as an
exploration's, and a subscribe landing while an exploration's videos came in left those videos
flagged on a channel that no longer was one. Both are held in the code now; this clears what they
left: the channel of a subscription is not an exploration's, and no video is an exploration's on a
channel that is not one. Data only, so the downgrade has nothing to undo.
"""
from alembic import op
revision: str = "0116_followed_explored_channels"
down_revision: str | None = "0115_takeout_channels"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute(
"UPDATE channels SET from_explore = false WHERE from_explore"
" AND EXISTS (SELECT 1 FROM subscriptions s WHERE s.channel_id = channels.id)"
)
op.execute(
"UPDATE videos SET via_explore = false WHERE via_explore"
" AND channel_id IN (SELECT id FROM channels WHERE NOT from_explore)"
)
def downgrade() -> None:
pass
+73 -25
View File
@@ -1,9 +1,8 @@
# Backend conventions
The rules below are already followed across `backend/app` — they lived only in a few module
docstrings (`demo.py`, `userscope.py`, `routes/_common.py`) until this file collected them (E3 S3,
B-core-20). When you add code, match these so the next reader doesn't have to guess "which pattern do
I copy".
docstrings (`userscope.py`, `routes/_common.py`) until this file collected them. When you add code,
match these so the next reader doesn't have to guess "which pattern do I copy".
## Module layering
@@ -13,35 +12,81 @@ Three layers, by *who* runs the code — not by feature:
response models, status-code mapping, serialization. It owns logic that runs *only* on a request.
Business logic that a request happens to trigger does **not** live here if anything else runs it too.
2. **`plex/`, `downloads/`, `sync/` — domain logic shared by more than one caller.** Anything the
**scheduler or the worker** runs (not just a request) lives in one of these domain packages, so the
route, the worker, and the scheduler share one definition. Examples from this codebase:
`downloads/service.py` owns enqueue + the job state transitions (routes call it); `plex/sync.py` is
run by both the admin route and the scheduler; `downloads/pipeline.py` is the worker's child pipeline.
2. **The units and the other domain packages — logic shared by more than one caller.** A unit's
database work (below), and anything the **scheduler or the worker** runs (not just a request), lives
in a domain package, so the route, the worker, and the scheduler share one definition. Examples from
this codebase: `downloads/service.py` owns enqueue + the job state transitions (routes call it);
`plex/sync.py` is run by both the admin route and the scheduler; `downloads/pipeline.py` is the
worker's child pipeline; `issues/bundle.py` is the one definition of a bug report's files, for the
zip export and the tracker.
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth`, `sysconfig`, `quota`, `audit`,
`notifications`, `userscope`, `realtime`, `ratelimit`, `progress`, … are single-file primitives many
layers import. They depend on nothing above them (a core module must never import a `routes/…`
module).
3. **Flat `app/*.py` core modules — cross-cutting primitives.** `auth` (with `apitokens`, a machine's
folder-scoped way into the Files library), `sysconfig`, `quota`, `audit`, `jobruns` (every
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
module must never import a `routes/…` module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
*second* caller needs it (a worker as well as a route; a scheduler as well as a route), extract it down
a layer — into the relevant domain package, or a flat core module if it's cross-cutting. Don't
pre-abstract for a single caller; don't leave shared logic duplicated once there are two.
## Units
The domain is four units, each owning its tables (`models.py` in its package):
| Unit | Package | Its routes |
| --- | --- | --- |
| Capture | `downloads/`, the worker | `downloads.py`, `files.py`, `public.py` |
| Reader | `reader/`, `discover/`, `feedquery.py`, the playlist sync and tag suggestions in `sync/` | `feed.py`, `playlists.py`, `saved_views.py`, `discover.py`, `tags.py` |
| Catalog | `catalog/`, the ingests in `sync/` | `channels.py`, `search.py`, `sync.py` |
| Plex | `plex/` | `plex/` |
- **A unit's route does no database work.** It validates, reads through the unit's `queries.py`,
writes through its `service.py`, serializes, and commits. It may name a unit's model as a type
(under `TYPE_CHECKING`), never use one as a value, and it passes a kind (`"movie"`), not a model, to
a query.
- **A unit's code keeps to its edges.** It reads another unit's tables only along a declared edge
(Capture and Reader read Catalog's and Plex's: the feed joins `Video` with `VideoState` in one
query) and writes them only through the owner's service. Core code reaches a unit's tables through
that unit's `queries`.
- **Whose a row is, `userscope` says** — in every module, core included: `owned_by(Model.user_id,
user)` for a filter, `owned_row(db, Model, id, user)` for a row by id (someone else's row is the
same answer as a missing one). A unit's `queries.own_x` returns None and its route answers
`not_found`; a core route calls `owned_or_404`. A rule `owned_row` cannot say — a job its owner
shared, a built-in profile nobody owns, a folder read under `FOR SHARE` — is written out in the
unit's own code, beside the reason.
All of it is held by `tests/test_unit_fences.py`, with no exemptions: the unit map, each unit's
packages and edges (`UNIT_PACKAGES`, `READS`) and what may commit (`UNIT_COMMITS`, `CORE_COMMITS`)
are listed there. It reads the code's AST, so it holds the forms written here: an owner column
compared inline (a model's, an import alias's or an `aliased()` one's, by `==`, `!=`, `.in_()`, or a
`filter_by(user_id=…)`), a model built or an `insert`/`update`/`delete` (under any import name, of a
model or its `__table__`) across units, and a `.commit()` call off the lists all fail it. What an AST
cannot tell is review's to catch: an ORM write on another unit's loaded row (an attribute, a
`db.delete`), SQL in `text()`, a column or model reached another way (bound to a variable, through
`__table__.c` or a module path), and a commit made by a `begin()` block or a `commit` handed in.
## Transaction ownership
This follows CLAUDE.md's canonical rule verbatim — **"the handler/job-wrapper commits; helpers flush
(only `quota` owns its session)"** — so read it that way, not as an exception to it. The commit belongs
to the **operation owner**; anything called *inside* another transaction only stages its change
This follows CLAUDE.md's canonical rule — **the handler commits; a unit's `service` and every helper
flush** — so read it that way, not as an exception to it. The commit belongs to the **operation
owner**; anything called *inside* another transaction only stages its change
(`db.add` / `flush`) and lets the owner commit, so the whole operation is one transaction.
`audit.record()` is the reference: it adds the audit row and the caller commits it together with the
mutation it records, so an audit entry can never exist without its change (or vice-versa).
- The **operation owner** is usually the route handler. A `service.*` operation the handler delegates a
whole transition to (e.g. `service.cancel`) *is* that owner and commits its one transition — it isn't
a mid-operation helper. A helper that owns its OWN short-lived session (the worker's `jobdb` writes,
the same carve-out CLAUDE.md grants `quota`) commits that session. Everything else flushes.
- The **operation owner** is the route handler, or a job. A unit's `service` flushes even when it
makes a whole transition (`service.cancel`): the handler commits it, together with whatever else the
request changed. What owns its OWN session commits it: `quota` and `jobruns`' run rows, the
scheduler's jobs, a unit's jobs (the worker's `jobdb` writes, the syncs and ingests that commit per
page or batch so a later failure loses nothing already stored), and a few core steps that must land
whatever their caller does next (a token's last-used stamp, a refreshed YouTube grant) — the fence's
`UNIT_COMMITS` and `CORE_COMMITS` name each with its reason, `test_route_conventions` a route's own
committing helpers. Everything else flushes.
- `SessionLocal` is `expire_on_commit=False`: after a raw `UPDATE` + commit, an ORM object you already
loaded is **stale**. Read the fresh value from the row (RETURNING, or a re-`SELECT`), not the ORM
attribute — see `service.decr_ref_count`.
@@ -63,13 +108,16 @@ mutation it records, so an audit entry can never exist without its change (or vi
- **`app.config.settings`** — boot/env config (DB URL, download root, worker concurrency): read once at
startup, not user-tunable at runtime.
- **`sysconfig.SPECS`** — the runtime admin knobs (DB-override-or-default). A new admin-tunable value
goes in `SPECS`; don't invent a fourth store. (`state.py`'s `maintenance_batch` predates this and is
slated to fold into `sysconfig` — do not add more like it.)
goes in `SPECS`; don't invent a fourth store. Each spec states when a change takes effect
(`applies`, required): `live` keys are read where they are used, never at import (test-held). The
scheduler's job intervals are `SPECS` keys too, edited on the Scheduler page, not the Configuration page.
## The gate
Everything below must be green before a change is done (run via `siftlode check`):
`ruff` (backend) · `tsc` ×3 + `eslint` + `prettier` + `knip` (frontend) · `vitest` · `pytest`. The
pytest lane runs in `backend/docker-compose.test.yml` — an ephemeral Postgres + the backend tree
bind-mounted — so the DB-backed integration tests (the `db` fixture) actually run; `SIFTLODE_TEST_
REQUIRE_DB=1` there turns a would-be silent skip into a hard failure. See `docs/development.md`.
`ruff` + `basedpyright` (backend) · the lock check + `tsc` ×3 + `eslint` + `prettier` + `knip` +
`vite build` (frontend) · `vitest` · `pytest`. The pytest lane runs in `backend/docker-compose.test.yml`
— an ephemeral Postgres + the backend tree bind-mounted — so the DB-backed integration tests (the `db`
fixture) actually run; a missing database is a hard failure, not a silent skip, unless a hand run asks
for pure-logic only with `SIFTLODE_TEST_REQUIRE_DB=0`. See `docs/development.md`, and `docs/gates.md`
for what each gate deliberately leaves out.
+276
View File
@@ -0,0 +1,276 @@
"""Machine access to the Files library: API tokens, and what each one may do where.
A user makes a token for a machine of theirs — their hub, a script — and it reaches their library as
`Authorization: Bearer sfl_…`. Only on `/api/files`: the routes there take `files_caller` instead of
`current_user`, and nothing else does (a test holds the route table to that), so a token can never
manage the account, the tokens, or anything outside the library. And only in the folders its grants
name, with the rights they give: list, read, write, delete. Every route asks the one question,
`FilesCaller.may(folder_id, right)`; a signed-in user may everything in their own folders.
The token is `sfl_` + 32 random bytes in base62 + a CRC32 of them: a mistyped or cut-off token is
refused without a database read, and a secret scanner can tell one by its shape. Only its SHA-256 is
kept (`security.hash_token`) — the user sees it once, when it is made — with its first characters to
tell tokens apart by. It is looked up by that hash on every request: no signed claims, so a deleted
token stops working at once. A token about to run out warns its user once, a week before
(`warn_expiring`), so a machine does not lose its key unannounced.
"""
import secrets
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from fastapi import Depends, Request
from sqlalchemy import delete, or_, select, update
from sqlalchemy.orm import Session
from app.userscope import owned_by
from app import audit, debugsession, logs
from app.audit import AuditAction
from app.auth import current_user
from app.db import get_db
from app.models import ApiToken, ApiTokenGrant, User
from app.notifications import create_notification
from app.ratelimit import RateLimiter, client_key
from app.refusal import refused
from app.security import hash_token
from app.utils import iso
log = logs.get_logger(__name__)
PREFIX = "sfl_"
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_BODY_BYTES = 32
_BODY_LEN = 43 # 62**43 > 2**256: every 32-byte value fits
_CHECK_LEN = 6 # 62**6 > 2**32: every CRC32 fits
TOKEN_LEN = len(PREFIX) + _BODY_LEN + _CHECK_LEN
# What a token is shown by once it is made: enough to tell a user's tokens apart, far too little to
# narrow a guess (8 of its 43 random characters).
SHOWN_LEN = len(PREFIX) + 8
# A machine that runs away (a loop in a script) is held back per token; a caller trying tokens is
# held back per client, counting only the refusals.
_calls = RateLimiter(600, 60)
_bad = RateLimiter(20, 300)
# How stale `last_used_at` may get before a request writes it again: a busy script costs one write a
# minute, not one per request.
_TOUCH_EVERY = timedelta(minutes=1)
# How long before a token runs out its user is told (`warn_expiring`); the token list marks it
# "expires soon" for the same span.
EXPIRY_WARNING = timedelta(days=7)
class Right(StrEnum):
LIST = "list"
READ = "read"
WRITE = "write"
DELETE = "delete"
# Each right's column on a grant. A dict over the enum, so a new right without a column fails here.
GRANT_COLUMNS = {
Right.LIST: "can_list",
Right.READ: "can_read",
Right.WRITE: "can_write",
Right.DELETE: "can_delete",
}
def _b62(value: int, width: int) -> str:
out = []
for _ in range(width):
value, digit = divmod(value, 62)
out.append(_ALPHABET[digit])
return "".join(reversed(out))
def _check(body: str) -> str:
return _b62(zlib.crc32(body.encode()), _CHECK_LEN)
def mint() -> str:
"""A new token, in full. The caller stores its hash and shows it to the user, once."""
body = _b62(int.from_bytes(secrets.token_bytes(_BODY_BYTES)), _BODY_LEN)
return PREFIX + body + _check(body)
def well_formed(raw: str) -> bool:
"""Whether `raw` has a token's shape and its checksum adds up."""
if len(raw) != TOKEN_LEN or not raw.startswith(PREFIX):
return False
body, check = raw[len(PREFIX) : -_CHECK_LEN], raw[-_CHECK_LEN:]
# The check too: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it.
if any(c not in _ALPHABET for c in body + check):
return False
return secrets.compare_digest(_check(body), check)
def grant_rights(grant: ApiTokenGrant) -> frozenset[Right]:
return frozenset(r for r, col in GRANT_COLUMNS.items() if getattr(grant, col))
@dataclass(frozen=True)
class FilesCaller:
"""Who is asking the Files library, and where they may do what. `grants` is None for a signed-in
user (everything in their own folders); for a token, folder id → its rights there."""
user: User
token_id: int | None = None
token_prefix: str | None = None
grants: Mapping[int, frozenset[Right]] | None = None
def may(self, folder_id: int | None, right: Right) -> bool:
if self.grants is None:
return True
return folder_id is not None and right in self.grants.get(folder_id, frozenset())
def folders_with(self, right: Right) -> set[int] | None:
"""The folders `right` is granted in; None = any of the user's own."""
if self.grants is None:
return None
return {f for f, rights in self.grants.items() if right in rights}
@property
def via(self) -> str | None:
"""How an audit row says a token did it, None when the user did."""
return None if self.token_prefix is None else f"api token {self.token_prefix}…"
def bearer_token(request: Request) -> str | None:
"""The token of an `Authorization: Bearer` header; None when the request carries none (another
scheme counts as none — it is not ours to judge)."""
scheme, _, value = request.headers.get("authorization", "").partition(" ")
if scheme.lower() != "bearer":
return None
return value.strip()
def session_caller(request: Request, db: Session = Depends(get_db)) -> User | None:
"""The signed-in user, or None when the request brings a token instead: a token wins over a
cookie, so a machine's request is judged by its grants alone, whatever else it carries."""
if bearer_token(request) is not None:
return None
return current_user(request, db)
def files_caller(
request: Request,
db: Session = Depends(get_db),
user: User | None = Depends(session_caller),
) -> FilesCaller:
"""The `/api/files` caller: a signed-in user, or a valid token's user with its grants."""
if user is not None:
return FilesCaller(user=user)
return _token_caller(request, db, bearer_token(request) or "")
def _refuse_token(key: str, reason: str) -> Exception:
_bad.allow(key)
log.info(
"a request brought an API token that does not hold",
extra=logs.event("apitoken.refused", reason=reason),
)
return refused(401, "not_authenticated")
def _token_caller(request: Request, db: Session, raw: str) -> FilesCaller:
key = client_key(request)
if _bad.exhausted(key):
raise refused(429, "api_token_rate_limited")
if not well_formed(raw):
raise _refuse_token(key, "malformed")
token = db.execute(
select(ApiToken).where(ApiToken.token_hash == hash_token(raw))
).scalar_one_or_none()
if token is None:
raise _refuse_token(key, "unknown")
now = datetime.now(timezone.utc)
if token.expires_at is not None and token.expires_at <= now:
raise _refuse_token(key, "expired")
user = db.get(User, token.user_id)
if user is None or not user.is_active or user.is_suspended:
raise _refuse_token(key, "account")
if not _calls.allow(str(token.id)):
raise refused(429, "api_token_rate_limited")
grants = {
g.folder_id: grant_rights(g)
for g in db.execute(select(ApiTokenGrant).where(ApiTokenGrant.token_id == token.id))
.scalars()
.all()
}
caller = FilesCaller(user=user, token_id=token.id, token_prefix=token.prefix, grants=grants)
db.execute(
update(ApiToken)
.where(
ApiToken.id == token.id,
or_(ApiToken.last_used_at.is_(None), ApiToken.last_used_at < now - _TOUCH_EVERY),
)
.values(last_used_at=now)
.execution_options(synchronize_session=False)
)
# Committed here, ahead of the route: a read-only route never commits, and a write route must
# not carry this along into a rollback of its own.
db.commit()
logs.set_user(user.id, *debugsession.level_for(user, None))
request.state.user_id = user.id
return caller
def revoke_all(db: Session, user_id: int) -> int:
"""Revoke every token of `user_id`, each with its audit row (NOT committed). A token is a
credential like a session, so whatever ends every session of an account — a password reset or
change, "log out everywhere", a revoked untrusted password — ends its tokens too: one minted
from a stolen session must not outlive the owner's answer to that theft. Returns how many."""
gone = (
db.execute(
delete(ApiToken).where(owned_by(ApiToken.user_id, user_id)).returning(ApiToken.id)
)
.scalars()
.all()
)
for token_id in gone:
audit.record(
db,
user_id,
AuditAction.API_TOKEN_REVOKE,
target_type="api_token",
target_id=token_id,
summary="revoked with every session of the account",
)
return len(gone)
def expiring(token: ApiToken, now: datetime) -> bool:
"""Whether `token` runs out within the warning span (and has not yet)."""
return token.expires_at is not None and now < token.expires_at <= now + EXPIRY_WARNING
def warn_expiring(db: Session) -> int:
"""Tell each user whose token runs out within `EXPIRY_WARNING` — once per token: the mark and the
notice commit together, and a token made again (a new row) warns afresh. Returns how many."""
now = datetime.now(timezone.utc)
hits = db.execute(
update(ApiToken)
.where(
ApiToken.expiry_warned.is_(False),
ApiToken.expires_at.is_not(None),
ApiToken.expires_at > now,
ApiToken.expires_at <= now + EXPIRY_WARNING,
)
.values(expiry_warned=True)
.returning(ApiToken.id, ApiToken.user_id, ApiToken.name, ApiToken.expires_at)
.execution_options(synchronize_session=False)
).all()
for token_id, user_id, name, expires_at in hits:
create_notification(
db,
user_id,
"api_token_expiring",
title=f"API token {name} expires soon", # English fallback; the client names it from data
data={"token_id": token_id, "name": name, "expires_at": iso(expires_at)},
commit=False,
)
db.commit()
return len(hits)
+34 -14
View File
@@ -1,9 +1,9 @@
"""Admin/scheduler audit trail: an append-only record of who changed what and when.
"""Admin audit trail: an append-only record of who changed what and when.
Design (locked): log at ACTION / job-run granularity, NEVER per-row — a scheduler run that
touches thousands of videos is ONE audit row summarising the outcome. Automatic job runs are
only logged when they actually changed something (or errored); all admin actions and all manual
runs are always logged. Secrets are never stored (config changes log the key name only).
Design (locked): log at ACTION granularity, NEVER per-row — an action that touches thousands of
rows is ONE audit row summarising the outcome. Background runs are not actions: they have their
own history (app/jobruns.py). Secrets are never stored (config changes log the key name only).
Each row keeps the request that made it, so the request's log lines are one click away.
`record()` only db.add()s the row — the caller commits it in the SAME transaction as the
mutation it describes, so the trail can't drift from reality (and a rolled-back mutation drops
@@ -15,6 +15,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from sqlalchemy.orm import Session
from app import logs
from app.models import AuditLog
@@ -27,7 +28,7 @@ class AuditAction:
USER_SUSPEND = "user_suspend"
USER_UNSUSPEND = "user_unsuspend"
USER_DELETE = "user_delete"
# E9.5 S5: an admin let a stuck registrant in, or re-sent their confirmation link. Both are
# An admin let a stuck registrant in, or re-sent their confirmation link. Both are
# recorded because both are the admin acting ON someone's access, not merely reading a list.
USER_ACTIVATE = "user_activate"
USER_VERIFY_RESEND = "user_verify_resend"
@@ -36,12 +37,9 @@ class AuditAction:
INVITE_DENY = "invite_deny"
INVITE_ADD = "invite_add"
INVITE_DELETE = "invite_delete"
# E9.5 S6: the admin chose to tell someone their address is whitelisted. Adding the address is
# The admin chose to tell someone their address is whitelisted. Adding the address is
# silent, so this is the act that actually reaches a person — and the one worth a record.
INVITE_SEND = "invite_send"
DEMO_ADD = "demo_add"
DEMO_REMOVE = "demo_remove"
DEMO_RESET = "demo_reset"
DISCOVERY_PURGE = "discovery_purge"
# Config
CONFIG_SET = "config_set"
@@ -51,13 +49,35 @@ class AuditAction:
MAINTENANCE_TUNE = "maintenance_tune"
SYNC_PAUSE = "sync_pause"
SYNC_RESUME = "sync_resume"
# Scheduler job outcome / run-summary (actor_id set = a manual "run now"; NULL = automatic)
JOB_RUN = "job_run"
# Per-user download cookie/auth credential set or removed (the value is NEVER recorded)
DOWNLOAD_CREDENTIAL_SET = "download_credential_set"
DOWNLOAD_CREDENTIAL_DELETE = "download_credential_delete"
# The log itself was cleared (a single row survives the clear as tamper-evidence)
AUDIT_CLEAR = "audit_clear"
# A user sent a debug session's issue to the tracker — the one act that takes their log
# off this instance, so it is recorded (target = the session, summary = the issue's URL).
ISSUE_CREATE = "issue_create"
# The Files library (`routes.files`): what a user put in, took out, kept or reorganised. Ids
# only — a file's or folder's name is the user's own text, not the admin's to read here.
FILE_UPLOAD = "file_upload"
FILE_DELETE = "file_delete"
FILE_KEEP = "file_keep"
FILE_UNKEEP = "file_unkeep"
FILE_MOVE = "file_move"
FILE_FOLDER_CREATE = "file_folder_create"
FILE_FOLDER_RENAME = "file_folder_rename"
FILE_FOLDER_RETENTION = "file_folder_retention"
FILE_FOLDER_DELETE = "file_folder_delete"
# A machine's token to the library (`routes.tokens`): made (its folders and rights) or revoked.
# What a token then did is the file rows above, their summary naming it.
API_TOKEN_CREATE = "api_token_create"
API_TOKEN_REVOKE = "api_token_revoke"
# A public `/watch` link to a file or a folder (`routes.downloads`): made, its settings changed,
# revoked.
# Never its token or password — only whether it has one.
DOWNLOAD_LINK_CREATE = "download_link_create"
DOWNLOAD_LINK_UPDATE = "download_link_update"
DOWNLOAD_LINK_REVOKE = "download_link_revoke"
def record(
@@ -83,16 +103,16 @@ def record(
summary=(summary or None) and summary[:255],
before=before,
after=after,
request_id=logs.current_request_id(),
)
)
def gc(db: Session, retention_days: int) -> int:
"""Delete audit rows older than `retention_days`. Returns the number removed. A
retention_days <= 0 disables pruning (keep forever)."""
retention_days <= 0 disables pruning (keep forever). Flushes; the job commits."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
result = db.execute(delete(AuditLog).where(AuditLog.created_at < cutoff))
db.commit()
return result.rowcount or 0
+88 -100
View File
@@ -1,19 +1,21 @@
import logging
import httpx
from fastapi import BackgroundTasks, Depends, HTTPException, Request
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from starlette.requests import HTTPConnection
from app.userscope import owned_by
from app import debugsession
from app import email as email_mod
from app import logs
from app import mailtext
from app import sysconfig
from app.config import settings
from app.db import get_db
from app.models import Invite, User
from app.models import Invite, LogEvent, User
from app.security import decrypt_optional
from app.utils import canonical_email
from app.refusal import refused
PASSWORD_MIN_LEN = 10
@@ -23,10 +25,6 @@ def operator_contact() -> str | None:
return next(iter(sorted(settings.admin_email_set)), None)
DEMO_GOOGLE_SUB = "__demo__"
DEMO_EMAIL = "demo@siftlode.local"
MAX_SESSION_ACCOUNTS = 6
# OAuth scope vocabulary — the atoms + the request bundles, kept together as core. has_*_scope check
# the atoms against a user's stored grant; the *_SCOPES bundles are what the login/callback endpoints
# request (imported by app/routes/auth.py).
@@ -37,7 +35,7 @@ READ_SCOPES = f"{BASE_SCOPES} {READ_SCOPE}"
WRITE_SCOPES = f"{BASE_SCOPES} {READ_SCOPE} {WRITE_SCOPE}"
_YOUTUBE_SCOPES = frozenset({READ_SCOPE, WRITE_SCOPE})
log = logging.getLogger("siftlode.auth")
log = logs.get_logger(__name__)
def google_enabled(db: Session) -> bool:
@@ -68,7 +66,7 @@ def has_write_scope(user: User) -> bool:
def env_allowed_canonicals() -> set[str]:
"""The env bootstrap allow-list, in CANONICAL form — the only shape it may be compared in.
S3 canonicalised the Invite branch of `is_allowed` and left this one matching the literal string,
Canonicalising only the Invite branch of `is_allowed` left this one matching the literal string,
which is the 2026-09-03 lockout with a different source of truth: an operator who writes
`first.last@gmail.com` into ALLOWED_EMAILS is refused when Google answers with the undotted
primary, and a `+tag`ged entry can never match at all, because a Google account cannot contain a
@@ -128,7 +126,7 @@ def can_activate(db: Session, user: User) -> bool:
Lives in core so the admin surface and the /api/me badge count answer from ONE definition; a
button that appears and a number that counts must never disagree about who is stuck.
"""
if user.is_demo or user.is_suspended or user.is_active:
if user.is_suspended or user.is_active:
return False
return is_allowed(db, user.email)
@@ -146,7 +144,6 @@ def count_pending_activations(db: Session) -> int:
rows = (
db.execute(
select(User).where(
User.is_demo.is_(False),
User.is_suspended.is_(False),
User.is_active.is_(False),
)
@@ -187,24 +184,6 @@ def find_user_by_email(db: Session, email: str) -> User | None:
return candidates[0] if len(candidates) == 1 else None
def get_or_create_demo_user(db: Session) -> User:
"""The one shared demo user, created on first use. No OAuth token / YouTube scope, so
has_read_scope/has_write_scope are False and every YouTube-touching path is closed to it."""
user = db.query(User).filter(User.is_demo.is_(True)).one_or_none()
if user is None:
user = User(
google_sub=DEMO_GOOGLE_SUB,
email=DEMO_EMAIL,
display_name="Demo",
role="user",
is_demo=True,
preferences={"language": "en"},
)
db.add(user)
db.commit()
return user
def revoke_google_token(token: str | None) -> None:
"""Best-effort revocation of a Google OAuth grant at Google's endpoint. Revoking the refresh
token tears down the whole grant, so a later sign-in starts from a clean consent (no scopes
@@ -220,7 +199,7 @@ def revoke_google_token(token: str | None) -> None:
timeout=10,
)
log.info("Google token revoke: status=%s", resp.status_code)
except Exception:
except Exception: # noqa: BLE001 — a revoke we cannot make must not fail the sign-out
log.warning("Google token revoke failed", exc_info=True)
@@ -245,6 +224,12 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
)
# Raw delete so Postgres applies ON DELETE CASCADE / SET NULL on every dependent table.
db.execute(delete(User).where(User.id == user_id))
# Stored log events name the account by id with no foreign key to cascade from (models.LogEvent),
# so they go explicitly — they are request history, i.e. personal data. Forgotten FIRST: this very
# request's access line, a queued row, a later background task would otherwise put rows back
# under the id after this DELETE (logs.forget_user).
logs.forget_user(user_id)
db.execute(delete(LogEvent).where(owned_by(LogEvent.user_id, user_id)))
# Every canonically-equal invite goes, not just the literal spelling: is_allowed matches on the
# canonical form, so leaving a sibling row behind would keep granting access to a mailbox whose
# account was just erased — and prod really does carry approved pairs for one mailbox.
@@ -266,62 +251,74 @@ def purge_user(db: Session, user: User, background: BackgroundTasks) -> None:
background.add_task(email_mod.send_account_deleted, email, lang, operator_contact())
ACTIVE_ACCOUNT_HEADER = "x-siftlode-account"
def session_user_id(conn: HTTPConnection) -> int | None:
"""The account this browser session signed in as, read from the signed cookie alone (no DB).
One account per session: signing in as another replaces it (`routes.auth._establish_session`)."""
return conn.session.get("user_id")
def resolved_user_id(conn: HTTPConnection) -> tuple[int | None, bool]:
"""Which account this connection acts as, and whether that's the session's default account.
Takes any HTTPConnection — an HTTP Request OR a WebSocket — so the WS push channel shares this
exact per-tab resolution instead of re-implementing it.
def stamp_session(request: Request, user: User) -> None:
"""Record the account's current `session_epoch` in the signed cookie, so `current_user` can later
reject the cookie once the epoch moves (password reset/change, or "log out everywhere")."""
request.session["epoch"] = user.session_epoch or 0
The signed cookie holds the *wallet* (`account_ids`, every account signed into this browser)
plus a default `user_id`. A caller may override the default with the X-Siftlode-Account header,
but ONLY for an account already in the wallet — so a tab can't impersonate an account that never
authenticated here. Everything else (WebSocket, plain navigations) keeps using the cookie default.
"""
default_id = conn.session.get("user_id")
wallet = conn.session.get("account_ids") or []
# Header for normal XHR; ?account= for contexts that can't set headers (WebSocket, and a
# plain <a> file download). Both are wallet-gated below, so neither can impersonate an
# account that never signed into this browser.
hdr = conn.headers.get(ACTIVE_ACCOUNT_HEADER) or conn.query_params.get("account")
if hdr:
try:
hid = int(hdr)
except ValueError:
hid = None
if hid is not None and hid in wallet:
return hid, hid == default_id
return default_id, True
# The account a tab was built for, sent with every request it makes (`identityHeaders`, the
# frontend). The cookie alone says who is asking, and one browser holds one account: a sign-in as
# someone else in another tab replaces it under every open tab. A tab still showing the previous
# account would otherwise act as the new one — its player checkpoints, pref saves, Sign out and
# Delete account landing there — until it noticed. It never selects an account; a mismatch is
# refused, and the tab reloads as whoever is signed in now.
EXPECTED_ACCOUNT_HEADER = "x-siftlode-expect-account"
def built_for_another_account(user_id: int, expected: str | None) -> bool:
"""Whether a tab naming `expected` is showing another account than the one signed in. No name is
no claim (a caller from before this check, a script): only a mismatch is refused, and the
session stays — it is the other tab's sign-in, and it holds. The one comparison for every way a
tab names its account: this header on a request, `expect` on a navigation."""
if expected is None or expected == str(user_id):
return False
log.info(
"a request from a tab built for another account was refused",
extra=logs.event("auth.session.account_changed", user_id=user_id),
)
return True
def current_user(request: Request, db: Session = Depends(get_db)) -> User:
user_id, is_default = resolved_user_id(request)
user = _session_user(request, db, session_user_id(request))
if built_for_another_account(user.id, request.headers.get(EXPECTED_ACCOUNT_HEADER)):
raise refused(401, "account_changed")
return user
def _session_user(request: Request, db: Session, user_id: int | None) -> User:
if not user_id:
raise HTTPException(status_code=401, detail="Not authenticated")
raise refused(401, "not_authenticated")
user = db.get(User, user_id)
if user is None or not user.is_active or user.is_suspended:
# Suspended/deactivated mid-session → drop the session so the block takes effect at once.
# But only nuke the whole session when the SESSION DEFAULT identity is gone; a stale
# per-tab header account just 401s that one tab (other tabs' default may still be valid).
if is_default:
request.session.clear()
raise HTTPException(status_code=401, detail="Not authenticated")
# SA4: reject a cookie whose recorded epoch is behind the account's current one (revoked by a
# password reset/change or "log out everywhere"). A cookie with no recorded epoch (pre-SA4, or an
# account signed in before this shipped) is treated as epoch 0 — so the FIRST bump revokes it too.
epochs = request.session.get("epochs") or {}
if int(epochs.get(str(user_id), 0)) != (user.session_epoch or 0):
if is_default:
request.session.clear()
raise HTTPException(status_code=401, detail="Not authenticated")
# Always keep the session default in the switchable list — covers sessions created before
# multi-session existed (their account_ids was never seeded), so the switcher isn't empty.
default_id = request.session.get("user_id")
accounts = request.session.get("account_ids") or []
if default_id and default_id not in accounts:
accounts.append(default_id)
request.session["account_ids"] = accounts[-MAX_SESSION_ACCOUNTS:]
# Suspended/deactivated mid-session → end it so the block takes effect at once.
request.session.clear()
raise refused(401, "not_authenticated")
# A cookie whose epoch is behind the account's (revoked by a password reset/change or "log out
# everywhere") is refused — and so is one with no epoch at all, which only a build before the
# one-account session wrote: its holder signs in once more.
epoch = request.session.get("epoch")
if epoch is None or int(epoch) != (user.session_epoch or 0):
request.session.clear()
raise refused(401, "not_authenticated")
# Attribute this request's log lines to the account. Here, not in the request middleware:
# this is the one place a session becomes an account, and the middleware runs outside both the
# session middleware and this task's context. `request.state` is shared across middleware layers
# (it lives on the ASGI scope), so the access-log line can read it back. The user's consented
# diagnostics level rides along: their requests are logged — and stored — in that detail.
# So does a debug session the request names: only the caller's own open one counts, and a
# header naming anything else is ignored rather than refused — a stale tab must keep working.
session = debugsession.from_header(db, user, request.headers.get(debugsession.HEADER))
level, until = debugsession.level_for(user, session)
logs.set_user(user.id, level, until, session.id if session else None)
request.state.user_id = user.id
return user
@@ -329,36 +326,27 @@ def optional_current_user(request: Request, db: Session = Depends(get_db)) -> Us
"""Like `current_user` but returns None instead of raising 401 when there's no valid session.
Used by the bootstrap /api/me probe so the logged-out landing doesn't log a failed request to
the browser console (a non-2xx fetch is a console error no matter how the app handles it)."""
user_id = session_user_id(request)
if user_id is None:
return None
try:
return current_user(request, db)
return _session_user(request, db, user_id)
except HTTPException:
# A session that named an account, refused: the browser is signed out without being told
# why, so say it here.
log.info(
"a session was refused: its account no longer holds",
extra=logs.event("auth.session.probe_rejected", user_id=user_id),
)
return None
def require_human(user: User = Depends(current_user)) -> User:
"""Reject the shared demo account from actions that need a real Google/YouTube identity
or spend the shared quota. Most YouTube paths are already closed to it implicitly (it has
no token, so has_read_scope/has_write_scope are False); this makes the boundary explicit
where there's no scope check to lean on."""
if user.is_demo:
# Structured, because `req` leaves 403 to the caller and `notifyActionError` only shows a
# STRUCTURED detail — a plain string here reached the demo user as their caller's generic
# "couldn't do that", with no hint that the demo account is the reason (round 6). Affordances
# this gate sits behind are not all UI-gated on `is_demo` (the comfort shelf is not).
raise HTTPException(status_code=403, detail={"code": "demo"})
return user
def require_write_scope(user: User = Depends(require_human)) -> User:
def require_write_scope(user: User = Depends(current_user)) -> User:
"""Dependency: require the YouTube write scope (subscribe/unsubscribe, playlist export/edit).
Single source of truth for the write gate — replaces the inline `if not has_write_scope(user)`
checks that had drifted into three different 403 messages across channels/playlists. Layers on
`require_human` because writing to YouTube needs a real, non-demo identity."""
checks that had drifted into three different 403 messages across channels/playlists."""
if not has_write_scope(user):
raise HTTPException(
status_code=403,
detail="Enable YouTube editing in Settings first.",
)
raise refused(403, "youtube_editing_off")
return user
@@ -366,7 +354,7 @@ def admin_user(user: User = Depends(current_user)) -> User:
"""Dependency: require the signed-in user to be an admin. Single source of truth for the
admin gate — every admin route/action depends on this rather than re-checking the role."""
if user.role != "admin":
raise HTTPException(status_code=403, detail="Admin only")
raise refused(403, "admin_only")
return user
+239
View File
@@ -0,0 +1,239 @@
"""The global request-body cap (SEC-20).
**The hole.** FastAPI reads a declared body IN FULL before auth or any other dependency runs — the
body parameter has to be parsed to be passed in. So `POST /auth/password-login` drains a 32 MiB
upload from an anonymous caller and only then answers 401. 70 of the 72 body-declaring routes behave
the same way: one unauthenticated client can spend the process's memory and time at will.
**Why a middleware and not `capped_body` on every route.** `routes/_common.py:capped_body` is the
right tool where a route genuinely takes bulk bytes, and the two routes that use it use it
correctly. But putting it on all 70 others would (a) fight the canonical "Request body → a Pydantic
model" row, since capping means taking the body as raw bytes and parsing it by hand, and (b) leave
the NEXT route added uncovered, because nothing would remind its author. A default that has to be
opted into is not a default. This module inverts it: the cap applies to everything, and the few
routes that need more opt OUT, in one list, next to the code that reads it.
**Why raw ASGI and not `@app.middleware("http")`.** Starlette's BaseHTTPMiddleware cannot wrap
`receive`, so it can only read `Content-Length` — which a client may omit entirely (chunked) or
simply understate. The bytes have to be counted as they arrive, which is the same rule `capped_body`
already applies per-route, and counting needs the receive channel.
**Why the refusal is byte-identical to `capped_body`'s.** Two shapes for one fact is how the
`cancelled`/`canceled` bug happened. The 413 body here is built through the same `JSONResponse`
FastAPI renders `HTTPException(413, {"code": "too_large"})` with — not a hand-rolled `json.dumps`,
whose default separators differ by two bytes — and `tests/test_body_limit.py` compares the RAW BYTES
of the two against each other rather than against a literal, so they cannot drift apart. The same
reasoning decides where the middleware is registered (`main.py`): just inside CORS, so this 413
carries the same headers `capped_body`'s does.
**No admin knob, deliberately.** The ceiling is a constant, not a `sysconfig` value: an attacker-
facing limit that an admin can raise from the UI is a limit that can be raised by whoever gets the
admin session, and nobody has asked to tune it. If a self-hoster ever needs a different ceiling it
belongs in `config.py` (env, set before the process starts), not in the database.
"""
import re
from starlette.responses import JSONResponse
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from app import debugsession
from app.config import settings
# 1 MiB. The largest body any route on the default ceiling legitimately takes is 512 KiB (the client
# log sink, `routes/diagnostics.py:BODY_MAX_BYTES`), so this leaves 2x headroom — a ceiling that
# refuses a legitimate request would be a worse bug than the one this module fixes. That ratio is
# the load-bearing fact, so `test_body_limit.py` holds it rather than the figure above.
DEFAULT_MAX_BODY_BYTES = 1024 * 1024
# The routes that need MORE than the default, as (method, path TEMPLATE) → their own ceiling. The
# template, not a concrete path, so the list reads like the route decorators it mirrors and
# `test_body_limit.py` can assert that each one still resolves to a real route: a renamed route
# fails the gate instead of silently losing its allowance in prod.
#
# These are RAISED CEILINGS, NOT EXEMPTIONS, and the difference is the whole point. An exemption
# would mean "this route is counted by something else" — which is only true when the route's own
# limit applies to the bytes as they ARRIVE. That, in turn, is only true of a route with NO DECLARED
# BODY PARAMETER: FastAPI parses and spools a declared body BEFORE the first dependency runs, so a
# route that declares one has already stored the bytes by the time its own cap — or its auth — gets
# a say.
#
# Both listed routes now satisfy that. The screenshot route always did (`capped_body` streams it,
# and the owner check resolves first). The upload route did NOT at first: it declared
# `file: UploadFile = File(...)`, so an anonymous caller could make the process write the whole
# per-file allowance to temp before the auth dependency refused them — this middleware bounded that
# number but could not stop the writing. It now reads its own body from a dependency declared after
# the auth one, so its in-handler cap measures bytes as they arrive, like everything else here.
#
# That rule is no longer only written here: `test_body_limit.py` parses the routes and fails the
# gate if any route in this dict declares a body parameter. A comment could not have stopped the
# defect it describes; the test can.
#
# A route belongs here ONLY IF ITS OWN CAP IS LARGER THAN THE DEFAULT. That is why
# `POST /api/diagnostics/events` is absent: its `capped_body` limit is 512 KiB, already below the
# default, so listing it could only widen what it accepts.
# A Google Takeout file (`reader.takeout`). A real watch history measured 3.7 MB for 9500 views, ~390
# bytes a view, so 32 MiB holds ~86k views. YouTube keeps the history 36 months by default on an
# account made since mid-2020 and for good on an older one, so a heavy viewer's can pass 16 MiB. The
# file's text is decoded a window at a time and read an entry at a time (`takeout._json_list`), so
# it costs its bytes and a window, not a tree of it; the route takes two at once across every user.
TAKEOUT_MAX_BYTES = 32 * 1024 * 1024
ROUTE_MAX_BODY_BYTES: dict[tuple[str, str], int] = {
# Ingesting a local file. The body IS the file now — no multipart envelope to allow for —
# so this ceiling and the route's own cap count exactly the same bytes. `sysconfig`'s
# `download_upload_max_bytes` is bounded by this same constant (see sysconfig.SPECS), so an
# admin can only ever LOWER the route's cap below the edge, never raise it past one.
#
# WHICH LAYER ANSWERS, measured rather than assumed:
# * knob LOWERED (an override below the default) — the route's cap is the smaller number, so
# `capped_stream_to` trips first and answers 422 `upload_too_large` carrying the admin's own
# figure, after draining the rest of the body so the client can read it.
# * knob at the DEFAULT — the two numbers are EQUAL, and the edge always wins: a declared
# Content-Length over the cap is refused here before a byte is read, and an undeclared body
# trips this counter on the same chunk the route would have tripped on. The route's 422 is
# unreachable in that configuration, and no amount of headroom would change it — the
# declared-length check fires first for any body big enough to matter, and the route's drain
# pushes the count past the edge for the rest.
#
# So this 413 is a refusal a real user meets on a default install, not a theoretical edge, and
# it has to be as readable as the route's own: the client localizes `too_large` (`errors.tooLarge`)
# exactly as it does `upload_too_large`, and the upload dialog refuses an over-cap file before
# sending it at all, off the `upload_max_bytes` that `GET /api/files/usage` reports.
("POST", "/api/files/upload"): settings.download_upload_max_bytes,
("POST", "/api/debug-sessions/{sid}/screenshots"): debugsession.SCREENSHOT_MAX_BYTES,
("POST", "/api/takeout"): TAKEOUT_MAX_BYTES,
}
# The refusal, byte-for-byte what `capped_body`'s `HTTPException(413, {"code": "too_large"})` renders
# as — built through the same JSONResponse so the separators match too, and asserted against a real
# route's raw bytes in `test_body_limit.py` rather than against a literal here.
TOO_LARGE_BODY = JSONResponse({"detail": {"code": "too_large"}}, status_code=413).body
_PATH_PARAM = re.compile(r"\{[^/{}]+\}")
def template_to_regex(template: str) -> re.Pattern[str]:
"""`/api/debug-sessions/{sid}/screenshots` → a regex matching one concrete path.
A path parameter matches `[^/]+` — one segment, never across a `/`, so `{sid}` cannot swallow a
deeper path and hand an exemption to a route that was never granted one.
"""
literals = [re.escape(part) for part in _PATH_PARAM.split(template)]
return re.compile("^" + "[^/]+".join(literals) + "$")
def _declared_length(scope: Scope) -> int | None:
"""The `Content-Length` header as an int, or None when absent or unparseable (a chunked body has
none, and a malformed one is not a number we may trust — both fall through to counting)."""
for name, value in scope.get("headers", ()):
if name == b"content-length":
try:
return int(value)
except ValueError:
# silent-ok: a malformed Content-Length is the caller's doing, not a fault of ours,
# and it is about to be answered anyway — returning None drops the request into the
# counting path, which refuses on the real bytes. Logging it would hand an anonymous
# client a free line in our log for every request it cares to malform.
return None
return None
async def _send_too_large(send: Send) -> None:
await send(
{
"type": "http.response.start",
"status": 413,
"headers": [
(b"content-type", b"application/json"),
(b"content-length", str(len(TOO_LARGE_BODY)).encode()),
],
}
)
await send({"type": "http.response.body", "body": TOO_LARGE_BODY})
class BodyLimitMiddleware:
"""Refuses any request body over its route's ceiling, before the application sees it.
Registered just inside CORS in `main.py` — so the 413 is decorated like every other response
and a browser can actually read it — but outside the session, the setup gate and every auth
dependency, none of which run for a request that is already too large.
Every request gets a finite ceiling: `ROUTE_MAX_BODY_BYTES` raises it for the two routes that
genuinely take bulk bytes, and there is no path that skips counting altogether.
"""
def __init__(
self,
app: ASGIApp,
*,
limit: int = DEFAULT_MAX_BODY_BYTES,
routes: dict[tuple[str, str], int] | None = None,
) -> None:
self.app = app
self.limit = limit
self._routes = tuple(
(method, template_to_regex(path), cap)
for (method, path), cap in (ROUTE_MAX_BODY_BYTES if routes is None else routes).items()
)
def limit_for(self, method: str, path: str) -> int:
"""This request's ceiling: the route's own if it has one, else the global default. Every
request gets a finite number — there is no path through here that is simply not counted."""
for m, rx, cap in self._routes:
if m == method and rx.match(path):
return cap
return self.limit
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
limit = self.limit_for(scope.get("method", ""), scope.get("path", ""))
declared = _declared_length(scope)
if declared is not None and declared > limit:
# Refused on the declaration alone: nothing downstream is called, and not one byte of
# the body is read.
await _send_too_large(send)
return
seen = 0
refused = False
started = False
async def counting_receive() -> Message:
nonlocal seen, refused
message = await receive()
if message["type"] != "http.request":
return message
seen += len(message.get("body", b""))
if seen > limit:
# The app is blocked on this receive, so it has not answered yet. Telling it the
# client went away is the only in-band way to stop it reading: `request.stream()`
# turns an `http.disconnect` into ClientDisconnect, which unwinds the handler. Our
# own 413 goes out below, once the app has finished unwinding.
refused = True
return {"type": "http.disconnect"}
return message
async def guarded_send(message: Message) -> None:
nonlocal started
if refused and not started:
return # discard the app's answer to the disconnect; the 413 below is the response
started = True
await send(message)
try:
await self.app(scope, counting_receive, guarded_send)
except Exception: # only swallowed when WE caused it, see below
# ClientDisconnect (or whatever a handler raises on a truncated read) is the expected
# consequence of the disconnect we just faked, and the caller must get the 413 rather
# than a 500. Anything raised when we did NOT refuse is a real failure and is re-raised
# untouched.
if not refused:
raise
if refused and not started:
await _send_too_large(send)
View File
+268
View File
@@ -0,0 +1,268 @@
"""Catalog's tables: the shared YouTube world every user reads — channels and their videos — and
each user's footprint in it: what they subscribe to, block, explored or found by search. Only this
unit writes them; another unit reads them along a declared edge, or asks this unit's service."""
from datetime import datetime
from typing import TYPE_CHECKING
from sqlalchemy import (
BigInteger,
Boolean,
Computed,
DateTime,
ForeignKey,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db import Base
from app.models import TimestampMixin
if TYPE_CHECKING:
from app.models import User
class Channel(Base, TimestampMixin):
"""A YouTube channel. Shared across all users (one channel's videos are the same
for everyone), so its expensive metadata is fetched and stored only once."""
__tablename__ = "channels"
id: Mapped[str] = mapped_column(String(32), primary_key=True) # "UC..." channel id
title: Mapped[str | None] = mapped_column(String(255))
handle: Mapped[str | None] = mapped_column(String(255))
description: Mapped[str | None] = mapped_column(Text)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
uploads_playlist_id: Mapped[str | None] = mapped_column(String(64))
subscriber_count: Mapped[int | None] = mapped_column(BigInteger)
video_count: Mapped[int | None] = mapped_column(BigInteger)
# Channel "About" metadata (fetched with the same channels.list call, no extra quota):
# lifetime view count, the channel's join date, banner image, and external links.
total_view_count: Mapped[int | None] = mapped_column(BigInteger)
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
banner_url: Mapped[str | None] = mapped_column(String(1024))
external_links: Mapped[list | None] = mapped_column(JSON)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
country: Mapped[str | None] = mapped_column(String(8))
# Creator keyword tags (brandingSettings.channel.keywords) — a single space-separated string,
# multi-word tags quoted; the client parses it into chips for the About tab.
keywords: Mapped[str | None] = mapped_column(Text)
# Sync bookkeeping.
details_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
recent_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_rss_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# When the newest uploads were last read through the Data API because the RSS feed would not
# answer (`sync/runner.py` `run_feed_fallback`): the once-per-`rss_fallback_hours` clock.
feed_fallback_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
backfill_cursor: Mapped[str | None] = mapped_column(String(128))
backfill_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this channel only entered the catalog because a live YouTube search
# surfaced one of its videos (nobody subscribes to it / it has no organic upload sync).
# Lets the Library hide search-only "noise" by default.
from_search: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
# True when this channel exists only because a user EXPLORED it (opened its page without
# subscribing) — an ephemeral, per-explorer addition. After a grace period the cleanup job
# hard-deletes its via_explore videos nobody keeps (no subscription, no live ExploredChannel
# row, no per-user state), and then the channel row itself once it holds no videos — unless a
# Takeout watch history shows it watched: the "watched but not followed" list names it from
# that row. Cleared when someone subscribes, here or on YouTube (the "keep" signal promotes it
# to organic).
from_explore: Mapped[bool] = mapped_column(
Boolean, default=False, server_default="false", index=True
)
videos: Mapped[list["Video"]] = relationship(back_populates="channel")
subscriptions: Mapped[list["Subscription"]] = relationship(back_populates="channel")
class Subscription(Base, TimestampMixin):
"""Per-user link to a channel, with the user's own priority/visibility overrides."""
__tablename__ = "subscriptions"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
# YouTube's subscription resource id, needed to unsubscribe via the API.
yt_subscription_id: Mapped[str | None] = mapped_column(String(128))
priority: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
hidden: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Per-user opt-in to full-history (deep) backfill of this channel. Default off so a
# new user's unique channels don't trigger a big shared-quota burst; the deep
# scheduler only picks up channels at least one user has requested.
deep_requested: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
subscribed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
user: Mapped["User"] = relationship(back_populates="subscriptions")
channel: Mapped["Channel"] = relationship(back_populates="subscriptions")
# Video.live_status values that mean "currently a live/upcoming broadcast" — transient states
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
LIVE_OR_UPCOMING = ("live", "upcoming")
class Video(Base, TimestampMixin):
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
__tablename__ = "videos"
id: Mapped[str] = mapped_column(String(16), primary_key=True) # video id
# No standalone channel_id index: the composite `ix_videos_channel_pub_cover`
# (channel_id, published_at DESC NULLS LAST, id) from migration 0095 leads with channel_id, so it
# serves every channel_id lookup (FK cascade, sync upsert, channel-detail) and the per-channel feed
# sort. Its INCLUDEd columns make the channel manager's sums and the feed sidebar's counts Index Only
# Scans (no heap): a column those queries start reading belongs in it (test_channel_list_index_only).
# Like the search_vector GIN, this composite is migration-defined, not an inline `index=True`.
channel_id: Mapped[str] = mapped_column(ForeignKey("channels.id", ondelete="CASCADE"))
title: Mapped[str | None] = mapped_column(Text) # normalized for display (see app.titles)
original_title: Mapped[str | None] = mapped_column(Text) # raw YouTube title, preserved
description: Mapped[str | None] = mapped_column(Text)
# NOT NULL (migration 0068): in reality every video carries a publish date, and a nullable key
# forced the feed's keyset predicate into a non-sargable `... OR published_at IS NULL` disjunction
# that blocked the ix_videos_published_at_desc range (deep pages scanned from the top and filtered —
# PERF-15). server_default now() is the safety net for the rare date-less ingest (the enrich fills
# the real date); the ix_videos_published_at index stays for the `oldest` sort.
published_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), index=True, nullable=False, server_default=func.now()
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
# `feedquery.order_by` by test_feed_sort_indexes.
duration_seconds: Mapped[int | None] = mapped_column(Integer)
view_count: Mapped[int | None] = mapped_column(BigInteger)
like_count: Mapped[int | None] = mapped_column(BigInteger)
category_id: Mapped[int | None] = mapped_column(Integer)
topic_categories: Mapped[list | None] = mapped_column(JSON)
default_language: Mapped[str | None] = mapped_column(String(16))
detected_language: Mapped[str | None] = mapped_column(String(16))
# Creator-supplied keyword tags (snippet.tags joined by spaces) — fetched for free with the
# snippet we already request. Indexed into search_vector so the feed search matches the
# uploader's own keywords, not just words in the title.
keywords: Mapped[str | None] = mapped_column(Text)
# Accumulated distinct live-search queries that surfaced this video (across all users) —
# folds YouTube's own relevance judgement into the local search: a video YouTube returned
# for "ti amo magyarul" becomes findable by that query locally even if its title lacks it.
search_terms: Mapped[str | None] = mapped_column(Text)
# No btree here or on the flag columns below: migration 0069 (PERF-3) dropped the never-picked
# boolean btrees and replaced the useful ones with partial indexes over the target subset
# (via_search=true, NOT shorts_probed, unavailable_since IS NOT NULL) — migration-defined, like the
# search_vector GIN.
is_short: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Whether the youtube.com/shorts/<id> probe has run for this video.
shorts_probed: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video first entered the catalog via a live YouTube search (not an
# organic channel/playlist sync). Used to hide search-discovered "noise" from the
# Library by default; left False by the organic ingest paths.
via_search: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# True when this video entered the catalog because a user EXPLORED its (un-subscribed)
# channel. Like via_search it's hidden from the shared Library by default, but it's
# additionally PER-USER private: a via_explore video is only visible to users who
# explored or subscribe to its channel (a gate in the feed query), so one user's
# curiosity never floods everyone else's catalog. Left False by the organic ingest paths.
via_explore: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# none | live | upcoming | premiere | was_live
live_status: Mapped[str] = mapped_column(
String(16), default="none", server_default="none", index=True
)
enriched_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# videos.list?part=status fields (populated during enrichment); used by the
# maintenance/validation job to decide whether a video is playable anywhere.
embeddable: Mapped[bool | None] = mapped_column(Boolean)
privacy_status: Mapped[str | None] = mapped_column(String(16)) # public|unlisted|private
upload_status: Mapped[str | None] = mapped_column(
String(16)
) # processed|failed|rejected|deleted
# Maintenance/validation lifecycle. last_checked_at drives the rolling re-validation
# (least-recently-checked first); unavailable_since marks a video as currently
# unplayable (hidden from the feed immediately, hard-deleted after a grace period);
# unavailable_reason records why (deleted|private|paywalled|abandoned|stuck_live).
# last_checked_at: no plain btree — migration 0069 (PERF-18) replaced it with the partial
# ix_videos_recheck_order (last_checked_at NULLS FIRST WHERE unavailable_since IS NULL) that the
# revalidate job's exact ORDER BY rides; unavailable_since is a partial WHERE IS NOT NULL (PERF-3).
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
# weights so title matches outrank description matches. Backed by a GIN index (migration
# 0032). The accent-insensitive `unaccent_simple` config comes from migration 0031.
search_vector: Mapped[object | None] = mapped_column(
TSVECTOR,
Computed(
"setweight(to_tsvector('public.unaccent_simple', coalesce(title, '')), 'A') || "
"setweight(to_tsvector('public.unaccent_simple', coalesce(keywords, '') || ' ' || "
"coalesce(search_terms, '')), 'B') || "
"setweight(to_tsvector('public.unaccent_simple', left(coalesce(description, ''), 1000)), 'C')",
persisted=True,
),
)
channel: Mapped["Channel"] = relationship(back_populates="videos")
class SearchFind(Base, TimestampMixin):
"""Per-user record of a video the user surfaced via their own live YouTube search.
Lets "videos I searched for" appear in that user's own (Mine) feed and the Mine Source
filter — distinct from the GLOBAL `Video.via_search` provenance, which marks a video as
search-discovered by *anyone* and drives the shared Library's Source filter."""
__tablename__ = "search_finds"
__table_args__ = (UniqueConstraint("user_id", "video_id", name="uq_user_search_video"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
video_id: Mapped[str] = mapped_column(ForeignKey("videos.id", ondelete="CASCADE"), index=True)
class ExploredChannel(Base, TimestampMixin):
"""Per-user record of a channel the user opened the page of WITHOUT subscribing.
Exploring an un-subscribed channel ingests its recent uploads (flagged via_explore) so the
user can browse them, but keeps them out of everyone else's catalog. This row (a) gates the
explorer's access to those ephemeral videos in the feed query and (b) drives the grace-period
purge: of a from_explore channel with no live ExploredChannel row (and no subscription) the
cleanup job hard-deletes the videos without per-user state, then the channel once it holds none
and no Takeout watch history shows it watched. Subscribing is the "keep"
signal — it promotes the channel to organic and these rows become irrelevant. `created_at`
(from TimestampMixin) is the grace clock; a re-visit refreshes it."""
__tablename__ = "explored_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_explored_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
class BlockedChannel(Base, TimestampMixin):
"""Per-user channel blocklist. Videos from a blocked channel are excluded from this user's
live YouTube search (and not ingested on their behalf) and hidden from their feed / explore
views — a personal "never show me this channel again" that doesn't affect other users."""
__tablename__ = "blocked_channels"
__table_args__ = (UniqueConstraint("user_id", "channel_id", name="uq_user_blocked_channel"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
channel_id: Mapped[str] = mapped_column(
ForeignKey("channels.id", ondelete="CASCADE"), index=True
)
+309
View File
@@ -0,0 +1,309 @@
"""Catalog's reads: channels and videos, and each user's footprint in them (subscriptions, blocks,
explorations), for the HTTP layer and for the core code that asks about them.
A user's row is read through `userscope`, so someone else's is the same answer as a missing one.
Catalog does not read another unit's tables: what it needs from Reader (a playlist's channels, a
user's tag links) it asks `reader.queries`.
"""
from datetime import datetime
from sqlalchemy import Select, and_, case, func, or_, select
from sqlalchemy.orm import Session
from app.catalog.models import (
LIVE_OR_UPCOMING,
BlockedChannel,
Channel,
ExploredChannel,
Subscription,
Video,
)
from app.collation import collated
from app.models import User
from app.sync.explore import followed_channel
from app.userscope import owned_by
# --- channels --------------------------------------------------------------------------------
def channel(db: Session, channel_id: str) -> Channel | None:
return db.get(Channel, channel_id)
def channel_title(db: Session, channel_id: str) -> str | None:
return db.scalar(select(Channel.title).where(Channel.id == channel_id))
def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
"""Each channel's title by id; a channel with no title yet is left out."""
if not ids:
return {}
rows = db.execute(select(Channel.id, Channel.title).where(Channel.id.in_(ids)))
return {cid: title for cid, title in rows.tuples() if title is not None}
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
select(
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
)
def aggregates(db: Session, channel_ids: list[str]) -> list[tuple[str, int, datetime | None, int]]:
"""`channel_aggregates`, run: (channel id, stored videos, last upload, total seconds)."""
if not channel_ids:
return []
return list(db.execute(channel_aggregates(channel_ids)).tuples())
def video_counts(db: Session, channel_id: str) -> tuple[int, int, int]:
"""A channel's stored videos, its Shorts and its live/upcoming ones, in one grouped pass."""
n_all, n_short, n_live = db.execute(
select(
func.count(Video.id),
func.count(Video.id).filter(Video.is_short.is_(True)),
func.count(Video.id).filter(Video.live_status.in_(LIVE_OR_UPCOMING)),
).where(Video.channel_id == channel_id)
).one()
return int(n_all or 0), int(n_short or 0), int(n_live or 0)
def deep_queued(db: Session, channel_ids: list[str]) -> set[str]:
"""Which of these channels at least one user (any user) has asked full history for."""
if not channel_ids:
return set()
return set(
db.execute(
select(Subscription.channel_id)
.where(
Subscription.channel_id.in_(channel_ids),
Subscription.deep_requested.is_(True),
)
.distinct()
).scalars()
)
def followed(db: Session, channel_id: str) -> bool:
"""Whether anyone subscribes to the channel."""
return (
db.execute(
select(Subscription.id).where(Subscription.channel_id == channel_id).limit(1)
).first()
is not None
)
# --- a user's subscriptions, blocks and explorations ----------------------------------------
def subscriptions(db: Session, user: User, lang: str) -> list[tuple[Subscription, Channel]]:
"""The user's subscriptions with their channels, highest priority first, then by title in the
reader's alphabet."""
return list(
db.execute(
select(Subscription, Channel)
.join(Channel, Channel.id == Subscription.channel_id)
.where(owned_by(Subscription.user_id, user))
.order_by(Subscription.priority.desc(), collated(Channel.title, lang))
).tuples()
)
def subscription(db: Session, user: User, channel_id: str) -> Subscription | None:
return db.execute(
select(Subscription).where(
owned_by(Subscription.user_id, user), Subscription.channel_id == channel_id
)
).scalar_one_or_none()
def subscribed_channels(user: User):
"""The channel ids the user subscribes to, as a subquery."""
return select(Subscription.channel_id).where(owned_by(Subscription.user_id, user))
def exploration(db: Session, user: User, channel_id: str) -> ExploredChannel | None:
return db.execute(
select(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
).scalar_one_or_none()
def is_blocked(db: Session, user: User, channel_id: str) -> bool:
return (
db.execute(
select(BlockedChannel.id).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
).first()
is not None
)
def blocked_ids(db: Session, user: User) -> set[str]:
return set(
db.scalars(select(BlockedChannel.channel_id).where(owned_by(BlockedChannel.user_id, user)))
)
def followed_among(db: Session, user: User, channel_ids: set[str]) -> set[str]:
"""Which of these channels the user subscribes to."""
if not channel_ids:
return set()
return set(
db.scalars(
select(Subscription.channel_id).where(
owned_by(Subscription.user_id, user), Subscription.channel_id.in_(channel_ids)
)
)
)
def blocked_channels(db: Session, user: User, lang: str) -> list[Channel]:
return list(
db.execute(
select(Channel)
.join(BlockedChannel, BlockedChannel.channel_id == Channel.id)
.where(owned_by(BlockedChannel.user_id, user))
.order_by(collated(Channel.title, lang))
).scalars()
)
# --- sync progress ---------------------------------------------------------------------------
def catalog_totals(db: Session, user: User) -> dict:
"""The instance's catalog size and this user's subscription count (the sync status card)."""
return {
"subscriptions": db.scalar(
select(func.count())
.select_from(Subscription)
.where(owned_by(Subscription.user_id, user))
),
"channels_total": db.scalar(select(func.count()).select_from(Channel)),
"channels_backfilling": db.scalar(
select(func.count()).select_from(Channel).where(Channel.backfill_done.is_(False))
),
"videos_total": db.scalar(select(func.count()).select_from(Video)),
"pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
),
}
def my_progress(db: Session, user: User) -> tuple[int, int, int, int]:
"""Over the channels this user subscribes to: how many, how many have details, how many their
recent uploads, how many their whole history."""
total, details_synced, recent_synced, all_deep_done = db.execute(
select(
func.count(Channel.id),
func.count(Channel.details_synced_at),
func.count(Channel.recent_synced_at),
func.sum(case((Channel.backfill_done.is_(True), 1), else_=0)),
)
.select_from(Channel)
.join(
Subscription,
and_(Subscription.channel_id == Channel.id, owned_by(Subscription.user_id, user)),
)
).one()
return total or 0, details_synced or 0, recent_synced or 0, int(all_deep_done or 0)
def my_deep_requested(db: Session, user: User) -> list[Channel]:
"""The channels this user asked full history for."""
return list(
db.execute(
select(Channel).join(
Subscription,
and_(
Subscription.channel_id == Channel.id,
owned_by(Subscription.user_id, user),
Subscription.deep_requested.is_(True),
),
)
).scalars()
)
def my_video_count(db: Session, user: User) -> int:
"""The videos of the channels this user subscribes to and has not hidden."""
return (
db.scalar(
select(func.count(Video.id)).join(
Subscription,
and_(
Subscription.channel_id == Video.channel_id,
owned_by(Subscription.user_id, user),
Subscription.hidden.is_(False),
),
)
)
or 0
)
def video_total(db: Session) -> int:
return db.scalar(select(func.count(Video.id))) or 0
def deep_backlog(db: Session) -> list[Channel]:
"""Every channel someone asked full history for that does not have it yet."""
deep_requested = (
select(Subscription.id)
.where(Subscription.channel_id == Channel.id, Subscription.deep_requested.is_(True))
.exists()
)
return list(
db.execute(
select(Channel).where(Channel.backfill_done.is_(False), deep_requested)
).scalars()
)
def work_queue(db: Session) -> dict:
"""What the catalog's jobs still have queued, instance-wide (the scheduler card)."""
return {
# The same selection run_recent_backfill works through, or the card counts channels the
# job will never touch and never reaches zero.
"channels_recent_pending": db.scalar(
select(func.count())
.select_from(Channel)
.where(Channel.recent_synced_at.is_(None), followed_channel())
)
or 0,
"videos_pending_enrich": db.scalar(
select(func.count()).select_from(Video).where(Video.enriched_at.is_(None))
)
or 0,
"videos_pending_shorts": db.scalar(
select(func.count())
.select_from(Video)
.where(Video.shorts_probed.is_(False), Video.enriched_at.is_not(None))
)
or 0,
"videos_live_refresh": db.scalar(
select(func.count())
.select_from(Video)
.where(
or_(
Video.live_status.in_(("live", "upcoming")),
and_(Video.live_status == "was_live", Video.duration_seconds.is_(None)),
)
)
)
or 0,
}
+244
View File
@@ -0,0 +1,244 @@
"""Live YouTube search, the catalog's side: what a page of results leaves in the shared catalog.
A result becomes a catalog row so it renders with the normal feed cards and the in-app player and
gains per-user state. The ingest runs in steps with a YouTube call between each, and each step's
rows are committed before the next call: an enrich that fails rolls back to the stubs, which stay
(the FK targets for what follows, backfilled later). Results are flagged (`videos.via_search`,
`channels.from_search`) so the Library can hide this search-discovered noise; Shorts and
live/upcoming videos are never kept. The HTTP layer (`routes/search.py`) runs the paging, the caps
and the refusals.
"""
from collections.abc import Sequence
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from sqlalchemy import delete, func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app import logs, quota, sysconfig
from app.catalog import queries
from app.catalog.models import LIVE_OR_UPCOMING, Channel, SearchFind, Subscription, Video
from app.models import User
from app.reader import queries as reader_queries
from app.sync.subscriptions import apply_channel_details
from app.sync.videos import apply_video_details, insert_stubs, parse_dt
from app.userscope import owned_by
from app.utils import like_escape
from app.youtube.client import YouTubeClient, YouTubeError
from app.youtube.shorts import make_client, probe_is_short
log = logs.get_logger(__name__)
def classify_shorts(db: Session, videos: Sequence[Video]) -> None:
"""Confirm/deny Shorts for the freshly-ingested videos so none slip into the feed.
Mirrors the scheduler's run_shorts_classification, but scoped to just these rows and run
synchronously (the search is an explicit, awaited action). Non-candidates (too long, no
duration, or live) are finalised without a probe; the rest hit youtube.com/shorts (no
quota). Undetermined probes are left for the scheduler to retry."""
probe_max = sysconfig.get_int(db, "shorts_probe_max_seconds")
to_probe: list[Video] = []
for v in videos:
if v.shorts_probed:
continue
if v.enriched_at is None:
# Enrichment failed (proxy blip) or YouTube omitted the row — duration is unknown,
# so DON'T finalise shorts_probed off a null duration; leave it for the scheduler
# (which only classifies enriched rows) to retry. Otherwise a real Short would leak.
continue
if v.live_status != "none" or v.duration_seconds is None or v.duration_seconds > probe_max:
v.is_short = False
v.shorts_probed = True
else:
to_probe.append(v)
if to_probe:
with make_client() as client:
def work(v: Video):
return v, probe_is_short(client, v.id)
with ThreadPoolExecutor(max_workers=16) as pool:
for v, result in pool.map(work, to_probe):
if result is None:
continue
v.is_short = result
v.shorts_probed = True
db.commit()
def ingest_candidates(
db: Session, yt: YouTubeClient, user: User, candidates: list[dict]
) -> list[str]:
"""Materialise one page of search candidates into the catalog (channel stubs + enrich,
video stubs flagged via_search, enrich, Shorts classification) and return the KEEPABLE
video ids — non-Short, non-live — in the input relevance order. A videos.list that fails
raises its YouTubeError: with nothing enriched, nothing could be kept."""
ids = [it["id"] for it in candidates]
if not ids:
return []
# 1) Channels are the FK target — create stubs for unknown ones (flagged from_search),
# then enrich the new ones (channels.list, 1 unit). Stubs survive an enrich failure.
channel_ids = list({it["channel_id"] for it in candidates})
existing = set(db.scalars(select(Channel.id).where(Channel.id.in_(channel_ids))).all())
new_channel_ids = [c for c in channel_ids if c not in existing]
if new_channel_ids:
titles = {it["channel_id"]: it["channel_title"] for it in candidates}
for cid in new_channel_ids:
db.add(Channel(id=cid, title=titles.get(cid), from_search=True))
db.commit()
try:
with quota.attribute(user.id, quota.QuotaAction.CHANNELS_DISCOVER):
details = yt.get_channels(new_channel_ids)
apply_channel_details(db, details, new_channel_ids)
db.commit()
except YouTubeError:
# The stubs remain and the details can be backfilled later — but until they are, those
# channels sit in the results with no thumbnail and no subscriber count, looking like
# channels that have neither.
log.warning(
"the channels found by a search could not be enriched",
extra=logs.event("search.channel_enrich_failed", channels=len(new_channel_ids)),
)
db.rollback() # details can be backfilled later; the stubs remain
# 2) Video stubs, flagged via_search. Existing rows keep their flags (ON CONFLICT DO
# NOTHING), so an organically-synced video stays organic.
# published_at is NOT NULL (migration 0068); the scrape source exposes no publish date
# (only a relative "3 years ago"), so fall back to now() to satisfy the constraint — the
# enrich in step 3 overwrites it with the video's real snippet date.
now = datetime.now(timezone.utc)
insert_stubs(
db,
[
{
"id": it["id"],
"channel_id": it["channel_id"],
"title": it["title"],
"published_at": parse_dt(it["published_at"]) or now,
"thumbnail_url": it["thumbnail_url"],
"via_search": True,
}
for it in candidates
],
)
# 3) Enrich (videos.list, 1 unit/50) so cards have duration/stats and we can judge live.
videos = db.scalars(select(Video).where(Video.id.in_(ids))).all()
try:
with quota.attribute(user.id, quota.QuotaAction.VIDEOS_ENRICH):
detail_by_id = {it["id"]: it for it in yt.get_videos(ids)}
except YouTubeError:
# An unenriched video cannot be judged live or Short, so none of this page could be kept:
# swallowed, the failure answered as a page with nothing on it — "no results" for a search
# that never ran (the shared budget spent, no API key). The stubs stay (`insert_stubs`
# committed them), as the channels' do above; the caller answers the failure (its refusal
# says which).
log.warning(
"the videos found by a search could not be enriched",
extra=logs.event("search.video_enrich_failed", candidates=len(ids)),
)
raise
enriched_at = datetime.now(
timezone.utc
) # when THIS enrich ran (distinct from the stub placeholder)
for v in videos:
item = detail_by_id.get(v.id)
if item is None:
continue
apply_video_details(v, item)
v.enriched_at = enriched_at
db.commit()
# 4) Confirm/deny Shorts (no quota) so none enter via search.
classify_shorts(db, videos)
keep = {
v.id
for v in videos
if v.enriched_at is not None and not v.is_short and v.live_status not in LIVE_OR_UPCOMING
}
return [vid for vid in ids if vid in keep]
def _fold_term(db: Session, video_ids: list[str], term: str) -> None:
"""Append `term` to each video's space-joined `search_terms`, unless it is already one of them.
"Already one of them" is a whole-term match, as a literal: a substring test skipped `cat` on a
video already found for `category`, and an unescaped one read the `_` in `my_channel` as a
wildcard — either way the term was never folded in, and local search lost that relevance."""
padded = func.concat(" ", func.coalesce(Video.search_terms, ""), " ")
db.execute(
update(Video)
.where(Video.id.in_(video_ids))
.where(padded.notilike(f"% {like_escape(term)} %"))
.values(
search_terms=func.trim(
func.coalesce(Video.search_terms, "").op("||")(" ").op("||")(term)
)
)
)
def new_first(db: Session, user: User, ids: list[str]) -> list[str]:
"""Reorder results so genuinely-new discoveries come first and things the user already has
(a subscribed channel, or a video they've watched / in-progress / saved / playlisted) sink
to the back — keeping the relevance order within each group."""
if not ids:
return ids
have = set(
db.scalars(
select(Video.id).where(
Video.id.in_(ids), Video.channel_id.in_(queries.subscribed_channels(user))
)
)
) | reader_queries.held_by(db, user, ids)
return [i for i in ids if i not in have] + [i for i in ids if i in have]
def record_finds(db: Session, user: User, video_ids: list[str], term: str) -> None:
"""These results as THIS user's search finds (the Mine feed, the Source filter), and the term
folded into each one's `search_terms` so local full-text search inherits YouTube's relevance.
Idempotent."""
db.execute(
pg_insert(SearchFind)
.values([{"user_id": user.id, "video_id": vid} for vid in video_ids])
.on_conflict_do_nothing(index_elements=["user_id", "video_id"])
)
_fold_term(db, video_ids, term)
def clear(db: Session, user: User, video_ids: list[str]) -> int:
"""Discard these results "as if never added": drop this user's finds of them, then delete the
ones nobody kept — a `via_search` video no reader holds (`reader.queries.kept_by_a_reader`),
whose channel nobody subscribes to, and that no other user still has as a find. Returns how
many videos went.
Only the ones this user found: a video in the catalog with no find at all (a related list's,
which records none) is in someone's list right now, and an id any user could name would let
them delete it from under that list."""
mine = list(
db.scalars(
delete(SearchFind)
.where(owned_by(SearchFind.user_id, user), SearchFind.video_id.in_(video_ids))
.returning(SearchFind.video_id)
)
)
if not mine:
return 0
subscribed = select(Subscription.id).where(Subscription.channel_id == Video.channel_id).exists()
other_find = select(SearchFind.id).where(SearchFind.video_id == Video.id).exists()
result = db.execute(
delete(Video).where(
Video.id.in_(mine),
Video.via_search.is_(True),
~reader_queries.kept_by_a_reader(Video.id),
~subscribed,
~other_find,
)
)
return result.rowcount if isinstance(result, CursorResult) else 0
+144
View File
@@ -0,0 +1,144 @@
"""Catalog's writes a request makes: a user's subscription settings, what they explore or block, a
subscription recorded or dropped, a channel's sync markers reset.
Every function here flushes and leaves the commit to its caller (the handler). The YouTube side of a
subscribe or an unsubscribe is the handler's (it needs the user's client and its refusals); what
lands here is the local record of it. The ingest jobs (`sync/`, `catalog/search`) write the
catalog's shared rows themselves.
"""
from datetime import datetime, timezone
from sqlalchemy import delete, update
from sqlalchemy.engine import CursorResult
from sqlalchemy.orm import Session
from app.catalog import queries
from app.catalog.models import BlockedChannel, Channel, ExploredChannel, Subscription, Video
from app.models import User
from app.userscope import owned_by
def remember_own_channel(user: User, channel_id: str) -> None:
"""The user's own YouTube channel, so discovery never offers it to them."""
user.yt_channel_id = channel_id
def update_subscription(
db: Session,
sub: Subscription,
*,
priority: int | None = None,
hidden: bool | None = None,
deep_requested: bool | None = None,
) -> bool:
"""Change the settings given (None leaves one as it is). Returns whether full history was just
turned ON — the caller then pulls the channel now rather than at the scheduler's next tick."""
turned_on = False
if priority is not None:
sub.priority = priority
if hidden is not None:
sub.hidden = hidden
if deep_requested is not None:
# Turning it off won't undo already-fetched videos; it stops further deep paging if nobody
# else still wants it.
turned_on = deep_requested and not sub.deep_requested
sub.deep_requested = deep_requested
db.flush()
return turned_on
def set_deep_all(db: Session, user: User, on: bool) -> int:
"""Every one of the user's channels in (or out of) full-history backfill; how many rows moved."""
result = db.execute(
update(Subscription).where(owned_by(Subscription.user_id, user)).values(deep_requested=on)
)
return result.rowcount if isinstance(result, CursorResult) else 0
def explore(db: Session, user: User, channel: Channel) -> None:
"""Record (or refresh the grace clock of) the user's exploration of a channel, and flag the
channel ephemeral unless someone follows it — the cleanup job's keep rule reads both."""
existing = queries.exploration(db, user, channel.id)
if existing is None:
db.add(ExploredChannel(user_id=user.id, channel_id=channel.id))
else:
existing.created_at = datetime.now(timezone.utc)
if not channel.from_explore and not queries.followed(db, channel.id):
channel.from_explore = True
db.flush()
def reset_backfill(db: Session, channel: Channel) -> None:
"""Fetch the channel again from scratch: its backfill markers cleared, and full history asked
for on EVERY subscription to it — the deep scheduler gates on an EXISTS over those, so the admin
who resets it need not subscribe themselves."""
channel.backfill_done = False
channel.backfill_cursor = None
channel.recent_synced_at = None
db.execute(
update(Subscription)
.where(Subscription.channel_id == channel.id)
.values(deep_requested=True)
)
db.flush()
def record_subscription(db: Session, user: User, channel: Channel, yt_subscription_id: str) -> None:
"""The subscription YouTube now holds, recorded here. Subscribing is the "keep" signal: an
explored channel becomes an organic one, so the cleanup job leaves it alone and its videos join
the shared Library like any subscribed channel's uploads."""
db.add(
Subscription(
user_id=user.id,
channel_id=channel.id,
yt_subscription_id=yt_subscription_id or None,
subscribed_at=datetime.now(timezone.utc),
)
)
db.flush()
settle_explored(db, channel)
def settle_explored(db: Session, channel: Channel) -> None:
"""Hold the rule that a channel's videos are an exploration's only while nobody follows it,
decided by who follows it now, not by the flag held in memory: a subscribe in another request
may have cleared that while YouTube answered this one, and the videos that came in meanwhile
would stay flagged. A channel someone follows, however the follow came (here or pulled in from
YouTube), is the catalog's own, and so are its videos."""
if channel.from_explore and not queries.followed(db, channel.id):
return
channel.from_explore = False
db.execute(
update(Video)
.where(Video.channel_id == channel.id, Video.via_explore.is_(True))
.values(via_explore=False)
)
db.flush()
def drop_subscription(db: Session, sub: Subscription) -> None:
db.delete(sub)
db.flush()
def block(db: Session, user: User, channel_id: str) -> None:
"""Block a channel for this user (idempotent), and end any exploration of it so the cleanup job
can reclaim it."""
if queries.is_blocked(db, user, channel_id):
return
db.add(BlockedChannel(user_id=user.id, channel_id=channel_id))
db.execute(
delete(ExploredChannel).where(
owned_by(ExploredChannel.user_id, user), ExploredChannel.channel_id == channel_id
)
)
db.flush()
def unblock(db: Session, user: User, channel_id: str) -> None:
db.execute(
delete(BlockedChannel).where(
owned_by(BlockedChannel.user_id, user), BlockedChannel.channel_id == channel_id
)
)
+83
View File
@@ -0,0 +1,83 @@
"""Alphabetical order in the reader's language.
Why not the database default: the Postgres image is Alpine, and musl's `en_US.utf8` collation is BYTE
order — every accented initial sorted after "Z" ("Zorró < Életrajzi < Ötödik"), in every A→Z list the
server orders. Postgres ships the ICU collations on the same image (dev, test, prod alike), so a text
sort names one: Hungarian puts "Ö" after "O" and "Cs" after "Cu", English folds the accents in.
`collated` wraps an ORDER BY expression — and because a keyset cursor compares with the SAME expression
(`feedquery.sort_keys`), the cursor pages in the same order it sorted in. `sorted_words` orders a Python
list the same way through the database, so a list built in Python matches one built in SQL exactly.
The feed's title sorts ride indexes built in these collations (migration 0096), and an index keeps the
order of the ICU that built it. `postgres:16-alpine` is a floating tag: an image update can bring a newer
ICU, after which those indexes disagree with the comparisons a query makes (a keyset page skips or
repeats a title). `heal_drift` rebuilds them at startup once Postgres reports the version moved."""
from typing import Any
from sqlalchemy import Engine, SQLColumnExpression, column, func, select, text, values
from sqlalchemy.orm import Session
from sqlalchemy.sql.elements import ColumnElement
from sqlalchemy.types import String
from app import logs
from app.mailtext import normalize
log = logs.get_logger(__name__)
ICU: dict[str, str] = {"en": "en-x-icu", "hu": "hu-x-icu"}
def collation(lang: str | None) -> str:
"""The ICU collation for a UI language; anything unknown sorts as English."""
return ICU[normalize(lang)]
def collated(expr: SQLColumnExpression[Any], lang: str | None) -> ColumnElement[Any]:
"""`expr` compared in the reader's alphabet. Case-folded, so "apple" and "Apple" sit together."""
return func.lower(expr).collate(collation(lang))
def sorted_words(db: Session, words: list[str], lang: str | None) -> list[str]:
"""`words` in the reader's alphabetical order, by the same ICU collation as `collated`."""
if not words:
return []
w = values(column("w", String), name="words").data([(x,) for x in words])
return list(db.scalars(select(w.c.w).order_by(collated(w.c.w, lang), w.c.w)))
def heal_drift(engine: Engine) -> list[str]:
"""Rebuild every index on an ICU collation whose library version moved, then record the new version.
Returns the rebuilt indexes. CONCURRENTLY: the feed keeps reading while it runs."""
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
drifted = conn.execute(
text(
"SELECT oid, collname FROM pg_collation WHERE collname = ANY(:names) "
"AND collversion IS DISTINCT FROM pg_collation_actual_version(oid)"
),
{"names": list(ICU.values())},
).all()
if not drifted:
return []
indexes = list(
conn.execute(
text(
"SELECT DISTINCT d.objid::regclass::text FROM pg_depend d "
"JOIN pg_class c ON c.oid = d.objid AND c.relkind = 'i' "
"WHERE d.classid = 'pg_class'::regclass "
"AND d.refclassid = 'pg_collation'::regclass AND d.refobjid = ANY(:oids)"
),
{"oids": [row.oid for row in drifted]},
).scalars()
)
log.warning(
"ICU collation version changed (%s): rebuilding %d indexes",
", ".join(row.collname for row in drifted),
len(indexes),
)
for name in indexes:
conn.execute(text(f"REINDEX INDEX CONCURRENTLY {name}"))
for row in drifted:
conn.execute(text(f'ALTER COLLATION "{row.collname}" REFRESH VERSION'))
return indexes
+125 -26
View File
@@ -1,6 +1,7 @@
from pathlib import Path
from typing import Literal, get_args, get_origin
from pydantic import model_validator
from pydantic import AliasChoices, Field, ValidationInfo, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
# Shipped placeholder; production must override it. Used by the startup guard below.
@@ -15,11 +16,40 @@ def _version_from_file() -> str:
v = p.read_text(encoding="utf-8").strip()
if v:
return v
# silent-ok: this runs at import, and `app.logs` imports THIS module — a log call here
# would be a circular import. The loop tries the other location, then falls back to "dev".
except OSError:
continue
return "dev"
# Closed value sets. A setting typed with one of these is a CHOICE: pydantic rejects anything else at
# boot, and sysconfig derives the admin picker's options from the same Literal — so the list lives in
# exactly one place and a consumer never has to guess what a typo should fall back to.
SearchSource = Literal["scrape", "api"]
DownloadLayout = Literal["plex", "flat"]
LogLevel = Literal["TRACE", "DEBUG", "INFO", "WARNING", "ERROR"]
LogFormat = Literal["json", "text"]
TranscodeHw = Literal["auto", "on", "off"]
# libx264's presets, fastest first (`placebo` left out on purpose: no measurable gain for its cost).
TranscodePreset = Literal[
"ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow"
]
def choices_of(annotation: object) -> tuple[str, ...]:
"""The allowed values of a choice-typed setting, or () for any other type."""
return tuple(get_args(annotation)) if get_origin(annotation) is Literal else ()
def canonical_choice(options: tuple[str, ...], raw: object) -> str | None:
"""The option `raw` names, in its canonical spelling (any letter case, surrounding space
ignored), or None. The ONE matching rule for a choice value from the env and from the admin
API, so a spelling one accepts the other cannot refuse."""
folded = str(raw).strip().casefold()
return next((option for option in options if option.casefold() == folded), None)
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
@@ -57,7 +87,7 @@ class Settings(BaseSettings):
# VPS Caddy front door. A request from any other peer (someone hitting the app port directly)
# is NOT trusted and its raw socket IP is used, so it can't spoof its rate-limit identity via
# XFF. Empty = trust no proxy (use the direct peer) — correct for a directly-exposed / local-dev
# deploy. Comma-separated. See routes.auth._client_ip + the deploy docs.
# deploy. Comma-separated. See ratelimit.client_key + the deploy docs.
trusted_proxy_ips: str = ""
# --- Outbound email (onboarding: access-request + approval notices) ---
@@ -83,12 +113,20 @@ class Settings(BaseSettings):
# each search.list costs 100 units, so the shared budget only affords ~80-90/day total; with
# the zero-quota scrape source it's a pure anti-abuse rate limit. Admin-tunable.
search_daily_limit_per_user: int = 70
# Per-user cap on related-video requests per day — a list opened and each "Show more" (the
# InnerTube watch page, no quota; its enrich is up to ~8 units: up to 4 pages, each a videos.list
# and a channels.list). A failed one gives its slot back. 0 turns related videos off.
# Admin-tunable.
related_daily_limit_per_user: int = 200
# Source for live YouTube search: "scrape" (InnerTube, no API quota) or "api" (search.list,
# 100 units/page). Scrape is the default; flip to "api" if scraping is blocked/breaks.
search_source: str = "scrape"
search_source: SearchSource = "scrape"
# Recent-first backfill: how far back to fetch on the first pass per channel.
backfill_recent_max_videos: int = 100
backfill_recent_max_days: int = 365
# A channel whose RSS feed has not answered for this many hours gets its newest uploads read
# through the Data API instead (1 unit), at most once per this many hours. 0 = never.
rss_fallback_hours: int = 2
# Shorts are confirmed by probing youtube.com/shorts/<id>. Only videos at or below
# this duration (and not livestreams) are probed; longer videos are never Shorts.
@@ -113,8 +151,6 @@ class Settings(BaseSettings):
autotag_interval_minutes: int = 30
subscriptions_resync_minutes: int = 360
playlist_sync_minutes: int = 360
# How often the shared demo account is auto-reset to a clean baseline (communal sandbox).
demo_reset_minutes: int = 720
# --- Maintenance / validation job ---
# Runs once a day by default (admin-tunable via the Scheduler dashboard). Grace periods
# are in days because that's the granularity that matters; the rolling re-validation
@@ -140,9 +176,13 @@ class Settings(BaseSettings):
feed_default_hidden_live: str = "live,upcoming"
# --- Download center (yt-dlp) ---
# Root directory (inside the container) for downloaded media — a Docker volume/bind mount.
# Root directory (inside the container) for the files library — a Docker volume/bind mount.
# Env-only (a mount path can't live in the DB). The worker lays a Plex-style tree under it.
download_root: str = "/downloads"
# DOWNLOAD_ROOT is its old name: an install still on the old compose file (mounted at
# /downloads, DOWNLOAD_ROOT=/downloads) keeps finding its files after it pulls a new image.
files_root: str = Field(
default="/files", validation_alias=AliasChoices("FILES_ROOT", "DOWNLOAD_ROOT", "files_root")
)
# Whether THIS process runs the download worker loop. On for the dedicated worker
# container, off for the API (which keeps the scheduler). Mirrors scheduler_enabled.
worker_enabled: bool = False
@@ -150,12 +190,18 @@ class Settings(BaseSettings):
# for YouTube Proof-of-Origin tokens (bypasses bot-detection + unlocks high-quality
# formats). Empty disables it (yt-dlp still works, but may hit "confirm you're not a bot").
download_pot_base_url: str = "http://bgutil-pot:4416"
# yt-dlp player clients (comma-separated). PRIMARY = android_vr: high-quality DASH, no PO
# token / JS runtime needed, reliable for normal use. On a bot/DRM/format failure (YouTube's
# per-client responses are flaky, and android_vr can get bot-flagged under heavy load) the
# worker retries with the FALLBACK set — the POT-capable web clients (web needs the token +
# Deno) plus android for audio. Both admin-tunable when YouTube shifts what works.
download_player_clients: str = "android_vr"
# yt-dlp player clients (comma-separated). PRIMARY = tv_simply: measured 2026-10-10 as the one
# client that is offered 1080p+ AND finishes the fetch, its stream token minted by the bgutil
# sidecar. android_vr (the primary until then) was still offered 2160p, but every fetch died
# with HTTP 403 — it now needs a stream PO token the sidecar cannot mint — and the fallback
# below served 360p only, so every download came out at 360p without failing. On a
# bot/DRM/format failure the worker retries with the FALLBACK set — the POT-capable web clients
# (web needs the token + Deno) plus android, whose 360p is the last resort. Both admin-tunable
# (env) when YouTube shifts what works.
# A SIGNED-IN download (the user added YouTube cookies) keeps only the cookie-capable clients
# of these two sets — yt-dlp drops the rest, so android_vr / android / ios never run there.
# See adapters.YouTubeAdapter._COOKIE_CAPABLE.
download_player_clients: str = "tv_simply"
download_player_clients_fallback: str = "web_safari,web,android"
@staticmethod
@@ -208,10 +254,14 @@ class Settings(BaseSettings):
download_default_max_bytes: int = 5_368_709_120 # 5 GiB
download_default_max_concurrent: int = 2
download_default_max_jobs: int = 50
# Per-file cap for local uploads (E7). Unlike a download (whose size is unknown until the worker
# Per-file cap for local uploads. Unlike a download (whose size is unknown until the worker
# finishes), an upload's size is known at ingest, so this bounds a single file directly; the
# user's overall byte footprint is still governed by download_default_max_bytes.
download_upload_max_bytes: int = 524_288_000 # 500 MiB
# ge=1: this number is also the EDGE ceiling (`bodylimit.ROUTE_MAX_BODY_BYTES`), where 0 is a
# real limit of zero rather than the route's "no per-file cap" sentinel — an instance set to 0
# would refuse every upload, including the admin page's own way back. To lift the cap in
# practice, set a large number.
download_upload_max_bytes: int = Field(default=524_288_000, ge=1) # 500 MiB
# Retention: an asset expires this many days after its last access; the GC job then
# deletes it (once no job references it) after an additional grace window.
download_retention_days: int = 30
@@ -220,10 +270,32 @@ class Settings(BaseSettings):
# Admin audit log: how long to keep rows (days; 0 = keep forever) + how often the GC runs.
audit_retention_days: int = 180
audit_gc_minutes: int = 1440
# YouTube quota history (quota_events): days kept (0 = forever). A year, so the admin usage view's
# longest window (365 days) stays whole; the same daily cleanup job prunes it.
quota_retention_days: int = 365
# Background-run history (job_runs): days kept (0 = forever). The audit log's 180, which is how
# long a run lived there before it moved here: ~1.8k rows a day at the default intervals
# (measured 2026-10-17), ~320k at the limit. The same daily cleanup job prunes it.
job_run_retention_days: int = 180
# On-disk layout: "plex" (Channel/Season YYYY/… [id].ext + .nfo/poster) or "flat".
download_layout: str = "plex"
# Default SponsorBlock (skip sponsor segments) for new custom profiles.
sponsorblock_default: bool = False
download_layout: DownloadLayout = "plex"
# --- Diagnostics / logging ---
# BOOTSTRAP level only: the running level is the `log_level` admin knob (sysconfig), applied at
# startup and on change without a restart. This is what a process logs at before it can reach
# the database — and the only knob the worker/runner children read, since they are short-lived.
# TRACE (below DEBUG) is the "Teljes" diagnostics level; see app/logs.py.
log_level: LogLevel = "INFO"
# "json" (one object per line — what Loki and the log_events sink ingest) or "text" (a readable
# line for a local terminal). Redaction applies to both.
log_format: LogFormat = "json"
# Also write events to the `log_events` table (app/logsink.py): WARNING and above, the request
# lines, and — once they exist — client batches. Off only where no table is wanted (the tests).
log_db_sink: bool = True
# Days a `log_events` row is kept (the privacy policy promises this), and how often the log_gc job
# enforces it — hourly, so a row never outlives the promise by more than an hour.
log_retention_days: int = 7
log_gc_minutes: int = 60
# --- Plex integration (optional) ---
# The Plex module plays the LOCAL physical media file directly (our own ffmpeg/direct-serve);
@@ -241,7 +313,7 @@ class Settings(BaseSettings):
plex_path_map: str = ""
# Which Plex library section keys are exposed (comma-separated). Empty = all enabled sections.
plex_libraries: str = ""
# HU-overlay mapping (E5 S5): comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# HU-overlay mapping: comma-separated `huSectionKey:enSectionKey` pairs. Each named HU
# section is synced as a non-browsable overlay whose Hungarian metadata is written onto the EN
# twin section's rows (matched by guid). Empty = no overlays. E.g. "3:4,5:1".
plex_overlay_map: str = ""
@@ -251,7 +323,7 @@ class Settings(BaseSettings):
plex_sync_interval_min: int = 360
# How often the incremental Plex→Siftlode watch-state reconcile runs, in minutes (Phase C):
# a cheap history + on-deck pull plus a re-push of any states that failed their immediate push.
# 15 min (S2): with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# 15 min: with Plex now the source of truth (prompt write-back on the ~10s checkpoint +
# Plex-authoritative pull), a faster incremental pull keeps Siftlode close to Plex-side changes.
plex_watch_sync_interval_min: int = 15
# How often the FULL watch-state reconcile runs, in minutes (Phase C). A whole-section rescan
@@ -262,11 +334,15 @@ class Settings(BaseSettings):
# re-encode is CPU-heavy (the VAAPI hardware path is far cheaper and could sustain more); direct-serve
# and remux (stream-copy) are not capped by this.
plex_max_transcodes: int = 4
# --- Transcode (playable="transcode") re-encode tuning (E6 S1) ---
# Max concurrent playback sessions of ANY kind — remux (stream-copy) included. Cheap on the CPU but
# each one is a whole segmented copy of the film on the scratch volume, written faster than
# realtime. One account holds at most two regardless (plex/stream.py `_MAX_SESSIONS_PER_USER`).
plex_max_sessions: int = 8
# --- Transcode (playable="transcode") re-encode tuning ---
# Hardware-encoder policy: "auto" = VAAPI (h264_vaapi) when the iGPU render node is present AND the
# source codec is hardware-decodable, else libx264 software; "on" = force VAAPI wherever the render
# node exists (skip the decodability gate); "off" = always software. See plex/stream.py `_select_encode`.
plex_transcode_hw: str = "auto"
plex_transcode_hw: TranscodeHw = "auto"
# Software-fallback libx264 tuning (IGNORED on the VAAPI path). `preset` trades CPU for size; `crf`
# trades quality for size (lower = better + bigger). `max_height` caps the SOFTWARE transcode's output
# height (0 = keep source) so a CPU re-encode of a tall source stays affordable — the GPU path always
@@ -276,16 +352,16 @@ class Settings(BaseSettings):
# call, not a fix. Only affects files that go through remux/transcode — a direct-played file is
# served untouched, but those are browser-native stereo anyway, not the 5.1 mixes this is for.
plex_audio_loudnorm: bool = False
plex_transcode_preset: str = "veryfast"
plex_transcode_preset: TranscodePreset = "veryfast"
plex_transcode_crf: int = 20
plex_transcode_max_height: int = 1080
# Where on-the-fly HLS remux segments are written (transient, reaped). Empty → a `.plex-hls`
# dir under download_root. On localdev the download_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's download_root is fast
# dir under files_root. On localdev the files_root is a slow Windows bind-mount, so point
# this at a fast container-local path (e.g. /var/tmp/plex-hls); prod's files_root is fast
# local disk, so the default is fine there.
plex_hls_dir: str = ""
# --- TMDB enrichment (E5 S2; optional — enriches the Plex catalog from TMDB) ---
# --- TMDB enrichment (optional — enriches the Plex catalog from TMDB) ---
# TMDB API key (v3 api_key, 32 hex chars). Server-side only; stored encrypted when set via the
# admin Config page. Empty → the TMDB enrichment job no-ops. Feeds keywords / crew / budget /
# revenue / TMDb score / recommendations / backdrops onto the mirrored Plex rows, matched by the
@@ -294,10 +370,22 @@ class Settings(BaseSettings):
tmdb_api_base: str = "https://api.themoviedb.org/3"
# Metadata language for TMDB fetches (keyword/overview locale). Stored EN by default.
tmdb_language: str = "en-US"
# --- Bug reports: where a debug session's issue goes. All three empty = no tracker; the
# composer still exports a zip. The repository must be PRIVATE — an issue carries the reporter's
# log — and app/issues/gitea.py refuses a public one. Runtime knobs (sysconfig, admin UI). ---
issues_gitea_url: str = ""
issues_gitea_repo: str = ""
issues_gitea_token: str = ""
# How often the TMDB enrichment job fetches a batch of not-yet-enriched Plex titles, in minutes.
tmdb_enrich_interval_min: int = 120
# Titles fetched per TMDB enrichment run (TMDB is fast + generously rate-limited).
tmdb_enrich_batch: int = 100
# How often the people job fetches a batch of person bios from TMDB and portraits through Plex,
# in minutes, and how many of each per run: ~18k linked people at 300 per 10 min is about ten hours
# once, then only the newcomers.
people_enrich_interval_min: int = 10
people_enrich_batch: int = 300
@property
def allowed_email_set(self) -> set[str]:
@@ -325,6 +413,17 @@ class Settings(BaseSettings):
https OAUTH_REDIRECT_URL as the signal for 'real deployment' vs local dev."""
return self.oauth_redirect_url.lower().startswith("https://")
@field_validator("*", mode="before")
@classmethod
def _canonical_choice(cls, value: object, info: ValidationInfo) -> object:
"""A choice setting accepts its value in any letter case (`LOG_LEVEL=debug`) and stores the
canonical spelling; an unknown value still fails the boot with pydantic's own message."""
field = cls.model_fields.get(info.field_name or "")
options = choices_of(field.annotation) if field else ()
if options and isinstance(value, str):
return canonical_choice(options, value) or value
return value
@model_validator(mode="after")
def _enforce_production_secrets(self) -> "Settings":
"""Fail fast rather than boot a public instance with the shipped placeholder
+55 -2
View File
@@ -1,6 +1,6 @@
from collections.abc import Iterator
from sqlalchemy import create_engine
from sqlalchemy import create_engine, text
from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
from app.config import settings
@@ -12,7 +12,9 @@ class Base(DeclarativeBase):
# Pool sized above the sync-endpoint threadpool (~40) so a burst of short DB sessions can't starve on
# checkout. The image proxies deliberately DON'T hold a connection during their slow external fetch
# (see routes/plex/), so this headroom is only for the many brief sessions, not long ones.
# (see routes/plex/), so this headroom is only for the many brief sessions, not long ones. The one
# long hold — a playlist sync's advisory claim, for a whole push — opens its own unpooled connection
# (sync/playlists.py `_claim_engine`) instead of drawing a third one from here.
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
@@ -20,6 +22,10 @@ engine = create_engine(
max_overflow=30,
pool_timeout=30,
future=True,
# A database error renders its bound parameters into its message — a title, an uploader, a
# file path, a URL — and that message is logged and stored as a job's error text. The SQL
# stays; the values are data, and the ids an operator needs are on the line already.
hide_parameters=True,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False)
@@ -30,3 +36,50 @@ def get_db() -> Iterator[Session]:
yield db
finally:
db.close()
def release_read_connection(db: Session) -> None:
"""End a request's READ transaction so its pooled connection goes back BEFORE a long await.
SQLAlchemy 2.0 autobegins on the first SELECT, so a request that has merely authenticated is
already holding one of the connections above — `idle in transaction` to Postgres, and nothing
times it out. A route that then awaits something slow (a request body still arriving, an
external fetch) pins it for the whole wait, which is precisely the invariant the pool is sized
on in the comment above. Measured: the next checkout after this call reuses the very backend
that was released, and a 1.8 s upload goes from one sustained `idle in transaction` to none.
Every route that reads its own body from a dependency needs this, and `test_body_limit.py`
holds that — the three that do are the ones with a raised (or self-imposed) body ceiling, and
they got here one at a time, which is how the upload route ended up the only one fixed.
The caller must have NOTHING PENDING, and that is checked rather than described. The session is
shared with every dependency that ran before, so a rollback here would silently discard a write
one of them staged — `service.enqueue` already learned that lesson the expensive way and uses a
SAVEPOINT for it. CLAUDE.md: a rule stated in a comment must also be enforced in code.
TWO checks, because the obvious one is blind to the shapes this codebase actually writes in.
`db.new`/`deleted`/`dirty` hold only UNFLUSHED unit-of-work: the moment a helper calls
`db.flush()` — CLAUDE.md's canonical split is "the handler commits; helpers flush" — the object
is persistent with no unflushed history and all three sets are empty again, and Core DML
(`db.execute(update(...))`) never enters them at all. Measured: after `add`+`flush` the sets are
empty, the old check passed, and the rollback lost the row. So the transaction is asked
directly. Postgres assigns a transaction id on the first WRITE and never for a read, so
`pg_current_xact_id_if_assigned()` is NULL for exactly the state this function is safe in — and
it sees a raw `text("UPDATE …")` too, which no session-level bookkeeping would.
"""
if db.new or db.deleted or db.dirty:
raise RuntimeError(
"release_read_connection() with pending work: something ahead of this dependency "
"staged a write, which ending the transaction would discard. Commit it first, or do "
"it after the await."
)
if (
db.in_transaction()
and db.execute(text("SELECT pg_current_xact_id_if_assigned()")).scalar() is not None
):
raise RuntimeError(
"release_read_connection() on a transaction that has already WRITTEN: something ahead "
"of this dependency flushed a row or ran an UPDATE/INSERT/DELETE, which ending the "
"transaction would discard. Commit it first, or do it after the await."
)
db.rollback()
+484
View File
@@ -0,0 +1,484 @@
"""Debug sessions: Start → reproduce → Stop, and the evidence a bug report is built from.
While a session is open its user's browser names it in `X-Debug-Session` on every request. The auth
dependency accepts the header only for the caller's OWN open session (anything else is ignored, never
an error — a stale tab must not break the app) and then logs that request at the session's level with
every line tagged by its id; the browser tags its own events the same way. The issue is later built
from exactly the rows carrying the tag, so what the review list shows is what gets sent.
Starting a session IS the consent for its level, for its duration: the start dialog carries the
consent text. It needs no change to the user's own diagnostics level, and none is made — when the
session ends the level in force is simply the user's own again.
A session is capped at `MAX_MINUTES`, set by the server at Start. An open one past its cap is over
whether or not anything has closed it yet: it reads as over at once, a new Start closes it, and the
hourly log job sweeps the rest. Sessions go with the logs they point at, after `log_retention_days`.
Core layer: imports `models`, `diagnostics`.
"""
import secrets
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from typing import Literal, get_args
from uuid import uuid4
from sqlalchemy import and_, delete, func, not_, select, update
from sqlalchemy.orm import Session, defer
from app.userscope import owned_by, owned_row
from app import diagnostics
from app.models import DebugScreenshot, DebugSession, LogEvent, User
HEADER = "x-debug-session"
MAX_MINUTES = diagnostics.SESSION_MINUTES
SCREENSHOT_MAX_BYTES = 10 * 1024 * 1024
SCREENSHOTS_MAX = 8
# What one user may keep in screenshots across ALL their sessions (until retention takes them): the
# per-session cap alone would let Start → 8 images → Stop repeat without end.
SCREENSHOT_USER_MAX_BYTES = 100 * 1024 * 1024
# The screenshot types kept, and the extension each is exported and sent under — one table, so an
# accepted type always has a name.
IMAGE_TYPES = {"image/png": "png", "image/jpeg": "jpg", "image/webp": "webp"}
# What a screenshot is called in the zip and on the issue. `issues/bundle.py` builds the names from
# this and `IMAGE_TYPES`; this module and `issues/gitea.py` read them back to tell a screenshot's
# link from a log's. One definition, because three copies of a name is how the next rename breaks
# `images_pending` into silence.
SCREENSHOT_PREFIX = "screenshot-"
# Rows an issue can carry. A 60-minute session at Teljes is a few thousand rows; past this the
# bundle says how many it left out rather than growing without bound.
BUNDLE_MAX = 5_000
SessionLevel = Literal["DEBUG", "TRACE"]
SESSION_LEVELS: tuple[str, ...] = get_args(SessionLevel)
# Detail order of the diagnostics levels: a session raises the level in force, never lowers it.
_DETAIL = {"INFO": 0, "DEBUG": 1, "TRACE": 2}
_ID_LEN = 16
class EndReason(StrEnum):
STOPPED = "stopped"
EXPIRED = "expired"
def is_live(session: DebugSession, now: datetime | None = None) -> bool:
"""Open and within its cap — THE liveness rule: an open session past its cap is over."""
return session.ended_at is None and session.until > (now or datetime.now(timezone.utc))
def open_session(db: Session, user: User) -> DebugSession | None:
"""The user's live session, or None. Read-only: one past its cap is over and reads as None;
`start` and the hourly sweep are what close it."""
session = _unclosed(db, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
def _unclosed(db: Session, user: User) -> DebugSession | None:
return db.scalar(
select(DebugSession).where(
owned_by(DebugSession.user_id, user), DebugSession.ended_at.is_(None)
)
)
def start(db: Session, user: User, level: SessionLevel) -> DebugSession:
"""Open a session, or return the live one — a second tab pressing Start joins the recording
rather than ending it. One past its cap is closed first: the one-open-per-user index would
refuse the new row beside it. Flushes; the caller commits. Two concurrent Starts can still both
get here: the index refuses the second, and the caller joins the one that won."""
now = datetime.now(timezone.utc)
current = _unclosed(db, user)
if current is not None:
if is_live(current, now):
return current
end(current, EndReason.EXPIRED)
db.flush()
session = DebugSession(
id=secrets.token_hex(_ID_LEN // 2),
user_id=user.id,
level=level,
started_at=now,
until=now + timedelta(minutes=MAX_MINUTES),
)
db.add(session)
db.flush()
return session
def stop(session: DebugSession) -> None:
"""The user's Stop. Past its cap the session had already ended on its own, and a late Stop must
not rename that."""
end(session, EndReason.STOPPED if is_live(session) else EndReason.EXPIRED)
def end(session: DebugSession, reason: EndReason) -> None:
"""Close a session. An expired one is dated at its cap, not at whenever it was noticed."""
if session.ended_at is not None:
return
session.ended_at = (
session.until
if reason is EndReason.EXPIRED
else min(datetime.now(timezone.utc), session.until)
)
session.ended_reason = reason
def from_header(db: Session, user: User, value: str | None) -> DebugSession | None:
"""The session a request names, if it is the caller's own and still live; else None."""
if not value or len(value) != _ID_LEN or not all(c in "0123456789abcdef" for c in value):
return None
session = owned_row(db, DebugSession, value, user)
if session is None or not is_live(session, datetime.now(timezone.utc)):
return None
return session
def level_for(user: User, session: DebugSession | None) -> tuple[str, datetime | None]:
"""The diagnostics level (and its expiry) a request is logged at: the user's own, raised to the
session's while one is named — whichever records more.
A named session also caps the expiry at its own: this pair is what the process tracks under the
session (app/logs.py), and an expiry outliving the recording would hold the root logger down
after it ended."""
own = diagnostics.effective_level(user)
until = user.diagnostics_until if own != diagnostics.BASE else None
if session is None:
return own, until
if _DETAIL[session.level] <= _DETAIL[own]:
return own, min(until, session.until) if until else session.until
return session.level, session.until
def lock(db: Session, session: DebugSession) -> None:
"""Re-read the session under a row lock, held until the caller commits. Everything that changes
what an issue is made of — the draft, the screenshots, the send itself — takes it first: a send
in flight sees none of them change under it, and each of them sees a send that committed."""
db.refresh(session, with_for_update=True)
def save_draft(session: DebugSession, draft: dict) -> None:
"""Keep the composer's draft (the caller validated, locked and commits). Shape: `title`, `body`
(the rendered markdown an export or an issue carries), `excluded` (log row ids taken out of the
evidence) and `form` (the composer's own fields, to reopen it as it was). A delivery's progress
is not the composer's to replace, and is kept."""
progress = (session.draft or {}).get("delivery")
session.draft = {**draft, "delivery": progress} if progress else draft
# --- a delivery's progress -----------------------------------------------------------------
# Written by app/issues/gitea.py into the draft; read HERE, so the send's decision and what the
# composer shows are one reading of it.
def delivery(session: DebugSession) -> dict:
return dict((session.draft or {}).get("delivery") or {})
def not_uploaded(session: DebugSession) -> list[str]:
"""The files a send that stopped half-way has still to upload — by name, as the issue carries
them. What the composer names to the user."""
return list(delivery(session).get("pending") or [])
def images_pending(session: DebugSession) -> bool:
"""Whether the issue's body still has to be edited to show the screenshots it already carries
(every upload went, the edit that shows them did not)."""
state = delivery(session)
shown = state.get("images_shown")
return not shown and any(n.startswith(SCREENSHOT_PREFIX) for n in state.get("links") or {})
def unfinished(session: DebugSession) -> bool:
"""Whether a send owes the issue anything — the retry's question."""
return bool(not_uploaded(session)) or images_pending(session)
# A send claims the session while it talks to the tracker, instead of holding the row's lock across
# those calls: a Stop, an autosave or a screenshot would queue behind a lock for the whole send.
# They see the claim and are refused at once instead. A claim not renewed for this long was left by
# a request that died mid-send (the process restarted), and the next send takes it over — which is
# why a live send RENEWS it every time it reports progress (`set_delivery`). A claim is a lease, and
# a send slower than the lease must not lose it while it is still running.
SEND_CLAIM_SECONDS = 600
def _claim(session: DebugSession) -> dict:
claim = delivery(session).get("sending")
return claim if isinstance(claim, dict) else {}
def _stamp(token: str) -> dict:
return {"at": datetime.now(timezone.utc).isoformat(), "by": token}
def sending(session: DebugSession) -> bool:
"""Whether a send is under way right now (a live claim)."""
at = _claim(session).get("at")
if not at:
return False
return (datetime.now(timezone.utc) - datetime.fromisoformat(at)).total_seconds() < (
SEND_CLAIM_SECONDS
)
def claim_send(session: DebugSession) -> str:
"""Mark the session as being sent, and return the token that OWNS this claim. The caller holds
the lock and commits, which is what makes the claim visible to everyone else."""
token = uuid4().hex
_write_delivery(session, {**delivery(session), "sending": _stamp(token)})
return token
def release_send(session: DebugSession, token: str) -> None:
"""Give back our OWN claim, and only ours: one that aged out and was taken over belongs to the
send holding it now, and handing that one back would let a third send in beside it."""
if _claim(session).get("by") != token:
return
_write_delivery(session, {k: v for k, v in delivery(session).items() if k != "sending"})
class SendClaimLost(Exception):
"""This send's claim is no longer on the row: it aged out and another send took it over. The
send must stop — what it uploads from here it can no longer record, so a retry would send it
again — and it must not write, because the row now holds the other send's progress."""
def claim_owner(db: Session, session: DebugSession) -> str | None:
"""Who holds the row's claim RIGHT NOW, read from the ROW and not from the object.
`expire_on_commit` is off (app/db.py), so a checkpoint does not re-read: a send that stalled
past its lease still carries its own dead token in memory long after another send took over.
A targeted scalar, not `lock`/`db.refresh` — a refresh here would discard the `issue_url` that
`gitea.deliver` assigns one line before its first progress report, and that link is the one
thing the send must not lose."""
return db.scalar(
select(DebugSession.draft["delivery"]["sending"]["by"].as_string()).where(
DebugSession.id == session.id
)
)
def set_delivery(db: Session, session: DebugSession, state: dict, token: str) -> None:
"""Write a delivery's progress (app/issues/gitea.py keeps it; this module reads it), and renew
the claim while doing it — a send still reporting progress is still alive, so one slower than
the lease cannot expire while it is still running.
The claim is not the caller's to carry: `gitea.deliver` threads ONE snapshot of this dict
through a whole send, so a progress report that carried the claim wrote a stale one back at
every step. It is stamped from `token` here instead, and only after the ROW confirms the claim
is still ours. Without that check the write is a full-column overwrite with no WHERE: a send
that lost its lease came back and stomped the take-over send's committed links, then put its
own dead token back on an idle session. `release_send` has always made
the same check; this is the other half of it."""
if claim_owner(db, session) != token:
raise SendClaimLost(session.id)
state = {k: v for k, v in state.items() if k != "sending"}
state["sending"] = _stamp(token)
_write_delivery(session, state)
def _write_delivery(session: DebugSession, state: dict) -> None:
# A new dict, so the JSONB column sees the change.
session.draft = {**(session.draft or {}), "delivery": state}
def serialize(session: DebugSession | None, screenshots: int = 0) -> dict | None:
if session is None:
return None
return {
"id": session.id,
"level": session.level,
"started_at": session.started_at.isoformat(),
"until": session.until.isoformat(),
"ended_at": session.ended_at.isoformat() if session.ended_at else None,
"ended_reason": session.ended_reason,
"issue_url": session.issue_url,
# A send that stopped half-way: the files it has still to upload, and whether the issue's
# body has still to be edited to show the screenshots it already carries. Both are true of a
# send that is RUNNING too — every step commits its progress — so `sending` goes with them:
# without it a reload or a second tab renders a live send as a failed one, and offers a
# retry that is certain to be refused.
"pending_files": not_uploaded(session),
"images_pending": images_pending(session),
"sending": sending(session),
"screenshots": screenshots,
"draft": session.draft,
}
def view(db: Session, session: DebugSession) -> dict:
"""The session as the API serves it — one shape wherever it is served."""
return serialize(session, screenshot_count(db, session)) or {}
def screenshot_count(db: Session, session: DebugSession) -> int:
return (
db.scalar(
select(func.count())
.select_from(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
)
or 0
)
# --- screenshots ---------------------------------------------------------------------------
# By content, not by the declared type: what is stored is served back to a browser as an image.
_SIGNATURES = (
(b"\x89PNG\r\n\x1a\n", "image/png"),
(b"\xff\xd8\xff", "image/jpeg"),
)
def sniff(data: bytes) -> str | None:
"""The image type `data` really is — one of IMAGE_TYPES — or None."""
for magic, mime in _SIGNATURES:
if data.startswith(magic):
return mime
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return "image/webp"
return None
def lock_user(db: Session, user_id: int) -> None:
"""Hold this user's own lock for the rest of the transaction. A session's row lock bounds what
THAT session may keep; the byte cap below is the user's across all of them, and a lock on one
session cannot serialize a check that reads them all — two uploads to two sessions both read the
total before either commits, and both pass."""
db.execute(select(func.pg_advisory_xact_lock(user_id)))
def stored_screenshot_bytes(db: Session, user_id: int) -> int:
return (
db.scalar(
select(func.coalesce(func.sum(DebugScreenshot.size), 0))
.join(DebugSession, DebugSession.id == DebugScreenshot.session_id)
.where(owned_by(DebugSession.user_id, user_id))
)
or 0
)
class ScreenshotRefused(Exception):
"""A screenshot that cannot be kept; `code` is the refusal's machine word."""
def __init__(self, code: str) -> None:
super().__init__(code)
self.code = code
def add_screenshot(db: Session, session: DebugSession, data: bytes) -> DebugScreenshot:
"""Keep one screenshot. Locks the user and the session, and flushes; the caller commits.
Refused while a send is under way or once the issue was sent (it can no longer carry it), past
the per-session count or the user's storage, or when it is not an image we can show."""
if not data:
raise ScreenshotRefused("empty")
mime = sniff(data)
if mime is None:
raise ScreenshotRefused("not_an_image")
# Under both locks: two uploads at once cannot both pass the count (the session's), nor both
# pass the user's byte cap from two different sessions (the user's). Always in this order.
lock_user(db, session.user_id)
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if session.issue_url is not None:
raise ScreenshotRefused("already_sent")
if screenshot_count(db, session) >= SCREENSHOTS_MAX:
raise ScreenshotRefused("too_many")
if stored_screenshot_bytes(db, session.user_id) + len(data) > SCREENSHOT_USER_MAX_BYTES:
raise ScreenshotRefused("storage_full")
shot = DebugScreenshot(
session_id=session.id,
created_at=datetime.now(timezone.utc),
mime=mime,
size=len(data),
data=data,
)
db.add(shot)
db.flush()
return shot
def remove_screenshot(db: Session, session: DebugSession, shot: DebugScreenshot) -> None:
"""Delete one screenshot, under the lock. Refused while a send is under way or unfinished: a
delivery names the screenshots by their order, and removing one would renumber what it still
has to send. Once the issue has everything, this only drops OUR copy — the issue keeps its
own. The caller commits."""
lock(db, session)
if sending(session):
raise ScreenshotRefused("sending")
if unfinished(session):
raise ScreenshotRefused("already_sent")
db.delete(shot)
db.flush()
def screenshots(db: Session, session: DebugSession) -> list[DebugScreenshot]:
"""The session's screenshots, oldest first, WITHOUT their bytes loaded."""
return list(
db.scalars(
select(DebugScreenshot)
.where(DebugScreenshot.session_id == session.id)
.options(defer(DebugScreenshot.data))
.order_by(DebugScreenshot.id)
)
)
# --- the evidence --------------------------------------------------------------------------
# The browser's own log upload (`POST` here, every few seconds while recording) is how the evidence
# arrives, not part of it: its server lines would fill the bundle with the recording's plumbing. The
# browser leaves its copies out for the same reason (frontend lib/diagnostics `isOwnRequest`).
INGEST_PATH = "/api/diagnostics/events"
def _evidence(session: DebugSession):
upload_line = and_(
LogEvent.event == "http.request", LogEvent.attrs["path"].astext == INGEST_PATH
)
# `IS TRUE`, not the bare condition: for a row with no event or no attrs it is NULL, and
# `NOT NULL` would drop exactly the rows that are evidence.
return (LogEvent.debug_session_id == session.id, not_(upload_line.is_(True)))
def bundle(
db: Session, session: DebugSession, *, drop_excluded: bool = False
) -> tuple[list[LogEvent], int]:
"""The session's log rows, browser and server interleaved in time, capped at BUNDLE_MAX — and
how many rows it has in all, so a cut bundle can say so. `drop_excluded` leaves out the rows
the user took out in the composer: what an export or an issue carries."""
where = list(_evidence(session))
excluded = (session.draft or {}).get("excluded") or [] if drop_excluded else []
if excluded:
where.append(LogEvent.id.not_in(excluded))
rows = list(
db.scalars(
select(LogEvent).where(*where).order_by(LogEvent.ts, LogEvent.id).limit(BUNDLE_MAX)
)
)
total = len(rows)
if total == BUNDLE_MAX:
total = db.scalar(select(func.count()).select_from(LogEvent).where(*where)) or total
return rows, total
# --- housekeeping (the hourly log job) -----------------------------------------------------
def sweep(db: Session) -> int:
"""Stage closing every open session past its cap; returns how many."""
now = datetime.now(timezone.utc)
result = db.execute(
update(DebugSession)
.where(DebugSession.ended_at.is_(None), DebugSession.until <= now)
.values(ended_at=DebugSession.until, ended_reason=EndReason.EXPIRED.value)
.execution_options(synchronize_session=False)
)
return result.rowcount or 0
def gc(db: Session, retention_days: int) -> int:
"""Stage deleting sessions (and by cascade their screenshots) started before the log retention
window — the rows they would bundle are gone by then too. Returns how many."""
cutoff = datetime.now(timezone.utc) - timedelta(days=max(1, retention_days))
result = db.execute(delete(DebugSession).where(DebugSession.started_at < cutoff))
return result.rowcount or 0
-82
View File
@@ -1,82 +0,0 @@
"""Demo-account service: reset the shared demo sandbox to a clean, seeded baseline.
Shared by the admin's manual reset endpoint (routes/admin.py) and the scheduled demo-reset job
(scheduler.py). Lives in the service layer, not routes, so the scheduler can call it without an
`app.routes -> app.scheduler` import cycle (the reason its import used to be lazy).
"""
from sqlalchemy import delete, func, select
from sqlalchemy.orm import Session
from app.models import Playlist, PlaylistItem, Subscription, User, Video, VideoState
def _seed_demo_playlists(db: Session, demo: User) -> int:
"""Seed a few sample playlists from the shared catalog so a freshly-reset demo isn't
empty — gives visitors something to play with. Best-effort: skips silently if the
catalog is too small."""
recent = (
db.execute(
select(Video.id)
.where(Video.is_short.is_(False), Video.live_status == "none")
.order_by(Video.published_at.desc().nullslast())
.limit(60)
)
.scalars()
.all()
)
if not recent:
return 0
samples = [
("Fresh picks", recent[0:10]),
("Quick watch", recent[10:18]),
("Saved for later", recent[18:26]),
]
created = 0
for pos, (name, vids) in enumerate(samples):
if not vids:
continue
pl = Playlist(user_id=demo.id, name=name, position=pos)
db.add(pl)
db.flush()
for ipos, vid in enumerate(vids):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=ipos))
created += 1
return created
def _seed_demo_subscriptions(db: Session, demo: User, n: int = 14) -> int:
"""Seed the demo with a handful of catalog channels (the ones with the most videos) so its
Channel manager isn't empty — gives visitors something to explore and matches the landing-page
screenshot. Idempotent: clears the demo's subscriptions first, then re-adds the top N."""
db.execute(delete(Subscription).where(Subscription.user_id == demo.id))
top = (
db.execute(
select(Video.channel_id)
.where(Video.channel_id.is_not(None))
.group_by(Video.channel_id)
.order_by(func.count().desc())
.limit(n)
)
.scalars()
.all()
)
for channel_id in top:
db.add(Subscription(user_id=demo.id, channel_id=channel_id))
return len(top)
def reset_demo_state(db: Session, demo: User) -> int:
"""Wipe the demo account's per-user state (watch/save/hide states, playlists, preferences)
back to a clean baseline and re-seed a few sample playlists + channel subscriptions. Returns
the seeded playlist count. Shared by the admin's manual reset button and the scheduled demo
reset (see scheduler)."""
db.execute(delete(VideoState).where(VideoState.user_id == demo.id))
# Playlist items cascade on playlist delete (ondelete="CASCADE").
db.execute(delete(Playlist).where(Playlist.user_id == demo.id))
demo.preferences = {"language": "en"}
db.commit()
seeded = _seed_demo_playlists(db, demo)
_seed_demo_subscriptions(db, demo)
db.commit()
return seeded
+64
View File
@@ -0,0 +1,64 @@
"""A user's diagnostics level: what they consented to, and for how long.
Three levels, named in the UI Alap / Részletes / Teljes:
* INFO — the base, for everyone, no consent. The browser keeps its log in memory; nothing extra
is sent or stored.
* DEBUG — consented. The server logs this user's requests in detail and stores every line of
them, and the browser streams its own log to `POST /api/diagnostics/events`.
* TRACE — consented. As DEBUG, plus what the user typed into a search.
A raised level always lapses after `ELEVATION_HOURS`: the expiry is computed HERE at consent time,
never taken from the client, and an expired level reads as INFO without anything having to clear it.
Core layer: imports `models` and `sysconfig`.
"""
from datetime import datetime, timedelta, timezone
from typing import Literal, get_args
from sqlalchemy.orm import Session
from app import sysconfig
from app.models import User
DiagnosticsLevel = Literal["INFO", "DEBUG", "TRACE"]
LEVELS: tuple[str, ...] = get_args(DiagnosticsLevel)
BASE: DiagnosticsLevel = "INFO"
ELEVATION_HOURS = 24
# How long a debug session records at most (app/debugsession.py). Here, beside the other
# consent window, because the start dialog's consent text states it and reads it from `state`.
SESSION_MINUTES = 60
def effective_level(user: User, now: datetime | None = None) -> DiagnosticsLevel:
"""The level in force for `user` right now: the consented one until it expires, else BASE."""
level, until = user.diagnostics_level, user.diagnostics_until
if level not in LEVELS or level == BASE or until is None:
return BASE
if until <= (now or datetime.now(timezone.utc)):
return BASE
return level # type: ignore[return-value] — narrowed by the LEVELS check above
def state(db: Session, user: User) -> dict:
"""The `/api/me` view of it: the level in force, when a raised one lapses, and how long stored
log lines are kept — the consent text states that number, so it must be the instance's own."""
level = effective_level(user)
return {
"level": level,
"until": user.diagnostics_until.isoformat() if level != BASE else None,
"retention_days": sysconfig.get_int(db, "log_retention_days"),
"session_minutes": SESSION_MINUTES,
}
def set_level(user: User, level: DiagnosticsLevel) -> None:
"""Record a consent (or its withdrawal). The caller commits. Raising the level again restarts
the clock — each consent is for the next ELEVATION_HOURS, not an extension of an older one."""
if level == BASE:
user.diagnostics_level = None
user.diagnostics_until = None
return
user.diagnostics_level = level
user.diagnostics_until = datetime.now(timezone.utc) + timedelta(hours=ELEVATION_HOURS)
+1 -1
View File
@@ -1,4 +1,4 @@
"""E24 — the read-model behind the Home surface ("Mit nézzek most?").
"""The read-model behind the Home surface ("Mit nézzek most?").
The anti-paralysis answer layer: one hero suggestion, a cross-source resume rail, and the user's own
comfort shelf, all served from our own Postgres with no upstream call at serve time (the same rule
+1 -1
View File
@@ -1,4 +1,4 @@
"""The time budget — E24's replacement for "what mood are you in?".
"""The time budget — the replacement for "what mood are you in?".
`random-gondolatok` 15/D asked for a mood questionnaire before the recommendations. The same file's
later BA-interview note (#23) says the mood half is not what makes this work: the friction is
+2 -1
View File
@@ -22,7 +22,8 @@ to the client, which has both the native payload and the translations.
"""
from app.discover.budget import remaining_seconds
from app.models import Channel, Playlist
from app.catalog.models import Channel
from app.reader.models import Playlist
# Plex artwork is a poster for the things you browse and a landscape still for the things you play.
_POSTER_KINDS = ("movie", "show")
+9 -4
View File
@@ -48,6 +48,7 @@ def pick(
shelf: list[dict] | None = None,
exclude: set[str] | None = None,
start: int = 0,
lang: str,
) -> dict | None:
"""`{"tier": ..., "card": ...}`, or None when nothing at all fits.
@@ -70,8 +71,8 @@ def pick(
skip = exclude or set()
def _resume() -> dict | None:
for card in rails.continue_watching(db, user) if resume is None else resume:
playable = rails.playable(db, user.id, card)
for card in rails.continue_watching(db, user, lang=lang) if resume is None else resume:
playable = rails.playable(db, user.id, card, lang=lang)
# Excluded on the card the USER saw (the show), not only on what it resolves to, so a
# rerolled-past show does not come straight back as its own next episode.
if cards.key(card) in skip or (playable is not None and cards.key(playable) in skip):
@@ -81,7 +82,11 @@ def pick(
return None
def _comfort() -> dict | None:
for card in pins.playables(pins.shelf(db, user) if shelf is None else shelf):
if shelf is None:
own = pins.shelf(db, user, include_plex=rails.plex_available(db), lang=lang)
else:
own = shelf
for card in pins.playables(own):
if cards.key(card) not in skip and fits(card["remaining_seconds"], budget):
return card
return None
@@ -89,7 +94,7 @@ def pick(
def _surprise() -> dict | None:
# A wider draw than the one card we need: the shuffle is seeded, so without the headroom an
# excluded pick would leave the tier empty rather than move past it.
for card in rails.surprise(db, user, budget, seed, limit=SURPRISE_DRAW):
for card in rails.surprise(db, user, budget, seed, limit=SURPRISE_DRAW, lang=lang):
if cards.key(card) not in skip:
return card
return None
+153 -28
View File
@@ -8,22 +8,28 @@ second one.
Every shelf card carries a resolved `next`: the thing that actually plays when you click the tile —
a show's in-progress-or-next episode, a channel's newest unwatched video, a playlist's first
unwatched item. It costs a query per non-leaf pin, which is affordable precisely because this shelf is
short and hand-curated, and it buys the property the whole surface exists for: one click, no
intermediate page, no second decision.
unwatched item. That buys the property the whole surface exists for: one click, no intermediate page,
no second decision. Channels and playlists are resolved for the whole shelf at once (`_Batch`), so the
cost does not grow a query or two per pin.
Domain module, so nothing here raises HTTP. `resolve_ref` returns None for "no such thing, or not
yours" and the route turns that into its 404 — core never imports routes.
"""
from dataclasses import dataclass, field
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.discover import cards, queries
from app.models import Channel, ComfortPin, PlexItem, PlexShow, Playlist, User, Video
from app.models import User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.reader.models import ComfortPin, Playlist
from app.plex import rollup
from app.plex.cards import leaf_card, show_card
from app.plex.cards import SHOW_NEW, leaf_card, show_card
from app.plex.lookups import by_rating_key
from app.userscope import owned_by, owned_row
# kind -> the ComfortPin column that holds it. The client's vocabulary and the schema's, in one place.
PIN_COLUMNS: dict[str, str] = {
@@ -44,8 +50,8 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
translation belongs here rather than in five call sites.
The COLUMN half of every answer comes from `PIN_COLUMNS`; only the LOOKUP differs per kind
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before the
E24 review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
(a direct `db.get`, a rating-key translation, an int parse plus an ownership check). Before a
review the branches spelled the column names out again and `PIN_COLUMNS` had no readers at
all — a constant that called itself the one vocabulary while nothing spoke it."""
col = PIN_COLUMNS.get(kind)
if col is None:
@@ -62,20 +68,60 @@ def resolve_ref(db: Session, user: User, kind: str, ref: str) -> tuple[str, obje
return (col, sh.id) if sh is not None else None
try:
pid = int(ref)
# silent-ok: validating a reference the client supplied. "Not an integer" is answered as "no
# such pin target", which is the same answer this function gives for one that does not exist.
except ValueError:
return None
pl = db.get(Playlist, pid)
# A playlist is per-user, unlike the other four (which are shared catalog rows whose PRIVACY
# lives in the state tables). Someone else's list must read as "no such thing", not as a 403 —
# the same reason `owned_or_404` 404s rather than 403s.
return (col, pl.id) if pl is not None and pl.user_id == user.id else None
pl = owned_row(db, Playlist, pid, user)
return (col, pl.id) if pl is not None else None
def reorder(db: Session, user: User, ids: list[int]) -> None:
"""Put the listed pins in the listed order, WITHIN the slots they already hold.
The client sends the order it can SEE, which is not always the whole shelf: with Plex off the Home
rail hides the Plex pins, and a stale tab may not know a pin added elsewhere. Numbering only the
listed ids 0..n-1 collided with every pin left out — two tiles on one position, and the hidden
ones jumped about when they came back. So the pins left out keep their slots, and the listed ones
are dealt into the slots the listed set occupies. Ids that are not this user's, or repeated, are
ignored rather than failing a stale replay wholesale. Flushes; the route commits."""
# Locked: the rows it leaves alone are part of the answer too. Two overlapping reorders each
# skipped the rows that already matched in their own snapshot and interleaved into two pins on
# one slot; the second now waits and deals from the first one's result. Locked in id order and
# sorted HERE: with `ORDER BY position … FOR UPDATE` Postgres sorts before it locks, so the one
# that waited got the fresh rows in its stale order and dealt the pins it left out back to their
# old slots — and one global lock order means two reorders cannot deadlock.
locked = db.execute(
select(ComfortPin)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
current = sorted(locked, key=lambda p: (p.position, p.id))
owned = {p.id for p in current}
listed: list[int] = []
for i in ids:
if i in owned and i not in listed:
listed.append(i)
chosen = set(listed)
queue = iter(listed)
order = [next(queue) if p.id in chosen else p.id for p in current]
by_id = {p.id: p for p in current}
for position, pin_id in enumerate(order):
if by_id[pin_id].position != position:
by_id[pin_id].position = position
db.flush()
def list_pins(db: Session, user: User) -> list[ComfortPin]:
return list(
db.execute(
select(ComfortPin)
.where(ComfortPin.user_id == user.id)
.where(owned_by(ComfortPin.user_id, user.id))
.order_by(ComfortPin.position.asc(), ComfortPin.id.asc())
)
.scalars()
@@ -90,18 +136,72 @@ def next_position(db: Session, user: User) -> int:
deletes the row, and every FK cascade silently removes a pin when the thing it points at goes
away — so past the first hole a count under-counts and would drop the new pin into the MIDDLE of
the shelf, or onto a position another pin already holds (nothing in the schema forbids that).
Same idiom as `routes/playlists._next_playlist_position`."""
Same idiom as a new playlist's place in the rail (`reader.service`)."""
return (
db.scalar(
select(func.coalesce(func.max(ComfortPin.position), -1)).where(
ComfortPin.user_id == user.id
owned_by(ComfortPin.user_id, user.id)
)
)
+ 1
)
def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
def own_pin(db: Session, user: User, pin_id: int) -> ComfortPin | None:
return owned_row(db, ComfortPin, pin_id, user)
def pin_of(db: Session, user: User, column: str, value: object) -> ComfortPin | None:
"""The user's pin of this thing (`column`/`value` from `resolve_ref`), if it is on the shelf."""
return db.execute(
select(ComfortPin).where(
owned_by(ComfortPin.user_id, user), getattr(ComfortPin, column) == value
)
).scalar_one_or_none()
def add(db: Session, user: User, column: str, value: object) -> ComfortPin:
"""A new pin at the end of the shelf. A second pin of the same thing (another tab, at the same
moment) raises `IntegrityError` at the flush: the unique constraint is the guard."""
pin = ComfortPin(user_id=user.id, position=next_position(db, user), **{column: value})
db.add(pin)
db.flush()
return pin
def remove(db: Session, pin: ComfortPin) -> None:
db.delete(pin)
db.flush()
@dataclass
class _Batch:
"""What the shelf resolves for every pinned channel and playlist in one go."""
channels: dict[str, Channel] = field(default_factory=dict)
channel_summary: dict[str, tuple[int, dict | None]] = field(default_factory=dict)
playlists: dict[int, Playlist] = field(default_factory=dict)
playlist_summary: dict[int, tuple[int, dict | None]] = field(default_factory=dict)
@classmethod
def load(cls, db: Session, user: User, pins: list[ComfortPin]) -> "_Batch":
ch_ids = [p.channel_id for p in pins if p.channel_id is not None]
pl_ids = [p.playlist_id for p in pins if p.playlist_id is not None]
b = cls()
if ch_ids:
b.channels = {
c.id: c for c in db.scalars(select(Channel).where(Channel.id.in_(ch_ids)))
}
b.channel_summary = queries.channels_summary(db, user, ch_ids)
if pl_ids:
b.playlists = {
p.id: p for p in db.scalars(select(Playlist).where(Playlist.id.in_(pl_ids)))
}
b.playlist_summary = queries.playlists_summary(db, user, pl_ids)
return b
def _shelf_card(db: Session, user: User, pin: ComfortPin, batch: _Batch, lang: str) -> dict | None:
"""One pin as a unified card with its resolved `next`. None when the pinned row has gone (the FK
cascade removes the pin itself, so this is only the window between a delete and a refetch)."""
card: dict | None = None
@@ -115,11 +215,11 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
nxt = dict(card) # a video plays itself
elif pin.channel_id is not None:
ch = db.get(Channel, pin.channel_id)
ch = batch.channels.get(pin.channel_id)
if ch is None:
return None
card = cards.from_channel(ch, queries.channel_unwatched_count(db, user, ch.id))
row = queries.channel_next_video(db, user, ch.id)
count, row = batch.channel_summary.get(ch.id, (0, None))
card = cards.from_channel(ch, count)
nxt = cards.from_video(row) if row else None
elif pin.plex_item_id is not None:
@@ -127,23 +227,23 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
if it is None:
return None
st = rollup.item_state_map(db, user.id, [it.id]).get(it.id)
card = cards.from_plex(leaf_card(it, st))
card = cards.from_plex(leaf_card(it, st, lang=lang))
nxt = dict(card) # a movie or a single episode plays itself
elif pin.plex_show_id is not None:
sh = db.get(PlexShow, pin.plex_show_id)
if sh is None:
return None
status = rollup.state_map(db, user.id, [sh.id]).get(sh.id, "new")
card = cards.from_plex(show_card(sh, status))
ep = queries.next_episode(db, user.id, sh.id)
state = rollup.state_map(db, user.id, [sh.id]).get(sh.id, SHOW_NEW)
card = cards.from_plex(show_card(sh, state, lang=lang))
ep = queries.next_episode(db, user.id, sh.id, lang=lang)
nxt = cards.from_plex(ep) if ep else None
elif pin.playlist_id is not None:
pl = db.get(Playlist, pin.playlist_id)
pl = batch.playlists.get(pin.playlist_id)
if pl is None:
return None
count, row = queries.playlist_summary(db, user, pl.id)
count, row = batch.playlist_summary.get(pl.id, (0, None))
card = cards.from_playlist(pl, count, row.get("thumbnail_url") if row else None)
nxt = cards.from_video(row) if row else None
@@ -157,11 +257,24 @@ def _shelf_card(db: Session, user: User, pin: ComfortPin) -> dict | None:
return card
def shelf(db: Session, user: User) -> list[dict]:
"""The whole shelf, in the user's own order."""
_PLEX_COLUMNS = ("plex_item_id", "plex_show_id")
def shelf(db: Session, user: User, *, include_plex: bool = True, lang: str) -> list[dict]:
"""The whole shelf, in the user's own order.
`include_plex=False` is the Home surface with the deployment's Plex toggle off (the owner's call,
2026-09-19): the rails beside it go YouTube-only, and a Plex tile there would offer a Play that
lands on a gated module. The pins themselves are kept — the pin menus still read the full shelf
(`GET /pins`), and switching Plex back on brings the tiles back in their old slots
(`reorder` keeps a hidden pin's place)."""
out = []
for pin in list_pins(db, user):
card = _shelf_card(db, user, pin)
pins = list_pins(db, user)
if not include_plex:
pins = [p for p in pins if all(getattr(p, c) is None for c in _PLEX_COLUMNS)]
batch = _Batch.load(db, user, pins)
for pin in pins:
card = _shelf_card(db, user, pin, batch, lang)
if card is not None:
out.append(card)
return out
@@ -169,5 +282,17 @@ def shelf(db: Session, user: User) -> list[dict]:
def playables(shelf_cards: list[dict]) -> list[dict]:
"""The `next` of every shelf entry that has one — what the hero may draw from. A pinned channel
with nothing unwatched left, or an empty playlist, simply is not an answer right now."""
return [c["next"] for c in shelf_cards if c.get("next")]
with nothing unwatched left, or an empty playlist, simply is not an answer right now.
Each carries `pinned_from`: the pin it was resolved from. A show's next EPISODE is not itself
pinned, so without it the hero's menu looked the episode up, missed, and offered "Pin" under a
kicker that says "From your shelf" — and taking it inserted a second pin. With it, the hero's
pin row acts on the shelf entry the card came from (the owner's call, 2026-09-19: "Unpin")."""
return [
{
**c["next"],
"pinned_from": {k: c[k] for k in ("source", "kind", "id", "title")},
}
for c in shelf_cards
if c.get("next")
]
+83 -59
View File
@@ -6,7 +6,7 @@ Two rules shape this module:
**Reuse the query layer, not the route.** The YouTube half calls `feedquery.filtered_query` /
`sort_keys`, so the Home surface inherits the feed's own definitions of subscribed, blocked,
explore-visible and in-progress for free, and cannot drift from them. The Plex half calls
`plex.rollup` and the `plex.cards` serializers rather than `routes/plex/discover.py`: that route
`plex.rollup` and the `plex.cards` serializers rather than `plex/discover.py`: that read-model
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
sidebar and wrong for a landing page that does not.
@@ -29,22 +29,18 @@ from app.feedquery import (
serialize,
sort_keys,
)
from app.models import (
Channel,
PlaylistItem,
PlexItem,
PlexShow,
PlexState,
User,
Video,
VideoState,
)
from app.models import User
from app.plex.models import PlexItem, PlexShow, PlexState
from app.catalog.models import Channel, Video
from app.reader.models import PlaylistItem, VideoState
from app.plex import rollup
from app.plex.cards import leaf_card, movie_card, show_card
from app.plex.cards import display_title, leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
from app.userscope import owned_by
# How many rows a single rail may carry. Matches `routes/plex/discover.py`'s RAIL_LIMIT on purpose —
# the two surfaces should feel like the same app, and a rail is a rail.
# How many rows a single rail may carry — Home's rails and the Plex module's Discover rails alike
# (`plex/discover.py` imports it): the two surfaces should feel like the same app, and a rail
# is a rail. One definition, because a comment was all that held two copies in step.
RAIL_LIMIT = 20
# The feed parameters a Home rail runs with. Spelled out rather than borrowed from the route's
@@ -97,7 +93,7 @@ def yt_in_progress(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[tup
q = (
select(*feed_columns(user, st), st.progress_updated_at.label("progressed_at"))
.join(Channel, Channel.id == Video.channel_id)
.join(st, and_(st.video_id == Video.id, st.user_id == user.id))
.join(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(
st.position_seconds > 0,
st.status.notin_(("watched", "hidden")),
@@ -129,18 +125,6 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
return [r for r in out if not r["position_seconds"]][:limit]
def channel_next_video(db: Session, user: User, channel_id: str) -> dict | None:
"""The newest unwatched video of a pinned channel — what "put on that channel" resolves to."""
# `library_source="all"`: a pinned channel means the whole channel, including videos that only
# ever arrived through a search. The feed's "organic" default is a feed-scoping choice, not a
# statement about what belongs to a channel.
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("newest", 0, rank_expr)
row = db.execute(query.order_by(*order_by(keys)).limit(1)).first()
return serialize(row) if row is not None else None
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
"""One video as a feed row, with this user's watch state.
@@ -152,37 +136,61 @@ def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
row = db.execute(
select(*feed_columns(user, st))
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(Video.id == video_id)
).first()
return serialize(row) if row is not None else None
def channel_unwatched_count(db: Session, user: User, channel_id: str) -> int:
"""How many unwatched videos a pinned channel is holding — the tile's only honest promise, since a
channel has no runtime of its own."""
params = dict(_FEED_BASE, channel_ids=[channel_id], scope="all", library_source="all")
query, _ = filtered_query(db, user, **params)
return int(db.execute(select(func.count()).select_from(query.subquery())).scalar() or 0)
def channels_summary(
db: Session, user: User, channel_ids: list[str]
) -> dict[str, tuple[int, dict | None]]:
"""For each pinned channel: (unwatched count, its newest unwatched video) — the tile's only honest
promise, since a channel has no runtime of its own, and what "put on that channel" resolves to.
ONE query for the whole shelf, not two per channel: the count and the first row per channel are
window functions over the same filtered set. `library_source="all"`: a pinned channel means the
whole channel, including videos that only ever arrived through a search — the feed's "organic"
default is a feed-scoping choice, not a statement about what belongs to a channel."""
if not channel_ids:
return {}
params = dict(_FEED_BASE, channel_ids=list(channel_ids), scope="all", library_source="all")
query, rank_expr = filtered_query(db, user, **params)
newest = order_by(sort_keys("newest", 0, rank_expr))
per_channel = {"partition_by": Video.channel_id}
sub = query.add_columns(
func.row_number().over(**per_channel, order_by=newest).label("_rn"),
func.count().over(**per_channel).label("_total"),
).subquery()
rows = db.execute(select(sub).where(sub.c._rn == 1)).all()
return {r.channel_id: (int(r._total), serialize(r)) for r in rows}
def playlist_summary(db: Session, user: User, playlist_id: int) -> tuple[int, dict | None]:
"""(item count, the row to play next) for a pinned playlist — its first not-yet-watched item, or
its first item once the whole list is watched. The same row doubles as the tile's artwork, so a
shelf of playlists is scannable instead of a row of blank boxes."""
def playlists_summary(
db: Session, user: User, playlist_ids: list[int]
) -> dict[int, tuple[int, dict | None]]:
"""For each pinned playlist: (item count, the row to play next) — its first not-yet-watched item,
or its first item once the whole list is watched. The same row doubles as the tile's artwork, so
a shelf of playlists is scannable instead of a row of blank boxes. One query for the whole shelf."""
if not playlist_ids:
return {}
st = aliased(VideoState)
rows = db.execute(
select(*feed_columns(user, st), PlaylistItem.position)
select(*feed_columns(user, st), PlaylistItem.playlist_id.label("_playlist"))
.join(PlaylistItem, PlaylistItem.video_id == Video.id)
.join(Channel, Channel.id == Video.channel_id)
.outerjoin(st, and_(st.video_id == Video.id, st.user_id == user.id))
.where(PlaylistItem.playlist_id == playlist_id)
.order_by(PlaylistItem.position.asc())
.outerjoin(st, and_(st.video_id == Video.id, owned_by(st.user_id, user)))
.where(PlaylistItem.playlist_id.in_(playlist_ids))
.order_by(PlaylistItem.playlist_id, PlaylistItem.position.asc())
).all()
if not rows:
return 0, None
nxt = next((r for r in rows if (r.status or "new") not in ("watched", "hidden")), rows[0])
return len(rows), serialize(nxt)
grouped: dict[int, list] = {}
for r in rows:
grouped.setdefault(r._playlist, []).append(r)
out: dict[int, tuple[int, dict | None]] = {}
for pid, items in grouped.items():
nxt = next((r for r in items if (r.status or "new") not in ("watched", "hidden")), items[0])
out[pid] = (len(items), serialize(nxt))
return out
# ── Plex ──────────────────────────────────────────────────────────────────────────────────────────
@@ -196,7 +204,7 @@ def _unseen_movie_conds(uid: int):
.select_from(PlexState)
.where(
PlexState.item_id == PlexItem.id,
PlexState.user_id == uid,
owned_by(PlexState.user_id, uid),
or_(
PlexState.status.in_(("watched", "hidden")),
# Started counts as "not a surprise" too — see the note in `yt_surprise`.
@@ -208,10 +216,12 @@ def _unseen_movie_conds(uid: int):
return ~seen
def plex_in_progress(db: Session, uid: int, limit: int = RAIL_LIMIT) -> list[tuple[object, dict]]:
def plex_in_progress(
db: Session, uid: int, limit: int = RAIL_LIMIT, *, lang: str
) -> list[tuple[object, dict]]:
"""In-progress movies + partially-watched shows as (progressed_at, card).
Mirrors `routes/plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
Mirrors `plex/discover.py::_continue_watching` — same `PlexState` predicate for movies (a
NULL status still counts as started, so `notin_` alone would drop it), and the same shared
`rollup.agg_subquery` + `apply_show_state_filter` for shows, so the two rails cannot disagree on
what "in progress" means. What it does NOT inherit is that route's `LibraryFilters` narrowing:
@@ -222,7 +232,9 @@ def plex_in_progress(db: Session, uid: int, limit: int = RAIL_LIMIT) -> list[tup
if movie_libs:
mq = (
select(PlexItem, PlexState)
.join(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.join(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
@@ -236,26 +248,32 @@ def plex_in_progress(db: Session, uid: int, limit: int = RAIL_LIMIT) -> list[tup
.limit(limit)
)
for it, st in db.execute(mq).all():
out.append((st.progress_updated_at, movie_card(it, st)))
out.append((st.progress_updated_at, movie_card(it, st, lang=lang)))
if show_libs:
prog = rollup.agg_subquery(db, uid, include_resumed=True).subquery()
sq = apply_show_state_filter(
select(PlexShow, prog.c.resumed_at).join(prog, prog.c.show_id == PlexShow.id),
select(PlexShow, prog.c.resumed_at, prog.c.total, prog.c.watched, prog.c.inprog).join(
prog, prog.c.show_id == PlexShow.id
),
prog.c.total,
prog.c.watched,
prog.c.inprog,
"in_progress",
).where(PlexShow.library_id.in_(show_libs))
for sh, resumed_at in db.execute(
for sh, resumed_at, total, watched, inprog in db.execute(
sq.order_by(prog.c.resumed_at.desc().nullslast()).limit(limit)
).all():
out.append((resumed_at, show_card(sh, "in_progress")))
out.append(
(resumed_at, show_card(sh, rollup.show_state(total, watched, inprog), lang=lang))
)
return out
def plex_surprise(db: Session, uid: int, budget: str, seed: int, limit: int) -> list[dict]:
def plex_surprise(
db: Session, uid: int, budget: str, seed: int, limit: int, *, lang: str
) -> list[dict]:
"""Unseen playable movies inside the budget, in seeded-shuffle order.
Movies only. A show is not a thing you can start playing, and resolving every candidate show to
@@ -287,14 +305,14 @@ def plex_surprise(db: Session, uid: int, budget: str, seed: int, limit: int) ->
q = q.where(PlexItem.duration_s <= hi)
items = db.execute(q).scalars().all()
states = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, states.get(it.id)) for it in items]
return [movie_card(it, states.get(it.id), lang=lang) for it in items]
# Watch states that take an episode out of the running for "what would play next".
_DONE = ("watched", "hidden")
def next_episode(db: Session, uid: int, show_id: int) -> dict | None:
def next_episode(db: Session, uid: int, show_id: int, *, lang: str) -> dict | None:
"""What "put that show on" means: the episode in progress, else the first unwatched one, else the
first — over the same `(season, episode)` broadcast ordering `detail.py::_episode_nav` uses.
@@ -310,7 +328,9 @@ def next_episode(db: Session, uid: int, show_id: int) -> dict | None:
)
base = (
select(PlexItem, PlexState)
.outerjoin(PlexState, and_(PlexState.item_id == PlexItem.id, PlexState.user_id == uid))
.outerjoin(
PlexState, and_(PlexState.item_id == PlexItem.id, owned_by(PlexState.user_id, uid))
)
.where(PlexItem.show_id == show_id, PlexItem.kind == "episode")
.limit(1)
)
@@ -344,5 +364,9 @@ def next_episode(db: Session, uid: int, show_id: int) -> dict | None:
ep, st = pick
show = db.get(PlexShow, show_id)
return leaf_card(
ep, st, show_title=show.title if show else None, show_rk=show.rating_key if show else None
ep,
st,
show_title=display_title(show, lang) if show else None,
show_rk=show.rating_key if show else None,
lang=lang,
)
+17 -9
View File
@@ -11,7 +11,8 @@ from sqlalchemy.orm import Session
from app import sysconfig
from app.discover import cards, queries
from app.discover.queries import RAIL_LIMIT
from app.models import PlexShow, User
from app.models import User
from app.plex.models import PlexShow
from app.plex.lookups import by_rating_key
from app.plex.rollup import NO_TS
@@ -22,7 +23,7 @@ def plex_available(db: Session) -> bool:
return sysconfig.get_bool(db, "plex_enabled")
def playable(db: Session, uid: int, card: dict) -> dict | None:
def playable(db: Session, uid: int, card: dict, *, lang: str) -> dict | None:
"""The thing that would actually START from this card — itself for anything with a runtime, and
for a Plex SHOW the episode that is next (in progress, else the first unwatched, else the first).
@@ -40,11 +41,11 @@ def playable(db: Session, uid: int, card: dict) -> dict | None:
sh = by_rating_key(db, PlexShow, card["id"])
if sh is None:
return None
ep = queries.next_episode(db, uid, sh.id)
ep = queries.next_episode(db, uid, sh.id, lang=lang)
return cards.from_plex(ep) if ep else None
def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[dict]:
def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT, *, lang: str) -> list[dict]:
"""One resume rail across both sources, most recently progressed first.
A Plex SHOW enters as a show CARD — that is what the Plex rail shows, and it is the artwork the
@@ -57,20 +58,25 @@ def continue_watching(db: Session, user: User, limit: int = RAIL_LIMIT) -> list[
(ts, cards.from_video(r)) for ts, r in queries.yt_in_progress(db, user, limit)
]
if plex_available(db):
rows += [(ts, cards.from_plex(c)) for ts, c in queries.plex_in_progress(db, user.id, limit)]
rows += [
(ts, cards.from_plex(c))
for ts, c in queries.plex_in_progress(db, user.id, limit, lang=lang)
]
rows.sort(key=lambda r: r[0] or NO_TS, reverse=True)
out = [card for _, card in rows[:limit]]
# Resolve AFTER the slice: only the rows that survive it are worth a query.
for card in out:
if card["kind"] == "show":
card["next"] = playable(db, user.id, card)
card["next"] = playable(db, user.id, card, lang=lang)
return out
def surprise(db: Session, user: User, budget: str, seed: int, limit: int = 3) -> list[dict]:
def surprise(
db: Session, user: User, budget: str, seed: int, limit: int = 3, *, lang: str
) -> list[dict]:
"""The hero's draw pool: unseen things that fit the budget, from both sources under ONE seed.
Interleaved rather than concatenated — the same alternation `routes/plex/discover.py` uses for its
Interleaved rather than concatenated — the same alternation `plex/discover.py` uses for its
movie/show split — so a short pool never comes back all-YouTube or all-Plex just because one side
happened to be queried first.
@@ -83,7 +89,9 @@ def surprise(db: Session, user: User, budget: str, seed: int, limit: int = 3) ->
(cards.from_video, queries.yt_surprise(db, user, budget, seed, limit)),
(
cards.from_plex,
queries.plex_surprise(db, user.id, budget, seed, limit) if plex_available(db) else [],
queries.plex_surprise(db, user.id, budget, seed, limit, lang=lang)
if plex_available(db)
else [],
),
]
if seed % 2:
+161 -7
View File
@@ -11,9 +11,22 @@ Resolution works both before extraction (from source_kind / URL host) and after
`extractor_key` in the info dict), so a non-YouTube "url" job still lands on the right adapter.
"""
import re
from urllib.parse import urlparse
import yt_dlp
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
from app.downloads.stated import StatedFailure
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
# below and the failure catalog (errors.py) decide on it, so the wording exists once.
FACEBOOK_PARSE_WALL = "cannot parse data"
# A source's own verdicts on a link, raised from `resolve_url`. errors.py classifies them; it imports
# them from here because this module must not import errors (errors → credentials → adapters).
POST_WITHOUT_VIDEO_MESSAGE = "No single video was found in this post."
POST_NOT_SHOWN_MESSAGE = "Facebook did not show this post to the signed-in session."
class SourceAdapter:
@@ -22,9 +35,21 @@ class SourceAdapter:
key = "generic"
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
def client_sets(self) -> list:
"""Player-client sets to attempt, in order. A generic source needs no client override, so a
single default attempt (`[None]`) — the per-client fallback is a YouTube-only concept."""
def resolve_url(self, url: str, ydl_opts: dict) -> str:
"""The link to hand yt-dlp for `url`, resolved BEFORE the first attempt. `ydl_opts` are the
download's own options (cookie, timeouts, address family) for any page this needs to read.
Raises a RuntimeError carrying one of this module's verdicts when the page read shows the
download cannot go ahead: the content is there but holds no single video
(`POST_WITHOUT_VIDEO_MESSAGE`), or the page is not the content at all — the login page a
refused cookie gets (`POST_NOT_SHOWN_MESSAGE`), never the no-video verdict. A failed page read
is raised as itself, so it is classified like any other download error. Generic: `url`."""
return url
def client_sets(self, authed: bool = False) -> list:
"""Attempts to make, in order — each entry is handed back to `ydl_opts_extra`, and the next one
is tried when `retriable` says so. A generic source makes a single default attempt (`[None]`).
`authed` is whether the acting user's cookie file goes with the download."""
return [None]
def ydl_opts_extra(self, clients) -> dict:
@@ -37,6 +62,10 @@ class SourceAdapter:
return False
# --- metadata -----------------------------------------------------------------------------
def title(self, info: dict) -> str | None:
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
return info.get("title")
def uploader_url(self, info: dict) -> str | None:
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
@@ -65,12 +94,47 @@ class YouTubeAdapter(SourceAdapter):
"Requested format is not available",
"Only images are available",
"Sign in to confirm",
# Measured 2026-09-16: a gated video whose EXTRACTION succeeds and whose media fetch is then
# refused by the CDN ("unable to download video data: HTTP Error 403"). The wall moves
# between the player response and the stream, so this is the same per-client flakiness one
# step later — and it used to end the job on attempt 1, because the text classifies as
# `network` and nothing here matched it. The format URLs belong to the client that minted
# them, so the next client set is a real second chance.
"unable to download video data",
)
# Measured in the worker against a REAL account cookie (2026-09-16, yt-dlp 2026.07.04): with
# `player_client=android_vr` yt-dlp logs `Skipping client "android_vr" since it does not support
# cookies`, drops it, and the request that is left with no client fails with "Requested format is
# not available" — which is already a retriable marker above, so the cost is a wasted attempt
# rather than a hard failure. Our primary set is tv_simply ALONE, which cannot carry cookies either
# (measured on yt-dlp 2026.08.19: `SUPPORTS_COOKIES` False), so without this filter every
# signed-in download would burn its first attempt that way before reaching the POT-backed web
# fallback the cookie exists for. The list is yt-dlp's own INNERTUBE_CLIENTS[...]
# ["SUPPORTS_COOKIES"], pinned by test_adapters against the installed yt-dlp, so a version bump
# that moves the table turns the gate red instead of silently un-authing every YouTube download.
_COOKIE_CAPABLE = frozenset(
{
"mweb",
"tv",
"tv_downgraded",
"web",
"web_creator",
"web_embedded",
"web_music",
"web_safari",
}
)
# Where a signed-in download lands when the admin's knobs hold no cookie-capable client at all.
# Both clients in ONE set: yt-dlp queries each and merges their formats, which is the cover for
# the per-client flakiness that the two-tier ladder gives a signed-out download.
_AUTHED_DEFAULT = ("web_safari", "web")
def client_sets(self) -> list:
return [
c for c in (settings.player_client_list, settings.player_client_fallback_list) if c
] or [None]
def client_sets(self, authed: bool = False) -> list:
sets = [c for c in (settings.player_client_list, settings.player_client_fallback_list) if c]
if authed:
kept = [[c for c in s if c in self._COOKIE_CAPABLE] for s in sets]
return [s for s in kept if s] or [list(self._AUTHED_DEFAULT)]
return sets or [None]
def ydl_opts_extra(self, clients) -> dict:
# == --extractor-args "youtube:player_client=…;youtubepot-bgutilhttp:base_url=…" (list values).
@@ -93,6 +157,63 @@ class YouTubeAdapter(SourceAdapter):
class FacebookAdapter(SourceAdapter):
key = "facebook"
# Measured (yt-dlp 2026.07.04): with the user's cookie, some reels answer a plain request
# with "Cannot parse data" and parse only when the request looks like Chrome. Impersonation is a
# SECOND attempt behind that one error, not the default: the links that already download with the
# cookie keep their proven path, and only the ones stuck at the wall go through curl_cffi. Without
# a cookie the same error is the login wall, which no disguise gets past.
_IMPERSONATE = "impersonate"
# A post page is ~4–5 MB of inline JSON; anything far past that is not a post page.
_POST_PAGE_MAX_BYTES = 16 * 1024 * 1024
def resolve_url(self, url: str, ydl_opts: dict) -> str:
# Measured: given a group post's link, yt-dlp either finds no formats or — worse —
# downloads ANOTHER video from the page's feed data. The post's own video is the one Video node
# in the page scripts that carry the post id; its watch link downloads (plain first, then the
# Chrome retry above). Needs the cookie: signed out, the page is the login wall.
post_id = facebook_group_post_id(url)
if post_id is None or not ydl_opts.get("cookiefile"):
return url
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
page = raw.decode("utf-8", "replace")
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
# form, no post id at all), which is not a post without a video — the user's cookie is the cause.
if _quoted(post_id) not in page:
raise StatedFailure(POST_NOT_SHOWN_MESSAGE)
video_id = facebook_post_video_id(page, post_id)
if video_id is None:
# Never fall back to the post link: that is exactly where the wrong video comes from.
raise StatedFailure(POST_WITHOUT_VIDEO_MESSAGE)
return f"https://www.facebook.com/watch/?v={video_id}"
def client_sets(self, authed: bool = False) -> list:
return [None, self._IMPERSONATE] if authed else [None]
def ydl_opts_extra(self, clients) -> dict:
if clients == self._IMPERSONATE:
return {"impersonate": ImpersonateTarget.from_str("chrome")}
return {}
def retriable(self, error_text: str) -> bool:
return FACEBOOK_PARSE_WALL in error_text.lower()
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
# signed-out read titles the page with that same text behind the engagement counts, cut short.
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
def title(self, info: dict) -> str | None:
raw = info.get("title")
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
return raw
line = next(
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
)
# The whole first line, never cut here with an ellipsis: normalize_title's NFKC turns "…" into
# "..." and then collapses it to ".", which read as a finished sentence ("… live at."). The
# stored title is unbounded; display_name and the file name clip it themselves.
return line or raw # no description either (the 18+ reel): nothing better on the page
def uploader_url(self, info: dict) -> str | None:
# Facebook gives neither channel_url nor uploader_url, but exposes a numeric `uploader_id`
@@ -107,6 +228,39 @@ class FacebookAdapter(SourceAdapter):
return True
_FB_GROUP_POST_PATH = re.compile(r"^/groups/[^/]+/(?:permalink|posts)/(\d+)/?$")
_SCRIPT_BLOCK = re.compile(r"<script[^>]*>(.*?)</script>", re.S)
_VIDEO_NODE = re.compile(r'"__typename":"Video","id":"(\d+)"')
def _quoted(post_id: str) -> str:
# How the page JSON carries the id: quoted, so a longer id that merely starts with it never counts.
return f'"{post_id}"'
def facebook_group_post_id(url: str) -> str | None:
"""The post id of a Facebook group post link (`/groups/<group>/permalink|posts/<id>/`), else None."""
parsed = urlparse(url)
if not _host_is(parsed.netloc.lower(), "facebook.com"):
return None
match = _FB_GROUP_POST_PATH.match(parsed.path)
return match.group(1) if match else None
def facebook_post_video_id(page: str, post_id: str) -> str | None:
"""The id of the one video attached to `post_id`, read from its page: the Video nodes of the script
blocks that carry the post id (the page's other ~10 videos — feed, reels — sit in other blocks).
None for no video AND for more than one: guessing between two would be the wrong-video bug again."""
quoted = _quoted(post_id)
ids = {
vid
for block in _SCRIPT_BLOCK.findall(page)
if quoted in block
for vid in _VIDEO_NODE.findall(block)
}
return ids.pop() if len(ids) == 1 else None
class RedditAdapter(SourceAdapter):
key = "reddit"
+29 -9
View File
@@ -11,24 +11,30 @@ Run once per environment:
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
"""
import logging
from urllib.parse import urlsplit
import yt_dlp
from sqlalchemy import select
from app import logs
from app.config import settings
from app.db import SessionLocal
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, service
from app.models import MediaAsset
from app.downloads.models import MediaAsset
log = logging.getLogger("siftlode.backfill")
log = logs.get_logger(__name__)
def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
needs the POT/player-client strategy even just to extract)."""
with formats.hiding(url):
return _extract_meta_scoped(url, adapter, asset_id)
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
for clients in adapter.client_sets():
opts = formats.build_ydl_opts(
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
@@ -40,7 +46,19 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter) -> dict | None:
if info:
return info
except Exception as exc: # noqa: BLE001 — best-effort backfill, keep going
log.info("meta extract failed for %s (%s): %s", url, clients, str(exc)[:100])
# The asset id, not the URL. A source URL is a row of this user's download
# history, and `redact_text` only takes credential params out of it — the rest
# of it goes in whole. The id names the same asset to anyone holding the
# database and nothing at all to anyone who is not; the host is kept because it
# is what decides which client set was tried. yt-dlp's message goes through
# `log_safe` for the same reason: it opens with the URL or the video id.
log.info(
"meta extract failed for asset %s on %s (%s): %s",
asset_id,
urlsplit(url).hostname or "?", # log-pii-ok: the host, never the path or query
clients,
formats.log_safe(exc)[:100],
)
return None
@@ -60,20 +78,22 @@ def run() -> dict:
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
if a.uploader_url is None and adapter.wants_metadata_backfill():
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
info = _extract_meta(url, adapter)
info = _extract_meta(url, adapter, a.id)
uu = adapter.uploader_url(info) if info else None
if uu:
a.uploader_url = uu
filled_url += 1
log.info("asset %s uploader_url <- %s", a.id, uu)
log.info(
"asset %s uploader_url filled", a.id
) # the URL names what they downloaded
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
rel = editmod.ensure_poster(a, settings.download_root)
rel = editmod.ensure_poster(a, settings.files_root)
if rel:
a.poster_path = rel
filled_poster += 1
log.info("asset %s poster <- %s", a.id, rel)
log.info("asset %s poster filled", a.id) # the path carries the title
db.commit()
result = {
"assets": len(assets),
+289
View File
@@ -0,0 +1,289 @@
"""Ad-hoc clips: a marked range of a YouTube video or a Plex item, cut into a new per-user asset.
A clip never needs the whole source first. A YouTube clip fetches only its range (yt-dlp's
`download_ranges`; measured: 3-7 s whether the video runs four minutes or an hour, and the fetched
range starts on the marked frame), and a Plex clip reads the library file in place. Either way the
range then goes through ONE re-encode built here, so every clip has the same shape whatever its
source: frame-exact at both ends, H.264 at most 1080p (an HDR source tonemapped to SDR — kept as
10-bit PQ it played washed out), stereo AAC, in an .mp4 that plays on any phone.
A clip may also crop the picture to a frame the user drew over the playing video. The frame is
given in fractions of the box it was drawn in, with that box's aspect: the YouTube iframe never
tells its picture's size, so the frame there is drawn over the player's 16:9 box (a short's 9:16),
and the cut maps it onto the real picture (`crop_filter`) — what lies over a letterbox bar is dropped.
The spec is stored on the job (`DownloadJob.edit_spec`) and is the clip's whole identity: its
signature is the asset's cache key, so marking the same range again is an instant cache hit, and a
different range or audio track is a different clip.
"""
import hashlib
import json
import math
import re
from pathlib import Path
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
from app.plex.sync import HDR_TRANSFERS
SOURCES = ("youtube", "plex")
# Owner's cap: a clip is a moment, not a way to export a whole film to a public link.
MAX_CLIP_S = 600
MIN_CLIP_S = 1
MAX_HEIGHT = 1080
# What yt-dlp fetches the range in (formats.normalize fills the rest): the clip's own ceiling, so the
# re-encode never upscales and never throws away a better source than it keeps.
YOUTUBE_FETCH_SPEC = {"mode": "av", "max_height": MAX_HEIGHT, "container": "mp4"}
# Bump when the spec → output-bytes mapping changes, so a cached clip re-derives.
_SIG_VERSION = 1
_YOUTUBE_ID = re.compile(r"[A-Za-z0-9_-]{11}")
_RATING_KEY = re.compile(r"[0-9]{1,32}")
_MAX_AUDIO_TRACK = 63
# HDR → SDR as measured on the owner's 4K HDR files (2026-10-10): linearise, tonemap (hable),
# convert the primaries and transfer to BT.709 and back to 8-bit 4:2:0.
_TONEMAP = (
"zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable:desat=0,"
"zscale=t=bt709:m=bt709:r=tv,format=yuv420p"
)
_SCALE = f"scale=-2:'min({MAX_HEIGHT},ih)'"
# A crop frame is at least this share of its box on each side (the player's frame holds the same
# floor), and what is left of it on the real picture at least _MIN_VISIBLE of it.
MIN_CROP = 0.1
_MIN_VISIBLE = 0.05
# The box aspects a frame may be drawn in: a 9:16 short to a 21:9 film, with room either side.
_CROP_ASPECTS = (0.25, 4.0)
# Float slack on the frame's far edges (x + w, y + h), which the client rounds to 4 decimals.
_EDGE_SLACK = 1e-3
class ClipError(Exception):
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
message names (the cap itself, so the client never says a stale one)."""
def __init__(self, reason: str, **params: float):
self.reason = reason
self.params = params
super().__init__(reason)
def normalize(
source: str,
ref: str,
start_s: float,
end_s: float,
audio_track: int | None = None,
crop: dict | None = None,
) -> dict:
"""The validated, canonical clip spec. Times are rounded to the millisecond and a crop to four
decimals, so the same marks always give the same cache key. Raises ClipError."""
if source not in SOURCES:
raise ClipError("bad_source")
pattern = _YOUTUBE_ID if source == "youtube" else _RATING_KEY
if not isinstance(ref, str) or not pattern.fullmatch(ref):
raise ClipError("bad_source")
if not (math.isfinite(start_s) and math.isfinite(end_s)) or start_s < 0:
raise ClipError("bad_range")
start, end = round(start_s, 3), round(end_s, 3)
if end - start < MIN_CLIP_S:
raise ClipError("too_short")
if end - start > MAX_CLIP_S:
raise ClipError("too_long", limit_s=MAX_CLIP_S)
spec: dict = {"source": source, "ref": ref, "start_s": start, "end_s": end}
if source == "plex" and audio_track is not None:
if not 0 <= audio_track <= _MAX_AUDIO_TRACK:
raise ClipError("bad_audio")
spec["audio_track"] = audio_track
frame = _crop(crop)
if frame is not None:
spec["crop"] = frame
return spec
def _crop(raw: dict | None) -> dict | None:
"""A crop frame, canonical — or None for no crop, which a frame over the whole box also is (the
same clip as one cut without a frame, so the same cache key)."""
if raw is None:
return None
try:
x, y, w, h, aspect = (float(raw[k]) for k in ("x", "y", "w", "h", "aspect"))
except (KeyError, TypeError, ValueError) as e:
raise ClipError("bad_crop") from e
if not all(math.isfinite(v) for v in (x, y, w, h, aspect)):
raise ClipError("bad_crop")
lo, hi = _CROP_ASPECTS
if not lo <= aspect <= hi or x < 0 or y < 0:
raise ClipError("bad_crop")
if w < MIN_CROP - _EDGE_SLACK or h < MIN_CROP - _EDGE_SLACK:
raise ClipError("bad_crop")
if x + w > 1 + _EDGE_SLACK or y + h > 1 + _EDGE_SLACK:
raise ClipError("bad_crop")
frame = {"x": x, "y": y, "w": min(w, 1 - x), "h": min(h, 1 - y), "aspect": aspect}
frame = {k: round(v, 4) for k, v in frame.items()}
if (frame["x"], frame["y"], frame["w"], frame["h"]) == (0, 0, 1, 1):
return None
return frame
def duration(spec: dict) -> float:
return spec["end_s"] - spec["start_s"]
def sig(spec: dict) -> str:
"""The clip's cache signature: the spec itself, versioned."""
body = json.dumps({**spec, "_v": _SIG_VERSION}, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(body.encode()).hexdigest()[:24]
def asset_ref(user_id: int, spec: dict) -> str:
"""The asset's `source_ref`. The user id makes a clip private to whoever cut it, like an edit:
never dedup-shared, never cache-served to someone else."""
return f"{user_id}:{spec['source']}:{spec['ref']}"
def reference_url(spec: dict) -> str | None:
"""Where a YouTube clip came from: its video at the clip's start. A Plex clip has no public page."""
if spec.get("source") != "youtube" or not spec.get("ref"):
return None
return f"https://www.youtube.com/watch?v={spec['ref']}&t={int(spec.get('start_s') or 0)}s"
def is_hdr(probe: dict) -> bool:
"""Whether the first video stream of an ffprobe `-show_streams` JSON carries an HDR transfer."""
for s in probe.get("streams") or []:
if s.get("codec_type") == "video":
# PQ / HLG: an 8-bit SDR encode must tonemap them (the one set the Plex stream uses).
return s.get("color_transfer") in HDR_TRANSFERS
return False
def display_aspect(probe: dict) -> float | None:
"""The picture's shape as a player shows it, from an ffprobe `-show_streams` JSON: the first video
stream's width × its sample aspect ratio over its height, turned a quarter when the stream says
it is rotated (ffmpeg turns such a stream upright before any filter of ours, so a crop is cut
from the upright picture). None when the stream has no size."""
for s in probe.get("streams") or []:
if s.get("codec_type") != "video":
continue
w, h = s.get("width"), s.get("height")
if not isinstance(w, int) or not isinstance(h, int) or w <= 0 or h <= 0:
return None
dar = w * (_ratio(s.get("sample_aspect_ratio")) or 1.0) / h
return 1 / dar if _quarter_turned(s) else dar
return None
def _ratio(v: object) -> float | None:
"""'32:27' → 1.185…; '0:1', 'N/A', a missing or malformed value → None."""
if not isinstance(v, str) or ":" not in v:
return None
num, _, den = v.partition(":")
try:
n, d = float(num), float(den)
except ValueError: # silent-ok: a malformed ratio reads as none given; the caller takes 1:1
return None
return n / d if n > 0 and d > 0 else None
def _quarter_turned(stream: dict) -> bool:
rotation: object = (stream.get("tags") or {}).get("rotate")
for sd in stream.get("side_data_list") or []:
if "rotation" in sd:
rotation = sd["rotation"]
try:
return round(float(str(rotation))) % 180 == 90
except ValueError: # silent-ok: no (None) or a malformed rotation: the stream is upright
return False
def crop_filter(crop: dict, dar: float) -> str | None:
"""The ffmpeg `crop` for a frame drawn in a box of `crop["aspect"]`, over a picture of display
aspect `dar` fitted into that box as a player fits it (whole, centred, bars on the short sides).
The frame's part over a bar is dropped: "" when what is left is the whole picture, None when too
little of the frame lies on the picture at all.
In fractions of the input's own width and height, so a sample aspect ratio other than 1:1 (an
anamorphic DVD) needs nothing of its own, and floored to even pixels: an even 4:2:0 frame, and a
window that never runs past the picture's far edge."""
box = crop["aspect"]
pic_w, pic_h = (1.0, box / dar) if dar > box else (dar / box, 1.0)
def span(start: float, size: float, extent: float) -> tuple[float, float]:
origin = (1 - extent) / 2
a = min(1.0, max(0.0, (start - origin) / extent))
b = min(1.0, max(0.0, (start + size - origin) / extent))
return a, b - a
x, w = span(crop["x"], crop["w"], pic_w)
y, h = span(crop["y"], crop["h"], pic_h)
if w < _MIN_VISIBLE or h < _MIN_VISIBLE:
return None
if w > 1 - 1e-4 and h > 1 - 1e-4:
return ""
def px(axis: str, f: float) -> str:
return f"trunc({axis}*{f:.6f}/2)*2"
return f"crop={px('iw', w)}:{px('ih', h)}:{px('iw', x)}:{px('ih', y)}"
def audio_track_count(probe: dict) -> int:
return sum(1 for s in probe.get("streams") or [] if s.get("codec_type") == "audio")
def build_cmd(
src: Path,
dest: Path,
start_s: float,
dur_s: float,
*,
audio_track: int | None,
hdr: bool,
threads: int,
crop: str = "",
) -> list[str]:
"""ffmpeg for one clip: an input seek (decodes from the keyframe before `start_s` and drops the
frames up to it, so the clip starts on the marked frame), the first video stream, the chosen audio
track (the first when none was chosen; none at all is fine), no subtitles, data or chapters.
A `crop` (crop_filter) comes first: the tonemap and the scale then work on the frame it keeps."""
vf = ",".join([*([crop] if crop else []), *([_TONEMAP] if hdr else []), _SCALE, EVEN_FRAME])
audio = f"0:a:{audio_track}" if audio_track is not None else "0:a:0?"
return [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-ss",
f"{start_s:.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-t",
f"{dur_s:.3f}",
"-map",
"0:v:0",
"-map",
audio,
"-sn",
"-dn",
# A source's chapters would ride into the .mp4 as a data track of their own (an mkv's
# chapter list came out as `bin_data`); a clip of a moment has no chapters to keep.
"-map_chapters",
"-1",
"-vf",
vf,
*X264_ARGS,
"-threads",
str(threads),
*AAC_ARGS,
"-ac",
"2",
"-movflags",
"+faststart",
str(dest),
]
+103 -25
View File
@@ -1,15 +1,17 @@
"""Per-user, per-host download cookies — the single source of truth for "does the acting user's
cookie apply to this download, and how does that keep the result private".
Two call sites depend on ONE consistent answer (house rule: an invariant lives in shared code, not
Two sides depend on ONE consistent answer (house rule: an invariant lives in shared code, not
in prose):
* `enqueue` salts the MediaAsset cache signature with the user id WHEN a cookie applies, so the
authed download becomes a PRIVATE asset that is never cache-served to another user.
authed download becomes a PRIVATE asset that is never cache-served to another user. Retry
(`service.resume`) and Re-download (`service.redownload`) ask the same question again, so a
cookie the user added after a failure goes with the next attempt.
* the worker attaches the cookie file to yt-dlp ONLY when a cookie still applies AND the asset it
is downloading into is that private, salted identity — so a cookie can never poison a shared
asset (the load-bearing cross-user privacy fix).
Both go through `cookie_salt` / `authed_format_sig` / `host_key_for_url` here, so they can't drift.
Both key the private identity on `cookie_salt` / `host_key_for_url` here, so they can't drift.
Security controls enforced here:
* **host scope (SEC-12)** — `host_key_for_url` matches the download URL's netloc against a curated
@@ -21,19 +23,21 @@ Security controls enforced here:
logged, or written to a job's error/snapshot.
"""
import logging
import os
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app import logs
from app.downloads import clip as clipmod
from app.downloads import formats
from app.downloads.adapters import _host_is
from app.models import DownloadCredential
from app.downloads.models import DownloadCredential, MediaAsset
from app.security import decrypt_optional, encrypt
from app.userscope import owned_by
log = logging.getLogger("siftlode.downloads.credentials")
log = logs.get_logger(__name__)
class HostSpec:
@@ -53,6 +57,11 @@ class HostSpec:
# all derive from this, so there is nothing else to wire up.
SUPPORTED_HOSTS: tuple[HostSpec, ...] = (
HostSpec("facebook", "Facebook", ("facebook.com", "fb.watch")),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273: every player client fails on that one video while its neighbours download fine, and
# the newest yt-dlp fails the same way — a signed-in session is the only way through). Without a
# row here the `login_required` copy sent the user to a dropdown that could not help them.
HostSpec("youtube", "YouTube", ("youtube.com", "youtu.be", "youtube-nocookie.com")),
HostSpec("instagram", "Instagram", ("instagram.com",)),
HostSpec("twitter", "X (Twitter)", ("twitter.com", "x.com")),
HostSpec("tiktok", "TikTok", ("tiktok.com",)),
@@ -99,11 +108,38 @@ def cookie_salt(user_id: int, host_key: str) -> str:
def authed_format_sig(spec: dict, user_id: int, host_key: str) -> str:
"""The salted `format_sig` an authed download for (user, host) resolves to — the private asset
identity. Both enqueue (to create it) and the worker (to recognise it) compute it HERE."""
"""The salted `format_sig` an authed download for (user, host) resolves to under the current
signature version — the private asset enqueue creates, and one of the identities `is_authed_asset`
recognises. Neither calls it; the tests use it to hold the two to each other."""
return formats.format_sig(spec, salt=cookie_salt(user_id, host_key))
def rebuild_salt(asset_id: int, cookie: str | None) -> str:
"""The salt of a Re-download's fresh asset: its OWN row id (so every rebuild is a new identity no
enqueue can ever compute), prefixed with the user's `cookie_salt` when a cookie applies. The id is
what makes it recognisable — the worker has the asset row, so `is_authed_asset` can recompute this
one, where a salt built from the job's previous asset id could not be."""
return f"{cookie}:rebuild:{asset_id}" if cookie else f"rebuild:{asset_id}"
def is_authed_asset(asset: MediaAsset, spec: dict, user_id: int, url: str | None) -> bool:
"""Whether an asset is `user_id`'s private, cookie-salted identity for `url`'s host — the one
kind of asset a cookie is ever attached to: the one enqueue creates (`authed_format_sig`), or a
Re-download's rebuild of it (`rebuild_salt`). The worker's cookie gate asks it before a download,
and the failure diagnosis asks it after one (a failure on this asset happened WITH the cookie).
Either signature version counts (`formats.format_sigs`): a job queued before a deploy that bumped
the version still downloads into the asset it was given."""
host_key = host_key_for_url(url)
if host_key is None:
return False
salt = cookie_salt(user_id, host_key)
if asset.format_sig in formats.format_sigs(spec, salt=salt):
return True
return asset.id is not None and asset.format_sig in formats.format_sigs(
spec, salt=rebuild_salt(asset.id, salt)
)
def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
"""At enqueue: the cache salt to apply if this user has a cookie for `url`'s host, else None (a
normal shared download). Returns the salt string so the caller passes it straight to
@@ -122,7 +158,7 @@ def salt_for_enqueue(db: Session, user_id: int, url: str | None) -> str | None:
def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredential | None:
return db.execute(
select(DownloadCredential).where(
DownloadCredential.user_id == user_id, DownloadCredential.host_key == host_key
owned_by(DownloadCredential.user_id, user_id), DownloadCredential.host_key == host_key
)
).scalar_one_or_none()
@@ -179,7 +215,7 @@ def status_list(db: Session, user_id: int) -> list[dict]:
have = {
c.host_key: c
for c in db.execute(
select(DownloadCredential).where(DownloadCredential.user_id == user_id)
select(DownloadCredential).where(owned_by(DownloadCredential.user_id, user_id))
).scalars()
}
out = []
@@ -199,43 +235,85 @@ def status_list(db: Session, user_id: int) -> list[dict]:
# --- worker: resolve + write the cookie file --------------------------------------------------
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset_format_sig: str,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download, or None.
def usable_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, spec: dict
) -> str | None:
"""The user's cookie text a download of `url` into `asset` goes with, or None — the ONE decision
whether a download is signed in. The worker child writes it out for yt-dlp (`resolve_cookiefile`);
the worker asks it again for a job that inherits a failed asset. That second ask sees the
credential as it is when the job is claimed: a cookie the user removed or added since the failed
download changes this job's explanation to fit the cookie its Retry would go with.
The double gate that prevents cross-user leakage:
1. a credential must still exist for `url`'s host (host-scoped via `host_key_for_url`), AND
2. the asset being downloaded into must BE this user's private, salted identity
(`asset_format_sig == authed_format_sig(...)`).
(`is_authed_asset`).
If (2) fails the asset is the SHARED (unsalted) one — attaching the cookie would poison the
global cache with a private-group video, so we attach nothing. An unreadable blob (rotated key →
InvalidToken) also yields None; the download then fails honestly on the login wall (S3's
`login_required`)."""
InvalidToken) also yields None; the download then fails honestly on the login wall (the
`login_required` category)."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset_format_sig != authed_format_sig(spec, user_id, host_key):
if not is_authed_asset(asset, spec, user_id, url):
return None # shared asset — never attach a cookie to a non-private identity
return _readable(cred, user_id, host_key)
def usable_clip_cookie(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict
) -> str | None:
"""`usable_cookie` for an ad-hoc clip's range fetch. The second gate is the clip's own identity:
a clip asset is private by construction (`clip.asset_ref` names its user, see downloads/clip),
so a cookie goes with it exactly when that asset is THIS user's clip of `clip_spec`. Without it a
video YouTube gates signed out could never be clipped, while its download goes through."""
host_key = host_key_for_url(url)
if host_key is None:
return None
cred = get_credential(db, user_id, host_key)
if cred is None:
return None
if asset.source_kind != "clip" or asset.source_ref != clipmod.asset_ref(user_id, clip_spec):
return None
return _readable(cred, user_id, host_key)
def _readable(cred: DownloadCredential, user_id: int, host_key: str) -> str | None:
raw = decrypt_optional(cred.cookies_enc)
if not raw:
log.warning(
"download cookie for user %s host %s could not be decrypted (rotated key?) — "
"proceeding without it",
"treated as no cookie",
user_id,
host_key,
)
return None
return _write_cookiefile(raw, staging)
return raw
def resolve_cookiefile(
db: Session,
user_id: int,
url: str | None,
asset: MediaAsset,
spec: dict,
staging: Path,
) -> Path | None:
"""In the worker child: the cookies.txt to hand yt-dlp for this download (`usable_cookie`), or None."""
raw = usable_cookie(db, user_id, url, asset, spec)
return _write_cookiefile(raw, staging) if raw else None
def resolve_clip_cookiefile(
db: Session, user_id: int, url: str | None, asset: MediaAsset, clip_spec: dict, staging: Path
) -> Path | None:
"""`resolve_cookiefile` for an ad-hoc clip (`usable_clip_cookie`)."""
raw = usable_clip_cookie(db, user_id, url, asset, clip_spec)
return _write_cookiefile(raw, staging) if raw else None
def _write_cookiefile(raw: str, staging: Path) -> Path:
+153 -132
View File
@@ -23,7 +23,6 @@ for a player hover-preview later.
import hashlib
import json
import logging
import math
import os
import queue
@@ -33,9 +32,11 @@ import threading
import time
from pathlib import Path
from app import logs
from app.config import settings
from app.downloads.stated import StatedFailure
log = logging.getLogger("siftlode.edit")
log = logs.get_logger(__name__)
def encode_thread_cap() -> int:
@@ -71,7 +72,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if crop:
try:
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
except (KeyError, TypeError, ValueError):
except (KeyError, TypeError, ValueError) as exc:
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
# and nothing anywhere called it a failure; this line is the only trace there is.
log.warning(
"the crop in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.crop_dropped", reason=type(exc).__name__),
)
c = None
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
out["crop"] = c
@@ -80,6 +87,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
segs = spec.get("segments")
if isinstance(segs, list) and segs:
norm: list[dict] = []
unreadable = 0
for s in segs:
try:
st = max(0.0, float((s or {}).get("start_s") or 0.0))
@@ -88,9 +96,18 @@ def normalize_edit_spec(spec: dict | None) -> dict:
continue
en = float(end_raw)
except (TypeError, ValueError):
# silent-ok: counted, not logged here — one cut-list can drop several, and the
# join that comes out is simply shorter than the one the user built. Reported
# once after the loop.
unreadable += 1
continue
if en > st:
norm.append({"start_s": round(st, 3), "end_s": round(en, 3)})
if unreadable:
log.warning(
"some segments of a cut list could not be read and were dropped",
extra=logs.event("edit.spec.segments_dropped", dropped=unreadable, kept=len(norm)),
)
norm.sort(key=lambda x: x["start_s"])
# SEC-13: an accurate join builds one ffmpeg `filter_complex` with two filters per segment,
# so an unbounded list becomes an unbounded command line and an ffmpeg that never finishes.
@@ -114,7 +131,13 @@ def normalize_edit_spec(spec: dict | None) -> dict:
start = max(0.0, float(trim.get("start_s") or 0.0))
end_raw = trim.get("end_s")
end = float(end_raw) if end_raw is not None else None
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# Either a 400 the user cannot act on, or — beside a crop — a full-length clip where
# they asked for a cut. Both are worth one line saying the spec was the problem.
log.warning(
"the trim in an edit spec could not be read and was dropped",
extra=logs.event("edit.spec.trim_dropped", reason=type(exc).__name__),
)
start, end = 0.0, None
t: dict = {"start_s": round(start, 3)}
if end is not None and end > start:
@@ -165,6 +188,51 @@ def clip_duration(spec: dict, source_duration: int | None) -> int | None:
# --- ffmpeg: trim / crop --------------------------------------------------------------------
# Every libx264 / aac re-encode in this package uses these, so a re-encoded file is always the
# phone-safe shape (`playable`): `-pix_fmt yuv420p` pins 8-bit 4:2:0 — without it libx264 keeps a
# 10-bit or 4:4:4 source's pixel format and writes H.264 High 10 / High 4:4:4, which browsers decode
# only patchily (the one safe H.264 is 8-bit 4:2:0).
X264_ARGS = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p"]
AAC_ARGS = ["-c:a", "aac", "-b:a", "192k"]
# The last step of every video filter chain a libx264 re-encode runs: 4:2:0 halves the chroma, so
# libx264 refuses an odd width or height ("width not divisible by 2") — measured on an odd source
# that is not 4:2:0 already (a 4:4:4 or RGB screen capture, 1365x767). A crop of a 4:2:0 source is
# rounded to its chroma grid by ffmpeg itself; this holds every other path to the encoder to the same
# even frame. Rounding down to even costs at most one pixel row/column.
EVEN_FRAME = "scale=trunc(iw/2)*2:trunc(ih/2)*2"
# The demuxers ffmpeg/ffprobe may use to open a library file — single-file containers only. Without
# it ffmpeg picks the demuxer from the file's CONTENT and NAME, and an upload's name is the user's:
# a playlist-type demuxer (HLS, DASH, concat) opens whatever local paths its entries name, so a
# crafted upload could copy another user's clip or a Plex file into its own converted asset. A file
# no demuxer here can open fails the probe, and an unprobeable upload is a download-only 'file'.
MEDIA_DEMUXERS = (
"mov", # mp4, m4a, mov, 3gp
"matroska", # mkv, webm
"mpegts",
"mpeg",
"avi",
"asf",
"flv",
"ogg",
"mp3",
"aac",
"ac3",
"eac3",
"flac",
"wav",
"w64",
"aiff",
"caf",
"amr",
)
INPUT_GUARD = ["-format_whitelist", ",".join(MEDIA_DEMUXERS)]
# A downloaded thumbnail is the remote site's bytes under the suffix yt-dlp gave it (`.png`, `.webp`,
# `.jpg` — `pipeline._THUMB_SUFFIXES`): only the single-image demuxers those open, so a "thumbnail"
# that is really a playlist cannot pull a local file into the poster. Anything else fails, and the
# poster falls back to a cut frame (`ensure_poster`).
THUMB_GUARD = ["-format_whitelist", "png_pipe,jpeg_pipe,webp_pipe"]
def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]:
"""ffmpeg command for a trim/crop edit. `-progress pipe:1` streams machine-readable progress.
@@ -191,7 +259,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
]
if start:
cmd += ["-ss", f"{start:.3f}"]
cmd += ["-i", str(src)]
cmd += [*INPUT_GUARD, "-i", str(src)]
if end is not None:
dur = end - start
if dur > 0:
@@ -199,21 +267,9 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
if reencode:
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
# the exact cut). A crop adds the filter; an accurate trim re-encodes with no filter.
if crop:
cmd += ["-vf", f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"]
cmd += [
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
]
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
else:
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -261,11 +317,14 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
parts, labels = [], []
for i, s in enumerate(segs):
st, en = s["start_s"], s["end_s"]
parts.append(f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf}[v{i}]")
parts.append(
f"[0:v]trim=start={st}:end={en},setpts=PTS-STARTPTS{cropf},{EVEN_FRAME}[v{i}]"
)
parts.append(f"[0:a]atrim=start={st}:end={en},asetpts=PTS-STARTPTS[a{i}]")
labels.append(f"[v{i}][a{i}]")
parts.append(f"{''.join(labels)}concat=n={len(segs)}:v=1:a=1[v][a]")
cmd = head + [
*INPUT_GUARD,
"-i",
str(src),
"-filter_complex",
@@ -274,26 +333,23 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
"[v]",
"-map",
"[a]",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"20",
"-c:a",
"aac",
"-b:a",
"192k",
*X264_ARGS,
*AAC_ARGS,
]
else:
listp = staging / "concat.txt"
lines = []
lines = ["ffconcat version 1.0"]
for s in segs:
lines.append(f"file '{_concat_quote(src)}'")
# ffmpeg hands the list's whitelist (which must name `concat` to read the list) down to
# the file it names, so a library file whose bytes are a concat list would be read as one
# and open the paths inside it. Each entry's own whitelist replaces the inherited one.
lines.append(f"option format_whitelist {INPUT_GUARD[1]}")
lines.append(f"inpoint {s['start_s']}")
lines.append(f"outpoint {s['end_s']}")
prep[listp] = "\n".join(lines) + "\n"
cmd = head + ["-f", "concat", "-safe", "0", "-i", str(listp), "-c", "copy"]
guard = ["-format_whitelist", "concat," + INPUT_GUARD[1]]
cmd = head + ["-f", "concat", "-safe", "0", *guard, "-i", str(listp), "-c", "copy"]
if out_ext in ("mp4", "m4a", "mov"):
cmd += ["-movflags", "+faststart"]
@@ -306,11 +362,14 @@ def _parse_out_time(line: str) -> float | None:
if line.startswith("out_time_us="):
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: one line of ffmpeg's `-progress` stream, several per second. None means "this
# line carried no usable time", which the caller already treats as "no progress yet".
except ValueError:
return None
if line.startswith("out_time_ms="): # (some builds mislabel this as microseconds)
try:
return int(line.split("=", 1)[1]) / 1_000_000
# silent-ok: as above, the other spelling of the same line.
except ValueError:
return None
return None
@@ -329,7 +388,9 @@ def _pump(stream, sink: "queue.Queue[str | None]") -> None:
try:
for line in stream:
sink.put(line)
except (ValueError, OSError): # pipe closed under us by the kill path
# silent-ok: the pipe was closed under us by our OWN kill path — the expected end of a
# cancelled or timed-out run, and the code that did the killing is the code that reports it.
except (ValueError, OSError):
pass
finally:
sink.put(None)
@@ -347,6 +408,8 @@ def _drain_tail(stream, sink: list[str]) -> None:
sink.append(line)
if len(sink) > 200:
del sink[:100]
# silent-ok: same closed pipe as `_pump` above, and the tail collected so far is still handed
# to whoever reports the failure.
except (ValueError, OSError):
pass
@@ -362,13 +425,17 @@ def escalate_kill(proc: subprocess.Popen, terminate, kill, grace_s: float, label
try:
proc.wait(timeout=grace_s)
return
# silent-ok: the escalation below IS the handling of this timeout, and surviving THAT is what
# the line under the second wait reports.
except subprocess.TimeoutExpired:
pass
kill()
try:
proc.wait(timeout=grace_s)
except subprocess.TimeoutExpired:
log.error("%s pid %s survived SIGKILL", label, proc.pid)
# TRY400: the exception IS the message here. A TimeoutExpired's traceback is this
# `wait` call and nothing else — it would add three frames and no fact the line lacks.
log.error("%s pid %s survived SIGKILL", label, proc.pid) # noqa: TRY400
def _stop(proc: subprocess.Popen) -> None:
@@ -404,6 +471,8 @@ def run_ffmpeg(
while True:
try:
line = lines.get(timeout=1.0)
# silent-ok: not a failure — the one-second timeout is how this loop gets a turn to
# poll for cancellation while ffmpeg is quiet.
except queue.Empty:
line = ""
if line is None: # stdout hit EOF — ffmpeg is done (or dying)
@@ -416,11 +485,11 @@ def run_ffmpeg(
if should_cancel():
raise EditAborted
if time.monotonic() - last_output > stall_timeout_s:
raise RuntimeError(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
raise StatedFailure(f"ffmpeg stalled: no progress for {int(stall_timeout_s)}s")
try:
ret = proc.wait(timeout=_FFMPEG_EXIT_GRACE_S)
except subprocess.TimeoutExpired: # pipes closed but the process lingers
raise RuntimeError("ffmpeg did not exit after closing its output")
raise StatedFailure("ffmpeg did not exit after closing its output") from None
except BaseException:
# ONE exit path stops the process, so the invariant is unconditional: nothing leaves this
# function with ffmpeg still running. Not just our own EditAborted/stall — `on_progress`
@@ -435,17 +504,10 @@ def run_ffmpeg(
if pipe:
pipe.close()
if ret != 0:
raise RuntimeError(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
raise StatedFailure(f"ffmpeg failed ({ret}): {''.join(err_tail)[-300:]}")
# --- ffmpeg: universal browser playability --------------------------------------------------
# A downloaded file must play in EVERY browser, whatever the source shipped. Universal = H.264
# video + AAC/MP3 audio in an mp4 container: H.264 is the only video codec all browsers decode
# (VP9/AV1 fail on Safari/iOS; HEVC fails on Chrome/Firefox), and mp4/AAC is the safe pairing.
# Anything else is fixed post-download: re-encode the offending stream(s), or just remux when the
# codecs are already fine but the container isn't.
_BROWSER_SAFE_VCODEC = ("h264",)
_BROWSER_SAFE_ACODEC = ("aac", "mp3")
# --- ffprobe ---------------------------------------------------------------------------------
def run_ffprobe_json(
@@ -457,10 +519,10 @@ def run_ffprobe_json(
) -> dict | None:
"""One ffprobe call → its parsed JSON dict, or None on any failure (timeout, non-zero exit,
missing binary, unparseable output). The SINGLE ffprobe-JSON-invocation point in the downloads
package: the browser-playability codec probe here and the shared media probe (`probe.probe_media`,
used by both the download worker's finalize and the upload ingest) both go through it, so the
package: the shared media probe (`probe.probe_json`: the stored facts and the phone-safe rule) goes
through it, so the
subprocess flags/timeout/error handling live in one place instead of drifting copies."""
args = ["ffprobe", "-v", "error"]
args = ["ffprobe", "-v", "error", *INPUT_GUARD]
if show_streams:
args.append("-show_streams")
if show_format:
@@ -471,39 +533,16 @@ def run_ffprobe_json(
try:
out = subprocess.run(args, capture_output=True, text=True, timeout=30, check=True).stdout
return json.loads(out)
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# None makes every field it would have filled read as absent, and a file with no streams
# and no duration is indistinguishable here from one that is not media at all.
log.warning(
"ffprobe could not read a file",
extra=logs.event("media.ffprobe_failed", reason=type(exc).__name__),
)
return None
def _codecs_from_streams(streams: list) -> tuple[str | None, str | None]:
"""(video_codec, audio_codec) lowercased from an ffprobe JSON stream list — the first of each kind.
Parsed from the JSON `streams` list, NOT ffprobe's flat `default=nw=1` output: there the fields
arrive in ffprobe's own struct order, which puts `codec_name` BEFORE `codec_type`, so a line-by-line
"remember the last codec_type" parser attributes every name to the PREVIOUS stream. Measured on real
downloads: an ordinary H.264+AAC mp4 probed as `v=aac a=None` — the browser-playability check then
believed the video stream was AAC and re-encoded a perfectly fine file on every download. Reading each
stream dict by its own `codec_type` is immune to that ordering."""
vcodec = acodec = None
for s in streams:
name = (s.get("codec_name") or "").lower() or None
ctype = (s.get("codec_type") or "").lower()
if ctype == "video" and vcodec is None:
vcodec = name
elif ctype == "audio" and acodec is None:
acodec = name
return vcodec, acodec
def probe_codecs(path: Path) -> tuple[str | None, str | None]:
"""Return (video_codec, audio_codec) lowercased via one ffprobe call; None for a missing
stream or if ffprobe fails (a failed probe is treated as 'leave it alone' upstream)."""
data = run_ffprobe_json(path, entries="stream=codec_type,codec_name")
if data is None:
return None, None
return _codecs_from_streams(data.get("streams") or [])
def probe_duration(path: Path) -> int:
"""Media duration in whole seconds via ffprobe, or 0 if unknown — a progress denominator for a
re-encode when the yt-dlp info dict carried no `duration` (some URL/live extractions)."""
@@ -513,6 +552,7 @@ def probe_duration(path: Path) -> int:
"ffprobe",
"-v",
"error",
*INPUT_GUARD,
"-show_entries",
"format=duration",
"-of",
@@ -525,53 +565,16 @@ def probe_duration(path: Path) -> int:
check=True,
).stdout.strip()
return int(float(out)) if out else 0
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError):
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError, ValueError) as exc:
# Zero is the denominator of the re-encode progress bar, so a failed probe leaves the user
# watching a bar that never moves rather than a job that says it went wrong.
log.info(
"the duration probe failed; progress for this job has no total",
extra=logs.event("media.duration_probe_failed", reason=type(exc).__name__),
)
return 0
def browser_transcode_flags(
vcodec: str | None, acodec: str | None, is_mp4: bool
) -> tuple[bool, bool, bool]:
"""Decide what (if anything) makes a file universally browser-playable. Returns
(needs_change, reencode_video, reencode_audio). needs_change=False => already H.264 + AAC/MP3
in mp4. A None codec (no such stream, or a failed probe) is never re-encoded."""
rv = bool(vcodec) and vcodec not in _BROWSER_SAFE_VCODEC
ra = bool(acodec) and acodec not in _BROWSER_SAFE_ACODEC
return (rv or ra or not is_mp4), rv, ra
def build_browser_safe_cmd(
src: Path, dest: Path, reencode_video: bool, reencode_audio: bool, threads: int | None = None
) -> list[str]:
"""ffmpeg command that lands `src` as a browser-safe mp4 (faststart): re-encode only the
stream(s) that need it (libx264 / aac), stream-copy the rest, always into mp4. `-progress
pipe:1` streams machine-readable progress for run_ffmpeg. `threads` caps the encoder so a
re-encode can't claim every core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
"-i",
str(src),
]
cmd += (
["-c:v", "libx264", "-preset", "veryfast", "-crf", "20"]
if reencode_video
else ["-c:v", "copy"]
)
cmd += ["-c:a", "aac", "-b:a", "192k"] if reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
cmd += ["-movflags", "+faststart", str(dest)]
return cmd
# --- filmstrip (editor scrub track) ---------------------------------------------------------
_SB_COLS = 12
@@ -607,6 +610,7 @@ def build_storyboard_cmd(src: Path, dest: Path, duration_s: int | None, geom: di
"-loglevel",
"error",
"-nostdin",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -630,6 +634,7 @@ def build_poster_cmd(src: Path, dest: Path, at_s: float) -> list[str]:
"-nostdin",
"-ss",
f"{max(0.0, at_s):.3f}",
*INPUT_GUARD,
"-i",
str(src),
"-frames:v",
@@ -651,6 +656,7 @@ def build_thumbnail_jpg_cmd(src: Path, dest: Path) -> list[str]:
"-loglevel",
"error",
"-nostdin",
*THUMB_GUARD,
"-i",
str(src),
"-frames:v",
@@ -670,15 +676,28 @@ def _run_ffmpeg_best_effort(cmd: list[str], timeout_s: int) -> Exception | None:
try:
subprocess.run(cmd, capture_output=True, timeout=timeout_s, check=True)
return None
# silent-ok: the exception is the RETURN VALUE. This helper exists so the three callers share
# one run/timeout policy, and each logs the failure with the context only it has.
except (subprocess.TimeoutExpired, subprocess.CalledProcessError, OSError) as exc:
return exc
def _step_failure(err: BaseException) -> str:
"""What a best-effort image step's failure line says: the kind and, for ffmpeg, its exit
status. Never `str(err)`: a CalledProcessError or TimeoutExpired quotes ffmpeg's argv, and an
OSError its file name — the source path either way, which is built from the channel and the
title."""
code = getattr(err, "returncode", None)
return f"{type(err).__name__} {code}" if code is not None else type(err).__name__
def _is_jpeg(path: Path) -> bool:
"""True if the file starts with the JPEG SOI marker (FF D8 FF)."""
"""True if the file starts with the JPEG SOI marker (0xFF 0xD8 0xFF)."""
try:
with path.open("rb") as f:
return f.read(3) == b"\xff\xd8\xff"
# silent-ok: False only means "do not copy this straight through" — the caller then goes the
# ffmpeg route, which opens the same file and reports it properly if it really is unreadable.
except OSError:
return False
@@ -699,19 +718,21 @@ def thumbnail_to_jpg(src: Path | None, dest: Path, timeout_s: int = 30) -> Path
shutil.copyfile(src, dest)
return dest
except OSError as exc:
log.info("thumbnail copy failed (%s); poster falls back to a cut frame", str(exc)[:120])
log.info(
"thumbnail copy failed (%s); poster falls back to a cut frame", _step_failure(exc)
)
return None
err = _run_ffmpeg_best_effort(build_thumbnail_jpg_cmd(src, dest), timeout_s)
if err is not None:
log.info(
"thumbnail->jpg conversion failed (%s); poster falls back to a cut frame",
str(err)[:120],
_step_failure(err),
)
return None
return dest if dest.exists() else None
def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
def ensure_poster(asset, files_root: str, timeout_s: int = 30) -> str | None:
"""Generate a poster frame for a ready asset that has no thumbnail (e.g. a direct media URL
like a reddit HLS manifest, which yt-dlp can't get a thumbnail for). Written as the media's
`<base>.jpg` sidecar — so Plex uses it too and `delete_asset_files` cleans it up — and returned
@@ -719,7 +740,7 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
short clip) dodges black intro frames."""
if not asset.rel_path:
return None
root = Path(download_root)
root = Path(files_root)
src = root / asset.rel_path
if not src.exists():
return None
@@ -730,12 +751,12 @@ def ensure_poster(asset, download_root: str, timeout_s: int = 30) -> str | None:
at = min(3.0, (asset.duration_s or 10) * 0.1)
err = _run_ffmpeg_best_effort(build_poster_cmd(src, dest, at), timeout_s)
if err is not None:
log.info("poster gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("poster gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
return rel if dest.exists() else None
def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict | None:
def ensure_storyboard(asset, files_root: str, timeout_s: int = 90) -> dict | None:
"""Return the filmstrip meta for a ready source asset, generating the sprite on first use.
Best-effort + time-bounded: for a very long/high-res source the one-pass decode can be slow,
@@ -747,7 +768,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
if not asset.rel_path:
return None
geom = storyboard_geometry(asset.duration_s)
root = Path(download_root)
root = Path(files_root)
rel = f".storyboards/asset-{asset.id}.jpg"
dest = root / rel
@@ -761,7 +782,7 @@ def ensure_storyboard(asset, download_root: str, timeout_s: int = 90) -> dict |
cmd = build_storyboard_cmd(src, dest, asset.duration_s, geom)
err = _run_ffmpeg_best_effort(cmd, timeout_s)
if err is not None:
log.info("storyboard gen skipped for asset %s: %s", asset.id, str(err)[:120])
log.info("storyboard gen skipped for asset %s: %s", asset.id, _step_failure(err))
return None
if not dest.exists():
return None
+214 -8
View File
@@ -1,4 +1,5 @@
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code.
"""Classify a raw yt-dlp / ffmpeg failure string into a stable category code — and, where a site's
failure has been measured, into that site's entry of the failure catalog.
The worker stores the raw exception text (truncated) for technical detail, but a raw
`ERROR: [youtube] <id>: Private video. Sign in…` is noise to a user. `classify()` maps that text
@@ -6,22 +7,67 @@ to one of a small set of category codes; the frontend turns the code into a loca
explanation (and decides whether a retry makes sense). The raw text stays available behind a
"details" disclosure. This is the same string-matching approach the worker already trusts for
`_RETRIABLE_MARKERS` — kept here as the single source of truth for "what kind of failure is this".
**The failure catalog** (`_CATALOG`). A category is generic: "Sign-in required" cannot tell a
Facebook user that the cookie they already gave was refused, and yt-dlp words several different
Facebook walls identically (`Cannot parse data`). A catalog entry is scoped to ONE host and may also
depend on whether a cookie was attached and on the kind of link, and names a `detail` code the UI has
its own sentence for — why it failed on that site, and what the user can do. `diagnose()` tries the
catalog first and falls back to the category. How an entry is added, every time a failure is reported:
1. reproduce it on dev (with and without the user's cookie) and keep the verbatim yt-dlp text;
2. add the entry below, scoped as narrowly as the measurement allows (a marker that is yt-dlp's
generic wording for every site must never match without its host; a marker another module also
decides on — the Facebook Chrome retry's FACEBOOK_PARSE_WALL — is imported, never retyped);
3. pin the real text in `tests/test_download_errors.py`;
4. add `downloads.errors.fail.catalog.<detail>.{short,body}` in EN and HU (a contract test checks it;
the group is not called `detail`, which `actionErrors.contract` reads as a raw `HttpError.detail`).
"""
# Two worker-authored strings that this module owns because both are coupled to `_RULES`.
from collections.abc import Callable
from dataclasses import dataclass
from typing import NamedTuple
from app.downloads.adapters import (
FACEBOOK_PARSE_WALL,
POST_NOT_SHOWN_MESSAGE,
POST_WITHOUT_VIDEO_MESSAGE,
facebook_group_post_id,
)
from app.downloads.credentials import host_key_for_url
# Worker-authored strings that this module owns because each is coupled to `_RULES`.
#
# QUOTA_FULL_MESSAGE is raised by the worker and must classify back to `quota_exceeded`; its rule
# below matches a fragment of it, and test_download_errors pins the pair, because a message and its
# rule drifting apart is exactly how a category silently becomes unreachable.
# QUOTA_FULL_MESSAGE and the three *_TOO_BIG_MESSAGEs are raised by the worker and must classify
# back to `quota_exceeded`; their rules below match a fragment of each, and test_download_errors pins
# each one, because a message and its rule drifting apart is exactly how a
# category silently becomes unreachable.
#
# NETWORK_REDACTED_MESSAGE is the opposite direction: it REPLACES a message after classification, so
# it deliberately has no rule. Nothing re-classifies a stored `job.error` today; if anything ever
# does, it must carry the original code rather than re-deriving one from this text.
# NETWORK_REDACTED_MESSAGE REPLACES a message after classification (`user_error_text`). It is written
# to the shared asset too, and a second job that inherits that failed asset re-diagnoses the stored
# text (`worker._process_job`), so it has a rule of its own that gives the category back.
#
# Other worker-raised texts are still inline literals at their raise sites. The ffmpeg ones and
# "produced no output file" have markers here; the urlguard refusal deliberately has none and
# classifies as `unknown` — it is a refusal, not a diagnosis, and no category would say more.
#
# A source adapter's verdicts on a link (`SourceAdapter.resolve_url`) are adapters.py's
# constants, imported above; each has a rule below that gives its category back, and a catalog entry
# that gives the site's sentence.
QUOTA_FULL_MESSAGE = "Your download storage is full — free up space, then try again."
# A download, a cut clip, or an upload's converted file, held to the space left, which is not zero:
# "full" would be untrue. All three classify by the one phrase they share.
DOWNLOAD_TOO_BIG_MESSAGE = (
"The video is larger than the download storage you have left — free up space, then try again."
)
CLIP_TOO_BIG_MESSAGE = (
"The clip is larger than the download storage you have left — "
"free up space or cut a shorter clip, then try again."
)
CONVERTED_TOO_BIG_MESSAGE = (
"The converted file is larger than the download storage you have left — "
"free up space, then try again."
)
UPLOAD_MISSING_MESSAGE = "The uploaded file is missing."
NETWORK_REDACTED_MESSAGE = "The server could not reach that address."
# (marker, category) — first match wins, so order most-specific → most-general. Matching is
@@ -30,6 +76,13 @@ _RULES: tuple[tuple[str, str], ...] = (
# Our own refusal: the worker found the user already at their storage cap (SEC-13). Placed
# first — it is our wording, so it can never be a false match on an upstream message.
("download storage is full", "quota_exceeded"),
("larger than the download storage you have left", "quota_exceeded"),
# Our own redaction of a transport failure (see NETWORK_REDACTED_MESSAGE) — our wording too.
(NETWORK_REDACTED_MESSAGE.lower(), "network"),
# Our own verdicts on a post link (adapters.py) — our wording too: its page holds no single video,
# or the site answered the signed-in read with something that is not the post at all.
(POST_WITHOUT_VIDEO_MESSAGE.lower(), "format_unavailable"),
(POST_NOT_SHOWN_MESSAGE.lower(), "login_required"),
# Login / age / members / bot-check — the user must be signed in (we deliberately are not).
("sign in to confirm your age", "login_required"),
("confirm your age", "login_required"),
@@ -61,6 +114,8 @@ _RULES: tuple[tuple[str, str], ...] = (
# An edit's source download went away — our own wording, and MORE specific than the generic
# "no longer available" below, so it must be matched first.
("source download is no longer available", "source_missing"),
# An upload's original went away before its conversion ran — our wording (UPLOAD_MISSING_MESSAGE).
(UPLOAD_MISSING_MESSAGE.lower().rstrip("."), "upload_missing"),
("source file is missing", "source_missing"),
# Private / deleted / removed / terminated.
("private video", "unavailable"),
@@ -89,6 +144,13 @@ _RULES: tuple[tuple[str, str], ...] = (
("postprocessing", "postprocess"),
("produced no output file", "postprocess"),
("no output file", "postprocess"),
# yt-dlp's own verdict that its extractor did not understand the site's answer: it appends this
# request for a bug report to an UNEXPECTED extraction failure, never to an expected one (a private,
# removed or geo-blocked video says so in words matched above). Measured 2026-10-12: a TikTok link
# failed with it on every attempt under yt-dlp 2026.07.04 and downloads under 2026.08.19 — the
# site changed and a newer yt-dlp is the cure. After the media-format and post-processing rules,
# whose texts can carry the same request; before the network ones, which are about the transport.
("please report this issue", "site_changed"),
# Network / transport.
("unable to download", "network"),
("http error", "network"),
@@ -101,6 +163,120 @@ _RULES: tuple[tuple[str, str], ...] = (
)
class Diagnosis(NamedTuple):
"""What a failure was: the generic `code` (category) and, when a catalog entry matched, its
host-specific `detail`."""
code: str
detail: str | None = None
@dataclass(frozen=True)
class _Entry:
"""One measured failure of one site. It matches when the download URL belongs to `host`
(`credentials.SUPPORTED_HOSTS`), every one of `markers` is in the message, the cookie state is
`cookie` (None: either), and — when `link` is given — it recognises the URL (a truthy result). A
link predicate is the adapter's own, so the catalog and the download agree on what a link is."""
host: str
markers: tuple[str, ...]
code: str
detail: str
cookie: bool | None = None
link: Callable[[str], object] | None = None
_CATALOG: tuple[_Entry, ...] = (
# Facebook answers a signed-out client with a page yt-dlp cannot parse — an 18+ reel, a closed
# group's post (measured 2026-09-13: prod jobs 145, 156, 256 download once the owner's cookie is
# attached). "Cannot parse data" is yt-dlp's generic wording for a broken extractor on EVERY site,
# which is why the entry is scoped to Facebook and never becomes a general rule.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.login_wall",
cookie=False,
),
# The same text WITH a cookie: an expired / signed-out cookie, or a page that also needs
# impersonation (prod job 257). Telling this user to add cookies would be wrong — they have.
_Entry(
host="facebook",
markers=("[facebook]", FACEBOOK_PARSE_WALL),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post read with a cookie that Facebook answered with its login page (measured with a
# refused cookie: no post id anywhere on the page) — the same refused cookie, found by the adapter.
_Entry(
host="facebook",
markers=(POST_NOT_SHOWN_MESSAGE.lower(),),
code="login_required",
detail="facebook.cookie_rejected",
cookie=True,
),
# A group post with no single video of its own. With a cookie the adapter reads the post's page
# and says so itself (POST_WITHOUT_VIDEO_MESSAGE); without one yt-dlp is given the post link and
# answers "No video formats found" (measured on prod jobs 66 and 145, whose videos are reshared
# clips). With a cookie yt-dlp only ever gets the post's VIDEO link, so the same text there is about
# that video, not the post — the category says it. Only for group post links (the adapter's own
# test of one): a reel with no formats is a different failure.
_Entry(
host="facebook",
markers=(POST_WITHOUT_VIDEO_MESSAGE.lower(),),
code="format_unavailable",
detail="facebook.group_post_video",
link=facebook_group_post_id,
),
_Entry(
host="facebook",
markers=("[facebook]", "no video formats found"),
code="format_unavailable",
detail="facebook.group_post_video",
cookie=False,
link=facebook_group_post_id,
),
# YouTube gates INDIVIDUAL videos behind "Sign in to confirm you're not a bot" (measured on prod
# job 273, a festival set: all 12 of yt-dlp's player clients fail on that one video while its
# neighbours download fine, the POT sidecar is live and answering, and the newest yt-dlp fails
# identically — so this is the video's own signed-out gate, not a broken stack or a stale yt-dlp).
# The marker is quote-agnostic on purpose: YouTube writes the apostrophe as U+2019, so
# "you're not a bot" with a straight quote never matches the real text.
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.bot_wall",
cookie=False,
),
# The same wall WITH a cookie. Not yet reproduced with a real YouTube cookie (this user's first
# one is being added now) — but "add cookies" is provably the wrong advice for someone who has,
# and yt-dlp names the likeliest cause itself: "The provided YouTube account cookies are no
# longer valid. They have likely been rotated in the browser as a security measure."
_Entry(
host="youtube",
markers=("[youtube]", "not a bot"),
code="login_required",
detail="youtube.cookie_rejected",
cookie=True,
),
# The same gate one step later: the player response comes back, and then the CDN refuses the
# stream itself ("unable to download video data: HTTP Error 403"). Measured 2026-09-16 on the
# very same video, ~90 minutes after it was answering with the bot message — signed out it
# 403'd on every attempt; with the owner's cookie attached it downloaded in full (519 MB). So
# the category this text carries generically, `network`, is a lie here: nothing was wrong with
# the network. Scoped to a cookie-less YouTube download, where that is what we measured.
_Entry(
host="youtube",
markers=("unable to download video data", "403"),
code="login_required",
detail="youtube.media_forbidden",
cookie=False,
),
)
def classify(msg: str | None) -> str:
"""Return a category code for a raw failure message (``"unknown"`` if nothing matches)."""
if not msg:
@@ -112,6 +288,36 @@ def classify(msg: str | None) -> str:
return "unknown"
def classify_conversion(msg: str | None) -> str:
"""The category of an upload's failed conversion. The generic ffmpeg categories speak of a
download ("the video downloaded but…"), which an upload never was: a failed encode or an
unrecognised failure of one is `convert_failed`; a category of its own (a full quota, a missing
original) stays as it is."""
code = classify(msg)
return "convert_failed" if code in ("postprocess", "unknown") else code
def diagnose(
msg: str | None, *, url: str | None = None, cookie_attached: bool = False
) -> Diagnosis:
"""The catalog entry for this failure of this link, else just its category (see the module doc).
`url` is the address yt-dlp was given; `cookie_attached` whether the acting user's cookie went
with it. Without a URL only the category can be known."""
host = host_key_for_url(url) if msg and url else None
if host is not None:
low = msg.lower() # type: ignore[union-attr] — `host` is only set when `msg` is truthy
for e in _CATALOG:
if (
e.host == host
and all(m in low for m in e.markers)
and (e.cookie is None or e.cookie == cookie_attached)
and (e.link is None or e.link(url)) # type: ignore[arg-type] — as `msg` above
):
return Diagnosis(e.code, e.detail)
return Diagnosis(classify(msg))
def user_error_text(source_kind: str, msg: str, code: str) -> str:
"""The failure text to STORE on the job, given the raw message and its category.
+345
View File
@@ -0,0 +1,345 @@
"""The folders of the Files library — where each of a user's files sits.
A folder is a row, never a directory: on disk a file stays where the download layout put it (one
shared asset can be held by several users), so a rename or a move is a column update. Three kinds:
- a SOURCE folder is made automatically, one per origin a user has files from (YouTube, Plex clips,
uploads, one per other site). Its `source_key` names the origin; the client renders its name
from the key, in the reader's language.
- an OWN folder is the user's, with the name they gave it.
- the HUB folder (one per user, made when they first give a machine a token for it, `hub_folder`) is
where they swap files with that machine. Its name, like a source folder's, is the client's; a file
may be moved in like into an own folder, and it keeps a file until it is deleted.
Every file of a user is in exactly one folder. The folder a new file lands in is decided here
(`source_key_for` + `source_folder`), at every place a job is created, so the rule lives once. A file
moves only into one of the user's own folders, their Hub folder, or back to its own source folder
(`move`): a YouTube file in the TikTok folder would be a lie the folder's name tells.
Retention is the folder's: a source folder keeps a file `download_retention_days` (the value when the
folder was made; the user can change it), Uploads, own folders and the Hub keep it until it is deleted. A
done file carries its own `expires_at`, stamped by `done_expiry` when it finishes — unless the user
kept it — and its `ready_at`, the moment it finished (`stamp_done`).
"""
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from urllib.parse import urlsplit
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app import sysconfig
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, FileFolder, MediaAsset
from app.userscope import owned_by
class FolderKind(StrEnum):
SOURCE = "source"
OWN = "own"
HUB = "hub"
class SourceKey(StrEnum):
"""The fixed origins. Any other site is `site:<host>` (see `site_key`)."""
YOUTUBE = "youtube"
PLEX = "plex"
UPLOAD = "upload"
SITE_PREFIX = "site:"
# A host's leading label that only says "the same site, another front door".
_FRONT_DOORS = ("www.", "m.", "mobile.", "web.")
_YOUTUBE_HOSTS = {"youtube.com", "youtu.be", "youtube-nocookie.com", "music.youtube.com"}
def site_key(url: str) -> str:
"""The source key of a page on some site: `site:<host>` without its front door (www., m.),
or the YouTube key for a YouTube address that arrived as a plain URL."""
host = (urlsplit(url).hostname or "").lower().rstrip(".")
for door in _FRONT_DOORS:
if host.startswith(door) and host.count(".") > 1:
host = host[len(door) :]
break
if host in _YOUTUBE_HOSTS:
return SourceKey.YOUTUBE
return f"{SITE_PREFIX}{host or 'unknown'}"
def source_key_for(db: Session, job: DownloadJob) -> str:
"""The origin a job's file came from — the source folder it belongs in.
The one recorded when it was placed, else: youtube → YouTube; a clip → the clip's source
(YouTube or Plex); an upload → Uploads; a URL → its site; an edit → wherever its parent's file
came from (its parent job, or, once that is deleted, the parent's asset)."""
if job.source_key:
return job.source_key
kind = job.source_kind
if kind == "youtube":
return SourceKey.YOUTUBE
if kind == "upload":
return SourceKey.UPLOAD
if kind == "clip":
return (
SourceKey.PLEX if (job.edit_spec or {}).get("source") == "plex" else SourceKey.YOUTUBE
)
if kind == "url":
return site_key(job.source_ref)
if kind == "edit":
parent = db.get(DownloadJob, job.source_job_id) if job.source_job_id else None
if parent is not None and parent.id != job.id:
return source_key_for(db, parent)
asset = db.get(MediaAsset, job.source_asset_id) if job.source_asset_id else None
if asset is not None:
if asset.source_kind == "youtube":
return SourceKey.YOUTUBE
if asset.source_kind == "upload":
return SourceKey.UPLOAD
if asset.source_kind == "url":
return site_key(asset.source_ref)
return SourceKey.UPLOAD
def source_folder(db: Session, user_id: int, key: str) -> FileFolder:
"""The user's source folder for `key`, made on first use. A race on the unique (user, key) is
settled by re-selecting, inside a SAVEPOINT so the caller's own pending writes survive."""
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.source_key == key
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
retention = (
None if key == SourceKey.UPLOAD else sysconfig.get_int(db, "download_retention_days")
)
folder = FileFolder(
user_id=user_id, kind=FolderKind.SOURCE, source_key=key, retention_days=retention
)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: the expected outcome of the race the savepoint exists for — another request made
# the same folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def hub_folder(db: Session, user_id: int) -> FileFolder:
"""The user's Hub folder, made on first use; a race on its one-per-user index is settled like
`source_folder`'s."""
stmt = select(FileFolder).where(
owned_by(FileFolder.user_id, user_id), FileFolder.kind == FolderKind.HUB
)
folder = db.execute(stmt).scalar_one_or_none()
if folder is not None:
return folder
folder = FileFolder(user_id=user_id, kind=FolderKind.HUB)
try:
with db.begin_nested():
db.add(folder)
db.flush()
# silent-ok: another request made the Hub folder first; re-selecting it is the whole handling.
except IntegrityError:
folder = db.execute(stmt).scalar_one()
return folder
def place(db: Session, job: DownloadJob) -> None:
"""Put a new job's file in its source folder, and record that origin on it. Called wherever a
job is created; the job must carry its `user_id`, `source_kind` and source refs already."""
key = source_key_for(db, job)
job.source_key = key
job.folder_id = source_folder(db, job.user_id, key).id
def done_expiry(db: Session, folder_id: int | None, kept: bool) -> datetime | None:
"""When a file that finishes now goes: its folder's retention from now, or never (kept, or a
folder that keeps files)."""
if kept or folder_id is None:
return None
# Read under a share lock, held to the caller's commit: a retention change (`set_retention` locks
# the folder) waits for this file and restamps it too, or this read waits for the change and
# takes the new value — never the old value stamped on a file the change did not see.
days = db.execute(
select(FileFolder.retention_days)
.where(FileFolder.id == folder_id)
.with_for_update(read=True)
).scalar()
return None if days is None else datetime.now(timezone.utc) + timedelta(days=days)
def stamp_done(db: Session, job: DownloadJob) -> None:
"""Stamp a job that just became done (in this session): when it did, and its expiry."""
job.ready_at = datetime.now(timezone.utc)
job.expires_at = done_expiry(db, job.folder_id, job.kept)
job.expiry_warned = False
class FolderError(Exception):
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
def __init__(self, reason: str) -> None:
super().__init__(reason)
self.reason = reason
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The user's folder `folder_id` that a file is about to be put in — held (FOR SHARE) until the
caller's transaction ends, so a delete of the folder waits for the file and then takes it home
with the rest (`delete_own` locks it FOR UPDATE), never failing on its foreign key. `hold=False`
only looks (a check ahead of a long wait, which must not hold a lock through it). Another user's
folder answers like a missing one."""
stmt = select(FileFolder).where(FileFolder.id == folder_id)
if hold:
stmt = stmt.with_for_update(read=True)
folder = db.execute(stmt.execution_options(populate_existing=True)).scalar_one_or_none()
if folder is None or folder.user_id != user_id:
raise FolderError("file_folder_gone")
return folder
def upload_target(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
"""The folder an upload asked to land in, if it may: one of the user's own folders, their Hub
folder, or their Uploads folder. A file put in a site's folder would claim an origin it does not
have."""
folder = own_folder(db, user_id, folder_id, hold=hold)
if folder.kind == FolderKind.SOURCE and folder.source_key != SourceKey.UPLOAD:
raise FolderError("file_folder_not_its_source")
return folder
def restamp(db: Session, job: DownloadJob) -> None:
"""Start a file's clock again from now: it moved to another folder, or its user stopped keeping
it. A file not done yet gets its expiry when it finishes (`stamp_done`), so it has none now."""
done = job.status == JobStatus.DONE
job.expires_at = done_expiry(db, job.folder_id, job.kept) if done else None
job.expiry_warned = False
def set_kept(db: Session, job: DownloadJob, kept: bool) -> None:
"""Keep a file until its user deletes it, or hand it back to its folder's retention."""
job.kept = kept
restamp(db, job)
def move(db: Session, job: DownloadJob, folder: FileFolder) -> None:
"""Put a file in `folder` — one of its user's own folders, their Hub folder, or the source folder
it came from.
The caller has checked that `folder` is the job's user's."""
if folder.kind == FolderKind.SOURCE and folder.source_key != source_key_for(db, job):
raise FolderError("file_folder_not_its_source")
if folder.id == job.folder_id:
return
job.folder_id = folder.id
restamp(db, job)
def _hold_for_change(db: Session, folder: FileFolder) -> None:
"""Lock `folder` FOR UPDATE and read it afresh into the same object: a change waits out a file
being put in it or stamped from it right now (`own_folder`, `done_expiry`), and works from what
another change that landed first left, not from what the caller read before the wait. The row is
consumed on purpose: an ORM result refreshes the object only as its rows are fetched."""
fresh = db.execute(
select(FileFolder)
.where(FileFolder.id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if fresh is None:
raise FolderError("file_folder_gone")
def set_retention(db: Session, folder: FileFolder, days: int | None) -> int | None:
"""Change how long `folder` keeps a file (None = until deleted). The done files already in it
move by the difference — each keeps the age it has — but none ends up closer than the GC's
warning window, so a shortened retention still warns before it takes anything. Returns the
retention it replaced (equal to `days` when nothing changed)."""
_hold_for_change(db, folder) # so the file arriving now is among the ones restamped below
old = folder.retention_days
if old == days:
return old
folder.retention_days = days
now = datetime.now(timezone.utc)
floor = now + timedelta(days=sysconfig.get_int(db, "download_gc_grace_days"))
# Locked as they are read: one moved out or deleted meanwhile is no longer this folder's to time.
jobs = db.execute(
select(DownloadJob)
.where(
DownloadJob.folder_id == folder.id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
)
.with_for_update()
.execution_options(populate_existing=True)
).scalars()
for job in jobs:
if days is None:
job.expires_at = None
elif old is None or job.expires_at is None:
job.expires_at = max(now + timedelta(days=days), floor)
else:
job.expires_at = max(job.expires_at + timedelta(days=days - old), floor)
job.expiry_warned = False
return old
def delete_own(db: Session, folder: FileFolder) -> int:
"""Delete one of a user's own folders. Its files are not deleted: each goes back to the source
folder it came from, its clock starting there now. Returns how many moved."""
if folder.kind != FolderKind.OWN:
raise FolderError("file_folder_not_own")
_hold_for_change(db, folder) # so the file arriving now is taken home too
# Locked as they are read: a file moved out meanwhile stays where it went, and one deleted
# meanwhile is not written back (its gone row would fail the delete).
jobs = (
db.execute(
select(DownloadJob)
.where(DownloadJob.folder_id == folder.id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
for job in jobs:
job.folder_id = source_folder(db, job.user_id, source_key_for(db, job)).id
restamp(db, job)
db.flush() # the files leave before the folder goes: its foreign key RESTRICTs the delete
db.delete(folder)
return len(jobs)
def create_own(db: Session, user_id: int, name: str) -> FileFolder:
"""A new folder of the user's own; FolderError("file_folder_name_taken") when they already have
one by that name (the unique (user, lower(name)))."""
folder = FileFolder(user_id=user_id, kind=FolderKind.OWN, name=name)
try:
with db.begin_nested():
db.add(folder)
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
return folder
def renamable(folder: FileFolder) -> bool:
"""Only a folder of the user's own making takes a name: a source folder's is its origin's."""
return folder.kind == FolderKind.OWN
def rename(db: Session, folder: FileFolder, name: str) -> None:
"""Rename one of the user's own folders (a source folder's name is its origin's)."""
if not renamable(folder):
raise FolderError("file_folder_not_own")
folder.name = name
try:
with db.begin_nested():
db.flush()
except IntegrityError as e:
raise FolderError("file_folder_name_taken") from e
+204 -18
View File
@@ -13,12 +13,162 @@ spec shape (all keys optional; defaults + browser-playability clamp applied by `
vcodec always coerced to null → H.264 selected (see `normalize`); no opt-out
embed_subs bool embed_chapters bool embed_thumbnail bool sponsorblock bool
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser (E7); the
`normalize` clamps container/vcodec/audio_format so EVERY download plays in-browser; the
`container`/`vcodec` keys survive only so an old stored profile deserializes without error.
"""
import contextlib
import contextvars
import hashlib
import json
import logging
import os
import re
import urllib.parse
from collections.abc import Iterator
from app import logs
log = logs.get_logger(__name__)
# yt-dlp names what it is working on in these shapes, and each one is a row of someone's download
# history: the source URL, the `[extractor] <id>:` prefix on nearly every warning and error, the
# file path it writes to (`storage.rel_path` builds it from the channel and the title), a
# playlist's title, and an id quoted mid-sentence. The job id is already on every line
# (`logs.set_job` in the download child), so none of them adds anything an operator needs. What
# stays: the host (it says which extractor is failing) and the extractor tag. Every pattern below
# is a shape measured in yt-dlp's own source (2026.07.04), pinned in test_ytdlp_logging.py.
_URL_RE = re.compile(r"https?://([^/\s]+)\S*")
# `Destination:` is yt-dlp's own word in `[download] Destination: <path>`, not an id — the path
# after it is what `_PATH_RE` takes out.
_ID_PREFIX_RE = re.compile(r"(\[[\w:.-]+\] )(?!Destination:)[^\s:\[\]]+:")
# The generic extractor's "id" is the URL's basename, percent-DECODED: `My%20Holiday.mp4` is
# `My Holiday` — spaces, brackets, colons and all, which no pattern can bound (`Our Trip: Day 1`
# ends at which `: `?). The job knows its own URL, so the exact name is taken out as a literal
# (`hiding`). This pattern is the fallback for a line logged outside that scope: up to the first
# `: `. Only this extractor: every other one names a site id, and letting the prefix run to `: `
# everywhere would take the words out of lines like `[VideoConvertor] Converting …; Destination:`.
_GENERIC_ID_RE = re.compile(r"(\[generic\] )(?!Destination:)[^\[\]\n]+?:(?= |$)", re.M)
# `[youtube:tab] Playlist <title>: Downloading 5 items` — the title is not an id, and has spaces.
_PLAYLIST_TITLE_RE = re.compile(r"(\] Playlist ).+?(?=: Downloading )")
# `[download] Finished downloading playlist: <title>`, `[download] Downloading playlist: <title>`,
# `… Skipping already downloaded playlist: <title>` — the title runs to the end of the line.
_TITLE_TAIL_RE = re.compile(r"(playlist: |\[download\] Downloading \w+: ).*", re.I)
# An id quoted mid-sentence, not as a prefix: `… not available for <id>`, and `--no-playlist`'s
# `Downloading just the video <id>` (`_yes_playlist`) / `Downloading just video <id>` (multifeed).
_BARE_ID_RE = re.compile(
r"((?:subtitles not available for|Downloading just (?:the )?\w+) )[^\s,;]+"
)
# Two segments or more, and not glued to a word: `www.youtube.com/…` is what the URL pattern
# leaves behind, and it must survive this one.
_PATH_RE = re.compile(r"(?<![\w.…])/(?:[^/\s\"']+/)+[^/\s\"']*")
_HIDDEN: contextvars.ContextVar[tuple[str, ...]] = contextvars.ContextVar(
"formats_hidden", default=()
)
def generic_id(url: str) -> str:
"""The name yt-dlp's generic extractor gives a URL: its basename, extension off, percent-
decoded — `InfoExtractor._generic_id` (2026.07.04), pinned against it in the tests."""
return urllib.parse.unquote(os.path.splitext(url.rstrip("/").split("/")[-1])[0])
@contextlib.contextmanager
def hiding(url: str | None) -> Iterator[None]:
"""For the work on one URL: `log_safe` also takes out the name yt-dlp's generic extractor
gives it, exactly, where yt-dlp puts it (`[generic] <name>:`). A context variable, so it
holds for yt-dlp's logger calls too — they run on the same thread as the extraction."""
name = generic_id(url) if url else ""
token = _HIDDEN.set((*_HIDDEN.get(), name) if name else _HIDDEN.get())
try:
yield
finally:
_HIDDEN.reset(token)
def log_safe(text: object) -> str:
"""yt-dlp's text (its own lines, or a DownloadError's message) with the parts that name a
person's download taken out. Every log line that carries yt-dlp's words goes through here,
not just the adapter's: yt-dlp raises the same text it has just logged, so a line that skipped
it would put back one line later exactly what the adapter took out."""
text = str(text)
for name in _HIDDEN.get():
text = text.replace(f"[generic] {name}:", "[generic] …:")
text = _URL_RE.sub(lambda m: f"{m.group(1)}/…", text)
text = _ID_PREFIX_RE.sub(r"\1…:", text)
text = _GENERIC_ID_RE.sub(r"\1…:", text)
text = _PLAYLIST_TITLE_RE.sub(r"\1…", text)
text = _TITLE_TAIL_RE.sub(r"\1…", text)
text = _BARE_ID_RE.sub(r"\1…", text)
return _PATH_RE.sub("…", text)
# yt-dlp's HTTP downloader, refusing a file announced larger than `max_filesize`, logs this line and
# RETURNS — no exception and no file (measured; test_formats holds it against the pinned yt-dlp).
_OVER_CAP_LINE = "File is larger than max-filesize"
_OVER_CAP: contextvars.ContextVar[list[bool] | None] = contextvars.ContextVar(
"formats_over_cap", default=None
)
@contextlib.contextmanager
def watching_size_cap() -> Iterator[list[bool]]:
"""While yt-dlp works inside it: a one-slot flag, set when it refused a file over the size cap
(`build_ydl_opts(max_bytes=…)`). It then returns as if it had downloaded, and all the caller
would find is no file — a processing fault, to anyone reading it, for what is the owner's
storage. A context variable, like `hiding`: yt-dlp's logger calls run on the extraction's thread."""
flag = [False]
token = _OVER_CAP.set(flag)
try:
yield flag
finally:
_OVER_CAP.reset(token)
class _YdlLog:
"""yt-dlp's logger protocol, pointed at ours.
Without it yt-dlp explains itself to nobody: `quiet` sends its output to a stdout we do not
read, so "Requested format is not available", a signature-extraction failure or a POT refusal
reached us only as whatever text ended up in the raised exception. Measured against yt-dlp
2026.07.04: with `logger` set it calls `debug` for extraction and progress lines and
`warning`/`error` for the rest, and `quiet`/`no_warnings` do NOT gate those calls.
`debug` goes to TRACE: it is several lines per second during a download, so it shows only
when the whole process runs at TRACE. (Not under a user's own Teljes consent — that lives in
the API process's request scope, and the download child that runs yt-dlp has none.)
`error` goes to WARNING, not ERROR. yt-dlp calls it from `trouble()` just before it raises,
on EVERY failed attempt — including the ones the pipeline then recovers from with its fallback
client set. Whether the job failed is the pipeline's verdict (`download.job.failed`), and that
is the one line whose level should say so."""
def _emit(self, level: int, msg: str) -> None:
if _OVER_CAP_LINE in msg and (over := _OVER_CAP.get()) is not None:
over[0] = True
# The line goes as an ARGUMENT, not as the format string. yt-dlp writes progress as
# `[download] 53.2% of 12.00MiB`, and while stdlib logging only applies `%` formatting to
# a record that has args, this way the question never arises — and it is the shape ruff's
# `G` rules ask for. (The sink's flood limiter groups a bare `%s` by the text itself, so
# distinct yt-dlp warnings do not share one bucket — `logsink._template`.)
log.log(level, "%s", log_safe(msg), extra=logs.event("download.ytdlp"))
def debug(self, msg):
self._emit(logs.TRACE, msg)
def info(self, msg):
self._emit(logging.DEBUG, msg)
def warning(self, msg):
self._emit(logging.WARNING, msg)
def error(self, msg):
self._emit(logging.WARNING, msg)
_YDL_LOG = _YdlLog()
# Fields that affect the produced file — and therefore the cache identity.
_SIG_KEYS = (
@@ -56,7 +206,13 @@ _DEFAULTS = {
# v4: EVERY download is clamped to the mp4/H.264/AAC compat profile (no more mkv/webm/VP9/AV1/Opus
# opt-out), so any asset cached under an old non-compat spec — or a stale non-H.264 mp4 that slipped
# the v3 net — must re-derive through the current browser-safe guarantee.
_SIG_VERSION = 4
# v5: the phone-safe rule (`playable`) also judges the pixel format, so a cached 10-bit / 4:4:4 H.264
# asset (which v4 let through as "H.264") re-derives as 8-bit 4:2:0 instead of being reused.
_SIG_VERSION = 5
# Audio-only downloads version on their own, so a change to the audio bytes doesn't re-download every
# cached video. v5: an extracted .mp3/.m4a is stored as itself instead of remuxed into an .mp4 that
# iPad's WebKit refuses to play, so a cached remuxed audio asset must re-derive.
_AUDIO_SIG_VERSION = 5
_SPONSORBLOCK_CATEGORIES = ["sponsor", "selfpromo", "interaction"]
@@ -74,21 +230,28 @@ def normalize(spec: dict) -> dict:
# rather than raising an unhandled 500 at enqueue.
try:
s["max_height"] = int(s["max_height"])
except (TypeError, ValueError):
except (TypeError, ValueError) as exc:
# "best" is the opposite of what a height cap asks for: the user set a ceiling to save
# bandwidth or disk, and the download quietly takes the largest rendition there is.
log.warning(
"the height cap on a download could not be read and was dropped",
extra=logs.event("download.format.max_height_dropped", reason=type(exc).__name__),
)
s["max_height"] = None
for b in ("embed_subs", "embed_chapters", "embed_thumbnail", "sponsorblock"):
s[b] = bool(s[b])
# Audio-only files can't embed subtitles/thumbnails as video; keep the flags meaningful.
if s["mode"] == "a":
s["embed_subs"] = False
# Browser-playability guarantee (E7): the Download module exists to download → share → PLAY
# Browser-playability guarantee: the Download module exists to download → share → PLAY
# in-browser, so every output must play in EVERY browser, iOS/Safari/WebKit included. The only
# universally-safe combo is mp4 + H.264 + AAC/MP3, so we CLAMP the spec to it HERE — the single
# point every path (picker, stored profile, raw API) flows through — instead of only hiding
# options in the UI. Non-mp4 containers (mkv doesn't play at all; webm/VP9/AV1 fail iOS) and any
# explicit vcodec are coerced to the compat profile; Opus audio (Safari-flaky as a bare file)
# falls back to AAC/m4a. A source that exposes no H.264 is still caught by the worker's
# browser-safe re-encode (`_ensure_browser_playable`), so playback is guaranteed end-to-end.
# universally-safe combo is mp4 + H.264 + AAC (an .mp3/.m4a for audio), so we CLAMP the spec to
# it HERE — the single point every path (picker, stored profile, raw API) flows through — instead
# of only hiding options in the UI. Non-mp4 containers (mkv plays nowhere on iOS; AV1 fails there,
# and webm/VP9 plays only on recent devices) and any explicit vcodec are coerced to the compat
# profile; Opus audio falls back to AAC/m4a. A source that exposes no H.264 (or a 10-bit one) is
# still caught by the worker's phone-safe step (`downloads.playable`), so playback is guaranteed
# end-to-end.
s["container"] = "mp4"
s["vcodec"] = None
if s["audio_format"] not in ("m4a", "mp3"):
@@ -96,20 +259,34 @@ def normalize(spec: dict) -> dict:
return s
def format_sig(spec: dict, salt: str | None = None) -> str:
def _sig_version(s: dict) -> int:
return _AUDIO_SIG_VERSION if s["mode"] == "a" else _SIG_VERSION
def format_sig(spec: dict, salt: str | None = None, *, version: int | None = None) -> str:
"""Stable short hash of the output-affecting spec — the MediaAsset cache key component.
`salt` forks a distinct, non-dedup identity for the same spec (used by force-redownload so it
always re-fetches into its own asset instead of sharing a cached file with other jobs)."""
always re-fetches into its own asset instead of sharing a cached file with other jobs).
`version` hashes under a signature version other than the current one (see `format_sigs`)."""
s = normalize(spec)
data = {k: s[k] for k in _SIG_KEYS}
data["_v"] = _SIG_VERSION
data["_v"] = _sig_version(s) if version is None else version
if salt is not None:
data["_salt"] = salt
payload = json.dumps(data, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode()).hexdigest()[:24]
def format_sigs(spec: dict, salt: str | None = None) -> tuple[str, str]:
"""`format_sig` under the current signature version and under the one before it. An asset keeps
the signature it was created with, so a job queued before a deploy that bumped the version and
run after it downloads into an asset carrying the previous one — which must still be recognised
as that user's private asset (`credentials.is_authed_asset`), or it downloads without its cookie."""
previous = _sig_version(normalize(spec)) - 1
return format_sig(spec, salt), format_sig(spec, salt, version=previous)
def build_ydl_opts(
spec: dict,
outtmpl: str,
@@ -124,12 +301,15 @@ def build_ydl_opts(
`extractor_args` with the player-client set + bgutil PO-token base_url); it is merged onto the
generic options below. Everything here is source-agnostic.
`cookiefile` is the acting user's per-host cookies.txt path (S6), attached ONLY by the worker
`cookiefile` is the acting user's per-host cookies.txt path, attached ONLY by the worker
after the host-scope + private-asset gate in `downloads.credentials`. It is set AFTER the `extra`
merge so a source adapter can never clobber it.
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp refuses a format larger
than it rather than filling the disk and only then being caught by the quota accounting."""
`max_bytes` (SEC-13) is the caller's remaining storage budget; yt-dlp's HTTP downloader refuses
a file announced larger than it rather than filling the disk and only then being caught by the
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
pipeline also holds every produced file to what is left by its real size."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -163,6 +343,12 @@ def build_ydl_opts(
if extra:
opts.update(extra)
# With a logger attached, `quiet` and `no_warnings` stop deciding whether yt-dlp explains itself
# and only decide where — so both stay on above and the explanation comes here instead of to a
# stdout nothing reads. AFTER the adapter merge, like `cookiefile` and the size cap: an adapter
# must not be able to silence the one account of what went wrong that a failed job leaves.
opts["logger"] = _YDL_LOG
# SEC-13: hard ceiling on a single download. AFTER the adapter merge, for the same reason
# `cookiefile` is — a source adapter must not be able to raise a limit the quota layer set.
#
@@ -173,7 +359,7 @@ def build_ydl_opts(
if max_bytes and max_bytes > 0:
opts["max_filesize"] = max_bytes
# The acting user's per-host cookie (S6). AFTER the adapter merge on purpose: an adapter must not
# The acting user's per-host cookie. AFTER the adapter merge on purpose: an adapter must not
# be able to override the authenticated session the worker resolved for this specific user.
if cookiefile:
opts["cookiefile"] = cookiefile
@@ -195,7 +381,7 @@ def build_ydl_opts(
# inside mp4 — a VP9/AV1-in-mp4 file shows a broken player there — so H.264 is selected at the
# SELECTION level, not merely preferred: a sort key can't conjure an H.264 stream a source
# doesn't expose as video-only. A source that ships only non-H.264 is caught after download by
# the worker's browser-safe re-encode (`_ensure_browser_playable`).
# the worker's phone-safe step (`downloads.playable`).
if s["mode"] == "v":
# Video-only: keep it audio-free. Prefer an H.264 video-only stream, then any video-only,
# and only fall to a progressive (audio-bearing) `best` as a last resort.
+246 -101
View File
@@ -1,84 +1,167 @@
"""Download retention garbage collector (a scheduler job).
Five passes each run:
1. Pre-expiry warning — for ready assets expiring within the grace window, notify each holder
once (gc_notified flag) so a download isn't silently deleted.
2. Expiry deletion — delete assets past their TTL (files + row). The FK SET NULL nulls the
holders' jobs' asset_id, so those jobs surface as "expired" (re-downloadable) in the UI.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed
ready assets until back under it.
Six passes each run:
1. Pre-expiry warning — a done file whose `expires_at` (its folder's retention, `folders`) falls
within the grace window, and that its user did not keep, warns its user once (`expiry_warned`).
2. Expiry — such a file past its `expires_at` leaves the library: the job is deleted and its hold
released, so the shared file goes with its last holder.
3. LRU eviction — if a total cache cap is set and exceeded, delete least-recently-accessed ready
assets (never an upload, never one a user kept) until back under it; each holder's file leaves
their library, with a notice.
4. Orphan reap — delete errored, fileless, unreferenced asset rows (they carry no TTL, so
passes 1-3 skip them; benign row-count hygiene).
5. Orphan FILE sweep — delete on-disk files/dirs no surviving asset owns (cruft left when a
re-download renames an asset, e.g. old accented channel dirs), skipping system trees and
anything newer than an hour so an in-flight download is never touched.
re-download renames an asset, e.g. old accented channel dirs, or an upload's stranded file).
It skips the dot-prefixed system trees (all but `.uploads`) and anything newer than an hour,
so an in-flight download is never touched.
6. Token expiry warning — a machine's token for the library about to run out warns its user once
(`apitokens.warn_expiring`): the library's other clock.
Runs in the API process (which mounts DOWNLOAD_ROOT). Pure disk/DB work — no YouTube quota.
Runs in the API process (which mounts FILES_ROOT). Pure disk/DB work — no YouTube quota.
"""
import logging
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
from sqlalchemy import func, select
from sqlalchemy import func, or_, select, update
from sqlalchemy.orm import Session
from app import sysconfig
from app import apitokens, logs, sysconfig
from app.config import settings
from app.downloads import storage
from app.models import DownloadJob, MediaAsset
from app.downloads import quota, service, storage
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, MediaAsset
from app.notifications import create_notification
log = logging.getLogger("siftlode.downloads.gc")
# Dot-prefixed top-level trees that are NOT part of the Plex/flat asset layout — the orphan sweep
# never touches them (user clips, Plex HLS/img caches, in-progress staging, editor storyboards).
_SYSTEM_TREE = {".edits", ".plex-hls", ".plex-img-cache", ".staging", ".storyboards"}
log = logs.get_logger(__name__)
# The one dot-prefixed top-level tree the orphan sweep enters. The others are not part of the asset
# layout (user clips, Plex HLS/img caches, in-progress staging, editor storyboards) and are skipped.
_UPLOAD_TREE = ".uploads"
_ORPHAN_MIN_AGE_S = 3600 # only sweep files older than this, so an in-flight download is never hit
def _holders(db: Session, asset_id: int) -> list[int]:
return list(
db.execute(
select(DownloadJob.user_id).where(DownloadJob.asset_id == asset_id).distinct()
).scalars()
def _notify(db: Session, job: DownloadJob, asset: MediaAsset | None, ntype: str) -> None:
title = job.display_name or (asset.title if asset else None) or "download"
create_notification(
db,
user_id=job.user_id,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={
"kind": ntype,
"title": title,
"video_id": asset.source_ref if asset else None,
"job_id": job.id,
},
commit=False,
)
def _notify_holders(db: Session, asset: MediaAsset, ntype: str) -> None:
title = asset.title or "download"
for uid in _holders(db, asset.id):
create_notification(
db,
user_id=uid,
type=ntype,
title=title, # English fallback; the client renders a translated message from data
data={"kind": ntype, "title": asset.title, "video_id": asset.source_ref},
commit=False,
def _relock_expired(
db: Session, job_id: int, asset_id: int | None, now: datetime
) -> DownloadJob | None:
"""Re-read an expired job under the lock order the user routes take (`own_job_for_update`: the
asset, then the job) and say whether it is still to go. Between the pass's SELECT and here a
user may have deleted or re-downloaded it (dropping it again would release its asset a second
time, freeing a file another job still holds) or kept or moved it (it is no longer expired)."""
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
still = (
job is not None
and job.asset_id == asset_id
and job.status == JobStatus.DONE
and not job.kept
and job.expires_at is not None
and job.expires_at <= now
)
return job if still else None
def _mark_warned(db: Session, job_id: int, expires_at: datetime) -> DownloadJob | None:
"""Mark one file warned if it still goes when the pass saw it go — not deleted, kept, moved or
re-timed since (each of those sets a fresh expiry, and its own warning to come) — and return it.
The UPDATE's row lock holds it so until the commit."""
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at == expires_at,
)
.values(expiry_warned=True)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
return db.get(DownloadJob, job_id) if hit is not None else None
def _delete_asset(db: Session, asset: MediaAsset, ntype: str) -> None:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
_notify_holders(db, asset, ntype)
db.delete(asset) # jobs.asset_id -> NULL via FK ON DELETE SET NULL
def _on_disk(db: Session) -> int:
"""The bytes the cache holds now (every asset whose file is on disk)."""
return (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).scalar()
or 0
)
def _sweep_orphan_files(db: Session, download_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under DOWNLOAD_ROOT that no `ready` asset
def _relock_evictable(db: Session, asset_id: int) -> tuple[MediaAsset | None, list[DownloadJob]]:
"""Re-read an eviction candidate and its holders under their locks (the asset, then its jobs —
the order the user routes take): between the pass's SELECT and here a holder may have kept it,
or every holder deleted it."""
asset = db.execute(
select(MediaAsset)
.where(MediaAsset.id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
holders = (
db.execute(
select(DownloadJob)
.where(DownloadJob.asset_id == asset_id)
.with_for_update()
.execution_options(populate_existing=True)
)
.scalars()
.all()
)
return asset, list(holders)
def _drop_job(db: Session, job: DownloadJob, ntype: str) -> None:
"""Take a file out of its user's library: tell them, release the job's hold on the asset (the
file goes with the last holder), delete the row (its shares and links cascade)."""
_notify(db, job, db.get(MediaAsset, job.asset_id) if job.asset_id else None, ntype)
service.release_asset(db, job)
db.delete(job)
def _sweep_orphan_files(db: Session, files_root: str) -> int:
"""Delete media/sidecar files (and now-empty dirs) under FILES_ROOT that no `ready` asset
owns — cruft left when a re-download changes an asset's on-disk name (e.g. an old accented
channel dir, or a poster from a superseded naming scheme). Skips the dot-prefixed system trees
and only removes files older than `_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download
is never deleted, and only files NOT owned by a surviving asset."""
root = Path(download_root)
channel dir, or a poster from a superseded naming scheme) — or, under `.uploads`, that no upload
row names at all. Skips the other dot-prefixed system trees and only removes files older than
`_ORPHAN_MIN_AGE_S`, so a just-completed or in-flight download is never deleted, and only files
NOT owned by a surviving asset."""
root = Path(files_root)
if not root.is_dir():
return 0
keep: set[Path] = set()
# An upload's file is on disk from ingest on, whatever its state (a queued conversion's original,
# a failed one kept for Retry), so every upload row keeps its file — not only a ready one.
for rel in db.execute(
select(MediaAsset.rel_path).where(
MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None)
or_(MediaAsset.status == "ready", MediaAsset.source_kind == "upload"),
MediaAsset.rel_path.is_not(None),
)
).scalars():
media = (root / rel).resolve()
@@ -99,8 +182,12 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
if pause_s and touched % batch == 0:
time.sleep(pause_s)
unreclaimed = 0
for entry in root.iterdir():
if entry.name.startswith("."): # .edits / .plex-* / .staging / .storyboards + any dotfile
# .edits / .plex-* / .staging / .storyboards + any dotfile — but `.uploads` is swept: a
# conversion stopped between placing the new file and the commit that points at it (or the
# unlink of the original after it) leaves a file no row names, and nothing else finds it.
if entry.name.startswith(".") and entry.name != _UPLOAD_TREE:
continue
files = [entry] if entry.is_file() else [p for p in entry.rglob("*") if p.is_file()]
for f in files:
@@ -111,7 +198,9 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
f.unlink()
removed += 1
except OSError:
pass
# silent-ok: counted, not logged per file — a broken mount would put one line
# under every orphan. The total is reported once at the end of the sweep.
unreclaimed += 1
if entry.is_dir(): # prune emptied dirs (deepest first), then the channel dir itself
for d in sorted(
(p for p in entry.rglob("*") if p.is_dir()),
@@ -120,12 +209,20 @@ def _sweep_orphan_files(db: Session, download_root: str) -> int:
):
try:
d.rmdir()
# silent-ok: `rmdir` refusing means the directory still holds files, which is
# the ordinary answer here — this prune only takes the ones that emptied.
except OSError:
pass
try:
entry.rmdir()
# silent-ok: the same prune one level up, at the channel directory.
except OSError:
pass
if unreclaimed:
log.warning(
"some orphaned download files could not be reclaimed",
extra=logs.event("download.gc.unreclaimed", files=unreclaimed, removed=removed),
)
return removed
@@ -134,74 +231,95 @@ def run_download_gc(db: Session) -> dict:
grace_days = sysconfig.get_int(db, "download_gc_grace_days")
warned = expired = evicted = reaped = 0
# 1. Pre-expiry warnings.
# 1. Pre-expiry warnings — one per file, the first run inside the grace window. Each mark is
# conditional on the expiry the SELECT saw and committed on its own: a file deleted since is
# passed over (a write to its gone row would fail the whole run), and one kept, moved or re-timed
# since keeps its fresh expiry unwarned.
warn_before = now + timedelta(days=grace_days)
soon = (
db.execute(
select(MediaAsset).where(
MediaAsset.status == "ready",
MediaAsset.gc_notified.is_(False),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at > now,
MediaAsset.expires_at <= warn_before,
)
soon = db.execute(
select(DownloadJob.id, DownloadJob.expires_at).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expiry_warned.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at > now,
DownloadJob.expires_at <= warn_before,
)
.scalars()
.all()
)
for asset in soon:
_notify_holders(db, asset, "download_expiring")
asset.gc_notified = True
warned += 1
if soon:
).all()
db.rollback() # the pass-1 snapshot holds no locks; each mark takes its own below
for job_id, expires_at in soon:
job = _mark_warned(db, job_id, expires_at)
if job is not None:
asset = db.get(MediaAsset, job.asset_id) if job.asset_id else None
_notify(db, job, asset, "download_expiring")
warned += 1
db.commit()
# 2. Expiry deletion (ready assets past TTL, and any errored leftovers with a TTL).
dead = (
db.execute(
select(MediaAsset).where(
MediaAsset.status.in_(("ready", "error")),
MediaAsset.expires_at.is_not(None),
MediaAsset.expires_at <= now,
)
# 2. Expiry — a file past its folder's retention leaves the library (warned in pass 1).
# Each file is re-checked under its locks and committed on its own: the release unlinks the
# file at once, so it must never act on what the SELECT saw a while ago.
dead = db.execute(
select(DownloadJob.id, DownloadJob.asset_id).where(
DownloadJob.status == JobStatus.DONE,
DownloadJob.kept.is_(False),
DownloadJob.expires_at.is_not(None),
DownloadJob.expires_at <= now,
)
.scalars()
.all()
)
for asset in dead:
_delete_asset(db, asset, "download_expired")
expired += 1
if dead:
).all()
db.rollback() # the pass-2 snapshot holds no locks; each file takes its own below
for job_id, asset_id in dead:
job = _relock_expired(db, job_id, asset_id, now)
if job is not None:
_drop_job(db, job, "download_expired")
expired += 1
db.commit()
# 3. LRU eviction over the total cache cap.
# 3. LRU eviction over the total cache cap: the disk's safety net. Never an upload (no source to
# fetch again) and never a file anyone kept; an evicted file leaves every holder's library.
cap = sysconfig.get_int(db, "download_total_max_bytes")
if cap > 0:
total = (
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.status == "ready"
)
).scalar()
or 0
)
if total > cap:
candidates = (
if _on_disk(db) > cap:
kept_assets = select(DownloadJob.asset_id).where(
DownloadJob.kept.is_(True), DownloadJob.asset_id.is_not(None)
)
candidate_ids = (
db.execute(
select(MediaAsset)
.where(MediaAsset.status == "ready")
select(MediaAsset.id)
.where(
MediaAsset.status == "ready",
MediaAsset.source_kind != "upload",
MediaAsset.id.not_in(kept_assets),
)
.order_by(MediaAsset.last_access_at.asc().nulls_first())
)
.scalars()
.all()
)
for asset in candidates:
if total <= cap:
db.rollback()
for asset_id in candidate_ids:
# Counted afresh before each: a file deleted since (this one, an upload, a candidate
# further down) has freed its space, and no other user's file goes for that space.
if _on_disk(db) <= cap:
break
total -= asset.size_bytes or 0
_delete_asset(db, asset, "download_evicted")
# Locked and re-read like an expiry: since the SELECT a holder may have kept it,
# every holder deleted it, or someone asked for it again (a fresh cache hit).
asset, holders = _relock_evictable(db, asset_id)
if (
asset is None
or asset.status != "ready"
or any(j.kept for j in holders)
or (asset.last_access_at is not None and asset.last_access_at > now)
):
db.commit()
continue
for job in holders:
_notify(db, job, asset, "download_evicted")
db.delete(job)
if asset.rel_path:
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
evicted += 1
db.commit()
db.commit()
# 4. Reap orphaned errored rows (no file, no holder). get_or_create_asset resets a still-
# referenced errored row to pending on re-enqueue, so these only accumulate once every
@@ -223,9 +341,35 @@ def run_download_gc(db: Session) -> dict:
if orphans:
db.commit()
# 4b. Reap unheld uploads. An upload carries no expires_at, is never evicted and never re-enqueued,
# so the passes above never free one that lost its last holder outside `release_asset` — a
# conversion that finished after its job was cancelled, or one cancelled mid-run (the asset
# was `downloading` then). Its file, ready or still the unconverted original, is at `rel_path`.
unheld = (
db.execute(
select(MediaAsset).where(
MediaAsset.source_kind == "upload",
MediaAsset.ref_count == 0,
MediaAsset.status != "downloading",
)
)
.scalars()
.all()
)
for asset in unheld:
if asset.rel_path: # no holder is left to tell, so no notice — just the file and the row
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
reaped += 1
if unheld:
db.commit()
# 5. Orphan FILE sweep — remove on-disk cruft no surviving asset owns (old accented dirs, stale
# posters). Runs last so the keep-set reflects the assets that survived passes 2-4.
swept = _sweep_orphan_files(db, settings.download_root)
swept = _sweep_orphan_files(db, settings.files_root)
# 6. A machine's token about to run out.
tokens_warned = apitokens.warn_expiring(db)
result = {
"warned": warned,
@@ -233,6 +377,7 @@ def run_download_gc(db: Session) -> dict:
"evicted": evicted,
"reaped": reaped,
"swept": swept,
"tokens_warned": tokens_warned,
}
log.info("download_gc %s", result)
return result
+3 -3
View File
@@ -6,7 +6,7 @@ from sqlalchemy import text
from app.config import settings
from app.db import SessionLocal
from app.models import DownloadJob, MediaAsset
from app.downloads.models import DownloadJob, MediaAsset
# --- small DB helpers (short-lived sessions; the download itself holds no transaction) -------
@@ -68,8 +68,8 @@ def _reset_asset_pending(asset_id: int) -> None:
def _staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"asset-{asset_id}"
return Path(settings.files_root) / ".staging" / f"asset-{asset_id}"
def _edit_staging_dir(asset_id: int) -> Path:
return Path(settings.download_root) / ".staging" / f"edit-{asset_id}"
return Path(settings.files_root) / ".staging" / f"edit-{asset_id}"
+196 -12
View File
@@ -1,27 +1,91 @@
"""Public "share by link" helpers for the download center.
A DownloadLink is a capability URL: the unguessable `token` grants anyone read access to one
download's file on the login-free `/watch/{token}` page. Password-protected links additionally
require a short-lived signed **grant** on the file request — because a `<video src>` can't send a
header, the password is exchanged once at `/unlock` for an HMAC grant appended to the file URL,
so the raw password never rides in the media URL (or server logs).
A DownloadLink is a capability URL: the unguessable token grants anyone read access to one
download's file, or to the files in one of its owner's own folders (`folder_files`), on the
login-free `/watch/{token}` page. So the token is never stored as written:
the link is found by its SHA-256, and the owner's copy is Fernet-encrypted — a database copy alone
opens nothing. Without that copy (no TOKEN_ENCRYPTION_KEY, or a key since changed) the link still
opens; only its address cannot be shown again, and the owner makes a new link to share.
Password-protected links additionally require a short-lived signed **grant** on the file request —
because a `<video src>` can't send a header, the password is exchanged once at `/unlock` for an HMAC
grant appended to the file URL, so the raw password never rides in the media URL (or server logs).
"""
import hashlib
import hmac
import secrets
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy import update as sql_update
from sqlalchemy.orm import Session
from app import audit, security
from app.config import settings
from app.models import DownloadLink
from app.downloads import storage
from app.downloads.status import JobStatus
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
from app.models import User
from app.utils import iso
from app.userscope import owned_by
_GRANT_TTL = 6 * 3600 # a watch session's grant is good for 6h, then the viewer re-unlocks
# The most files a folder link lists: as many as the owner's own Files list shows
# (`queries.MAX_LIST`; a test holds the two equal — `queries` imports this module).
FOLDER_MAX = 500
def new_token() -> str:
return secrets.token_urlsafe(24) # ~32 chars, 192 bits
def seal(link: DownloadLink, token: str) -> None:
"""Store `token` on `link` the only ways it is kept: its hash, and its encrypted copy."""
link.token_hash = security.hash_token(token)
link.token_enc = security.encrypt(token) if security.can_encrypt() else None
def find(db: Session, token: str) -> DownloadLink | None:
"""The link a `/watch` token names, by its hash."""
return db.execute(
select(DownloadLink).where(DownloadLink.token_hash == security.hash_token(token))
).scalar_one_or_none()
def recover(link: DownloadLink) -> str | None:
"""The link's token from its encrypted copy, or None when there is none this key can read."""
return security.decrypt_optional(link.token_enc)
def in_folder(folder: FileFolder) -> tuple:
"""Which jobs a link to `folder` reaches: the done files in it NOW, the folder owner's. A file
moved out stops opening through the link at its next request; one moved in starts."""
return (
DownloadJob.folder_id == folder.id,
owned_by(DownloadJob.user_id, folder.user_id),
DownloadJob.status == JobStatus.DONE,
)
def folder_files(db: Session, folder: FileFolder) -> list[tuple[DownloadJob, MediaAsset]]:
"""What a link to `folder` shows — its page lists exactly these, and its link card counts them:
each file `in_folder` whose asset is ready and whose bytes are on disk (so the list never offers
a dead row), newest first, at most `FOLDER_MAX`."""
rows = db.execute(
select(DownloadJob, MediaAsset)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(*in_folder(folder), MediaAsset.status == "ready")
.order_by(DownloadJob.ready_at.desc().nulls_last(), DownloadJob.id.desc())
.limit(FOLDER_MAX)
).all()
return [
(job, asset)
for job, asset in rows
if storage.safe_abs_path(settings.files_root, asset.rel_path) is not None
]
def is_expired(link: DownloadLink) -> bool:
return link.expires_at is not None and link.expires_at <= datetime.now(timezone.utc)
@@ -49,22 +113,28 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
exp_s, sig = grant.split(".", 1)
try:
exp = int(exp_s)
# silent-ok: an untrusted grant that does not parse is simply not a valid grant, and the
# honest answer to it — a password prompt — is the same as for one that expired.
except ValueError:
return False
if exp < datetime.now(timezone.utc).timestamp():
return False
return hmac.compare_digest(sig, _sign(f"{token}.{exp}.{password_version}"))
# As bytes: compare_digest refuses (TypeError, a 500) a str with anything non-ASCII in it, and
# `sig` is whatever the URL carried.
return hmac.compare_digest(sig.encode(), _sign(f"{token}.{exp}.{password_version}").encode())
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "") -> dict:
"""What the file owner sees when managing their links."""
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": link.token,
"url": f"{base_url}/watch/{link.token}",
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
@@ -104,3 +174,117 @@ def public_meta(
"file_url": file_url,
"poster_url": poster_url,
}
# --- the owner's writes --------------------------------------------------------------------
# Each flushes and records its audit row; the handler commits.
def expiry(days: int | None) -> datetime | None:
"""None/0/negative → no expiry; otherwise N days out, capped at ~10 years."""
if not days or days <= 0:
return None
return datetime.now(timezone.utc) + timedelta(days=min(days, 3650))
def audit_view(link: DownloadLink) -> dict:
"""A link's settings as the audit records them: whether it has a password, never which."""
return {
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": iso(link.expires_at),
}
def create(
db: Session,
user: User,
*,
password: str,
allow_download: bool,
expires_days: int | None,
**target: int,
) -> tuple[DownloadLink, str]:
"""A link to `target` (`job_id=` or `folder_id=`, already checked to be the user's), and the
token it is opened by — which is in hand only now."""
token = new_token()
link = DownloadLink(
**target,
created_by=user.id,
allow_download=allow_download,
password_hash=security.hash_password(password) if password else None,
expires_at=expiry(expires_days),
)
seal(link, token)
db.add(link)
db.flush()
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_CREATE,
target_type="download_link",
target_id=link.id,
after={**target, **audit_view(link)},
)
return link, token
def update(
db: Session,
user: User,
link: DownloadLink,
*,
allow_download: bool | None = None,
expires_days: int | None = None,
password: str | None = None,
sent: frozenset[str] = frozenset(),
) -> None:
"""Change what the caller sent (`sent` names the fields, so `expires_days=None` can mean "never
expires"). Setting, clearing or changing the password bumps its version, so any grant issued
under the previous one stops working now rather than living out its TTL."""
before = audit_view(link)
if "allow_download" in sent:
link.allow_download = bool(allow_download)
if "expires_days" in sent:
link.expires_at = expiry(expires_days)
if "password" in sent:
pw = (password or "").strip()
link.password_hash = security.hash_password(pw) if pw else None
link.password_version += 1
after = audit_view(link)
if after != before or "password" in sent:
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_UPDATE,
target_type="download_link",
target_id=link.id,
before=before,
after=after,
)
db.flush()
def revoke(db: Session, user: User, link: DownloadLink) -> None:
link_id = link.id
db.delete(link)
audit.record(
db,
user.id,
audit.AuditAction.DOWNLOAD_LINK_REVOKE,
target_type="download_link",
target_id=link_id,
after={"job_id": link.job_id} if link.job_id is not None else {"folder_id": link.folder_id},
)
db.flush()
def count_view(db: Session, link: DownloadLink) -> None:
"""One more opening of the link. In SQL, not `+= 1` on the loaded row: two opens at once would
both write the same count."""
db.execute(
sql_update(DownloadLink)
.where(DownloadLink.id == link.id)
.values(view_count=DownloadLink.view_count + 1)
.execution_options(synchronize_session=False)
)
+385
View File
@@ -0,0 +1,385 @@
"""Capture's tables: the downloaded files (`MediaAsset`), each user's jobs over them and the folders
they sit in, the shares and public links that hand them on, a user's quota, cookies and format
profiles, and where they stopped playing one. Only this unit writes them; another unit reads them
along a declared edge, or asks this unit's service."""
from datetime import date, datetime, timezone
from sqlalchemy import (
BigInteger,
Boolean,
CheckConstraint,
Date,
DateTime,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, validates
from app.db import Base
from app.models import DISPLAY_TEXT_MAX, TimestampMixin, UpdatedAtMixin
class LibraryState(Base):
"""Where a user stopped in a library item (their own download, or one shared with them) — the
library player's resume point. Keyed by the JOB, not the file: a job is the user's library entry,
a re-download keeps it, and deleting the job deletes the row; the shared file under it may outlive
both. A revoked share leaves its recipient's row behind, unread — only a job the user can still
reach is ever read or written — and it is back if the item is shared again. Rows exist only for a
position worth keeping (see `routes._common.kept_position`); no row means "from the start"."""
__tablename__ = "library_states"
__table_args__ = (UniqueConstraint("user_id", "job_id", name="uq_library_user_job"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
job_id: Mapped[int] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
# int4: a checkpoint body must stay under POSITION_MAX_S, or the flush is a DataError (a 500).
position_seconds: Mapped[int] = mapped_column(Integer, nullable=False)
progress_updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
class MediaAsset(Base, TimestampMixin):
"""A physical downloaded media file — the SHARED cache layer of the download center.
Mirrors the app's "shared catalog, private per-user state" model: one row per unique
``(source_kind, source_ref, format_sig)`` is the global cache. When a second user requests
the same video with a format that hashes to the same `format_sig`, their `download_job`
links to this ready asset instantly — no re-download, no cost. Only the SOURCE download is
shared; per-user edited derivatives (phase 2) are never cached here.
`rel_path` is relative to `settings.files_root` (the mount), so the physical filename is
system-decided and bound to the source id — never renamed (the user's custom name is display
metadata on `download_job`, applied only when downloading to their own machine). The naming
fields (`title`/`uploader`/`upload_date`) are denormalized so the Plex-tree path can be built
uniformly for both catalog videos and ad-hoc URLs. `ref_count` (active jobs pointing here) frees
the file with its last holder, and `last_access_at` orders the GC's LRU eviction. How long a file
stays is not the asset's: each holder's job carries its own `expires_at` (its library folder's)."""
__tablename__ = "media_assets"
__table_args__ = (
UniqueConstraint(
"source_kind", "source_ref", "format_sig", name="uq_media_asset_cache_key"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512)) # video id, or full URL for ad-hoc
format_sig: Mapped[str] = mapped_column(String(64)) # hash of the output-affecting spec
status: Mapped[str] = mapped_column(
String(16), default="pending", server_default="pending", index=True
) # "pending" | "ready" | "error"
rel_path: Mapped[str | None] = mapped_column(Text) # relative to files_root
storyboard_path: Mapped[str | None] = mapped_column(Text) # phase-2 filmstrip mosaic
# A locally-generated poster frame (ffmpeg), set only when the source had NO thumbnail so the
# card / watch page / link preview can still show an image. Relative to files_root.
poster_path: Mapped[str | None] = mapped_column(Text)
size_bytes: Mapped[int | None] = mapped_column(BigInteger)
duration_s: Mapped[int | None] = mapped_column(Integer)
width: Mapped[int | None] = mapped_column(Integer)
height: Mapped[int | None] = mapped_column(Integer)
container: Mapped[str | None] = mapped_column(String(16))
vcodec: Mapped[str | None] = mapped_column(String(32))
acodec: Mapped[str | None] = mapped_column(String(32))
# "video" | "audio" | "file" — the media-kind discriminator that drives which library actions
# apply (play/edit for media; download-only for a generic uploaded file). Null while an asset is
# still pending (kind is known once the file is probed at finalize). It is the single stored home
# of the invariant `downloads.mediakind.derive` used to compute — never re-derived in prose (an
# earlier ad-hoc `vcodec is None and acodec` check lived only in links.py; consolidated here).
media_kind: Mapped[str | None] = mapped_column(String(16))
nfo_written: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# Denormalized naming/metadata for the Plex tree + ad-hoc display.
title: Mapped[str | None] = mapped_column(Text)
uploader: Mapped[str | None] = mapped_column(String(255))
# yt-dlp's channel/uploader page URL (channel_url|uploader_url) — lets an auto-filled channel
# render as a clickable link. Null for sources without one (e.g. reddit) or older rows.
uploader_url: Mapped[str | None] = mapped_column(Text)
upload_date: Mapped[date | None] = mapped_column(Date)
thumbnail_url: Mapped[str | None] = mapped_column(Text)
# yt-dlp's canonical page URL for the source (webpage_url) — the clean "downloaded from"
# reference shown in the Files library. Null until the worker extracts it (older/queued rows
# fall back to a URL derived from source_kind+source_ref).
source_webpage_url: Mapped[str | None] = mapped_column(Text)
error: Mapped[str | None] = mapped_column(Text)
ref_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
last_access_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# The file's SHA-256 (hex), so a machine fetching it can check what it got. Hashed while an upload
# streams in, otherwise on the first ask (`routes.files`); NULL = not hashed yet.
sha256: Mapped[str | None] = mapped_column(String(64))
@validates("rel_path")
def _forget_hash(self, _key: str, rel_path: str | None) -> str | None:
# A hash names the bytes at `rel_path`: a new file there (a conversion, a retry) is not them.
self.sha256 = None
return rel_path
class DownloadProfile(Base, TimestampMixin):
"""A named download preset — a reusable format profile.
`user_id` NULL + `is_builtin` marks the shipped presets (Best MP4, 1080p, audio-only, …);
a user's saved custom profiles carry their `user_id`. `spec` is the format definition
(mode av/v/a, max_height, container, audio_format, codec prefs, embed subs/chapters/
thumbnail, sponsorblock) that `downloads.formats` turns into a yt-dlp selector + a cache
`format_sig`."""
__tablename__ = "download_profiles"
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int | None] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
) # NULL for a builtin preset
name: Mapped[str] = mapped_column(String(80))
spec: Mapped[dict] = mapped_column(JSON)
is_builtin: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
sort_order: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class FileFolder(Base, TimestampMixin):
"""A folder of the Files library (`app.downloads.folders`): a row, never a directory.
A SOURCE folder (`source_key` set: youtube / plex / upload / site:<host>) is made the first time a
user gets a file from that origin; an OWN folder (`name` set) is one the user made; the HUB folder
(neither set, one per user) is where they swap files with a machine of theirs. Every file —
a `DownloadJob` — sits in exactly one (`DownloadJob.folder_id`). `retention_days` is how long a
file stays once done (NULL = until deleted); a file's own `expires_at` is stamped from it."""
__tablename__ = "file_folders"
__table_args__ = (
UniqueConstraint("user_id", "source_key", name="uq_file_folder_source"),
Index(
"uq_file_folder_own_name",
"user_id",
func.lower(text("name")),
unique=True,
postgresql_where=text("name IS NOT NULL"),
),
# One Hub folder per user.
Index("uq_file_folder_hub", "user_id", unique=True, postgresql_where=text("kind = 'hub'")),
CheckConstraint(
"(kind = 'source' AND source_key IS NOT NULL AND name IS NULL)"
" OR (kind = 'own' AND name IS NOT NULL AND source_key IS NULL)"
" OR (kind = 'hub' AND name IS NULL AND source_key IS NULL)",
name="ck_file_folder_kind",
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
kind: Mapped[str] = mapped_column(String(16)) # folders.FolderKind
# "site:" + a host name (DNS caps one at 253 characters).
source_key: Mapped[str | None] = mapped_column(String(260))
name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
retention_days: Mapped[int | None] = mapped_column(Integer)
def _ready_on_insert(context) -> datetime | None:
"""A job inserted already done (an upload kept as it came) was ready the moment it was made; one
that becomes done later is stamped then (`folders.stamp_done`, `service.finalize_if_running`)."""
done = context.get_current_parameters().get("status") == "done"
return datetime.now(timezone.utc) if done else None
class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
"""A user's request to download one source — the per-user layer over `MediaAsset`.
The worker claims `queued` jobs with ``FOR UPDATE SKIP LOCKED`` and writes live
`progress`/`speed_bps`/`eta_s`/`phase` here (the frontend polls via useLiveQuery: there is
no server push channel, and the worker is a separate process anyway). `profile_snapshot`
freezes the spec at enqueue so later edits/deletion of the profile don't change this job.
`display_name` is the user's own name for the file (display + the Content-Disposition on
local download); the on-disk asset is never renamed. `asset_id` links to the shared file
(SET NULL if the asset is GC'd — the job then shows as expired).
Phase 2 (editor): `job_kind='edit'` jobs derive a new per-user clip from an already-downloaded
ready job. They still hang off a `MediaAsset` (with `source_kind='edit'`, a per-user cache key)
so all the file-serve/ref-count/GC/quota machinery is reused; the worker branches on the
asset's `source_kind` and runs ffmpeg instead of yt-dlp. `source_job_id`/`source_asset_id`
point at the parent download the clip was cut from; `edit_spec` is the (trim/crop) recipe."""
__tablename__ = "download_jobs"
__table_args__ = (
Index("ix_download_jobs_claim", "status", "queue_pos"),
# Every way a job becomes done stamps when (`folders.stamp_done`, `finalize_if_running`).
CheckConstraint(
"status <> 'done' OR ready_at IS NOT NULL", name="ck_download_jobs_ready_at"
),
)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL"), index=True
)
source_kind: Mapped[str] = mapped_column(String(16)) # "youtube" | "url"
source_ref: Mapped[str] = mapped_column(String(512))
# "download" (default), "edit" (a per-user trim/crop derivative of another job), "upload" (a
# user's own file, converted by the worker when it is not phone-safe — see `downloads.upload`) or
# "clip" (a range cut straight from a YouTube video or a Plex item — see `downloads.clip`).
job_kind: Mapped[str] = mapped_column(String(16), default="download", server_default="download")
# For edit jobs: the parent download this clip is cut from (SET NULL if the parent is deleted).
source_job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="SET NULL")
)
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("download_profiles.id", ondelete="SET NULL")
)
profile_snapshot: Mapped[dict] = mapped_column(JSON)
display_name: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
# Per-user display overrides + extras (like display_name — metadata only, never the file).
# `display_uploader`/`display_uploader_url` override the asset's auto channel name/link;
# `extra_links` is an optional list of extra reference URLs shown on the card + watch page.
display_uploader: Mapped[str | None] = mapped_column(String(DISPLAY_TEXT_MAX))
display_uploader_url: Mapped[str | None] = mapped_column(Text)
extra_links: Mapped[list | None] = mapped_column(JSON)
status: Mapped[str] = mapped_column(
String(16), default="queued", server_default="queued", index=True
) # queued | running | paused | done | error | canceled
progress: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
speed_bps: Mapped[int | None] = mapped_column(BigInteger)
eta_s: Mapped[int | None] = mapped_column(Integer)
phase: Mapped[str | None] = mapped_column(String(32))
error: Mapped[str | None] = mapped_column(Text) # raw yt-dlp/ffmpeg text (technical detail)
error_code: Mapped[str | None] = mapped_column(
String(24)
) # category, see downloads.errors.classify
# The host-specific failure-catalog entry on top of the category (`facebook.login_wall`), or NULL.
error_detail: Mapped[str | None] = mapped_column(String(48)) # see downloads.errors.diagnose
queue_pos: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
# The Files library: the folder this file sits in (`downloads.folders.place` at creation; an own
# folder is emptied back into the source folders before it is deleted, hence RESTRICT), whether
# the user keeps it past its folder's retention, and when it goes (stamped once done; NULL =
# never). `expiry_warned` marks the one warning the GC sends before it deletes the file.
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="RESTRICT"), index=True
)
# The origin the file came from (the source folder's key), recorded where it is placed: an edit
# traces its origin through its parent, and that parent may be deleted or expire later — the
# file must still know which source folder it may go back to. NULL only for a file placed before
# the column existed outside a source folder; `folders.source_key_for` works those out.
source_key: Mapped[str | None] = mapped_column(String(260))
kept: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True)
expiry_warned: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
# When the file became done — what the library's "Added" means: a download that failed for days
# and then went through is new to its user, so it sorts as new. `created_at` stays when it was
# asked for (the queue's order). Set again when a file is made anew (a retry, a re-download).
ready_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), default=_ready_on_insert
)
class DownloadShare(Base, TimestampMixin):
"""A user→user grant: `to_user` may access `from_user`'s downloaded file (private by
default). Since assets are already deduped/shared on disk, a share is just an ACL row."""
__tablename__ = "download_shares"
__table_args__ = (UniqueConstraint("job_id", "to_user_id", name="uq_download_share"),)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int] = mapped_column(ForeignKey("download_jobs.id", ondelete="CASCADE"))
from_user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True
)
to_user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
class DownloadQuota(Base, UpdatedAtMixin):
"""Per-user download limits (admin-managed). A row exists only when an admin customizes a
user; otherwise the sysconfig `download_default_*` values apply. `unlimited` bypasses the
byte cap entirely (e.g. for the admin's own account)."""
__tablename__ = "download_quotas"
user_id: Mapped[int] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
)
max_bytes: Mapped[int] = mapped_column(BigInteger)
max_concurrent: Mapped[int] = mapped_column(Integer)
max_jobs: Mapped[int] = mapped_column(Integer)
unlimited: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class DownloadLink(Base, TimestampMixin):
"""A public capability link to one download, or to one of its owner's own folders — the "share
by link" surface.
Anyone holding the unguessable token can watch the file on the login-free `/watch/{token}`
player page (Google-Drive style); a folder link opens the folder's list there, with what is in
the folder at the time it is opened. The token itself is never stored as written: a link is
found by its SHA-256 (`token_hash`), and `token_enc` is the Fernet-encrypted copy its owner
copies the link from again — absent without a key (`downloads/links.py`). Optional per-link
controls: `expires_at`, `allow_download` (stream-only vs downloadable), and a `password_hash`
(argon2, like a user password). Revoke = delete the row. Distinct from `DownloadShare`, which
grants a *registered* user access via the in-app "Shared with me" list."""
__tablename__ = "download_links"
__table_args__ = (
# A link opens exactly one thing: a file or a folder.
CheckConstraint("(job_id IS NULL) <> (folder_id IS NULL)", name="ck_download_link_target"),
)
id: Mapped[int] = mapped_column(primary_key=True)
job_id: Mapped[int | None] = mapped_column(
ForeignKey("download_jobs.id", ondelete="CASCADE"), index=True
)
folder_id: Mapped[int | None] = mapped_column(
ForeignKey("file_folders.id", ondelete="CASCADE"), index=True
)
token_hash: Mapped[str] = mapped_column(String(64), unique=True)
token_enc: Mapped[str | None] = mapped_column(Text) # Fernet ciphertext — never returned/logged
created_by: Mapped[int | None] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"))
allow_download: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
password_hash: Mapped[str | None] = mapped_column(Text)
# Bumped whenever the password changes; folded into the signed watch grant so rotating the
# password immediately invalidates any grants issued under the old one (instead of them living
# out the 6h grant TTL).
password_version: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
view_count: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
class DownloadCredential(Base, TimestampMixin, UpdatedAtMixin):
"""A user's stored login cookies for ONE auth-walled download host (Facebook, Instagram, …).
Per-user, per-host: each user pastes their OWN browser cookies.txt (Netscape format) for a host
where they're logged in (e.g. a closed Facebook group they belong to), and a download of a
yt-dlp-supported, auth-walled link on that host is retried with the ACTING user's cookie. The
blob is Fernet-encrypted at rest (`cookies_enc`) and is NEVER returned by any API/log/audit —
the routes are write-only (host + updated_at only). See `downloads.credentials`.
Two load-bearing security controls live around this row, NOT on it:
* **cache privacy** — a cookie-authed download SALTS its MediaAsset `format_sig` with the
user id, so the authed asset is private to that user and can never be cache-served to
another user (`downloads.credentials.cookie_salt`).
* **host scope (SEC-12)** — a cookie is attached ONLY when the download URL's netloc matches
this row's `host_key` (`downloads.credentials.host_key_for_url`), so an arbitrary `?url=`
can't exfiltrate the user's session to a look-alike host.
`host_key` is a short canonical key from the supported-host registry (e.g. "facebook"), not the
raw netloc — one credential can cover several domains of the same site (facebook.com/fb.watch)."""
__tablename__ = "download_credentials"
__table_args__ = (UniqueConstraint("user_id", "host_key", name="uq_download_credential"),)
id: Mapped[int] = mapped_column(primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
host_key: Mapped[str] = mapped_column(String(64))
cookies_enc: Mapped[str] = mapped_column(Text) # Fernet ciphertext — never returned/logged
+39 -10
View File
@@ -11,13 +11,12 @@ import html
import re
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import links as linksmod
from app.downloads import storage
from app.models import DownloadJob, DownloadLink, MediaAsset
from app.downloads.models import DownloadJob, DownloadLink, FileFolder, MediaAsset
_OG_BLOCK = re.compile(r"<!--OG:START-->.*?<!--OG:END-->", re.DOTALL)
@@ -66,28 +65,33 @@ def _jpeg_size(path: Path) -> tuple[int, int] | None:
width = (data[3] << 8) + data[4]
return width, height
f.seek(length - 2, 1) # skip this segment's payload
# silent-ok: None omits width/height from the unfurl card, which is what the card does for
# every poster whose dimensions were never asked for. Nothing is claimed that is not true.
except OSError:
return None
def _watch_tags(token: str, db: Session) -> str | None:
"""Build the OG/Twitter meta block for a watch token, or None to keep the generic card."""
link = db.execute(select(DownloadLink).where(DownloadLink.token == token)).scalar_one_or_none()
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected video"),
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
)
if folder:
return _folder_tags(db, link, url)
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
@@ -133,7 +137,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
if image_dims is None and image.endswith("poster.jpg") and asset.poster_path:
# safe_abs_path returns None if the poster file is missing (gc'd) or escapes the root —
# guard it, else _jpeg_size(None) would raise an uncaught AttributeError on this public page.
poster_abs = storage.safe_abs_path(settings.download_root, asset.poster_path)
poster_abs = storage.safe_abs_path(settings.files_root, asset.poster_path)
image_dims = _jpeg_size(poster_abs) if poster_abs else None
if image_dims:
tags.append(_tag("og:image:width", str(image_dims[0])))
@@ -145,13 +149,38 @@ def _watch_tags(token: str, db: Session) -> str | None:
return "\n ".join(tags)
def render_watch_html(token: str, index_html: Path, db: Session) -> str | None:
"""Return index.html with the OG block replaced for this watch token, or None to serve the
generic page unchanged (invalid/expired/not-ready link, or a template without the markers)."""
def _folder_tags(db: Session, link: DownloadLink, url: str) -> str | None:
"""A folder link's card: the folder's name and how many files it holds — never their names,
which the page shows only to whoever opens it."""
folder = db.get(FileFolder, link.folder_id)
if folder is None or not folder.name:
return None
count = len(linksmod.folder_files(db, folder)) # the page's own list, so the two agree
title = folder.name.strip()
about = f"{count} file{'' if count == 1 else 's'} · Shared via Siftlode"
return "\n ".join(
[
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", about),
_tag("og:type", "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
_tag("twitter:description", about, attr="name"),
_tag("twitter:card", "summary", attr="name"),
]
)
def render_watch_html(token: str, page_html: Path, db: Session) -> str | None:
"""Return the watch page (watch.html) with the OG block replaced for this token, or None to
serve the generic page unchanged (invalid/expired/not-ready link, or a template without the
markers)."""
tags = _watch_tags(token, db)
if tags is None:
return None
text = index_html.read_text(encoding="utf-8")
text = page_html.read_text(encoding="utf-8")
if "<!--OG:START-->" not in text:
return None
# A function replacement avoids re.sub interpreting backslashes/group refs in `tags`.
File diff suppressed because it is too large Load Diff
+103
View File
@@ -0,0 +1,103 @@
"""The phone-safe rule: what a library file must be so it plays wherever a share link lands.
A shared `/watch` link is opened by whoever got it, on whatever they hold: an iPhone's WebKit, an
Android Chrome, a desktop Firefox. The one shape every one of them plays:
* video → H.264, 8-bit 4:2:0 (`yuv420p`, or its full-range twin `yuvj420p`), AAC audio, in .mp4
* audio only → MP3 in .mp3, or AAC in .m4a
Measured on the owner's iPhone + iPad (2026-10-05): AV1, Matroska, HEVC tagged `hev1` and MP3
inside an .mp4 all fail there. VP9/WebM, HEVC `hvc1` and 10-bit H.264 played on those devices, but
Chrome and Firefox do not decode HEVC and browsers decode 10-bit / 4:4:4 H.264 only patchily, so the
rule is the shape every browser plays, not the shape one device plays.
Every file that becomes a ready library asset passes through here — a download, an upload (the
worker converts it, see `upload`), and an editor clip — so a link can only ever serve this shape.
`plan` says what (if anything) must change; `build_cmd` is the one ffmpeg command that changes it:
re-encode only the stream that fails, copy the rest, or just remux when only the container is wrong.
"""
from dataclasses import dataclass
from pathlib import Path
from app.downloads import edit as editmod
from app.downloads import probe
_SAFE_VCODEC = "h264"
_SAFE_PIX_FMTS = frozenset({"yuv420p", "yuvj420p"})
_VIDEO_EXT = "mp4"
# An audio-only file: the codec, and the extension it plays in. Anything else becomes AAC in .m4a.
_AUDIO_HOME = {"mp3": "mp3", "aac": "m4a"}
_AUDIO_FALLBACK_EXT = "m4a"
@dataclass(frozen=True)
class Plan:
"""What makes one file phone-safe. `ext` is the extension the safe file has (no dot)."""
ext: str
has_video: bool
reencode_video: bool
reencode_audio: bool
needs_change: bool
def plan(data: dict | None, ext: str) -> Plan | None:
"""The phone-safe plan for a file whose `probe.probe_json` is `data` and whose extension is
`ext`; None when it is not media at all (an unreadable probe, a still image, no audio or video
stream) — such a file is a download-only 'file' and never plays inline, so there is nothing to
make safe."""
if data is None:
return None
video, audio = probe.primary_streams(data)
ext = ext.lower().lstrip(".")
if video is not None:
rv = (video.get("codec_name") or "").lower() != _SAFE_VCODEC or (
video.get("pix_fmt") or ""
).lower() not in _SAFE_PIX_FMTS
ra = audio is not None and (audio.get("codec_name") or "").lower() != "aac"
return Plan(_VIDEO_EXT, True, rv, ra, rv or ra or ext != _VIDEO_EXT)
if audio is not None:
home = _AUDIO_HOME.get((audio.get("codec_name") or "").lower())
target = home or _AUDIO_FALLBACK_EXT
return Plan(target, False, False, home is None, home is None or ext != target)
return None
def build_cmd(src: Path, dest: Path, p: Plan, threads: int | None = None) -> list[str]:
"""The ffmpeg command that writes `src` to `dest` (whose suffix is `p.ext`) in the phone-safe
shape: the failing stream(s) re-encoded with the package's shared encoder settings, the rest
stream-copied. An audio-only result drops any cover-art stream. `-progress pipe:1`
streams machine-readable progress; `threads` caps the encoder so a re-encode cannot claim every
core on the shared box (see the worker-child de-prioritization)."""
cmd = [
"ffmpeg",
"-y",
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-progress",
"pipe:1",
"-nostats",
*editmod.INPUT_GUARD,
"-i",
str(src),
]
# Exactly the two streams `plan` judged: `V` is a video stream that is not cover art. Nothing
# else rides along — a subtitle or data track the .mp4 muxer cannot take would fail the run.
if p.has_video:
cmd += ["-map", "0:V:0", "-map", "0:a:0?"]
if p.reencode_video:
cmd += ["-vf", editmod.EVEN_FRAME, *editmod.X264_ARGS]
else:
cmd += ["-c:v", "copy"]
else:
cmd += ["-map", "0:a:0"]
cmd += editmod.AAC_ARGS if p.reencode_audio else ["-c:a", "copy"]
if threads and threads > 0:
cmd += ["-threads", str(threads)]
if p.ext in ("mp4", "m4a"):
cmd += ["-movflags", "+faststart"]
cmd += [str(dest)]
return cmd
+107
View File
@@ -0,0 +1,107 @@
"""What a library file previews as — the single home of the preview allow-list.
A generic file (`media_kind == "file"`; video and audio have their players) previews by its stored
`container`, the upload's cleaned extension, and ONLY through this table: an image is shown as an
image, a PDF in the browser's own viewer, and a text format as plain text. Anything else has no
preview. The served Content-Type comes from here too, never from a guess on the name, and HTML / SVG
sit in the TEXT list on purpose: rendered on this origin, an uploaded page would run with the
reader's session. The route adds `CSP: sandbox` on top, so even a mislabelled file (a "png" that is
really a page) gets no script and no origin.
The serializer hands the kind to the client (`asset.preview`), so the library and the server can
never disagree on which files open a preview.
"""
import codecs
from typing import Literal
from app.downloads import mediakind
PreviewKind = Literal["image", "pdf", "text"]
IMAGE_TYPES: dict[str, str] = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"avif": "image/avif",
"bmp": "image/bmp",
}
PDF_TYPE = "application/pdf"
# Read as text: notes, data, subtitles, config and source. HTML and SVG belong here (see above).
TEXT_EXTS: frozenset[str] = frozenset(
{
"txt", "text", "md", "markdown", "log", "csv", "tsv", "json", "jsonl", "ndjson", "xml",
"yaml", "yml", "toml", "ini", "conf", "cfg", "properties", "env", "srt", "vtt", "ass", "ssa",
"sub", "nfo", "cue", "m3u", "m3u8", "html", "htm", "xhtml", "svg", "css", "scss", "less",
"js", "mjs", "cjs", "ts", "tsx", "jsx", "py", "rb", "php", "pl", "lua", "go", "rs", "java",
"kt", "kts", "swift", "c", "h", "cc", "cpp", "hpp", "cs", "sh", "bash", "zsh", "fish", "ps1",
"bat", "cmd", "sql", "r", "dart", "diff", "patch", "tex", "rst", "adoc", "gradle", "dockerfile",
}
) # fmt: skip
TEXT_TYPE = "text/plain; charset=utf-8"
# A text preview is the file's head: enough to read, never a whole log loaded into a tab.
TEXT_PREVIEW_BYTES = 1 << 20
# A NUL byte in the first this-many bytes means the "text" is not text.
BINARY_SNIFF_BYTES = 8192
def kind(media_kind: str | None, container: str | None) -> PreviewKind | None:
"""The preview a library file gets, or None (a media file plays instead; others download)."""
if media_kind != mediakind.FILE:
return None
ext = (container or "").lower()
if ext in IMAGE_TYPES:
return "image"
if ext == "pdf":
return "pdf"
if ext in TEXT_EXTS:
return "text"
return None
def media_type(preview: PreviewKind, container: str | None) -> str:
"""The Content-Type a preview is served with: from this table, never guessed from the name."""
if preview == "image":
return IMAGE_TYPES[(container or "").lower()]
if preview == "pdf":
return PDF_TYPE
return TEXT_TYPE
# A text that is not UTF-8 is read as Central European Windows: what an old Hungarian subtitle or
# note is written in (its ő and ű are not in Latin-1).
FALLBACK_ENCODING = "cp1250"
_BOMS = (
(codecs.BOM_UTF8, "utf-8-sig"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
)
def looks_binary(head: bytes) -> bool:
"""Whether the start of a "text" file is binary after all (a NUL in it)."""
return b"\x00" in head[:BINARY_SNIFF_BYTES]
def text_utf8(head: bytes) -> bytes | None:
"""A text file's head as UTF-8, the one charset the preview is served in; None when it is not text.
A byte-order mark names its encoding (UTF-16 is full of NULs, so it is read before the binary
check); else UTF-8 is kept as it is — a character cut at the size cap included, the reader shows
that one as a replacement mark — and anything else is read as `FALLBACK_ENCODING`."""
for bom, encoding in _BOMS:
if head.startswith(bom):
return head.decode(encoding, errors="replace").encode()
if looks_binary(head):
return None
try:
head.decode("utf-8")
except UnicodeDecodeError as e: # silent-ok: not UTF-8 is an answer — read it as the fallback
if not (e.reason == "unexpected end of data" and e.end == len(head)):
return head.decode(FALLBACK_ENCODING, errors="replace").encode()
return head
+70 -32
View File
@@ -34,6 +34,28 @@ _IMAGE_FORMATS = {
}
def primary_streams(data: dict) -> tuple[dict | None, dict | None]:
"""(video stream, audio stream) — the first of each kind — from a NON-None ffprobe JSON dict, by
the same reading the stored facts use: a still image / image-sequence container has neither (it
is a generic 'file'), and a video stream flagged `attached_pic` (an mp3/m4a's cover art) is not a
video track. The phone-safe rule (`playable`) judges exactly these two streams."""
fmt_names = {
n.strip() for n in ((data.get("format") or {}).get("format_name") or "").split(",")
}
if fmt_names & _IMAGE_FORMATS:
return None, None
video = audio = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and video is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
video = stream
elif kind == "audio" and audio is None:
audio = stream
return video, audio
def _facts_from_probe(
data: dict,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
@@ -45,63 +67,79 @@ def _facts_from_probe(
fmt_names = {n.strip() for n in (fmt.get("format_name") or "").split(",")}
if fmt_names & _IMAGE_FORMATS:
return None, None, None, None, None
vcodec = acodec = None
width = height = None
for stream in data.get("streams", []):
kind = stream.get("codec_type")
if kind == "video" and vcodec is None:
if (stream.get("disposition") or {}).get("attached_pic"):
continue # cover art, not a video track
vcodec = stream.get("codec_name")
width = stream.get("width")
height = stream.get("height")
elif kind == "audio" and acodec is None:
acodec = stream.get("codec_name")
video, audio = primary_streams(data)
vcodec = video.get("codec_name") if video else None
width = video.get("width") if video else None
height = video.get("height") if video else None
acodec = audio.get("codec_name") if audio else None
duration = None
try:
raw = fmt.get("duration")
if raw is not None:
duration = int(float(raw))
# silent-ok: an unusable duration is stored as no duration, which the UI renders as unknown
# rather than as zero — the honest state, and the one a re-probe can still fill in.
except (TypeError, ValueError):
duration = None
return vcodec, acodec, width, height, duration
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (an unprobeable file — or a still image — is treated as a generic
non-media file, so it becomes a download-only 'file' rather than a broken 'video')."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def probe_json(path: Path) -> dict | None:
"""The full ffprobe JSON (streams + format) of `path`, or None when ffprobe could not read it —
the one call both the stored facts (`facts`) and the phone-safe rule read from."""
return editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
def facts(
data: dict | None,
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) of a `probe_json` result; all None when it is None
(an unprobeable file is treated as a generic non-media file, so it becomes a download-only 'file'
rather than a broken 'video')."""
if data is None:
return None, None, None, None, None
return _facts_from_probe(data)
def probe_media(path: Path) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) from one ffprobe call; all None if ffprobe fails
or the file isn't playable media (a still image is a generic 'file', see `primary_streams`)."""
return facts(probe_json(path))
def resolve_media_facts(
path: Path, info: dict
) -> tuple[str | None, str | None, int | None, int | None]:
"""(vcodec, acodec, width, height) for the stored download asset: PROBE the finished file on disk
(the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources (Facebook among
them), so deriving media_kind from it there labelled a real H.264 video a download-only 'file'.
) -> tuple[str | None, str | None, int | None, int | None, int | None]:
"""(vcodec, acodec, width, height, duration_s) for the stored download asset: PROBE the finished
file on disk (the ground truth). yt-dlp's `info` omits codec/dimension fields for some sources
(Facebook among them), so deriving media_kind from it there labelled a real H.264 video a
download-only 'file'. It omits the duration too (a Facebook reel, an Instagram reel), and a NULL
duration hides the Library row's trim action although the file plays for minutes.
Precedence is by whether the probe RAN, not per-field: if ffprobe read the file, its codec verdict
is authoritative — INCLUDING a deliberate None (cover-art-only audio, a still image). A per-field
`probe or info` fallback would let `info`'s stale vcodec resurrect a video track the file doesn't
have (yt-dlp reports a progressive `vcodec` for an audio-mode download whose video it then strips),
mislabelling an audio download as a broken in-app 'video'. Only when ffprobe could NOT read the
file at all do we fall back to `info` (the pre-probe behaviour, a safety net); the cosmetic
dimensions still gap-fill from `info` in either case."""
mislabelling an audio download as a broken in-app 'video'. When ffprobe could NOT read the file
at all, no codec is reported: the probe runs with the media demuxer whitelist, so a file it
cannot open is one the player and the editor cannot open either — and taking `info`'s codecs
for it would make it an editable 'video' whose bytes are whatever the remote site served. The
cosmetic dimensions and the duration still gap-fill from `info` in either case."""
data = editmod.run_ffprobe_json(path, show_streams=True, show_format=True)
info_v = info.get("vcodec") if info.get("vcodec") not in (None, "none") else None
info_a = info.get("acodec") if info.get("acodec") not in (None, "none") else None
info_w = int(info["width"]) if info.get("width") else None
info_h = int(info["height"]) if info.get("height") else None
info_d = int(info["duration"]) if info.get("duration") else None
if data is None:
# ffprobe could not read the file — fall back to the extractor's info entirely.
return info_v, info_a, info_w, info_h
pv, pa, pw, ph, _dur = _facts_from_probe(data)
# Unreadable to the guarded probe: a download-only 'file' (no codecs → media_kind FILE).
return None, None, info_w, info_h, info_d
pv, pa, pw, ph, pd = _facts_from_probe(data)
# Probe ran → its codecs win outright (a None means "no such track", not "unknown"); info only
# gap-fills the cosmetic width/height the probe couldn't read.
return pv, pa, pw if pw is not None else info_w, ph if ph is not None else info_h
# gap-fills the width/height/duration the probe couldn't read.
return (
pv,
pa,
pw if pw is not None else info_w,
ph if ph is not None else info_h,
pd if pd is not None else info_d,
)
+404
View File
@@ -0,0 +1,404 @@
"""Capture's reads: what the library, the download tray, the share dialogs and the public watch pages
show. Each returns rows or plain values and writes nothing; whose a row is goes through
`userscope.owned_by` / `owned_row`, so a row of someone else's reads as a missing one.
The HTTP layer (`routes.downloads`, `routes.files`, `routes.public`) asks here and serializes; the
writes are the service's (`service`, `folders`, `links`, `quota`)."""
from collections.abc import Sequence
from datetime import datetime, timezone
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.apitokens import FilesCaller, Right
from app.collation import collated
from app.downloads import folders, links, quota
from app.downloads.models import (
DownloadJob,
DownloadLink,
DownloadProfile,
DownloadQuota,
DownloadShare,
FileFolder,
LibraryState,
MediaAsset,
)
from app.downloads.status import FINISHED, JobStatus
from app.models import User
from app.userscope import owned_by, owned_row
from app.utils import like_escape
# Cap a listing so a heavy library can't return an unbounded result set (the old Download Center's
# own cap; far beyond any real library).
MAX_LIST = 500
# --- jobs and their files ------------------------------------------------------------------
def own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
return owned_row(db, DownloadJob, job_id, user)
def job(db: Session, job_id: int) -> DownloadJob | None:
"""A job by id, whoever's — for a public link, which already names its job."""
return db.get(DownloadJob, job_id)
def accessible_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""A file the caller may read: their own, or one shared with them."""
row = db.get(DownloadJob, job_id)
if row is None:
return None
if row.user_id == user.id or shared_with(db, job_id, user) is not None:
return row
return None
def readable_job(db: Session, caller: FilesCaller, job_id: int) -> DownloadJob | None:
"""A file `caller` may read: as `accessible_job` for the user; for a token, one of the user's
own in a folder it may read."""
if caller.grants is None:
return accessible_job(db, caller.user, job_id)
row = own_job(db, caller.user, job_id)
if row is None or not caller.may(row.folder_id, Right.READ):
return None
return row
def asset_of(db: Session, job: DownloadJob | None) -> MediaAsset | None:
"""The file under a job, if it has one yet."""
return db.get(MediaAsset, job.asset_id) if job is not None and job.asset_id else None
def assets_for(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, MediaAsset]:
ids = {j.asset_id for j in jobs if j.asset_id}
if not ids:
return {}
rows = db.execute(select(MediaAsset).where(MediaAsset.id.in_(ids))).scalars()
return {a.id: a for a in rows}
def positions_for(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, int]:
"""`user`'s resume point in each of `jobs` that has one (the library player's `library_states`)."""
ids = [j.id for j in jobs]
if not ids:
return {}
rows = db.execute(
select(LibraryState.job_id, LibraryState.position_seconds).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id.in_(ids)
)
).all()
return {job_id: pos for job_id, pos in rows}
def source_keys(db: Session, jobs: Sequence[DownloadJob]) -> dict[int, str]:
"""Where each file came from (its source folder's key), for the list's Source column, the
"back to its source folder" move and the download tray's site: a file in a source folder has its
folder's key; one moved into an own folder (or someone else's, shared) is traced back by
`folders.source_key_for`."""
ids = {j.folder_id for j in jobs if j.folder_id is not None}
by_id = (
{f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
if ids
else {}
)
out: dict[int, str] = {}
for j in jobs:
folder = by_id.get(j.folder_id) if j.folder_id is not None else None
if folder is not None and folder.kind == folders.FolderKind.SOURCE and folder.source_key:
out[j.id] = folder.source_key
else:
out[j.id] = folders.source_key_for(db, j)
return out
def queue(db: Session, user: User, recent: datetime) -> list[DownloadJob]:
"""The download tray: what is waiting, running, paused or failed, and what finished (done or
canceled) since `recent`. A done job is timed from when it became done, never from its last
write: a rename, a move or the GC's expiry warning touches an old file's row, and the tray would
announce it as just finished."""
return list(
db.execute(
select(DownloadJob)
.where(
owned_by(DownloadJob.user_id, user),
DownloadJob.status.not_in(FINISHED)
| ((DownloadJob.status == JobStatus.DONE) & (DownloadJob.ready_at >= recent))
| ((DownloadJob.status == JobStatus.CANCELED) & (DownloadJob.updated_at >= recent)),
)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
def statuses_by_source(db: Session, user: User, statuses: Sequence[str]) -> list[tuple[str, str]]:
"""(source_ref, status) of each of `user`'s jobs in one of `statuses`."""
return list(
db.execute(
select(DownloadJob.source_ref, DownloadJob.status).where(
owned_by(DownloadJob.user_id, user), DownloadJob.status.in_(tuple(statuses))
)
).tuples()
)
def library(
db: Session,
user: User,
*,
folder_id: int | None,
listable: set[int] | None,
shared: bool,
term: str,
sort: str,
order: str,
lang: str,
) -> list[DownloadJob]:
"""The caller's done files (optionally one folder's, or only in the folders a token may list),
or the ones shared with them; searched by name and channel, sorted by when they were added, by
name (in the reader's alphabet) or by size. A file is added to its owner's list when it is done,
not when it was asked for (a download that failed for days and then went through is new); one
shared with the caller, when it was shared."""
stmt = (
select(DownloadJob)
.join(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.status == JobStatus.DONE)
)
if shared:
stmt = stmt.join(DownloadShare, DownloadShare.job_id == DownloadJob.id).where(
owned_by(DownloadShare.to_user_id, user)
)
else:
stmt = stmt.where(owned_by(DownloadJob.user_id, user))
if folder_id is not None:
stmt = stmt.where(DownloadJob.folder_id == folder_id)
elif listable is not None:
stmt = stmt.where(DownloadJob.folder_id.in_(listable))
name = func.coalesce(DownloadJob.display_name, MediaAsset.title, "")
if term:
like = f"%{like_escape(term)}%"
stmt = stmt.where(
or_(
name.ilike(like),
func.coalesce(DownloadJob.display_uploader, MediaAsset.uploader, "").ilike(like),
)
)
key = {
"added": DownloadShare.created_at if shared else DownloadJob.ready_at,
"name": collated(name, lang),
"size": MediaAsset.size_bytes,
}[sort]
first = key.asc() if order == "asc" else key.desc()
tie = DownloadJob.id.asc() if order == "asc" else DownloadJob.id.desc()
return list(db.execute(stmt.order_by(first.nulls_last(), tie).limit(MAX_LIST)).scalars())
def recent_jobs(db: Session) -> list[DownloadJob]:
"""Everyone's newest jobs, for the admin's list."""
return list(
db.execute(
select(DownloadJob)
.order_by(DownloadJob.created_at.desc(), DownloadJob.id.desc())
.limit(MAX_LIST)
).scalars()
)
# --- shares --------------------------------------------------------------------------------
def shared_with(db: Session, job_id: int, user: User | int) -> DownloadShare | None:
"""The grant that hands this job to `user`, if there is one."""
return db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, user)
)
).scalar_one_or_none()
def shared_at(db: Session, user: User, jobs: Sequence[DownloadJob]) -> dict[int, datetime]:
rows = db.execute(
select(DownloadShare.job_id, DownloadShare.created_at).where(
owned_by(DownloadShare.to_user_id, user),
DownloadShare.job_id.in_([j.id for j in jobs]),
)
).all()
return {job_id: at for job_id, at in rows}
# --- folders -------------------------------------------------------------------------------
def own_folder(db: Session, user: User, folder_id: int) -> FileFolder | None:
return owned_row(db, FileFolder, folder_id, user)
def folders_by_id(db: Session, ids: set[int]) -> dict[int, FileFolder]:
"""These folders, by id — for a reader that already knows which ones (a token's grants)."""
if not ids:
return {}
return {f.id: f for f in db.execute(select(FileFolder).where(FileFolder.id.in_(ids))).scalars()}
def folders_with_totals(
db: Session, user: User, listable: set[int] | None
) -> list[tuple[FileFolder, int, int]]:
"""The user's folders (only `listable` ones, when given) with how many done files each holds and
their size. A source folder its files have all left stays (count 0)."""
done = (DownloadJob.folder_id == FileFolder.id) & (DownloadJob.status == JobStatus.DONE)
stmt = select(
FileFolder,
func.count(DownloadJob.id),
func.coalesce(func.sum(MediaAsset.size_bytes), 0),
).where(owned_by(FileFolder.user_id, user))
if listable is not None:
stmt = stmt.where(FileFolder.id.in_(listable))
rows = db.execute(
stmt.outerjoin(DownloadJob, done)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.group_by(FileFolder.id)
.order_by(FileFolder.kind.desc(), FileFolder.source_key, FileFolder.name, FileFolder.id)
).tuples()
return [(f, count, int(size or 0)) for f, count, size in rows]
def folder_totals(db: Session, folder_id: int) -> tuple[int, int]:
count, size = db.execute(
select(func.count(DownloadJob.id), func.coalesce(func.sum(MediaAsset.size_bytes), 0))
.select_from(DownloadJob)
.outerjoin(MediaAsset, MediaAsset.id == DownloadJob.asset_id)
.where(DownloadJob.folder_id == folder_id, DownloadJob.status == JobStatus.DONE)
).one()
return count, int(size)
def linked_folders(db: Session, folder_ids: list[int]) -> set[int]:
"""Which of these folders a live public link opens (one not past its expiry)."""
now = datetime.now(timezone.utc)
rows = db.execute(
select(DownloadLink.folder_id).where(
DownloadLink.folder_id.in_(folder_ids),
or_(DownloadLink.expires_at.is_(None), DownloadLink.expires_at > now),
)
).scalars()
return {folder_id for folder_id in rows if folder_id is not None}
# --- links ---------------------------------------------------------------------------------
def own_link(db: Session, user: User, link_id: int) -> DownloadLink | None:
"""A link to one of the user's files or folders."""
link = db.get(DownloadLink, link_id)
if link is None:
return None
target = (
owned_row(db, DownloadJob, link.job_id, user)
if link.job_id is not None
else owned_row(db, FileFolder, link.folder_id, user)
)
return link if target is not None else None
def links_to(
db: Session, *, job_id: int | None = None, folder_id: int | None = None
) -> list[DownloadLink]:
"""The links to one file or one folder, newest first."""
target = (
DownloadLink.job_id == job_id if job_id is not None else DownloadLink.folder_id == folder_id
)
return list(
db.execute(select(DownloadLink).where(target).order_by(DownloadLink.id.desc())).scalars()
)
def link_folder(db: Session, link: DownloadLink) -> FileFolder | None:
return db.get(FileFolder, link.folder_id) if link.folder_id is not None else None
def folder_job(db: Session, folder: FileFolder, job_id: int) -> DownloadJob | None:
"""One of the files a link to `folder` reaches (`links.in_folder`), while it is in it."""
return db.execute(
select(DownloadJob).where(DownloadJob.id == job_id, *links.in_folder(folder))
).scalar_one_or_none()
def owner_suspended(db: Session, job: DownloadJob) -> bool:
"""The job's owner is suspended (or gone): the worker leaves their queued jobs where they are."""
owner = db.get(User, job.user_id)
return owner is None or owner.is_suspended
# --- profiles ------------------------------------------------------------------------------
def profiles_for(db: Session, user: User) -> list[DownloadProfile]:
"""The built-in format profiles, then the user's own."""
return list(
db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True) | owned_by(DownloadProfile.user_id, user))
.order_by(
DownloadProfile.is_builtin.desc(), DownloadProfile.sort_order, DownloadProfile.id
)
).scalars()
)
def own_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""One of the user's own profiles (a built-in is no one's to change)."""
p = owned_row(db, DownloadProfile, profile_id, user)
return None if p is None or p.is_builtin else p
def usable_profile(db: Session, user: User, profile_id: int) -> DownloadProfile | None:
"""A profile the user may enqueue with: a built-in, or their own."""
p = db.get(DownloadProfile, profile_id)
if p is None or (not p.is_builtin and p.user_id != user.id):
return None
return p
def default_profile(db: Session) -> DownloadProfile | None:
"""The first built-in, for an enqueue that named no format."""
return db.execute(
select(DownloadProfile)
.where(DownloadProfile.is_builtin.is_(True))
.order_by(DownloadProfile.sort_order)
.limit(1)
).scalar_one_or_none()
# --- storage and quota (admin) -------------------------------------------------------------
def storage_totals(db: Session) -> tuple[int, int, int]:
"""(files on disk, their bytes, every asset row). On disk is `quota.ON_DISK`: ready files, and
uploads waiting for or failed in their conversion — the same disk the footprint and the GC cap
count."""
on_disk = db.execute(
select(func.count(), func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(quota.ON_DISK)
).one()
total_assets = db.execute(select(func.count()).select_from(MediaAsset)).scalar_one()
return on_disk[0], int(on_disk[1]), total_assets
def done_files_per_user(db: Session) -> dict[int, int]:
"""How many finished files each user's Files lists."""
return dict(
db.execute(
select(DownloadJob.user_id, func.count())
.where(DownloadJob.status == JobStatus.DONE)
.group_by(DownloadJob.user_id)
)
.tuples()
.all()
)
def custom_quota(db: Session, user_id: int) -> DownloadQuota | None:
"""The admin's own limits for this user, if any were set."""
return db.get(DownloadQuota, user_id)
+86 -13
View File
@@ -1,7 +1,9 @@
"""Per-user download limits + footprint accounting.
A user's *footprint* is the total size of the ready assets their library references (cache hits
count — it's "how much disk you're responsible for", not "how much you personally downloaded").
count — it's "how much disk you're responsible for", not "how much you personally downloaded"). An
upload counts from the moment it lands, ready or not: its original is on disk while its conversion
waits, and stays there when the conversion fails (for Retry) — bytes a ready-only sum never saw.
Limits come from a per-user DownloadQuota row if the admin set one, else the sysconfig defaults.
`unlimited` (e.g. the admin's own account) bypasses the byte cap.
@@ -13,21 +15,31 @@ the footprint as it stood when the job was created and earlier jobs may have fil
That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user each read the same
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. Edit jobs do not consult it at all (tracked as SEC-19).
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
"""
from dataclasses import dataclass
from sqlalchemy import func, select
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
from app import sysconfig
from app.models import DownloadJob, DownloadQuota, MediaAsset
from app.downloads.status import JobKind, JobStatus
from app.downloads.models import DownloadJob, DownloadQuota, MediaAsset
from app.userscope import owned_by
# Jobs a user is considered to be "holding" (occupy a job slot / footprint). Terminal failures
# (error, canceled) don't count.
_HOLDING = ("queued", "running", "paused", "done")
_RUNNING = ("queued", "running")
_HOLDING = (JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE)
_RUNNING = (JobStatus.QUEUED, JobStatus.RUNNING)
# An asset whose file is on disk: a ready one, or an upload in any state (its original lands at
# ingest, see the module docstring). Every byte sum — a user's footprint, the GC's total-cap pass,
# the admin's storage view — reads it, so they all count the same disk.
ON_DISK = or_(
MediaAsset.status == "ready",
and_(MediaAsset.source_kind == "upload", MediaAsset.rel_path.is_not(None)),
)
@dataclass
@@ -56,7 +68,8 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
# "at limit" checks — the worker's claim-window exclusion vs at_concurrency_limit — since a
# 0-limit user is "capped" with zero running jobs, which the SQL exclusion can't see. Keep
# them consistent by never letting the effective limit drop below 1. (To BLOCK a user's
# downloads, suspend them or set max_jobs=0, not max_concurrent=0.)
# downloads, suspend them — the worker leaves a suspended user's queued jobs unclaimed — or
# set max_jobs=0, not max_concurrent=0.)
return QuotaLimits(row.max_bytes, max(1, row.max_concurrent), row.max_jobs, row.unlimited)
return QuotaLimits(
max_bytes=sysconfig.get_int(db, "download_default_max_bytes"),
@@ -67,12 +80,17 @@ def resolve(db: Session, user_id: int) -> QuotaLimits:
def footprint(db: Session, user_id: int) -> int:
"""Total bytes of the distinct ready assets this user's active jobs reference."""
"""Total bytes of the distinct assets this user's jobs hold on disk: the ready ones their active
jobs reference, and every upload of theirs — including one whose conversion failed, which keeps
its original (see the module docstring)."""
held_assets = (
select(DownloadJob.asset_id)
.where(
DownloadJob.user_id == user_id,
DownloadJob.status.in_(_HOLDING),
owned_by(DownloadJob.user_id, user_id),
or_(
DownloadJob.status.in_(_HOLDING),
and_(DownloadJob.job_kind == JobKind.UPLOAD, DownloadJob.status == JobStatus.ERROR),
),
DownloadJob.asset_id.is_not(None),
)
.distinct()
@@ -80,7 +98,7 @@ def footprint(db: Session, user_id: int) -> int:
return int(
db.execute(
select(func.coalesce(func.sum(MediaAsset.size_bytes), 0)).where(
MediaAsset.id.in_(held_assets), MediaAsset.status == "ready"
MediaAsset.id.in_(held_assets), ON_DISK
)
).scalar()
or 0
@@ -92,7 +110,7 @@ def _count(db: Session, user_id: int, states) -> int:
db.execute(
select(func.count())
.select_from(DownloadJob)
.where(DownloadJob.user_id == user_id, DownloadJob.status.in_(states))
.where(owned_by(DownloadJob.user_id, user_id), DownloadJob.status.in_(states))
).scalar()
or 0
)
@@ -111,7 +129,7 @@ def at_concurrency_limit(db: Session, user_id: int) -> bool:
lim = resolve(db, user_id)
if lim.unlimited:
return False
running = _count(db, user_id, ("running",))
running = _count(db, user_id, (JobStatus.RUNNING,))
return running >= lim.max_concurrent
@@ -160,6 +178,18 @@ def check_upload(db: Session, user_id: int, add_bytes: int) -> None:
raise QuotaExceeded("max_bytes", lim.max_bytes, projected)
def fits_after_conversion(db: Session, user_id: int, old_bytes: int, new_bytes: int) -> bool:
"""Whether an upload converted from `old_bytes` (what it is charged at now) to `new_bytes` keeps
its owner within the cap. A re-encode can come out larger than the original, and nothing else
re-checks it once the upload was let in."""
if new_bytes <= old_bytes: # a smaller file never breaks a cap the larger one was let in under
return True
lim = resolve(db, user_id)
if lim.unlimited:
return True
return footprint(db, user_id) - old_bytes + new_bytes <= lim.max_bytes
def usage(db: Session, user_id: int) -> dict:
"""Compact usage snapshot for the UI (footprint + counts vs limits)."""
lim = resolve(db, user_id)
@@ -172,3 +202,46 @@ def usage(db: Session, user_id: int) -> dict:
"max_jobs": lim.max_jobs,
"unlimited": lim.unlimited,
}
def set_custom(
db: Session,
user_id: int,
*,
max_bytes: int | None = None,
max_concurrent: int | None = None,
max_jobs: int | None = None,
unlimited: bool | None = None,
) -> None:
"""The admin's own limits for a user: starts from what applies now, then each value given (a
size floored at 0, a count at 1). The handler commits."""
row = db.get(DownloadQuota, user_id)
if row is None:
cur = resolve(db, user_id)
row = DownloadQuota(
user_id=user_id,
max_bytes=cur.max_bytes,
max_concurrent=cur.max_concurrent,
max_jobs=cur.max_jobs,
unlimited=cur.unlimited,
)
db.add(row)
if max_bytes is not None:
row.max_bytes = max(0, max_bytes)
if max_concurrent is not None:
row.max_concurrent = max(1, max_concurrent)
if max_jobs is not None:
row.max_jobs = max(1, max_jobs)
if unlimited is not None:
row.unlimited = unlimited
db.flush()
def reset_custom(db: Session, user_id: int) -> bool:
"""Back to the defaults; False when the user had no limits of their own."""
row = db.get(DownloadQuota, user_id)
if row is None:
return False
db.delete(row)
db.flush()
return True
+53 -10
View File
@@ -11,18 +11,22 @@ writes the terminal job/asset state itself (done / error, or a cooperative pause
progress hook), then exits 0. An unexpected crash exits non-zero and the parent fails the job.
"""
import logging
import os
import shutil
import subprocess
import sys
import app.tables # noqa: F401 — a fresh interpreter: every table, before the first query
from app import logs
from app import logsink
from app.config import settings
from app.db import SessionLocal
from app.downloads import edit as editmod
from app.models import DownloadJob
from app.downloads import formats
from app.downloads.status import JobKind
from app.downloads.models import DownloadJob
log = logging.getLogger("siftlode.worker.runner")
log = logs.get_logger(__name__)
def _deprioritize() -> None:
@@ -52,12 +56,21 @@ def _deprioritize() -> None:
def main() -> None:
logging.basicConfig(level=logging.INFO)
logs.configure_logging("runner")
_deprioritize()
if len(sys.argv) != 2:
print("usage: python -m app.downloads.runner <job_id>", file=sys.stderr)
# T201: a usage line is for the person who typed the command. Logging is already
# configured here, and in a deployment it writes JSON records — the one reader of this
# line wants the bare sentence, not a record with a request id they do not have.
print( # noqa: T201
"usage: python -m app.downloads.runner <job_id>", file=sys.stderr
)
raise SystemExit(2)
job_id = int(sys.argv[1])
# This whole process exists to run ONE job — tag every line it prints with it, so the child's
# output joins up with the worker's and the API's view of the same download.
logs.set_job(job_id)
logsink.start() # flushed at exit (atexit) — this process has no shutdown hook of its own
with SessionLocal() as db:
job = db.get(DownloadJob, job_id)
@@ -71,14 +84,44 @@ def main() -> None:
if asset_id is None:
return
_run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id)
def _run(job_id, asset_id, job_kind, source_kind, source_ref, spec, user_id) -> None:
"""Run the job, and be the last stop before Python's own exception hook.
What gets past the pipeline is a failure it could not even record (the database refused the
write that fails the job). Python's hook would print it to stderr raw — and its context is the
download's own failure, whose text is the URL or ffmpeg's stderr naming the file. Here it is
one line with the whole chain scrubbed; the non-zero exit then has the worker fail the job
(`cause=child_exit`)."""
# Imported here (not at module top) so the child pays the yt-dlp import cost only when it has
# real work, and to keep the import graph obvious: runner is a thin entry over the pipeline.
from app.downloads.pipeline import _download, _process_edit
from app.downloads.pipeline import _convert_upload, _download, _process_clip, _process_edit
if job_kind == "edit":
_process_edit(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
with formats.hiding(source_ref):
try:
if job_kind == JobKind.EDIT:
_process_edit(job_id, asset_id)
elif job_kind == JobKind.UPLOAD:
_convert_upload(job_id, asset_id)
elif job_kind == JobKind.CLIP:
_process_clip(job_id, asset_id)
else:
_download(job_id, asset_id, source_kind, source_ref, spec, user_id)
except Exception as exc: # noqa: BLE001 — whatever it is, it is reported here or raw
# TRY400: the stack goes with it scrubbed (`scrubbed_stack`); `exc_info` is raw.
log.error( # noqa: TRY400
"the download child for job %s failed",
job_id,
extra={
**logs.event(
"download.child.failed", job_id=job_id, error_type=type(exc).__name__
),
**logs.scrubbed_stack(exc, formats.log_safe),
},
)
raise SystemExit(1) from None
if __name__ == "__main__":
+460 -44
View File
@@ -1,25 +1,42 @@
"""Enqueue + cache resolution for the download center (the per-user layer over MediaAsset).
"""Capture's writes: enqueue + cache resolution, the job transitions, and what the library's
dialogs change (profiles, resume points, shares, a file's display fields).
`enqueue` is the single entry point used by the API: it resolves (or creates) the shared
MediaAsset for the requested source+format, links a per-user DownloadJob to it, and — on a
cache hit (asset already ready) — marks the job done instantly with no re-download. The worker
handles everything else. State transitions used by the routes (pause/resume/cancel/delete) and
the worker live in `worker.py` / the routes; this module owns enqueue + asset bookkeeping.
handles everything else. Every function here flushes and leaves the commit to its caller (the
handler); the folder, link and quota writes are `folders`, `links` and `quota`.
"""
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete as delete_rows
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.config import settings
from app.downloads import adapters
from app.downloads import clip as clipmod
from app.downloads import credentials
from app.downloads import edit as editmod
from app.downloads import formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobStatus
from app.models import Channel, DownloadJob, MediaAsset, Video
from app.downloads import folders, formats, mediakind, quota, storage
from app.downloads.status import HOLDING, JobKind, JobStatus
from app.models import DISPLAY_TEXT_MAX, User
from app.plex.models import PlexItem, PlexShow
from app.catalog.models import Channel, Video
from app.downloads.models import (
DownloadJob,
DownloadProfile,
DownloadShare,
LibraryState,
MediaAsset,
)
from app.plex import lookups as plex_lookups
from app.plex import paths as plex_paths
from app.userscope import owned_by, owned_row
class EditError(Exception):
@@ -37,12 +54,15 @@ def source_url(source_kind: str, source_ref: str) -> str:
def reference_url(job, asset) -> str | None:
"""The clean "downloaded from" URL for a job — shared by the private Downloads page and the public
"""The clean "downloaded from" URL for a job — shared by the Files library and the public
watch page so both surfaces always agree. None for edit derivatives (a clip's source is the user's
own earlier download, not a web page); else the canonical page URL yt-dlp recorded; else one derived
from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == "edit":
own earlier download, not a web page) and for a Plex clip (the library is not a public page); a
YouTube clip links its video at the clip's start; else the canonical page URL yt-dlp recorded; else
one derived from source_kind+source_ref (covers queued/older rows before the worker fills it)."""
if job is None or job.job_kind == JobKind.EDIT:
return None
if job.job_kind == JobKind.CLIP:
return clipmod.reference_url(job.edit_spec or {})
if asset is not None and asset.source_webpage_url:
return asset.source_webpage_url
if job.source_kind in ("youtube", "url"):
@@ -50,6 +70,23 @@ def reference_url(job, asset) -> str | None:
return None
def library_left_behind(db: Session, files_root: str) -> bool:
"""True when the library names files and none of the newest of them is under `files_root` — a
fresh, empty volume mounted in the library's place (a compose file changed under an install, its
old volume left behind). The mount check cannot see that: the new volume is a real mount."""
rels = (
db.execute(
select(MediaAsset.rel_path)
.where(MediaAsset.status == "ready", MediaAsset.rel_path.is_not(None))
.order_by(MediaAsset.id.desc())
.limit(20)
)
.scalars()
.all()
)
return bool(rels) and not any(storage.abs_path(files_root, r).exists() for r in rels if r)
def get_or_create_asset(
db: Session, source_kind: str, source_ref: str, format_sig: str
) -> MediaAsset:
@@ -62,7 +99,12 @@ def get_or_create_asset(
MediaAsset.source_ref == source_ref,
MediaAsset.format_sig == format_sig,
)
asset = db.execute(stmt).scalar_one_or_none()
# Locked to the caller's commit: a release or the GC freeing this asset meanwhile (they lock it
# too) either waits for the new hold, or has deleted the row by the time this read returns — and
# the caller then gets a fresh one, never a job pointing at a row that is gone.
asset = db.execute(
stmt.with_for_update().execution_options(populate_existing=True)
).scalar_one_or_none()
if asset is not None:
# A previously-FAILED shared asset must be re-attempted for the new job about to hold it —
# reset it to pending (+clear the error) so the worker actually re-downloads instead of
@@ -80,11 +122,15 @@ def get_or_create_asset(
status="pending",
ref_count=0,
)
db.add(asset)
try:
db.flush()
# A SAVEPOINT: losing the race undoes only this insert, never the caller's own pending writes
# (Retry clears the job's error before it gets here; a whole-session rollback threw that away).
with db.begin_nested():
db.add(asset)
db.flush()
# silent-ok: the expected outcome of the race this savepoint exists for — two jobs wanting the
# same asset. Losing it means the other insert won, and re-selecting is the whole handling.
except IntegrityError:
db.rollback()
asset = db.execute(stmt).scalar_one()
return asset
@@ -174,29 +220,33 @@ def enqueue(
profile_id=profile_id,
profile_snapshot=spec,
display_name=display_name
or (asset.title[:255] if asset.status == "ready" and asset.title else None),
status="queued",
or (asset.title[:DISPLAY_TEXT_MAX] if asset.status == "ready" and asset.title else None),
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
# Cache hit: the file already exists — no worker round-trip needed.
if asset.status == "ready":
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
def _clip_title(source_title: str | None, display_name: str | None) -> str:
def clip_title(source_title: str | None, display_name: str | None) -> str:
# Clipped to DISPLAY_TEXT_MAX, not to a literal: this feeds `DownloadJob.display_name` through
# `enqueue_edit`, the same column `display_text` guards. A machine-made title, so it is clipped
# rather than run through `display_text` — blank here means "no title", not None.
if display_name:
return display_name[:255]
return display_name[:DISPLAY_TEXT_MAX]
base = source_title or "Clip"
return f"{base} (clip)"[:255]
return f"{base} (clip)"[:DISPLAY_TEXT_MAX]
def enqueue_edit(
@@ -231,7 +281,7 @@ def enqueue_edit(
ref = f"{user_id}:{src_asset.id}"[:512]
asset = get_or_create_asset(db, "edit", ref, sig)
if not asset.title:
asset.title = _clip_title(src_asset.title, display_name)
asset.title = clip_title(src_asset.title, display_name)
asset.uploader = src_asset.uploader
asset.thumbnail_url = src_asset.thumbnail_url
asset.upload_date = src_asset.upload_date
@@ -243,27 +293,116 @@ def enqueue_edit(
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="edit",
job_kind=JobKind.EDIT,
source_kind="edit",
source_ref=ref,
source_job_id=source_job.id,
source_asset_id=src_asset.id,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title,
status="queued",
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # identical clip already produced for this user → instant
job.status = "done"
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.commit()
db.refresh(job)
db.flush()
return job
def _clip_source_facts(db: Session, spec: dict) -> dict:
"""Title, uploader, thumbnail and length of what a clip is cut from, read from our own catalog.
Raises ClipError when a Plex item is unknown or its file cannot be read here — said at enqueue,
rather than as a job that fails in the queue a minute later."""
if spec["source"] == "youtube":
video = db.get(Video, spec["ref"])
if video is None: # not in the catalog: titled "Clip" until yt-dlp names it (worker)
return {"title": None, "uploader": None, "thumbnail_url": None, "length_s": None}
channel = db.get(Channel, video.channel_id)
return {
"title": video.title,
"uploader": channel.title if channel else None,
"thumbnail_url": video.thumbnail_url,
"length_s": video.duration_seconds,
}
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
if item is None:
raise clipmod.ClipError("not_found")
if plex_paths.local_media_path(db, item.file_path) is None:
raise clipmod.ClipError("media_missing")
title = item.title
show = db.get(PlexShow, item.show_id) if item.show_id else None
if show is not None:
episode = (
f"S{item.season_number}·E{item.episode_number}"
if item.season_number is not None and item.episode_number is not None
else None
)
title = " — ".join(p for p in (show.title, episode, item.title) if p)
return {
"title": title,
"uploader": None,
# Plex art is served through our own authenticated routes, never a public URL; the worker
# cuts the clip's poster from the clip itself.
"thumbnail_url": None,
"length_s": item.duration_s,
}
def enqueue_clip(
db: Session, user_id: int, spec: dict, display_name: str | None = None
) -> DownloadJob:
"""Queue an ad-hoc clip (`clipmod.normalize`d spec) for `user_id`: a NEW per-user asset cut
straight from the source — a YouTube range or a Plex library file (see downloads/clip).
Raises ClipError (unknown or unreachable source, a range past its end) or quota.QuotaExceeded."""
facts = _clip_source_facts(db, spec)
# One second of grace: a mark set on the last frame may round past the catalog's whole seconds.
if facts["length_s"] and spec["end_s"] > facts["length_s"] + 1:
raise clipmod.ClipError("past_end")
quota.check_enqueue(db, user_id)
ref = clipmod.asset_ref(user_id, spec)[:512]
asset = get_or_create_asset(db, "clip", ref, clipmod.sig(spec))
if not asset.title:
asset.title = clip_title(facts["title"], display_name)
asset.uploader = facts["uploader"]
asset.thumbnail_url = facts["thumbnail_url"]
asset.duration_s = round(clipmod.duration(spec))
asset.media_kind = mediakind.VIDEO
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind=JobKind.CLIP,
source_kind="clip",
source_ref=ref,
edit_spec=spec,
profile_snapshot={},
display_name=display_name or asset.title[:DISPLAY_TEXT_MAX],
status=JobStatus.QUEUED,
queue_pos=next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
incr_ref_count(db, asset.id)
if asset.status == "ready": # the same range was already cut for this user → instant
job.status = JobStatus.DONE
job.progress = 100
folders.stamp_done(db, job)
asset.last_access_at = datetime.now(timezone.utc)
db.flush()
return job
@@ -283,6 +422,8 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
if not fields:
return False
db.flush() # asset-then-job lock order: push pending asset writes before we lock the job row
if fields.get("status") == JobStatus.DONE:
return _finalize_done(db, job_id, {**fields, "ready_at": datetime.now(timezone.utc)})
# A Core UPDATE with the guard as a WHERE — same atomic "only while still running" write as the
# rest of the codebase's 2.0 statements, no hand-built SQL string. synchronize_session=False: this
# is a terminal write, so we don't pay to sync the (about-to-be-discarded) ORM identity map. Bound
@@ -296,6 +437,41 @@ def finalize_if_running(db: Session, job_id: int, **fields) -> bool:
return result.rowcount > 0
def _finalize_done(db: Session, job_id: int, fields: dict) -> bool:
"""`finalize_if_running` for a job that finished: done is when a file's retention starts, its
folder's unless the user kept it. The expiry is written only while the job is still where it was
read from — a move, a keep, or its folder's delete (which sends its files home) landing while the
folder's retention was read sends the read round again, never stamping a folder's retention on a
file now in another one."""
while True:
held = db.execute(
select(DownloadJob.folder_id, DownloadJob.kept, DownloadJob.status).where(
DownloadJob.id == job_id
)
).one_or_none()
if held is None or held.status != JobStatus.RUNNING:
return False
stamped = {
**fields,
"expires_at": folders.done_expiry(db, held.folder_id, held.kept),
"expiry_warned": False,
}
hit = db.execute(
update(DownloadJob)
.where(
DownloadJob.id == job_id,
DownloadJob.status == JobStatus.RUNNING.value,
DownloadJob.folder_id.is_not_distinct_from(held.folder_id),
DownloadJob.kept == held.kept,
)
.values(**stamped)
.returning(DownloadJob.id)
.execution_options(synchronize_session=False)
).scalar_one_or_none()
if hit is not None:
return True
def release_asset(db: Session, job: DownloadJob) -> None:
"""Drop this job's hold on its asset when it leaves a holding state. Once NO job holds the
asset anymore, delete the file + row immediately — a deleted download should free its disk,
@@ -310,24 +486,32 @@ def release_asset(db: Session, job: DownloadJob) -> None:
re-enqueuing the same (source,format) — get_or_create_asset would reset that row to `pending` and
+1 it, and deleting it under that would FK-null the new job's asset (a lost update on ref_count).
A ready asset is still freed at ref==0 (its file is the point); an orphaned errored row is cheap
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass."""
(no file) and gets reused+reset by the next enqueue, or reclaimed by a later GC pass.
An UPLOAD is the exception: it is never shared or re-enqueued (its ref is unique), and while it
waits for or failed its phone-safe conversion the original file already sits at `rel_path` — so
it is freed at ref==0 whatever its state, except mid-conversion (`downloading`: the worker child
still reads it; the GC upload pass takes it once the child has stopped)."""
if not job.asset_id or job.status not in HOLDING:
return
asset = db.get(MediaAsset, job.asset_id)
if asset is None:
return
new_count = decr_ref_count(db, asset.id)
if new_count == 0 and asset.status == "ready":
freeable = asset.status == "ready" or (
asset.source_kind == "upload" and asset.status != "downloading"
)
if new_count == 0 and freeable:
if asset.rel_path:
storage.delete_asset_files(settings.download_root, asset.rel_path)
storage.delete_asset_files(settings.files_root, asset.rel_path)
db.delete(asset)
# --- Job state transitions ---------------------------------------------------------------------
# The single home for the pause/resume/cancel/delete/redownload transitions + their asset
# bookkeeping (release, failed-asset reset, fresh-asset rebuild), so the routes are thin callers and
# the worker/GC could share the same definitions (B-sub-8). Each commits its own transition; the
# route keeps only auth + HTTP mapping + serialization.
# the worker/GC could share the same definitions (B-sub-8). Each flushes its transition and the caller
# commits it; the route keeps only auth + HTTP mapping + serialization.
def pause(db: Session, job: DownloadJob) -> None:
@@ -335,24 +519,57 @@ def pause(db: Session, job: DownloadJob) -> None:
in any other state."""
if job.status in (JobStatus.QUEUED, JobStatus.RUNNING):
job.status = JobStatus.PAUSED
db.commit()
db.flush()
def resume(db: Session, job: DownloadJob) -> None:
"""Requeue a paused/errored job for a fresh attempt, clearing its error. If the SHARED asset itself
failed, reset it to pending so the worker actually re-downloads — otherwise the requeued job would
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset)."""
instantly re-inherit the asset's stale error (the worker short-circuits an errored asset).
A cookie the user added AFTER the job was queued moves the job onto their private asset first
(`_follow_the_cookie`): the worker only ever attaches a cookie to that identity, so without the move
the Retry a login-wall message asks for would fail on the same wall."""
if job.status in (JobStatus.PAUSED, JobStatus.ERROR):
job.progress = 0
job.error = None
job.error_code = None
job.error_detail = None
if job.asset_id:
asset = db.get(MediaAsset, job.asset_id)
if asset is not None:
asset = _follow_the_cookie(db, job, asset)
if asset and asset.status == "error": # MediaAsset.status (no enum), not a JobStatus
asset.status = "pending"
asset.error = None
job.status = JobStatus.QUEUED
db.commit()
db.flush()
def _follow_the_cookie(db: Session, job: DownloadJob, asset: MediaAsset) -> MediaAsset:
"""Re-point a requeued job at the asset enqueue would pick NOW, when the user has a cookie for the
link's host but the job still holds an asset the worker won't attach it to (queued before the
cookie existed). Returns the asset the job holds afterwards.
Only a `pending` / `error` asset moves — neither has a file nor a download in flight. A `ready` one
already has the file, and a `downloading` one may be this job's own child still being stopped by
the pause; the next Retry after it settles moves the job then. Removing a cookie never moves a job
back: the worker attaches nothing without a credential, so the private asset is as good as any."""
if asset.status not in ("pending", "error"):
return asset
url = source_url(job.source_kind, job.source_ref)
salt = credentials.salt_for_enqueue(db, job.user_id, url)
spec = formats.normalize(job.profile_snapshot)
if salt is None or credentials.is_authed_asset(asset, spec, job.user_id, url):
return asset
target = get_or_create_asset(
db, job.source_kind, job.source_ref, formats.format_sig(spec, salt=salt)
)
populate_from_catalog(db, target)
release_asset(db, job) # drop the old hold BEFORE re-pointing — it keys off job.asset_id
incr_ref_count(db, target.id)
job.asset_id = target.id
return target
def cancel(db: Session, job: DownloadJob) -> None:
@@ -361,25 +578,43 @@ def cancel(db: Session, job: DownloadJob) -> None:
if job.status not in (JobStatus.DONE, JobStatus.CANCELED):
release_asset(db, job) # release while the job still counts as holding
job.status = JobStatus.CANCELED
db.commit()
db.flush()
def delete(db: Session, job: DownloadJob) -> None:
"""Delete a job, releasing its asset hold first (which frees the file iff this was the last holder)."""
release_asset(db, job)
db.delete(job)
db.commit()
db.flush()
def redownload(db: Session, job: DownloadJob) -> None:
"""Rebuild a finished/failed job's file into a FRESH, private asset (the cache sig is salted with
the job id + current asset id, so it never reuses the dedup pool) and requeue the SAME job row —
the new asset's own id, so it never reuses the dedup pool) and requeue the SAME job row —
so any public share link (DownloadLink -> job_id) keeps resolving to the newly-downloaded file.
The old hold is dropped first (freeing the old file iff this job was its last holder). The caller
has already rejected edit clips + still-in-progress jobs (HTTP 409)."""
has already rejected edit clips + still-in-progress jobs (HTTP 409).
When the user has a cookie for the link's host, the salt carries it (`credentials.rebuild_salt`),
so the worker recognises the rebuild as their private identity and attaches the cookie — a cookie
download, or one the user added after a failure, rebuilds with the cookie."""
spec = formats.normalize(job.profile_snapshot)
sig = formats.format_sig(spec, salt=f"{job.id}:{job.asset_id}")
asset = get_or_create_asset(db, job.source_kind, job.source_ref, sig)
cookie = credentials.salt_for_enqueue(
db, job.user_id, source_url(job.source_kind, job.source_ref)
)
# The salt names the row's own id, which exists only once the row does: insert it under a unique
# placeholder, then write the real signature in the same transaction (nobody sees the placeholder).
asset = MediaAsset(
source_kind=job.source_kind,
source_ref=job.source_ref,
format_sig=f"rebuild:{uuid.uuid4().hex}",
status="pending",
ref_count=0,
)
db.add(asset)
db.flush()
asset.format_sig = formats.format_sig(spec, salt=credentials.rebuild_salt(asset.id, cookie))
db.flush()
populate_from_catalog(db, asset)
# Drop the old hold BEFORE re-pointing the job — release_asset keys off the job's current asset.
release_asset(db, job)
@@ -389,10 +624,191 @@ def redownload(db: Session, job: DownloadJob) -> None:
job.progress = 0
job.phase = None
job.error = None
job.error_code = None # a rebuild that fails again gets its own diagnosis, never the old one
job.error_detail = None
job.speed_bps = None
job.eta_s = None
# Re-derive the display name from the freshly cleaned title on completion (else an older, un-cleaned
# name would survive the rebuild).
job.display_name = None
job.queue_pos = next_queue_pos(db) # tail of the queue, not its stale original slot
db.commit()
job.expires_at = None # stamped again when the rebuild is done
job.expiry_warned = False
db.flush()
# --- the library's writes ------------------------------------------------------------------
# What the HTTP layer changes for a user: each flushes, the handler commits.
def lock_own_job(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's jobs, re-read under a row lock, for a transition that moves its asset's ref
count (resume, cancel, delete, redownload) or writes the status those decide on (pause). Two
overlapping requests for one job run one after the other and the second decides on the first's
committed result — never both releasing the same hold, and never a pause turning a job a cancel
just released back into a holder; either can free a file another job still holds. None: not
theirs, or gone.
The asset is locked BEFORE the job: the order a finishing worker child takes them in
(`finalize_if_running`), so the two can never deadlock."""
while True:
owned = owned_row(db, DownloadJob, job_id, user)
if owned is None:
return None
asset_id = owned.asset_id
if asset_id is not None:
db.execute(select(MediaAsset.id).where(MediaAsset.id == asset_id).with_for_update())
job = db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update()
.execution_options(populate_existing=True)
).scalar_one_or_none()
if job is None:
return None
if job.asset_id == asset_id:
return job
db.rollback() # another request moved the job to another asset meanwhile: lock that one
def hold_for_share(db: Session, user: User, job_id: int) -> DownloadJob | None:
"""One of the user's files, held (FOR SHARE) for the rest of the transaction: a delete of it at
the same moment (the user's, or the retention sweep's) waits and takes the new link with it,
rather than failing the link's foreign key — or, done first, leaves None."""
if owned_row(db, DownloadJob, job_id, user) is None:
return None
return db.execute(
select(DownloadJob)
.where(DownloadJob.id == job_id)
.with_for_update(read=True)
.execution_options(populate_existing=True)
).scalar_one_or_none()
def create_profile(db: Session, user: User, name: str, spec: dict) -> DownloadProfile:
"""A format profile of the user's own, after their others."""
maxpos = db.execute(
select(func.coalesce(func.max(DownloadProfile.sort_order), 100)).where(
owned_by(DownloadProfile.user_id, user)
)
).scalar_one()
p = DownloadProfile(
user_id=user.id, name=name[:80], spec=spec, is_builtin=False, sort_order=maxpos + 1
)
db.add(p)
db.flush()
return p
def update_profile(profile: DownloadProfile, *, name: str | None, spec: dict | None) -> None:
"""Rename a profile and/or replace its spec; None leaves that one as it is."""
if name is not None:
profile.name = name[:80]
if spec is not None:
profile.spec = spec
def delete_profile(db: Session, profile: DownloadProfile) -> None:
db.delete(profile)
db.flush()
def set_position(db: Session, user: User, job_id: int, position: int) -> None:
"""The library player's resume point in a file the user can play; 0 clears it.
A clear is a statement, not a loaded row's delete: the periodic checkpoint and the pagehide beacon
of one stop may both clear it, and the second then deletes nothing rather than failing. A set is
an upsert, one row per (user, job) even when two checkpoints race, not a read-then-insert that
the second would fail on the unique key."""
if not position:
db.execute(
delete_rows(LibraryState).where(
owned_by(LibraryState.user_id, user), LibraryState.job_id == job_id
)
)
return
now = datetime.now(timezone.utc)
db.execute(
pg_insert(LibraryState)
.values(user_id=user.id, job_id=job_id, position_seconds=position, progress_updated_at=now)
.on_conflict_do_update(
constraint="uq_library_user_job",
set_={"position_seconds": position, "progress_updated_at": now},
)
)
def share(db: Session, job: DownloadJob, sender: User, recipient: User) -> bool:
"""Hand a file to another user, with a notice for them; False when it was already theirs."""
from app.notifications import create_notification
exists = db.execute(
select(DownloadShare.id).where(
DownloadShare.job_id == job.id, owned_by(DownloadShare.to_user_id, recipient)
)
).first()
if exists is not None:
return False
db.add(DownloadShare(job_id=job.id, from_user_id=sender.id, to_user_id=recipient.id))
create_notification(
db,
user_id=recipient.id,
type="download_shared",
title=(job.display_name or "A download") + " was shared with you",
# The client names the notice in the reader's language from `data`; the title is the
# English fallback only.
data={
"kind": "download_shared",
"from": sender.email,
"job_id": job.id,
"title": job.display_name,
},
commit=False,
)
db.flush()
return True
def unshare(db: Session, job_id: int, recipient: User) -> bool:
"""Take a file back from one user (its owner's act, or the recipient's own dismissal); False
when it was not theirs."""
row = db.execute(
select(DownloadShare).where(
DownloadShare.job_id == job_id, owned_by(DownloadShare.to_user_id, recipient)
)
).scalar_one_or_none()
if row is None:
return False
db.delete(row)
db.flush()
return True
# The display fields a user may set on their file; the file on disk keeps its own name.
DISPLAY_FIELDS = frozenset(
{"display_name", "display_uploader", "display_uploader_url", "extra_links"}
)
def set_display(job: DownloadJob, **fields: object) -> None:
"""Set a file's display fields (each already cleaned by the caller); a field not given is left."""
unknown = set(fields) - DISPLAY_FIELDS
if unknown:
raise ValueError(f"not a display field: {sorted(unknown)}")
for name, value in fields.items():
setattr(job, name, value)
def touch(asset: MediaAsset) -> None:
"""The file was just served: the cache's idle clock starts again."""
asset.last_access_at = datetime.now(timezone.utc)
def store_sha256(db: Session, asset_id: int, digest: str) -> None:
"""Keep a file's hash once it is known — only where none is, so a concurrent hash of the same
file never overwrites the first."""
db.execute(
update(MediaAsset)
.where(MediaAsset.id == asset_id, MediaAsset.sha256.is_(None))
.values(sha256=digest)
)
+13
View File
@@ -0,0 +1,13 @@
"""The failures the download pipeline raises itself, with the whole story in the message.
`ffmpeg failed (1): …`, `yt-dlp produced no output file`, a post that shows no video: the message
says what happened, and the stack is only the `except` that caught it. A failure line leaves that
stack off (`pipeline._failure_stack`) — but ONLY for these. A plain `RuntimeError` is also what
Python raises for a bug of ours (a dict changed size during iteration, a generator's
StopIteration, `RecursionError`, `NotImplementedError`), and that one's stack is the only lead.
"""
class StatedFailure(RuntimeError):
"""A failure we raised on purpose, whose message is its full explanation. A `RuntimeError` so
every `except RuntimeError` that caught these before still does."""
+21
View File
@@ -33,6 +33,27 @@ HOLDING = frozenset(
{JobStatus.QUEUED, JobStatus.RUNNING, JobStatus.PAUSED, JobStatus.DONE, JobStatus.ERROR}
)
class JobKind(StrEnum):
"""What a job produces, which decides the worker path that runs it and what the library may do
with the result (only a DOWNLOAD has a web source to fetch again)."""
DOWNLOAD = "download" # yt-dlp fetches a source into a shared, deduplicated asset
EDIT = "edit" # a per-user trim/crop of a file already in the library
UPLOAD = "upload" # a file the user sent; the worker only converts it
CLIP = (
"clip" # a per-user range of a YouTube video or a Plex item, cut straight from the source
)
# A job kind with no web source to fetch again: Re-download refuses these.
NOT_REFETCHABLE = frozenset({JobKind.EDIT, JobKind.UPLOAD, JobKind.CLIP})
# States a route sets to STOP a running job. The worker's supervisor must step in (kill the child, free
# the asset, leave the route's status intact) when it observes one of these on a job it is running.
STOPPED_BY_ROUTE = frozenset({JobStatus.PAUSED, JobStatus.CANCELED})
# A job that will not move again on its own: the queue lists it only briefly after it got here.
FINISHED = frozenset({JobStatus.DONE, JobStatus.CANCELED})
+114 -25
View File
@@ -11,6 +11,7 @@ a Plex library using a metadata agent picks up rich info. Channel = "show", vide
date-based — the community-standard mapping for YouTube-in-Plex.
"""
import os
import re
import shutil
import unicodedata
@@ -19,8 +20,11 @@ from datetime import date
from pathlib import Path
from xml.sax.saxutils import escape
from app import logs
from app.downloads import mediakind
log = logs.get_logger(__name__)
# Reserve room for the "_id.ext" suffix + parent dirs within the 255-char component limit.
_MAX_TITLE = 120
# User-facing (Content-Disposition) names can be longer than an on-disk component.
@@ -64,6 +68,57 @@ _FOLD_MAP = str.maketrans(
)
def _mount_points(mountinfo: str) -> list[str]:
"""The mount points a /proc/<pid>/mountinfo table lists (its fifth field, octal escapes undone)."""
points = []
for line in mountinfo.splitlines():
fields = line.split(" ")
if len(fields) > 4:
points.append(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), fields[4]))
return points
def _in_container() -> bool:
return Path("/.dockerenv").exists() or Path("/run/.containerenv").exists()
def _read_mountinfo() -> str | None:
try:
return Path("/proc/self/mountinfo").read_text()
# silent-ok: no mount table to read (not Linux) — the caller then cannot tell, and refuses nothing.
except OSError:
return None
def files_root_problem(
root: str, mountinfo: str | None = None, in_container: bool | None = None
) -> str | None:
"""Why `root` cannot be the files library's volume, or None when it can. In a container, a root
that is no mount of its own (nor inside one) is the image's empty directory: the library would
answer 404 for every file, and what a download writes would go when the container does — the
state an install is in when its compose file still mounts the library somewhere else.
Asked of the kernel's mount table, not by comparing devices: a storage driver that keeps the
container's root on the host's own disk (vfs, the fallback under Docker-in-LXC on older ZFS) gives
a mounted volume the same device as `/`. Outside a container a plain directory is a fine root."""
path = Path(root)
if not path.is_dir():
return f"{root} does not exist"
if in_container if in_container is not None else _in_container():
table = mountinfo if mountinfo is not None else _read_mountinfo()
if table is not None:
real = os.path.realpath(root)
mounted = any(
m != "/" and (real == m or real.startswith(m.rstrip("/") + "/"))
for m in _mount_points(table)
)
if not mounted:
return f"{root} is not a mounted volume (it is the container's own empty directory)"
if not os.access(path, os.W_OK):
return f"{root} is not writable"
return None
def _ascii_fold(text: str) -> str:
"""Transliterate to pure ASCII: apply the explicit map, NFKD-split accented letters and drop
the combining marks (á→a, ő→o), then strip any leftover non-ASCII (non-Latin scripts that have
@@ -119,7 +174,7 @@ def display_filename(name: str) -> str:
def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | None = None) -> str:
"""Path of the media file relative to DOWNLOAD_ROOT (forward slashes).
"""Path of the media file relative to FILES_ROOT (forward slashes).
`asset_id` disambiguates the file. A MediaAsset is unique on (source_kind, source_ref,
format_sig), so the SAME video downloaded in two formats is two distinct assets — but without
@@ -141,7 +196,7 @@ def rel_path(meta: MediaMeta, ext: str, layout: str = "plex", asset_id: int | No
def edit_rel_path(user_id: int, asset_id: int, ext: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a phase-2 editor derivative.
"""On-disk path (relative to FILES_ROOT) for a phase-2 editor derivative.
Kept in a per-user `.edits/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (these are user clips, not canonical episodes; there are no .nfo/poster
@@ -158,20 +213,32 @@ def _clean_upload_ext(filename: str) -> str:
def upload_rel_path(user_id: int, asset_id: int, filename: str) -> str:
"""On-disk path (relative to DOWNLOAD_ROOT) for a locally-uploaded file.
"""On-disk path (relative to FILES_ROOT) for a locally-uploaded file.
Kept in a per-user `.uploads/` tree — device-downloadable via the file endpoint, but NOT part of
the Plex library (an upload is arbitrary user content, not a canonical episode; no .nfo/poster
episode sidecars). The name is system-decided and bound to the asset id (`_a{id}` keeps two
same-named uploads apart); the original filename is display metadata only. The extension is
preserved (shell-safe, bounded) so the served file + its Content-Disposition carry a correct
suffix; an extensionless upload stays extensionless."""
stem = sanitize(Path(filename).stem, 80)
suffix; an extensionless upload stays extensionless.
The stem holds no dot, so the last dot of the name is always the extension's: every sibling
derived by swapping it (`with_suffix` — the poster, the converted file) keeps `_a{id}`. A dotted
stem with no extension ('Holiday.2024.ä') once lost it, and two such uploads shared one file."""
stem = sanitize(Path(filename).stem, 80).replace(".", "_")
ext = _clean_upload_ext(filename)
suffix = f".{ext}" if ext else ""
return f".uploads/{user_id}/{stem}_a{asset_id}{suffix}"
def with_ext(rel: str, old_ext: str | None, ext: str) -> str:
"""`rel` with its extension — `old_ext`, the stored `container`, None for a bare name — swapped
for `ext`. Not `Path.with_suffix`: on a bare name whose stem holds a dot (an upload placed before
the stems lost their dots) that would take the stem's tail for the extension, and `_a{id}` with it."""
base = rel[: -len(old_ext) - 1] if old_ext and rel.endswith(f".{old_ext}") else rel
return f"{base}.{ext}"
def upload_ext(filename: str) -> str | None:
"""The stored `container` for an uploaded file: its shell-safe lowercased extension, or None when
the upload has no extension (a bare name). Shares `_clean_upload_ext` with `upload_rel_path` so the
@@ -227,32 +294,39 @@ def served_media_type(media_kind: str | None, container: str | None) -> str | No
def download_filename(display_name: str, container: str | None, path: Path) -> str:
"""The Content-Disposition filename for a served download: the cleaned display name carrying a single
correct extension — the asset container, else the file's own suffix — without doubling it when the
name already ends in that extension. Shared by the private download and the public watch endpoints."""
name already ends in that extension. A file with no extension at all is served without one —
never as `name.` (Windows drops a trailing dot, so the saved file would not be the one named).
Shared by the private download and the public watch endpoints."""
ext = container or path.suffix.lstrip(".")
base = display_filename(display_name)
if not ext:
return base
if base.lower().endswith(f".{ext.lower()}"):
base = base[: -(len(ext) + 1)]
return f"{base}.{ext}"
def abs_path(download_root: str, rel: str) -> Path:
return Path(download_root) / rel
def abs_path(files_root: str, rel: str) -> Path:
return Path(files_root) / rel
def safe_abs_path(download_root: str, rel: str) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside DOWNLOAD_ROOT and exists on
disk — otherwise None. Centralizes the path-traversal + existence guard every file-serving endpoint
shares, so the containment check lives in exactly one place."""
root = Path(download_root).resolve()
path = abs_path(download_root, rel).resolve()
def safe_abs_path(files_root: str, rel: str | None) -> Path | None:
"""The resolved absolute path for `rel`, but ONLY if it stays inside FILES_ROOT and exists on
disk — otherwise None (no `rel` at all included: an asset without a file has nothing to serve).
Centralizes the path-traversal + existence guard every file-serving endpoint shares, so the
containment check lives in exactly one place."""
if not rel:
return None
root = Path(files_root).resolve()
path = abs_path(files_root, rel).resolve()
if root not in path.parents or not path.exists():
return None
return path
def place_file(staging_file: Path, download_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under DOWNLOAD_ROOT."""
dest = abs_path(download_root, rel)
def place_file(staging_file: Path, files_root: str, rel: str) -> None:
"""Move a completed staging file into its final location under FILES_ROOT."""
dest = abs_path(files_root, rel)
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(staging_file), str(dest))
@@ -286,32 +360,47 @@ def _sidecar(media: Path, ext: str) -> Path:
return media.with_suffix(ext)
def write_sidecars(download_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
def write_sidecars(files_root: str, rel: str, meta: MediaMeta, thumb_src: Path | None) -> bool:
"""Write `<basename>.nfo` + `<basename>.jpg` next to the media file. Returns True if written."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
try:
_sidecar(media, ".nfo").write_text(_nfo_xml(meta), encoding="utf-8")
if thumb_src and thumb_src.exists():
shutil.copyfile(str(thumb_src), str(_sidecar(media, ".jpg")))
return True
except OSError:
except OSError as exc:
# The download itself still completes, so the job reads as done while the library it was
# downloaded FOR has no .nfo to read and no poster to show.
log.warning(
"the sidecars beside a download could not be written",
extra=logs.event("download.sidecars.write_failed", reason=type(exc).__name__),
)
return False
def delete_asset_files(download_root: str, rel: str) -> None:
def delete_asset_files(files_root: str, rel: str) -> None:
"""Remove the media file and its sidecars; prune now-empty parent dirs (best effort)."""
media = abs_path(download_root, rel)
media = abs_path(files_root, rel)
for p in (media, _sidecar(media, ".nfo"), _sidecar(media, ".jpg")):
try:
p.unlink()
except OSError:
pass
except FileNotFoundError:
pass # silent-ok: already gone is the outcome this function wanted
except OSError as exc:
# The row goes and the bytes stay. Nothing else ever revisits this path, so the disk
# quietly keeps a file no account has any way to see, let alone delete.
log.info(
"a deleted download left a file behind",
extra=logs.event("download.asset.file_unlink_failed", reason=type(exc).__name__),
)
# Prune empty Season/Channel dirs up to (but not including) the root.
root = Path(download_root).resolve()
root = Path(files_root).resolve()
parent = media.parent
while parent != root and root in parent.parents:
try:
parent.rmdir() # only succeeds if empty
# silent-ok: this `rmdir` failing is the loop's stop condition — the parent still
# holds another download's files, so the prune has gone as far up as it may.
except OSError:
break
parent = parent.parent
+89 -24
View File
@@ -1,15 +1,24 @@
"""Local file ingest for the download center (E7 S5) — the non-yt-dlp path into the media library.
"""Local file ingest for the download center — the non-yt-dlp path into the media library.
A user-uploaded file joins the SAME library as a downloaded one, so it inherits the whole function
set already built (metadata chips, share links, and — for media — in-app play/edit). Unlike a
download it bypasses yt-dlp AND the worker entirely: the file is already on the host, so ingest
probes it, places it on disk, and creates an already-`ready` MediaAsset + `done` DownloadJob in one
step — the same "the file exists, mark it done now" shortcut `service.enqueue` takes on a cache hit.
download it bypasses yt-dlp: the file is already on the host, so ingest probes it and places it on
disk. A file that already is the phone-safe shape (`playable`) — or is not media at all — becomes an
already-`ready` MediaAsset + `done` DownloadJob in one step, the same "the file exists, mark it done
now" shortcut `service.enqueue` takes on a cache hit.
Media that is NOT phone-safe (an iPhone's HEVC .mov, a VP9 .webm, an .mkv) is placed as it came and
queued: the asset stays `pending` and the job `queued`, and the download worker converts it
(`pipeline._convert_upload`) — so a share link never hands a friend a file their phone cannot play.
Only the converted file is kept (the owner's choice: one file per upload). A 10-bit HDR original
becomes 8-bit H.264 with its colour tags kept — still tagged HDR (HLG/PQ, BT.2020), not tone-mapped
to SDR (the owner's choice too). Until then the original sits at the asset's `rel_path`, which is
what a failed conversion's Retry starts from again, and what a delete removes
(`service.release_asset`).
Kinds (the invariant lives in `mediakind`): a probe with a real video stream -> video, audio-only
-> audio, neither -> a generic 'file' (pdf/zip/docx/…) that is download-only and never rendered
inline (see the force-download serving in the file/watch routes). This module is the ingest seam a
future sub-function (e.g. convert) plugs a new adapter into, mirroring how it sidesteps the worker.
inline (see the force-download serving in the file/watch routes).
An upload does NOT get an `expires_at`: it's user-provided and may be irreplaceable (there's no
source to re-download), so it doesn't age out on the retention clock the way a re-fetchable download
@@ -22,10 +31,14 @@ from pathlib import Path
from sqlalchemy.orm import Session
from app import audit
from app.audit import AuditAction
from app.config import settings
from app.downloads import edit as editmod
from app.downloads import mediakind, probe, quota, service, storage
from app.models import DownloadJob, MediaAsset
from app.downloads import folders, mediakind, playable, probe, quota, service, storage
from app.downloads.status import JobKind, JobStatus
from app.models import display_text
from app.downloads.models import DownloadJob, MediaAsset
def _title_from_filename(filename: str) -> str:
@@ -43,16 +56,35 @@ def ingest_upload(
src_path: Path,
size: int,
display_name: str | None = None,
sha256: str | None = None,
folder_id: int | None = None,
via: str | None = None,
) -> DownloadJob:
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return the `done`
job. `src_path` is MOVED into place on success (the caller stages it to a temp file first).
"""Ingest an already-uploaded file at `src_path` into `user_id`'s library and return its job —
`done`, or `queued` for the worker's phone-safe conversion (see the module docstring). `src_path`
is MOVED into place on success (the caller stages it to a temp file first). `sha256` is the
hash of those bytes, when the caller took it on the way in; `folder_id` the folder it asked for
(`folders.upload_target`), else its Uploads folder; `via` what the audit row names as the doer
when it was not the user themselves (an API token).
Raises folders.FolderError when that folder is not one an upload may land in (or is gone).
Raises quota.QuotaExceeded (holdings/footprint cap) — checked BEFORE any disk write, so a
rejected upload leaves nothing behind and the caller just cleans up its temp file."""
quota.check_upload(db, user_id, size) # the file's size is known → charge it up front
if folder_id is not None:
# Refused before any work; held only once the job is about to name it (below), not through
# the probe and the poster, which a rename or delete of the folder would wait out.
folders.upload_target(db, user_id, folder_id, hold=False)
vcodec, acodec, width, height, duration = probe.probe_media(src_path)
data = probe.probe_json(src_path)
vcodec, acodec, width, height, duration = probe.facts(data)
kind = mediakind.derive(vcodec, acodec)
# Judged by the extension the file is PLACED under (and the worker re-judges), not the raw one:
# 'clip.mp4~' is stored as .mp4, and judging 'mp4~' queued a conversion with nothing to convert.
ext = storage.upload_ext(filename) or ""
safe = playable.plan(data, ext) if kind != mediakind.FILE else None
convert = safe is not None and safe.needs_change
# An upload never dedups (each file is its own asset): a random ref makes the cache key unique,
# so `format_sig` is a constant marker rather than a format hash.
@@ -64,19 +96,23 @@ def ingest_upload(
db.flush() # need asset.id for the on-disk path
rel = storage.upload_rel_path(user_id, asset.id, filename)
storage.place_file(src_path, settings.download_root, rel)
storage.place_file(src_path, settings.files_root, rel)
# From here the file is on disk under `.uploads/`. If the transaction fails before it commits, the
# asset/job rows roll back but the placed file would be ORPHANED — nothing references it, the
# per-user delete path keys off asset.rel_path (which is gone), and the GC orphan sweep skips the
# dot-prefixed `.uploads/` tree. So roll back the on-disk side too on any failure.
# asset/job rows roll back and nothing names the placed file: the per-user delete path keys off
# asset.rel_path (which is gone), and the GC's `.uploads` sweep reaches it only an hour later. So
# roll back the on-disk side too — unless the commit itself raised after it landed (see below).
asset_id = asset.id
try:
final = storage.abs_path(settings.download_root, rel)
final = storage.abs_path(settings.files_root, rel)
actual_size = final.stat().st_size if final.exists() else size
asset.status = "ready"
asset.status = "pending" if convert else "ready"
asset.rel_path = rel
# The hash names the bytes that arrived: a conversion replaces them, so it is kept only
# when the file stays as it came.
asset.sha256 = None if convert else sha256
asset.size_bytes = actual_size
asset.container = storage.upload_ext(filename)
asset.container = ext or None
asset.vcodec = vcodec
asset.acodec = acodec
asset.width = width
@@ -90,25 +126,44 @@ def ingest_upload(
# A frame poster gives the library row a thumbnail + the share link an og:image. Audio has
# no frame and a generic file no player, so neither gets one. Best-effort (ffmpeg decodes
# server side even for a codec the browser can't play).
asset.poster_path = editmod.ensure_poster(asset, settings.download_root)
asset.poster_path = editmod.ensure_poster(asset, settings.files_root)
job = DownloadJob(
user_id=user_id,
asset_id=asset.id,
job_kind="upload",
job_kind=JobKind.UPLOAD,
source_kind="upload",
source_ref=ref,
profile_snapshot={},
display_name=(display_name or "").strip() or None,
status="done",
progress=100,
display_name=display_text(display_name),
status=JobStatus.QUEUED if convert else JobStatus.DONE,
progress=0 if convert else 100,
queue_pos=service.next_queue_pos(db),
)
db.add(job)
folders.place(db, job)
if folder_id is not None:
job.folder_id = folders.upload_target(db, user_id, folder_id).id
if not convert:
folders.stamp_done(db, job)
service.incr_ref_count(db, asset.id)
db.flush()
audit.record(
db,
user_id,
AuditAction.FILE_UPLOAD,
target_type="file",
target_id=job.id,
summary=via,
)
db.commit()
except Exception:
db.rollback()
if _row_landed(db, asset_id) is not False:
# The commit reached the server and lost its answer, or the row cannot be read: the file
# may be the asset's now, so it is never removed blind — the sweep takes it if no row
# names it.
raise
# The rows are gone; remove the now-unreferenced file so no bytes leak. Clean the poster by its
# DETERMINISTIC path (rel with a .jpg suffix, the same path ensure_poster writes to) rather than
# ensure_poster's return value: it writes the frame directly to that path and returns None on an
@@ -116,7 +171,17 @@ def ingest_upload(
# return would skip that orphan. unlink is missing_ok, so the poster path is a harmless no-op
# when none was generated (audio/file, or a video whose poster never started).
for leaked in (rel, str(Path(rel).with_suffix(".jpg"))):
storage.abs_path(settings.download_root, leaked).unlink(missing_ok=True)
storage.abs_path(settings.files_root, leaked).unlink(missing_ok=True)
raise
db.refresh(job)
return job
def _row_landed(db: Session, asset_id: int) -> bool | None:
"""After the ingest transaction raised: is the asset row there anyway? A commit can reach the
server and lose its answer (the connection drops after COMMIT). None when it cannot be read."""
try:
return db.get(MediaAsset, asset_id) is not None
except Exception: # noqa: BLE001 — "cannot tell" is the answer, the caller acts on it
# silent-ok: the caller re-raises the ingest's own error.
return None

Some files were not shown because too many files have changed in this diff Show More