Compare commits

..
127 Commits
Author SHA1 Message Date
peter 45b5e19fb8 release: 0.118.0 (improvement/e59-small-residuals) 2026-11-08 01:23:33 +01:00
peter e08278fae2 chore(release): 0.118.0 2026-11-08 01:05:31 +01:00
peter 350913b6c2 test: a watch's weight by age, and each contrast test under its reason
`draw.recency` weighs every watch of both affinities: an undated watch
and a future one now have a test (deleting either branch fails it). The
over-the-picture contrast test moves below the form-on-chrome test, so
each sits under the comment that explains it.
2026-11-08 00:49:22 +01:00
peter 470caacfb0 fix(plex): the slow line says both rates in one unit
Each rate took its own unit, so a link under a megabit read "needs 1.4
Mbps, 800 kbps arrives". `fmtBitrates` picks the unit the largest takes
for all of them. The away-window header now says what ends an absence
outside hls.js (the answer) and after the error (the viewer's retry).
2026-11-08 00:49:22 +01:00
peter e8b9bdd7c1 fix(playlists): "+N without length" counts every video the sum left out
The label said "without length" but the count held only live videos, so
a video not fetched yet, or gone before it was, was left out of the sum
and named nowhere. The server's `Video.without_length` and the client's
`isWithoutLength` are now one rule (no length, whatever the live
status), and the field, the key and the helpers are named for what they
count. The comments that still quoted "+N live" say the current copy.
2026-11-08 00:49:21 +01:00
peter 55cbbfb6e4 fix(channels): the Length column shows the sum alone
The "+N without length" line did not fit the 4.5 rem column (88 px of
text in a 56 px cell) and spilled into Tags. The owner wants no such
line there: the cell shows the summed hours, and the channel list no
longer sends a count nothing reads. A playlist's total keeps it.
2026-11-08 00:10:28 +01:00
peter 4d058cba81 fix(plex): an error over the picture is readable on any frame
The error line kept the theme's red, which a light theme makes dark: 1.6:1 on the 75 % panel over a
white frame. The panel is now black 85 % under media-scope, so the error is the media red in both
themes (5.47:1 at worst, white text too); a contract test measures it on the composited panel.
2026-11-07 22:04:22 +01:00
peter 872b583f06 fix(card): a total's left-out videos are "without length", not "live"
The count holds archives and upcoming streams too, which a card names a stream or upcoming.
2026-11-07 21:58:36 +01:00
peter 90931b4750 fix(plex): a playlist that never answers is asked for again; the slow line reads this stream
A playlist timeout was resumed in the same session, but hls.js reloads no manifest on startLoad: no
request, no error, a spinner for good. It is now asked for again like a server away. An absence
also ended only on a fragment, which native HLS never reports, and a viewer trying again after the
error started with the spent window; both now start a fresh one. The slow-connection meter was
reset only when hls.js was built, so a direct answer showed the previous stream's numbers, and a
replaced session's late fragment still counted; its rates now use the stats overlay's units.
2026-11-07 21:58:36 +01:00
peter a6f1a26658 docs(plex): the transcode comments say the CPU-decode route 2026-11-07 21:48:39 +01:00
peter 4ed07388e8 test(feed): an off-catalog video's length, and async readers of the raw column
The YouTube lookup's length had no test, and the guard on who reads the raw duration skipped async
functions and module-level code.
2026-11-07 21:48:38 +01:00
peter bb91d0a22d refactor(discover): both affinities weigh a watch by one rule
The YouTube channel affinity and the Plex one each had the recency decay, the undated-watch rule and
the scale to the strongest; app.draw now holds them. Tests hold that another user's history leans
neither draw, the rail's blind slot, and that a show leans on show recommendations.
2026-11-07 21:48:38 +01:00
peter e7e8d8236e fix(downloads): the egress filter keeps what yt-dlp fetches itself, and shows no phase
f4m, ism, a bare m3u8 (native under hls_prefer_native) and the DASH generator fetch through the
proxy, but were dropped as if ffmpeg dialled them. The filter also took the job's postprocessor
hook, twice, and painted every guarded job "processing" before a byte arrived. Tests now hold an
allowed tunnel, the worker's own gate, the single-URL branch, and that the pipeline and the
backfill turn the gate on while YouTube does not.
2026-11-07 21:48:38 +01:00
peter fc65b8ab83 fix(playlists): a refused push spends none of the push allowance
The cooldown was checked before the list was found pushable and before its sync claim, so Watch later
or a list another sync held used up the push allowance. It now counts only a push that runs. The
docstring that called a YouTube mirror unpushable is corrected: its local edits sync back.
2026-11-07 21:48:38 +01:00
peter 26ed8855c9 fix(feed): an out-of-range seed or cursor is refused, not a 500
A seed past a float overflowed the shuffle's start point, and a cursor holding Infinity or a 400-digit
key overflowed its int/float read: both answered 500. The routes bound the seed (422), the start
point folds it to 32 bits, and the cursor read counts an overflow as a bad cursor (400). A plan test
holds that the walk reads ix_videos_shuffle_key with no sort over a realistic scope.
2026-11-07 21:48:15 +01:00
peter 9778bdf351 fix(plex): a codec the GPU cannot decode is decoded on the CPU and still encoded on the GPU
The prod iGPU has no VC-1 Advanced decode ("No support for codec vc1 profile 3"), so every VC-1
session died on the GPU and fell back to libx264, which needs ~17 s for the three warm-up segments:
a start or a seek took ~19.5 s. VC-1, XviD and DivX now take CPU decode -> hwupload -> h264_vaapi
(three segments in 5.5 s for a 1080p VC-1 remux, measured on prod hardware with this argv).

The transcode_hw facet now says whether the GPU encodes: VC-1/XviD/DivX count, and so does an HDR
film (the OpenCL tonemap runs on the GPU; the facet said software). Migration 0119 rewrites the
stored facet and re-queues 10-bit items for the HDR probe.
2026-11-07 18:39:15 +01:00
peter 2b7620ded9 fix(plex): text over the picture sits on a dark panel
The player's loading, stall and error text was bare white or red on the video,
unreadable over a bright frame. It now sits on the dark panel the subtitle
notice and the track OSD already used, one constant for all three. DESIGN.md
records it and the slow-connection line.
2026-11-07 18:01:15 +01:00
peter c0682aa572 fix(plex): a link too slow for the stream resumes in place, and says so
A request hls.js got no answer to in time (no status) was taken for a server
away: a new session, the same first segment lost again, a restart every half
minute that never ended, since every session answer reset the away window.
Measured on prod: a work network at ~4 Mbit/s against 11-16 MB remux segments.

A timeout now resumes loading in the same session; the away window ends only
when media arrives (or a direct stream answers), and runs out in a 'too slow'
error. A segment gets 30 s to its first byte. After 5 s of stalling on a link
measured slower than the stream, the player says so with both rates.
2026-11-07 15:09:22 +01:00
peter 25aed6635f perf(feed): a shuffle walks a stored random key instead of sorting the whole scope
Every video carries a random number drawn once (videos.shuffle_key, migration 0118, indexed with
id). A seed picks a point on that ring: the feed's Shuffle sort pages along it and wraps, and the
Home surprise reads a 500-video window of it and shuffles that window by the seed, so no video is
favoured for sitting after a wide gap. The md5(id || seed) order read and sorted the whole feed
scope on every call: ~350 ms on prod per surprise, 381-767 ms per Shuffle page on localdev, now
14 ms and 4 ms.
2026-11-07 04:00:00 +01:00
peter 9310ff85bf feat(discover): Surprise me leans on what you watch, and stays a surprise
The Home hero's surprise tier and the Plex Discover rail were uniform draws over the unseen. They
are now weighted draws (`app.draw`, Efraimidis-Spirakis over the same md5 shuffle): a title's
weight is 1 + its affinity. Plex: the genres of the latest watched titles (half-life 180 days, a
show counts once) and the TMDB recommendations stored on them (`plex.affinity`). YouTube: the
channels the user watches (half-life 90 days). One slot of three stays a blind draw, and with no
history the order is exactly the old one. No visible change.

The YouTube draw builds its pool once (a materialized CTE read twice): the scope is ~200k videos on
a prod-sized catalog, and a second ordered pass doubled the cost. Measured on localdev: a reroll
210 -> 292 ms.
2026-11-07 02:41:30 +01:00
peter 5b846136ad fix(catalog): a 24/7 stream archive's years-long length is unknown, not summed or sorted by
YouTube reports a 24/7 stream's archive with the years it has run (prod: 122,962,914 s), and one
such video made the Watch later total read 44,073 hours. A length past a day is now no length
(`Video.length_seconds`, LENGTH_MAX_S): the API sends none, the playlist and channel totals
leave it out and count it as `live_without_length` ("+N live" beside the sum), and the duration
filter, sort and Home budget never match it. Migration 0117 moves the duration sort indexes onto
the length expression. An AST test holds who may still read the raw column.

The migration 0110 test now ends its read before the way down: 0117 builds an index
CONCURRENTLY there, which waited for that open transaction forever.
2026-11-07 02:20:03 +01:00
peter 1bf3d6a592 fix(downloads): a download that hangs up mid-transfer is not an error at the egress gate
Seen live: yt-dlp closes a connection once it has read what it needed, and the
gate printed a traceback for each. A reset ends the tunnel quietly; anything
else the gate cannot finish is a log line with its code, not bare stderr.
2026-11-06 05:23:02 +01:00
peter 4e4fff87d4 fix(downloads): a user's download is judged at every connection it makes, not only at the URL typed
The URL guard judged the address the user typed; a public host's 3xx into our
network, a page's formats pointing inside, and a name rebound between the check
and the fetch all got past it, and the stored error told a live internal port
from a dead one. Every non-YouTube download now runs through an egress gate on
the worker's loopback: it resolves each host once, refuses any forbidden answer
before dialling, and connects to the address it vetted. Measured on the worker
image: yt-dlp's proxy reaches the Requests and CurlCFFI handlers, native HLS and
ffmpeg alike. Formats fetched over anything but http(s), which ffmpeg would dial
directly, are dropped, and one factory opens every yt-dlp so none skips it.
2026-11-06 04:54:56 +01:00
peter 995a493bd3 fix(playlists): the push side spends its own allowance, so a scripted loop cannot drain the shared YouTube budget
push-plan reads the live list and push reads and writes it, both against the
instance-wide budget, and neither had a limit. They now share one per-user
allowance of 20 calls per 10 minutes, far above a person's plan-then-push,
refused as playlist/push_cooldown with a sentence in both languages.
2026-11-06 04:11:36 +01:00
peter 5a93a9a8e7 release: 0.117.0 (improvement/e57-files-and-small-ideas) 2026-11-06 03:09:58 +01:00
peter 911fea3243 release: 0.117.0 2026-11-06 02:57:39 +01:00
peter 4aece8aff7 docs(design): log the Hungarian unlock button's noun form 2026-11-06 02:46:27 +01:00
peter adb5e94511 fix(watch): the Hungarian unlock button says a noun for a video too, as for a file
"Megnézem" (a first-person verb) and "Megnyitás" (a noun) labelled the same button by link kind;
the app's Hungarian buttons are nouns, so a video's reads "Megtekintés".
2026-11-06 02:46:18 +01:00
peter 673decf4d4 fix(channels): a pick in the +N fold keeps the focus on its chip after the fold closes
The closing popover hands the focus back to "+N" in a microtask it queues as it unmounts, after the
chip was focused (seen in Chrome; jsdom does not show it). A second focus, queued after it, wins.
2026-11-06 02:32:49 +01:00
peter 4d1cf66d96 fix: tag counts, chip focus, upload naming, share links and unsized clips hold after round 2
- A tag counts the followed channels the Subscriptions tab shows: hidden ones are left out, and a
  subscribe, unsubscribe, hide or tag toggle refreshes the count. The Tag manager's row and its
  hover list count the same channels as the chip; /api/tags sends both counts under their names.
- A chip's count keeps the chip's colour after a dot (a dimmed count fell under 4.5:1). A pick in
  the open "+N" fold closes it; a chip that leaves the list hands the focus to the one in its place.
  The fold is a dialog of toggle buttons, not an empty menu.
- Upload: a folder name left by a press is made once the press is over, if the dialog is still
  open, instead of guessing a dismissal from Modal's markup. While the folder is made, its place
  holds the focus and a live region says so.
- Share: an expired link has no Copy and a warn badge; the link made on open goes into the cached
  list, so a reopen makes no second one; the New link form survives the link made on open.
- Watch: a repeated Enter sends one unlock while one is on its way. See all buttons are named with
  their rail.
- An unsized clip of a rotated file gets the upright size its crop cut.
2026-11-06 02:20:52 +01:00
peter 2baad2ebae fix(files): a folder made from the upload dialog gives the keyboard back the target once it is made 2026-11-06 00:23:09 +01:00
peter 8383459f70 fix: share, upload, crop, auto-skip, tag chips and See all hold what they claim
- Share: a link that expired is never copied (the dialog and one-click Share pick a live one,
  else make one), says it expired, and the copy-on-open claim clears on the next copy.
- A password-protected file link asks to open a file, not to watch a video.
- Upload: Cancel, the close button or the backdrop while naming a folder makes no folder; a failed
  refetch keeps the target; a folder being made says so.
- Crop: an infinite number is dropped, not a 500; an unprobed clip's size follows a shown-size crop;
  the edit_spec docs name frame_w/frame_h; the join and rotated paths are tested.
- Auto-skip: the wait gives way on a marker too short for it, so the skip still fires.
- Tag chips count the followed channels (subscribed_count) and a toggled chip keeps the focus.
- Continue watching's See all sorts by last watched when in progress was already on.
2026-11-06 00:09:39 +01:00
peter c34f6f3297 fix(channels): picked tag chips lead the row, so the +N fold never hides an active filter 2026-11-05 22:34:12 +01:00
peter e4b6d918e4 fix(plex,share): Recently added's See all opens newest first, and a share link says open
See all on Recently added opened Browse in whatever sort was last used, by title on the account it
was seen on. The share dialog said anyone can watch a link that may hold a zip.
2026-11-04 19:43:02 +01:00
peter bf5cf011ce feat: tag chips count their channels, Plex shows a file's size, a skip waits, Continue watching opens whole
The channel manager's tag chips carry a muted count. A Plex item's Info page names the file's size.
An auto-skip's countdown starts 3 s into the marker, with its button there from the start and a delay of
0 still immediate. Continue watching has a See all, opening Browse on in progress, last watched first;
Recently added's sets the watch state back so it opens what it says.
2026-11-04 19:27:46 +01:00
peter 6e6e47652a feat(files): the upload dialog says where the file goes, and sharing copies on open
The upload dialog shows its target folder and can change it, with New folder in the same list; a
file never goes while a folder is being named or made. The share dialog puts the link on the clipboard
when it opens, says so only if the browser let it, and has Open in new tab beside Copy. A file's
format joins the line under its name.
2026-11-04 19:16:20 +01:00
peter 4c09a22add feat(share): a link's address says what it opens
A video stays on /watch/, anything else is given out on /file/ and a folder on /folder/. The token
is the whole key, so every path opens every link and the links already sent keep working. The new
paths are kept out of search indexes like /watch/, boot the player alone, and the unfurl card calls a
file a file.
2026-11-04 19:10:09 +01:00
peter 2bdf1ec76f fix(editor): a crop of a video with wide pixels keeps what its frame showed
The browser reports such a video by its shown size and ffmpeg's crop counts stored pixels, so a
frame over the middle half of a 320 stored, 640 shown video kept the whole width. A crop now carries
the size it was measured on and is scaled to the file's own pixels. A spec from before it did is cut
as it always was. Also drops a stale note in downloads/quota.py: edit jobs are held to the cap.
2026-11-04 19:01:49 +01:00
peter f7e036abef merge: e2e page loads retried on a machine out of buffers (improvement/e2e-goto-retry) 2026-11-04 00:53:23 +01:00
peter f2de8f73aa test(e2e): a page load the machine had no buffer for is tried again
net::ERR_NO_BUFFER_SPACE stopped a release at its first page with nothing else running, and the
unchanged rerun passed. Every page load of the suite now goes through one helper, held by a test.
2026-11-04 00:26:50 +01:00
peter 643f305814 release: 0.116.3 (bug/lost-stylesheet) 2026-11-04 00:19:42 +01:00
peter cbe1b3bd63 chore(release): 0.116.3 2026-11-04 00:05:30 +01:00
peter ae775b383c fix(boot): a stylesheet lost on the way is asked for again
A lost stylesheet left the page unstyled until a reload, and every e2e step that finds elements by
testid still passed on it. Only axe noticed: the one-off target-size failure on eight scattered
controls. Losing the main stylesheet on purpose gives those same eight.
2026-11-03 23:32:40 +01:00
peter 358d5af70b release: 0.116.2 (bug/e56-hls-reap) 2026-11-03 21:51:59 +01:00
peter c6831090b5 chore(release): 0.116.2 2026-11-03 21:35:19 +01:00
peter 38fc95d772 docs(plex): a quiet reclaim's replay is named as accepted 2026-11-03 21:22:17 +01:00
peter 3f6638bf96 fix(plex): a seek during a quiet reclaim is no longer undone
A quiet request leaves `live` alone, so a seek inside the old session's
range went native and the answer then jumped back to where it asked from.
`seek` now also goes by `pending`; the server-away retry had the same hole.

Comments that still named 404 for a lost session say 410, and the hls.js
retry comment now tells a finished playlist from a growing one.
2026-11-03 21:10:51 +01:00
peter 469c8aa481 test(e2e): the drag onto the rail waits for the rail's lists 2026-11-03 20:31:41 +01:00
peter df2997e9f3 fix(plex): a lost HLS session is reclaimed at the first answer
The route answers 410 for a session it no longer has and keeps 404 for a segment not there yet.
hls.js retries even a 4xx on a stream with one video variant, so the player waited about half a
minute before asking again, and minutes in a throttled page.
2026-11-03 19:20:22 +01:00
peter f69725503f fix(plex): a finished HLS session outlives a throttled heartbeat 2026-11-03 19:15:18 +01:00
peter d5bc7e8031 feat(plex): every HLS session reap logs its rule and idle time 2026-11-03 18:35:33 +01:00
peter efdffc82ae release: 0.116.1 (bug/e53-order-from-server) 2026-11-03 03:27:54 +01:00
peter e23544a46b chore(release): 0.116.1 2026-11-03 03:14:48 +01:00
peter 035ee937a0 fix(playlists): the list shows the order the server holds
A read with the same rows in another order never reached the screen: a
revert to YouTube, a sync or another device left the old order showing, and
the next write put it back on the server. With no write out on any list the
screen now takes it, and the undo history built on the old order goes. Own
reorders still keep theirs. This replaces the one flag all lists shared after
a rollback, and shows a reorder accepted after its list was picked again.

Also what the fourth review found: a list's reorders go one after the other,
a toast Undo asks for its place once the writes out have landed, an accepted
reorder keeps fresher rows from the cache, and a Play all disabled by the last
row going no longer strands the focus.
2026-11-03 03:01:17 +01:00
peter bbb4063362 release: 0.116.0 (improvement/e53-playlist-multiselect) 2026-11-03 01:02:33 +01:00
peter b4ba9d95b1 chore(release): 0.116.0 2026-11-03 00:48:36 +01:00
peter c18a2063c8 fix(playlists): a reorder counts among the writes, and what the third review found
A reorder was invisible to the overlap bookkeeping: a failed removal reset the
screen from a cache without it, and the membership guard never corrects an order.
It now counts as a write, a rollback leaves the order to the read after the last
write, and the accepted order keeps the rows it does not name, as the server does.
Undo places come from the order on screen. A stored position landing during a write
reaches the screen, and what a read brought is taken once the refresh is in. A write
that went through drops its rows from the cache again, past a read from before it.

Closing the player after Trash took the row it opened from no longer strands the
focus: the row in that place takes it, else the header's box, else the list's menu.
The page menu's grouping row closes its menu when it goes with the view. One hidden
row gets its own wording. /search/clear refuses a body without its list.
2026-11-02 23:43:34 +01:00
peter b636ef38bc fix(playlists): overlapping writes, and what the second review found
A read from before a write no longer brings a row back: a write calls off the
read on its way, what lands while one is out stays off the screen, and the last
write out does the refresh. A failure puts back its own rows, an accepted reorder
brings back none, and an Undo reads past the cache. A list the cache has dropped
is read before a removal, so its Undo has places.

undo-move refuses the same list twice, positions cannot repeat, and undo-move
and the Plex key bodies refuse a body without their videos. Focus goes to the
header's box when the last shown row or Save this order goes.
2026-11-02 22:15:13 +01:00
peter 195b5f3917 test(playlists): walk to the row with a key, as the keyboard does 2026-11-02 04:01:56 +01:00
peter 1a1e9d9542 fix(playlists): what the first review of bulk actions found
An Esc during a pointer drag is spent on the drag and leaves the selection.
A wheel over the rail re-reads the target each frame, so a drop lands on the
row under the pointer. After a bulk removal the focus goes to the next row
left, and to the list's menu when the list is emptied.

remove-bulk names the videos that went, and its Undo brings back only those.
An Undo of a move keeps each row (and when it was saved), leaves out a video
that moved on from the target, closes up its place, and says how many came
back. Delete with every tick hidden removes nothing and says why. A list gone
before the lock answers 404, and a body without its videos 422.

One removal path now: the single row goes through the bulk one, the same
toast and undo, which also retires the add-at-position route. railDrop reuses
the store and scrollParent, the drag image the overlay and the z-index token,
and the sensors keep their number. The count is read out from a region that
outlives the band. Tests fill the gaps the review named, each seen failing.
2026-11-02 03:41:08 +01:00
peter d09b5d6abb fix(dev): vite stops watching what an e2e run writes
On Windows a report file being rewritten answered the watcher with EBUSY, which
took the dev server down, and each run reloaded every open page.
2026-11-02 01:17:39 +01:00
peter 439f5cd67b test(playlists): a person's pause before the arrow in the keyboard drag 2026-11-02 00:57:58 +01:00
peter d79cb528fd feat(playlists): drag rows onto a list in the rail
A row's grip carries it, or every selected row with a selected one, onto another
list, under a sort or a filter too, and the toast undoes. The rail and the page
are separate React trees, so the page reads the rail by what lies under the
pointer. An unpinned rail peeks by the drag's own hover and scrolls at its edges.
Many rows dropped inside the list stay; a key still moves its own row.
2026-11-02 00:38:39 +01:00
peter 9b76d2f96c fix(playlists): the list's name keeps its room on a narrow window
Below xl Unsynced is its dot; on a phone the cover is gone, Play all is its
glyph, the band's count is its number and the undo pair stays out. The name was
0 px wide in Hungarian at 768 and on any phone, and a tick moved the list there.
2026-11-01 23:07:42 +01:00
peter bf54a3ce5f fix(playlists): the selection band keeps to one row on a tablet, and a tick moves no row
Below xl the band's buttons are glyphs named in their tooltips, and the row it
replaces stays in the same grid cell, invisible, so the first tick leaves the
list where the second tap expects it. The test's row menu asked for the pins
unmocked, and the offline toast could land before the check: mocked.
2026-11-01 22:44:32 +01:00
peter efbbb23525 feat(playlists): act on the selected rows from the toolbar, and undo either
While rows are selected the toolbar's second row is the band: the count,
the last target as one click, Move to, Remove, Clear selection. Both write
in one request and undo back to the places; a video the target held before
stays there. Delete takes the selection when there is one. A failure puts
the rows back ticked.

A row's own move goes through the same write, so it undoes too, and the
single-row move route and service function are gone. A playlist row drops
its grip below a 24rem table: the row menu keeps the moves.
2026-11-01 22:07:09 +01:00
peter 3b85881af4 docs(design): the selecting row, formatted 2026-11-01 20:55:28 +01:00
peter 8cf8c4139c feat(playlists): select rows with a box, Shift for a range, x and Esc
A box first in every row and one over the column for the rows shown, the
selected row tinted. A filter keeps what it hides selected; another list
starts over; a row that left the list drops out. Watch later has none.

A box no longer counts as a typing target: a click on one left the focus
there and the page's keys went dead.
2026-11-01 20:49:59 +01:00
peter cd773e8f33 feat(playlists): remove and move many videos at once, and undo either, in one transaction 2026-11-01 20:22:34 +01:00
peter 242f00671d release: 0.115.0 (improvement/e50-queue-skips) 2026-11-01 19:13:09 +01:00
peter 7f8d7cbe20 chore(release): 0.115.0 2026-11-01 19:01:45 +01:00
peter 8dd2f0c015 fix: review round 2 — Clear watch progress clears only the resume point, big view focus by the way it arrived, dialog name, reason tooltip, backfill locks 2026-11-01 18:26:33 +01:00
peter b3402d158f fix(player): the card under the big view is out of the Tab order, and the focus leaves it and comes back 2026-11-01 17:12:05 +01:00
peter c910cf2f20 fix: review round 1 — row press after a rename, related reset, dead items on removal, honest Prev/Next
- A press that begins while a field in the row is edited only ends the editing (rowOpen latches it
  at pointerdown); the card layout's row press is now tested.
- A related video reset in the player forgets the session's stored position.
- Removing the playing video passes over known-dead items and steps the way it went.
- Prev/Next off only because of dead items stay focusable and say why (HU/EN).
- The docked bar's scroller leaves room for focus rings.
- A focus-opened About card is not closed by its own scroll into view.
- EN "Clear watch progress" for the video reset, distinct from a page's "Reset view".
- The signed-out title read goes to the page yt-dlp read; the retitle writes are conditional.
- Tests: the cap of three errors, the bar's own Fullscreen, the group glyph, the backfill's reads.
2026-11-01 05:36:59 +01:00
peter 245c0a2019 test(e2e): the channel chrome's fixed-toolbar check holds by the watch-state pill
A channel page no longer shows a count pill; the toolbar's watch-state pill
rides in the same fixed chrome.
2026-11-01 04:44:37 +01:00
peter add7f7d167 feat(channel): on a tablet the header's words give their room to the tags
Below xl the action row drops "Prio" (the stepper's tooltip names it) and
the Subscribed button's word (a check glyph; its name stays in the
aria-label). On an iPad at 820 px with touch-sized targets the tag lane
grows from 29 px, a sliver of its control, to 130 px.
2026-11-01 04:22:20 +01:00
peter a29afd916d feat(feed): toolbar pills named by a glyph, Sort and View on the filters' row
The watch-state, content-type and sort pills on the feed, a channel page and
Plex Browse show a glyph where their group's word was; the word moves to the
tooltip and stays in the accessible name. Sort + View end the toolbar's row
instead of a row of their own, the chip row shows only with a chip, and a
channel page drops the count pill its header already carries.

Also logs the docked bar's glyph-only decision in DESIGN.md.
2026-11-01 04:18:01 +01:00
peter e9ce989998 feat(player): the big view's bar is glyphs only, and its way out stays in sight
On an iPad the bar's labelled row outgrew the screen and its last control,
leaving the big view, scrolled off the right. Every control is its glyph
now, named by its label and tooltip as in the windowed player; the controls
scroll on a narrow window while fullscreen and leave stay pinned right.
2026-11-01 03:59:17 +01:00
peter 7299e82c22 fix(downloads): a Facebook download titled only "Facebook" takes the post's text
Since 2026-10-28 a signed-in read of a reel gives the placeholder title and
no description, so the adapter had nothing to fall back on and every such
download (and its name) read "Facebook". A placeholder with nothing behind
it now gets one signed-out, metadata-only read of the page. The backfill
retitles the downloads already made, and every name that only repeated the
placeholder; a name its owner gave stays.
2026-11-01 03:52:16 +01:00
peter 41052eb4ea feat(files): a press anywhere on a file's row plays it
A table row (and its narrow card) can carry its object's default action:
a click or tap on the row runs it, a press on a control inside the row,
the click that ends a text selection and a drag do not.
2026-11-01 03:30:49 +01:00
peter 85fd9b6704 feat(channels): an Added column, and About cards that wait for a resting pointer
The channel manager lists when each follow reached the list (the
subscription import that first saw it, or a subscribe made here) as a
sortable Added date after Last upload. The About cell's card opens after the
pointer rests on it for half a second instead of at once, and a scroll puts
an open one away.
2026-11-01 01:30:30 +01:00
peter e06c0688d9 feat(video): Reset view on every YouTube video menu
The row that sets a video back to fresh (its state row and resume point both
go) was only on the feed's cards. It is now on Home's shelves, a Playlists
row's menu and the player's related list too, and is named "Reset view" /
"Megtekintés visszaállítása".
2026-11-01 01:20:20 +01:00
peter 15a3603513 fix(player): a step onto an unplayable video carries on the way it went
Previous onto an embed-blocked or removed video skipped forward again (the
auto-advance direction), straight back to the item just left. The player now
remembers the direction of the step that reached the item and the videos
YouTube refused this session: an error carries on that way, Previous / Next
pass over known-dead items without loading them, and go dark when only those
are left. A step of your own skips with auto-advance off too.
2026-11-01 01:14:50 +01:00
peter c430558ab3 release: 0.114.3 (improvement/e49-refusal-contract) 2026-11-01 00:30:51 +01:00
peter be38d652a5 chore(release): 0.114.3 notes 2026-11-01 00:18:59 +01:00
peter 31a75f4985 fix(i18n): write each refusal where it is sent, and read only that form
A refusal's detail now stands where it is sent (HTTPException's detail,
a JSONResponse's "detail") or is returned by a factory that only ever is,
with any spread before its code and reason, so no spread can replace
them. The scan follows no variable and no kwargs any more; what it cannot
read in a reason (a lambda's or comprehension's name, a nonlocal or
global, a local that reaches itself, a constant or map changed elsewhere,
an app import it cannot resolve) is reported. The detail guard reads the
same slots and fails on anything but a code dict or a factory's call.
2026-10-31 23:54:55 +01:00
peter f8a8e905da fix(i18n): read refusals along a closed list of shapes, report the rest
The scan followed many shapes and called the rest unread, but each shape it learned opened a silent
branch: a parameter rebound in the body, a second visit of a followed helper, an `import app.x`, a name
bound twice or under an `if`, a spread or a store that replaces the code, a detail rebuilt from its
variable, a None reason. It now reads only what the app writes and reports everything else: a
branch the app never takes is gone (an `or`, a default, a base class's attribute, string and union
annotations, a bare module constant), a spread is clean only from an untouched `**kwargs` no call puts
a `code` in, and the detail's variable may only grow by `|=` or `["reason"]`. The real tree reads
exactly as before; every exit and every kept branch has a self-test that a mutant of it fails.

The detail test also fails a dict without a `code`, a JSONResponse sentence and a detail-less
HTTPException; the two that are no refusal (the setup gate's 503, an unmatched api path's 404) are
listed with why. The contract's assertion that compared the reason map with itself is gone.
2026-10-31 18:45:02 +01:00
peter 5e8589bbf8 fix(i18n): read every refusal from the backend's syntax tree, not by regex
The contract read Python with regexes, and each guard added for a missed shape had shapes of its own
it missed: a reason after another key, a status passed by name, a second `e.reason` handler with the
same text, a pin naming a type nothing checked.

backend/tests/refusal_sites.py (stdlib `ast`, run by the contract on the host's Python) reads every
`{"code": ...}` detail and follows each reason to its values: literals, both arms of a conditional,
module constants, the `Literal` on the parameter or attribute it travels in (on the class an `except`
catches), every call of a `str` helper with its defaults, a map's values with its keys held to the
index's type. What it cannot follow fails the contract, unless no screen shows that code; no pins.

The reason attributes are typed with their Literal, so a subclass or an assignment cannot widen
them; InvalidSetting takes a SettingReason, and the settings map is held to it (a reason the map
lacked was a 500). The per-family reverse checks are one check by prefix (the admin actions
included), and the leaf-key filter applies only to the suffixed sign-in family.
2026-10-31 15:07:38 +01:00
peter ceadf69e43 fix(i18n): hold every refusal reason by type, and refuse what the scan cannot read
Review round 1 of the refusal contract: it claimed more than it held.

- The other reason carriers get their Literal too (BlockedUrl, ClipError,
  FolderError, the playlist and sign-in refusal helpers), so the type
  checker refuses a reason passed by variable or an unknown literal, and
  the contract reads one shape.
- A literal reason in a detail dict built anywhere counts as a reason of
  its code; a code the scan cannot read, a refused() whose reason it
  cannot read, and a new reason passed by variable each fail the contract
  until listed with what holds them.
- A sign-in refusal key the backend no longer sends is caught too.
- localizeDetail's numbers (download cap in GB, upload cap in bytes) and
  the shared-sentence fallbacks are tested in both languages.
- One Literal parser in the contract; a stale comment fixed.
2026-10-31 01:00:26 +01:00
peter 4379f76747 feat(i18n): every server refusal is said by a record the contract holds
A refusal code or reason with no sentence fell through to the generic
"couldn't carry out that action" unnoticed: the parity test is green when
EN and HU miss the same key. One did (an edit refused for a file that is
not video or audio).

- lib/api/refusalCodes.ts: which sentence answers each code, as data;
  localizeDetail says a refusal through it, unchanged for every reader.
- backendIds contract: every detail code the backend sends is in exactly
  one record, every reason it can carry has its sentence in EN and HU,
  and no family names a reason the backend cannot send.
- The reasons that travel in a variable are Literal types (edit, move,
  cookies, quota), so the type checker refuses an unknown one too.
- downloads.errors.edit.not_media in EN and HU.
2026-10-30 23:36:25 +01:00
peter b3be568e36 release: 0.114.2 (bug/joserfc-cve) 2026-10-30 22:39:07 +01:00
peter a8cbc7b338 chore(release): 0.114.2 2026-10-30 22:28:39 +01:00
peter cf8dc80428 fix(deps): joserfc 1.7.4 -> 1.7.6 for CVE-2026-119283 and CVE-2026-119284
One pin bumped in place (Authlib's dependency); pip check passes and pip-audit finds nothing.
2026-10-30 22:28:39 +01:00
peter da74b8b9bf release: 0.114.1 (bug/notice-language) 2026-10-30 03:10:42 +01:00
peter 6944622085 chore(release): 0.114.1 2026-10-30 02:58:51 +01:00
peter 2bdcebdb34 fix(notifications): a confirm names its user and channel again; t() and a notice's text are kept apart by a contract
The value sweep wrapped two plain t() values in raw(), so the delete-user and the YouTube subscribe
confirms read "[object Object]" where the email or the channel was. tsc cannot see that seam, so a
syntax contract now refuses a notice value handed to t() and an already-said text kept in a notice.
The raw() doc says what it is: text kept as fetched (a Plex title in the language it was fetched in).
seasonTitleText's no-number branches are tested.
2026-10-30 01:54:00 +01:00
peter 046dde44a4 test(notifications): the center re-says its rows after a switch, a repeat coalesces by content; e2e helper fails with a cause
The center's rows (title, message, refusal, live button) were never rendered across a language
switch, and nothing held the structural compare that coalescing now needs. The channel-link helper
swallowed whatever ended its poll into a bare false: a link that goes away mid-look now counts as
not hit, and no hittable link fails with the poll's own message.
2026-10-30 00:17:46 +01:00
peter 74f58ea001 fix(notifications): the English log formats its numbers in English; the ui/ contract sees msg() keys
A formatted value took the reader's language even when the notice was said in English for the
log. The platform-namespace contract matched only t("…"), so a domain key passed through msg()
in components/ui/ went unseen. Three comments pointed at localizeDetail's old home.
2026-10-30 00:15:22 +01:00
peter b515c16117 fix(notifications): a notice's button is said when shown, like its text
An action label was translated when the notice was raised, so after a language switch a live
toast or center row read in one language with its button in the other. The access-requests nudge
no longer re-issues on a switch for it. New tests: the button and a server refusal follow a
switch.
2026-10-30 00:10:38 +01:00
peter 4a0d4b0964 fix(notifications): a notice value is no plain string, so a translated name cannot freeze in one language
A name goes in as raw(), a translated word as its own msg(): tsc now lists every value, and the
ones that froze are said when shown — Watch later and an untitled channel in the pin and channel
notices, a Plex season's name in its removal toast (a new seasonTitleText, which seasonTitle says).
2026-10-30 00:03:48 +01:00
peter cb8b99e0e7 fix(notifications): keep the history under a key of its own, so an older build never reads its new shape
A build from before the change renders history rows as text; an object there crashed its
Notifications page after a rollback. The old key is read once while the new one does not exist,
and never written.
2026-10-29 23:54:48 +01:00
peter e5c75702e8 fix(views,tags): a rename onto a taken name names the taken one, not the row's old name
The rollback after the conflict puts the old name back on the row, so the refusal read
"you already have a view named 'hh'" when the taken name was 'gaming'.
2026-10-29 23:07:54 +01:00
peter 4ef772d37b fix(notifications): a notice keeps what to say and says it in the reader's language when shown 2026-10-29 22:37:14 +01:00
peter 6986eb1937 test(e2e): look for a hittable channel link until one is, not once on a fresh load 2026-10-29 22:37:14 +01:00
peter 41ad538edc release: 0.114.0 (feature/e13-notifications-order) 2026-10-29 02:18:56 +01:00
peter 7489cab3bc chore(release): 0.114.0 2026-10-29 02:06:52 +01:00
peter 91256b10b0 test(nav): pin the restore's last look and the clearing of the name it looks for 2026-10-29 01:31:41 +01:00
peter b22e903174 fix(notifications): Earlier opens on Back only for a row that can be nowhere else, and Mark read reads read at once
A namesake in Earlier while the row itself was still in Needs you or Recent opened the fold, which took the place match away and left the name twice on the page: the focus fell to <body>. Row names are trimmed as Back saves them, so a title ending in a space still matches. Mark read handed the focus to a row that still said it was unread until the write landed; the row now reads read in the press's own render, and unread again if the write is refused.
2026-10-29 01:31:40 +01:00
peter d6e72d7576 fix(notifications): Back to a row that folded away since opens Earlier on it 2026-10-29 00:47:52 +01:00
peter e3a4593acf fix(nav): Back looks for a moved item only in the page's lists, once its own list has had its chance, and its reveal stops the scroll restore
The cross-list match took a same-named row of the app-wide download tray, and took it at the first look while a virtualized list had yet to render the item. The reveal of a moved item ran before the scroll restore's first frame, which put the old offset back over it. The pending restore's name is readable, for a page that folds rows away.
2026-10-29 00:47:52 +01:00
peter fa271a31b9 fix(notifications): Mark all read waits for the fresh list, is not offered where it cannot act, and a row says it is unread
The mask over a stale read lifted when the write settled, before the refetch landed, and the band took the rows back. A band holding only what asks by its kind (an access-requests nudge) left the button enabled with nothing to do. The unread dot was only drawn: the spoken label now says it, apart from the name Back finds the row by, and follows the row while it holds the focus.
2026-10-29 00:38:34 +01:00
peter 947efaba6e docs(design): the notification order's decision row carries what it was chosen over and why 2026-10-29 00:38:34 +01:00
peter 22d873ed49 fix(nav): an item Back finds in another list is brought into view 2026-10-28 22:04:07 +01:00
peter d825aff30d fix(notifications): a row is named by what it is about, not its kind alone 2026-10-28 22:04:06 +01:00
peter 9dd56e94d4 docs(design): the notification center's order and its Earlier fold 2026-10-28 21:54:00 +01:00
peter 25ad2ee1ba test(ui): a self-closing dialog leaves the focus in a menu that stays open 2026-10-28 21:54:00 +01:00
peter c86ec7f7a5 fix(nav): Back finds an item that moved to another list by its name 2026-10-28 21:54:00 +01:00
peter 0469812499 fix(notifications): Mark all read empties a band of read rows, and a read landing mid-write cannot refill it 2026-10-28 21:54:00 +01:00
peter ee658b8bfc test(plex): pin the late-checkpoint race, its window, and the bulk mark's push supersede 2026-10-28 21:53:59 +01:00
peter 52b93d0014 fix(plex): a restart that fails to start still deletes the session it moved aside 2026-10-28 21:53:59 +01:00
peter c3ad7684ea test(ui): pin the notifications Earlier fold as a plain Pressable
Its text is the disclosure's own label, not a title to select.
2026-10-28 20:44:01 +01:00
peter a5666bfde9 style(notifications): the Earlier fold reads as a section heading, like its siblings
It was a ghost button in body type under two uppercase section labels. It is now
the heading itself, its text the disclosure's button (24px tall, past the
WCAG 2.5.8 floor).
2026-10-28 20:42:18 +01:00
peter f49dc6532f feat(notifications): what asks for you comes first, then one stream; old read rows fold
The center showed two fixed sections, the server's above this browser's, so an
admin's "access requests waiting" sat under every server row, week-old read
"Download ready" notices among them. One order now, for both sources
(lib/inboxOrder): "Needs you" (unread, an access-requests nudge, a notice
waiting on an interaction), then everything else newest first, and read rows
older than a week fold under "Earlier (n)", closed by default.

A row that asked in this visit keeps its place once read: marking it read no
longer moves it to another list under the reader. Mark all read empties the
band, the client notices of the visit included (the server rows read at once,
not when the write lands). Marking a row read from its button hands the focus
to the row instead of dropping it on <body> as the button goes.
2026-10-28 20:01:21 +01:00
peter 0d410592e2 fix(plex): a stale finish refusal no longer sticks, and a refusal keeps the player's detail
A refusal that landed after a replay raised the refusal flag for the finish that watch made later,
so leaving skipped the watched patch. Each finish now resets the flag and a refusal acts only for
the finish it answers.

A refusal while the player still sat on the item (the up-next card) removed the detail query it
observes, dropping the card and restarting its countdown, or losing it if the refetch failed. It is
now refetched under the player and dropped only when the player leaves.
2026-10-28 19:05:24 +01:00
peter 7e5d4a7d93 fix(ui): a closing dialog's focus restore reaches the dialog left standing above it, and survives a menu
A dialog closing on its own under a standing modal skipped its restore, and the standing modal's
own target was a control inside the closed one: closing it left the focus on <body>. The skipped
target is now handed to the standing dialog, which falls back to it.

Focus on a row of one of the dialog's own menus (the popover engine portals them out of the dialog
root) read as the user walking away, so Back with a menu open left the focus on <body>. A focus in
a popover portal now counts as held, and one still standing is not taken from.
2026-10-28 19:05:24 +01:00
peter 24190e4c16 fix(plex): a checkpoint sent just before the finish no longer lands on the watched row
The 10 s tick in flight when the player finishes (or a refusal rollback whose state write in fact
committed) wrote its mid-item position onto the freshly watched row and scheduled a resume push
after the scrobble, so Plex showed the item in progress again. The checkpoint is now a conditional
write that skips a row marked watched in the last 30 s and answers like a finish; a rewatch past
that is checkpointed as before. Marking an item (or a season/show) watched also supersedes resume
pushes still in flight, as the near-end finish already did.
2026-10-28 18:56:12 +01:00
peter e96a562f14 fix(plex): a segment name ffmpeg never spells is refused; teardown no longer holds the registry lock
The artifact guard parsed the sequence with int(), so a zero-padded or non-ASCII-digit name read
as a segment the session had reached and parked the sync route for the full 20 s wait. It now
also requires the canonical spelling segment_name() produces.

Stopping, replacing, culling or reaping a session waited up to 3 s for ffmpeg and deleted a whole
segmented film under _lock, which every segment request and checkpoint takes. Teardown now signals
the process and moves its directory aside under the lock, and waits/deletes after it. A new session
on the same path still waits for the dying ffmpeg first (it writes by path), and the orphan sweep
recognises a moved-aside directory left by a crash.
2026-10-28 18:48:21 +01:00
387 changed files with 20820 additions and 2857 deletions
+99
View File
@@ -7,6 +7,105 @@ edit the release notes and re-run the generator.
Upgrading, pinning a version and rolling back: [docs/self-hosting.md](docs/self-hosting.md#upgrading).
## 0.118.0 — 2026-11-08
### New
- Surprise me now leans on the channels and genres you watch, and one pick in three stays a blind draw, so it remains a surprise.
- On a connection too slow for a Plex video, playback waits and goes on where it was, and says how fast the video needs and how fast it arrives.
### Fixed
- A 24/7 stream's archive no longer adds years to a list's total: a total names the videos it could not count as "+N without length". The Feed sorted by Surprise me opens faster.
- Plex films the server's GPU cannot decode (VC-1, older DivX or XviD) start in a few seconds instead of about 20. Text over the Plex picture is readable on any frame.
## 0.117.0 — 2026-11-06
### New
- A share link's address now says what it opens: /file/ for a file, /folder/ for a folder, /watch/ for a video. Links you already sent keep working.
- The upload dialog shows which folder the file goes to, and you can pick another one or make a new folder there. Opening Share copies a link at once.
- The Channels tag chips show how many followed channels each holds. Continue watching in Plex has a See all, and a Plex item's Info shows the file size.
### Fixed
- An expired share link is marked as expired and is never copied. A crop of a video with wide pixels keeps exactly what its frame showed.
- A Plex auto-skip waits 3 seconds into the marker before its countdown starts.
## 0.116.3 — 2026-11-04
### Fixed
- If a stylesheet is lost on the way, the app now asks for it again, so a poor connection no longer leaves the page unstyled until a reload.
## 0.116.2 — 2026-11-03
### Fixed
- A Plex video paused or muted in a background tab no longer loses its session after a minute, and if a session is lost anyway, playback comes back at once instead of after half a minute.
## 0.116.1 — 2026-11-03
### Fixed
- A playlist now shows the order the server holds after a revert to YouTube, a sync or a change on another device, and the next drag no longer brings back the old one.
- An Undo on a removed or moved row puts it back in the right place while another change is still saving.
## 0.116.0 — 2026-11-03
### New
- In a playlist, tick rows with their box, Shift+click or x, then move or remove them together. An Undo puts them back at their places, and a single row's move has one too.
- Drag rows onto another list in the rail: one row, or all the selected ones, also under a sort or a filter.
### Fixed
- Removing a video from inside the player no longer loses your place in the list when the player closes.
## 0.115.0 — 2026-11-01
### New
- Clear watch progress is on every YouTube video's menu, on Home, in playlists and in the player's related videos. It forgets where you left a started video and changes nothing else.
- Channels have an Added column, and an About card opens only when the pointer rests on it.
- In Files, a press anywhere on a row plays the file.
- The feed's toolbar pills are named by a glyph, with Sort and View on the filters' row.
### Fixed
- In a play queue, Previous and Next pass over a video that can't play, in the direction you were going, and say why when only such videos are left.
- A Facebook download titled only “Facebook” now takes the post's text as its name.
- The player's big view keeps its way out in sight on a tablet, and Tab no longer lands on controls hidden under it.
## 0.114.3 — 2026-11-01
### Fixed
- Editing a file that is neither video nor audio now says why it can't be done, instead of a generic error.
## 0.114.2 — 2026-10-30
### Fixed
- Security: a library the sign-in relies on is updated, closing two published vulnerabilities.
## 0.114.1 — 2026-10-30
### Fixed
- Notifications follow a language switch: a notice on screen and every row in the notification center are said again in the new language, their buttons too.
## 0.114.0 — 2026-10-29
### New
- Notifications: what asks for you comes first, then everything else as one stream, newest first. Read notices older than a week fold under Earlier, and Back returns to the row you left.
### Fixed
- Plex: a progress save sent just before an episode ends no longer leaves it unwatched, and a refused finish no longer sticks.
- After a dialog closes, the keyboard focus returns to where it was, also inside a menu that stays open.
## 0.113.0 — 2026-10-28
### New
+3 -3
View File
@@ -65,7 +65,7 @@ how it gets built. Concretely:
`routes/*` (HTTP only: validation, auth-dependency, serialization) → domain packages `sync/`, `plex/`,
`downloads/`, `reader/`, `discover/`, `catalog/`, `youtube/`, `issues/` (shared by routes + scheduler + worker) → flat core (`auth`, `apitokens`, `quota`, `jobruns`,
`sysconfig`, `state`, `audit`, `notifications`, `userscope`, `ratelimit`, `bodylimit`, `progress`, `collation`,
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
`security`, `logs`, `logsink`, `diagnostics`, `debugsession`, `draw`) → `models/db/config`. **Dependency direction is one-way: core never imports routes.**
The domain is four **units**, each owning its tables: Capture (`downloads/`), Reader (`reader/`, `discover/`, the playlist
sync), Catalog (`catalog/`, the `sync/` ingests) and Plex (`plex/`). A unit's route does no database work: it reads through
the unit's `queries`, writes through its `service` (which flushes) and commits. A unit's code reads another unit's tables
@@ -74,7 +74,7 @@ only along a declared edge and writes them only through that unit's service; cor
cannot (an ORM write on another unit's loaded row, SQL in `text()`, a commit by `begin()`) is review's to catch, listed in
[`backend/app/README.md`](backend/app/README.md) with the conventions.
Frontend, by folder under `src/`: `main.tsx` provider tree → `shell/` (`App`, header, nav rail, palette,
boot; a shared `/watch` link boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
boot; a share link (`/watch`, `/file`, `/folder`) boots from its own entry, `watch.html` → `watchMain.tsx` → `watch/`, held
app-free by a contract test) → **`modules/<page>/`** (one folder per page: the page, its rail, its store, its tests,
and its `module.ts` definition — label, icon, gate, the lazy page and rail, its store — collected in nav order by
`features/navigation/moduleRegistry`, which also mounts every store at the root; a new page is its folder, its id in
@@ -105,7 +105,7 @@ CQRS, a repository/DI framework, a router library, a generic CRUD-router factory
| Ownership check | `userscope.owned_by` (a filter) / `owned_row` (a row by id): a unit's `queries.own_x` returns None and its route answers `not_found`; a core route `owned_or_404`. A composite rule (own or shared, built-in or own) says so in its `queries` | an inline owner filter — `Model.user_id == …`, an alias's, `.in_()`, `filter_by(user_id=…)` (fence-held, every module) — or a `db.get` and a compare where `owned_row` fits |
| Admin gate import | `app.auth` | the `app.routes.admin` re-export |
| Upstream failure status | `422` with JSON detail | `502` (reserve 502 for raw byte-proxies) |
| User-facing error | structured `detail` (`{code, reason}`) | a bare English string under a bilingual UI |
| User-facing error | structured `detail` (`{code, reason}`), its sentence named in `lib/api/refusalCodes.ts` (contract-held) | a bare English string under a bilingual UI |
| Mutation response | `{"ok": true, …}` (`{"deleted": id}`), test-held | a bare `{"unsubscribed"}`/`{"cleared": n}`, no `ok` |
| Transaction commit | the handler commits; a unit's `service` and every helper `flush` (fence-held, every module) | a helper that commits — only a listed one (what owns its session, or a core step that must land whatever its caller does next), each with its reason: the fence's `UNIT_COMMITS` / `CORE_COMMITS`, a route's `COMMITTING_HELPERS` (`test_route_conventions`) |
| Admin knob storage | `sysconfig` (values) | env / `app_state` / a table of its own (the job intervals had one) |
+1 -1
View File
@@ -1 +1 @@
0.113.0
0.118.0
-138
View File
@@ -185,118 +185,6 @@
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
"startColumn": 50,
"endColumn": 56,
"lineCount": 1
}
},
{
"code": "reportArgumentType",
"range": {
@@ -364,25 +252,7 @@
}
}
],
"./app/downloads/adapters.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 30,
"endColumn": 38,
"lineCount": 1
}
}
],
"./app/downloads/backfill.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportReturnType",
"range": {
@@ -413,14 +283,6 @@
}
],
"./app/downloads/pipeline.py": [
{
"code": "reportArgumentType",
"range": {
"startColumn": 34,
"endColumn": 38,
"lineCount": 1
}
},
{
"code": "reportAttributeAccessIssue",
"range": {
@@ -0,0 +1,76 @@
"""videos: the duration sorts ride an index on the video's length, not on the raw column
Revision ID: 0117_videos_length_sort_indexes
Revises: 0116_followed_explored_channels
Create Date: 2026-11-07
A YouTube length past a day is not a length (`catalog.models.LENGTH_MAX_S`): a 24/7 stream's archive
reports the years it has run. The feed's duration sorts now order by `Video.length_seconds`, which
is NULL for such a video, so "longest first" no longer opens on a stream archive. Migration 0096's
two duration indexes are on the raw column and no longer match that ORDER BY; they are replaced by
the same indexes over the length expression. The expression is spelled out here as SQLAlchemy emits
it, with the limit as a literal (an index matches a query's constant, never a bound parameter);
tests/test_feed_sort_indexes.py fails if the two drift apart.
CONCURRENTLY (autocommit_block), re-runnable, as in 0096.
"""
from alembic import op
from sqlalchemy import text
revision: str = "0117_videos_length_sort_indexes"
down_revision: str | None = "0116_followed_explored_channels"
branch_labels: str | None = None
depends_on: str | None = None
# Frozen with this migration: `catalog.models.LENGTH_MAX_S` is 24 h.
LENGTH = "(CASE WHEN (duration_seconds > 86400) THEN NULL ELSE duration_seconds END)"
NEW = {
"ix_videos_length_desc": f"{LENGTH} DESC NULLS LAST, id",
"ix_videos_length_asc": f"{LENGTH} ASC NULLS LAST, id",
}
OLD = {
"ix_videos_duration_desc": "duration_seconds DESC NULLS LAST, id",
"ix_videos_duration_asc": "duration_seconds ASC NULLS LAST, id",
}
def _valid(name: str) -> bool | None:
"""The index's `indisvalid`, or None when there is no index of that name."""
row = (
op.get_bind()
.execute(
text(
"SELECT i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid "
"WHERE c.relname = :name"
),
{"name": name},
)
.first()
)
return None if row is None else bool(row[0])
def _build(name: str, columns: str) -> None:
"""CREATE INDEX CONCURRENTLY, re-runnable: an INVALID leftover of a failed build is dropped first
(see 0096 `_build`)."""
if _valid(name) is False:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} ON videos ({columns})")
def upgrade() -> None:
with op.get_context().autocommit_block():
for name, columns in NEW.items():
_build(name, columns)
for name in OLD:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
def downgrade() -> None:
with op.get_context().autocommit_block():
for name, columns in OLD.items():
_build(name, columns)
for name in NEW:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
@@ -0,0 +1,40 @@
"""videos: a random key per video, so a seeded shuffle walks an index instead of sorting the scope
Revision ID: 0118_videos_shuffle_key
Revises: 0117_videos_length_sort_indexes
Create Date: 2026-11-07
The feed's Shuffle sort and the Home surprise ordered the whole feed scope by md5(id || seed) on every
call: ~350 ms on prod for a scope of ~213k videos, and a Shuffle page deeper in the list cost more
(767 ms on page 7 on localdev, against 4 ms for Newest). Each video now carries a random number drawn
once (`Video.shuffle_key`); a seed picks a point on that ring and the draw reads the index from there
(`feedquery.shuffle_walk`).
One ADD COLUMN with a volatile default rewrites the table and fills every row with its own value. On
localdev (278k videos) that took 28 s, against 84 s for an UPDATE of a nullable column, so the rewrite
it is: the table is locked for that time, and a deploy has the app down while it migrates anyway.
The index is (shuffle_key, id): the walk's keyset cursor compares the pair.
"""
from alembic import op
revision: str = "0118_videos_shuffle_key"
down_revision: str | None = "0117_videos_length_sort_indexes"
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
op.execute(
"ALTER TABLE videos ADD COLUMN IF NOT EXISTS shuffle_key double precision "
"NOT NULL DEFAULT random()"
)
with op.get_context().autocommit_block():
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
op.execute("CREATE INDEX CONCURRENTLY ix_videos_shuffle_key ON videos (shuffle_key, id)")
def downgrade() -> None:
with op.get_context().autocommit_block():
op.execute("DROP INDEX CONCURRENTLY IF EXISTS ix_videos_shuffle_key")
op.execute("ALTER TABLE videos DROP COLUMN IF EXISTS shuffle_key")
@@ -0,0 +1,65 @@
"""plex_items.transcode_hw: the facet says whether the GPU ENCODES, under the measured codec sets
Measured on prod (2026-11-07): the iGPU has no VC-1 Advanced decode, so a VC-1 film never ran on the
GPU although the facet said it did; VC-1, XviD and DivX now take the CPU-decode → GPU-encode path; and an
HDR film runs on the GPU through the OpenCL tonemap although the facet said software. The rule is
`plex.sync.transcode_hw_capable`; the two codec sets below are pinned to `plex.sync` by
tests/test_plex_playback_class.py.
Rewritten from the stored codec and bit depth. HDR is not stored, so a 10-bit HW-decodable item is set to
software here and its probe watermark cleared: the bit-depth backfill (`plex.sync._probe_media_batch`,
a bounded batch per Plex sync) reads its HDR flag again and moves an HDR one back to the GPU.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0119_plex_transcode_gpu"
down_revision: Union[str, None] = "0118_videos_shuffle_key"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
HW_DECODABLE_CODECS = ("h264", "hevc", "h265", "vp9", "mpeg2video")
CPU_DECODE_CODECS = ("vc1", "mpeg4", "msmpeg4v3")
def _in(codecs: tuple[str, ...]) -> str:
return ", ".join(f"'{c}'" for c in codecs)
def upgrade() -> None:
op.execute(
f"""
UPDATE plex_items
SET transcode_hw = CASE
WHEN lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
THEN coalesce(video_bit_depth, 8) <= 8
WHEN lower(codec_video) IN ({_in(CPU_DECODE_CODECS)})
THEN coalesce(video_bit_depth, 8) <= 8
ELSE false
END
WHERE playable = 'transcode'
"""
)
op.execute(
f"""
UPDATE plex_items
SET media_probed_at = NULL
WHERE playable = 'transcode'
AND lower(codec_video) IN ({_in(HW_DECODABLE_CODECS)})
AND video_bit_depth > 8
"""
)
def downgrade() -> None:
# The old rule: a decodable codec (VC-1 counted) at 8 bits; HDR and every other codec software.
op.execute(
f"""
UPDATE plex_items
SET transcode_hw = lower(codec_video) IN ({_in(HW_DECODABLE_CODECS + ("vc1",))})
AND coalesce(video_bit_depth, 8) <= 8
WHERE playable = 'transcode'
"""
)
+2 -2
View File
@@ -25,8 +25,8 @@ Three layers, by *who* runs the code — not by feature:
background run's durable row, its words in `runstatus`),
`notifications`, `userscope`, `ratelimit`, `progress`, `logs` (the one logging
configuration), `logsink` (its `log_events` table), `diagnostics` (a user's consented log
level), `debugsession` (a recording that tags its user's lines, E16) and `collation` (A→Z in the
reader's alphabet, E30), … are single-file primitives many layers import. They depend on nothing above them (a core
level), `debugsession` (a recording that tags its user's lines, E16), `collation` (A→Z in the
reader's alphabet, E30) and `draw` (the weighted, seeded draw "Surprise me" deals from), … are single-file primitives many layers import. They depend on nothing above them (a core
module must never import a `routes/…` module).
**Extraction trigger: the SECOND consumer.** Keep new logic inline in its first caller. The moment a
+53 -2
View File
@@ -10,16 +10,22 @@ from sqlalchemy import (
Boolean,
Computed,
DateTime,
Float,
ForeignKey,
Integer,
JSON,
String,
Text,
UniqueConstraint,
case,
func,
literal_column,
null,
)
from sqlalchemy.dialects.postgresql import TSVECTOR
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.sql.elements import ColumnElement
from app.db import Base
from app.models import TimestampMixin
@@ -114,6 +120,20 @@ class Subscription(Base, TimestampMixin):
# hidden from feeds/search and revisited by the refresh job until they settle into a VOD.
LIVE_OR_UPCOMING = ("live", "upcoming")
# A YouTube length past a day is not a length. A 24/7 stream's archive (`was_live`) reports the years
# the stream has run (measured on prod, 2026-11-06: 122,962,914 s), and one such video made a Watch
# later total read 44,073 hours. YouTube caps an upload at 12 hours, so nothing real is lost. Such a
# length is unknown wherever a length is used: the API sends none (`Video.length_seconds`), sums leave
# it out (a playlist's counts it as `without_length`), and the duration filter, sort and Home budget never
# match it. The raw `duration_seconds` stays for what it does prove — that the video is not a Short —
# and for the jobs that ask whether YouTube has answered yet (test_video_length holds who reads it).
LENGTH_MAX_S = 24 * 3600
def known_length(seconds: int | None) -> int | None:
"""`seconds` as a video's length, or None when it is not one (see LENGTH_MAX_S)."""
return seconds if seconds is not None and seconds <= LENGTH_MAX_S else None
class Video(Base, TimestampMixin):
"""A YouTube video. Shared across users; per-user state lives elsewhere."""
@@ -141,8 +161,8 @@ class Video(Base, TimestampMixin):
)
thumbnail_url: Mapped[str | None] = mapped_column(String(1024))
# The feed's views / duration / title sorts each ride a migration-defined index in their exact
# ORDER BY, one per direction (and per ICU collation for title): migration 0096, held to
# `feedquery.order_by` by test_feed_sort_indexes.
# ORDER BY, one per direction (and per ICU collation for title): migration 0096 (the duration
# pair since 0117, on `length_seconds`), held to `feedquery.order_by` by test_feed_sort_indexes.
duration_seconds: Mapped[int | None] = mapped_column(Integer)
view_count: Mapped[int | None] = mapped_column(BigInteger)
like_count: Mapped[int | None] = mapped_column(BigInteger)
@@ -197,6 +217,37 @@ class Video(Base, TimestampMixin):
last_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_since: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
unavailable_reason: Mapped[str | None] = mapped_column(String(24))
# A random number in [0, 1), drawn once when the row is written: the ring a seeded shuffle walks
# (`feedquery.shuffle_walk`, index ix_videos_shuffle_key on (shuffle_key, id), migration 0118).
shuffle_key: Mapped[float] = mapped_column(Float, nullable=False, server_default=func.random())
@hybrid_property
def length_seconds(self) -> int | None:
"""The video's length, or None when it is unknown — not fetched yet, or not a length at all
(`LENGTH_MAX_S`). What every reader of a length reads."""
return known_length(self.duration_seconds)
@length_seconds.inplace.expression
@classmethod
def _length_seconds_expression(cls) -> ColumnElement[int | None]:
# The limit is a literal, not a bound parameter: the duration sort indexes (migration 0117)
# are built on this very expression, and Postgres matches an index to a query's constant only.
return case(
(cls.duration_seconds > literal_column(str(LENGTH_MAX_S)), null()),
else_=cls.duration_seconds,
)
@hybrid_property
def without_length(self) -> bool:
"""A video whose length is unknown — live, an archive past a day, or not fetched yet: what a
sum of lengths leaves out and names as "+N without length". Every such video, so the count
and the sum never disagree about what was left out."""
return self.length_seconds is None
@without_length.inplace.expression
@classmethod
def _without_length_expression(cls) -> ColumnElement[bool]:
return cls.length_seconds.is_(None)
# Weighted full-text search document (DB-generated, never written by the app): title (A) >
# creator keywords + search queries (B) > truncated description (C). ts_rank honours the
+3 -2
View File
@@ -44,7 +44,8 @@ def channel_titles(db: Session, ids: set[str]) -> dict[str, str]:
def channel_aggregates(channel_ids: list[str]) -> Select:
"""Per-channel stored count, last upload and total duration, in one grouped pass. Every column it
"""Per-channel stored count, last upload and total duration (the lengths it knows: a video with
none is left out, `Video.length_seconds`), in one grouped pass. Every column it
reads is in `ix_videos_channel_pub_cover` (migration 0095), so Postgres answers it from the index
alone; a test holds that (test_channel_list_index_only)."""
return (
@@ -52,7 +53,7 @@ def channel_aggregates(channel_ids: list[str]) -> Select:
Video.channel_id,
func.count(Video.id),
func.max(Video.published_at),
func.coalesce(func.sum(Video.duration_seconds), 0),
func.coalesce(func.sum(Video.length_seconds), 0),
)
.where(Video.channel_id.in_(channel_ids))
.group_by(Video.channel_id)
+101 -29
View File
@@ -10,16 +10,25 @@ explore-visible and in-progress for free, and cannot drift from them. The Plex h
narrows EVERY rail by the Plex sidebar's `LibraryFilters`, which is right for a surface that HAS a
sidebar and wrong for a landing page that does not.
**One seed drives both halves.** The feed orders a shuffle by `md5(id || seed)` so a keyset cursor
stays valid across pages; the Plex surprise rail uses `ORDER BY random()`, which is fine for a single
un-paged draw but cannot be reproduced. A cross-source reroll needs both halves to move together —
otherwise "Mást kérek" re-draws the Plex picks while the YouTube ones stand still — so the Plex side
here takes the feed's seeded-md5 idiom instead of `random()`.
**One seed drives both halves.** The YouTube half starts on the ring of stored random keys at the
seed's point and shuffles that window by `md5(id || seed)`; the Plex surprise rail uses
`ORDER BY random()`, which is fine for a single un-paged draw but cannot be reproduced. A
cross-source reroll needs both halves to move together — otherwise "Mást kérek" re-draws the Plex
picks while the YouTube ones stand still — so the Plex side here orders by the same seeded md5
instead of `random()`.
"""
from sqlalchemy import and_, func, or_, select
from datetime import datetime, timezone
from sqlalchemy import (
and_,
func,
or_,
select,
)
from sqlalchemy.orm import Session, aliased
from app import draw
from app.discover.budget import band
from app.feedquery import (
feed_columns,
@@ -27,13 +36,14 @@ from app.feedquery import (
not_blocked_cond,
order_by,
serialize,
shuffle_walk,
sort_keys,
)
from app.models import User
from app.plex.models import PlexItem, PlexShow, PlexState
from app.catalog.models import Channel, Video
from app.reader.models import PlaylistItem, VideoState
from app.plex import rollup
from app.plex import affinity, rollup
from app.plex.cards import display_title, leaf_card, movie_card, show_card
from app.plex.query import apply_show_state_filter, content_libs
from app.userscope import owned_by
@@ -67,7 +77,8 @@ _FEED_BASE: dict = {
def _seeded(col, seed: int):
"""The feed's deterministic shuffle expression, reused for Plex so one seed moves both halves."""
"""The Home surprise's seeded order (md5 of the key and the seed), the same for Plex so one
seed moves both halves."""
return func.md5(func.concat(col, str(seed)))
@@ -112,17 +123,75 @@ def yt_surprise(db: Session, user: User, budget: str, seed: int, limit: int) ->
Shorts are opened up only for the shortest band: "I have ten minutes" is the one answer a Short is
actually a good response to, and the feed's own default keeps them out otherwise."""
lo, hi = band(budget)
params = dict(_FEED_BASE, min_duration=lo, max_duration=hi, include_shorts=budget == "short")
query, rank_expr = filtered_query(db, user, **params)
keys = sort_keys("shuffle", seed, rank_expr)
# Over-fetch, then drop anything already started. The feed's `unwatched` scope keeps watched and
# hidden out but deliberately keeps in-progress rows in (they belong in the feed). Here they must
# go: the resume tier has already had its chance at them, and it may have passed precisely because
# the REMAINING time did not fit the budget — offering the same item back as a surprise, whose
# runtime does fit, would label a half-finished thing "Meglepetés" and contradict the question.
rows = db.execute(query.order_by(*order_by(keys)).limit(limit * 4)).all()
out = [serialize(r) for r in rows]
return [r for r in out if not r["position_seconds"]][:limit]
params: dict = {
**_FEED_BASE,
"min_duration": lo,
"max_duration": hi,
"include_shorts": budget == "short",
}
query, _rank = filtered_query(db, user, **params)
# A window of the ring, not the whole scope: the seed's SURPRISE_WINDOW videos read along the
# stored random key (`shuffle_walk`, a millisecond), then shuffled by the seed. Dealing straight
# from the ring would favour for good every video that sits after a wide gap of it (the first
# video past a random point is the one whose gap the point fell into); shuffling the window makes
# each video's chance the window's share, even to within a few percent. A scope no larger than
# the window is the whole scope, in exactly the md5 order the draw always had.
#
# Anything already started is dropped. The feed's `unwatched` scope keeps watched and hidden out
# but deliberately keeps in-progress rows in (they belong in the feed). Here they must go: the
# resume tier has already had its chance at them, and it may have passed precisely because the
# REMAINING time did not fit the budget — offering the same item back as a surprise, whose runtime
# does fit, would label a half-finished thing "Meglepetés" and contradict the question.
window = [
r
for r in map(serialize, shuffle_walk(db, query, seed, SURPRISE_WINDOW))
if not r["position_seconds"]
]
blind = draw.shuffled(window, seed, key=_video_id)
channels = _channel_affinity(db, user)
if not channels:
return blind[:limit]
def weight(row: dict) -> float:
return 1.0 + CHANNEL_WEIGHT * channels.get(row["channel_id"], 0.0)
leaning = draw.leaning(window, seed, key=_video_id, weight=weight)
return draw.blend(leaning, blind, seed, limit, key=_video_id)
# How many videos of the ring a surprise draws from: wide enough that a channel the user watches has
# a few in it, small enough to read in a millisecond or two.
SURPRISE_WINDOW = 500
def _video_id(row: dict) -> str:
return row["id"]
# How much a channel the user watches most outweighs one they never watch (1 + this), and how far back
# the watched history counts: the latest WATCHED_RECENT watches, the recent ones weighing more.
CHANNEL_WEIGHT = 3.0
WATCHED_RECENT = 300
WATCHED_HALF_LIFE_DAYS = 90
def _channel_affinity(db: Session, user: User) -> dict[str, float]:
"""Each channel the user watched lately, scored 0..1 against the most watched: the sum of its
watches, each weighing half as much every WATCHED_HALF_LIFE_DAYS. Empty with no history."""
rows = db.execute(
select(Video.channel_id, VideoState.watched_at)
.join(Video, Video.id == VideoState.video_id)
.where(owned_by(VideoState.user_id, user), VideoState.status == "watched")
.order_by(VideoState.watched_at.desc().nullslast())
.limit(WATCHED_RECENT)
).all()
now = datetime.now(timezone.utc)
scores: dict[str, float] = {}
for channel_id, watched_at in rows:
r = draw.recency(watched_at, now, WATCHED_HALF_LIFE_DAYS)
scores[channel_id] = scores.get(channel_id, 0.0) + r
return draw.scaled(scores)
def yt_video_row(db: Session, user: User, video_id: str) -> dict | None:
@@ -289,21 +358,24 @@ def plex_surprise(
if not movie_libs:
return []
lo, hi = band(budget)
q = (
select(PlexItem)
.where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
_unseen_movie_conds(uid),
)
.order_by(_seeded(PlexItem.rating_key, seed))
.limit(limit)
q = select(PlexItem).where(
PlexItem.kind == "movie",
PlexItem.library_id.in_(movie_libs),
_unseen_movie_conds(uid),
)
if lo is not None:
q = q.where(PlexItem.duration_s >= lo)
if hi is not None:
q = q.where(PlexItem.duration_s <= hi)
items = db.execute(q).scalars().all()
shuffled = _seeded(PlexItem.rating_key, seed)
blind = db.execute(q.order_by(shuffled, PlexItem.id).limit(limit)).scalars().all()
leaning = affinity.for_user(db, uid)
if leaning:
key = draw.weighted_key(draw.seeded_u(PlexItem.rating_key, seed), leaning.weight(PlexItem))
weighted = db.execute(q.order_by(key, shuffled, PlexItem.id).limit(limit)).scalars().all()
items = draw.blend(list(weighted), list(blind), seed, limit, key=lambda it: it.id)
else:
items = list(blind)
states = rollup.item_state_map(db, uid, [it.id for it in items])
return [movie_card(it, states.get(it.id), lang=lang) for it in items]
+20 -3
View File
@@ -14,10 +14,10 @@ Resolution works both before extraction (from source_kind / URL host) and after
import re
from urllib.parse import urlparse
import yt_dlp
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.config import settings
from app.downloads import egress
from app.downloads.stated import StatedFailure
# Facebook's "Cannot parse data": the page yt-dlp got was a wall, not the content. Both the Chrome retry
@@ -33,6 +33,9 @@ class SourceAdapter:
"""Generic source: yt-dlp handles everything and the ref is already a fetchable URL."""
key = "generic"
# SEC-17: every fetch of this source's download goes through the egress gate (`egress`). A
# source's own host list is no guarantee — its pages, redirects and formats can point anywhere.
egress_guarded = True
# --- extraction (worker._extract_with_fallback / backfill) ---------------------------------
def resolve_url(self, url: str, ydl_opts: dict) -> str:
@@ -66,6 +69,11 @@ class SourceAdapter:
"""The download's raw title, before `titles.normalize_title` cleans it. Generic: yt-dlp's."""
return info.get("title")
def is_placeholder_title(self, title: str | None) -> bool:
"""Whether `title` is a stand-in the source gives in place of the real one (worth one more,
signed-out read of the page). Generic: never."""
return False
def uploader_url(self, info: dict) -> str | None:
"""The channel/uploader page URL. Generic: whatever yt-dlp exposes directly (YouTube gives
`channel_url`), else None (reddit / a raw media URL have no derivable channel page)."""
@@ -87,6 +95,9 @@ class SourceAdapter:
class YouTubeAdapter(SourceAdapter):
key = "youtube"
# Not through the gate: YouTube's fetches go to Google's fixed hosts, and the PO-token provider
# it needs is the sidecar on our internal network, which the gate exists to refuse.
egress_guarded = False
# Errors worth retrying with a different player-client set (YouTube per-client flakiness).
_RETRIABLE = (
"not a bot",
@@ -174,7 +185,7 @@ class FacebookAdapter(SourceAdapter):
post_id = facebook_group_post_id(url)
if post_id is None or not ydl_opts.get("cookiefile"):
return url
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
with egress.open_ydl(ydl_opts) as ydl:
raw = ydl.urlopen(url).read(self._POST_PAGE_MAX_BYTES)
page = raw.decode("utf-8", "replace")
# Measured: a refused or expired cookie gets the login page (~450 KB, a login
@@ -201,11 +212,17 @@ class FacebookAdapter(SourceAdapter):
# Measured (2026-09-14): a SIGNED-IN read gives the placeholder "Video" (plain and as Chrome), and an
# 18+ reel only readable as Chrome gives "Facebook"; the post's text is still in `description`. A
# signed-out read titles the page with that same text behind the engagement counts, cut short.
# Measured (2026-11-01, yt-dlp 2026.08.19): since 2026-10-28 every signed-in reel read gives
# "Facebook" and NO description, while a signed-out read of the same reel still carries the text —
# so a placeholder left with nothing behind it gets one signed-out read (`backfill.public_title_info`).
_PLACEHOLDER_TITLES = frozenset({"video", "facebook"})
def is_placeholder_title(self, title: str | None) -> bool:
return (title or "").strip().lower() in self._PLACEHOLDER_TITLES
def title(self, info: dict) -> str | None:
raw = info.get("title")
if (raw or "").strip().lower() not in self._PLACEHOLDER_TITLES:
if not self.is_placeholder_title(raw):
return raw
line = next(
(ln.strip() for ln in (info.get("description") or "").splitlines() if ln.strip()), ""
+63 -10
View File
@@ -5,7 +5,10 @@ auto-detected channel becomes a clickable link) and `poster_path` (an ffmpeg fra
thumbnail-less sources, so the card / watch page / link preview still shows an image). Best-effort
and safe to re-run: it only touches assets still missing the field, and only re-fetches YouTube /
Facebook sources (a direct media URL like a reddit HLS manifest carries no channel and its signed
URL may have expired — those just get a locally-generated poster instead).
URL may have expired — those just get a locally-generated poster instead). A Facebook download
titled with the source's placeholder ("Facebook", "Video") is retitled from a signed-out read of its
page, and so is each download name that only repeated the placeholder. That page read
(`public_title_info`) also titles every new download, from the worker.
Run once per environment:
docker exec siftlode-worker-1 python -c "from app.downloads import backfill; print(backfill.run())"
@@ -13,21 +16,23 @@ Run once per environment:
from urllib.parse import urlsplit
import yt_dlp
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app import logs
from app.config import settings
from app.db import SessionLocal
from app.downloads import adapters
from app.downloads import edit as editmod
from app.downloads import formats, service
from app.downloads.models import MediaAsset
from app.downloads import egress, formats, service
from app.downloads.models import DownloadJob, MediaAsset
from app.models import DISPLAY_TEXT_MAX
from app.titles import normalize_title
log = logs.get_logger(__name__)
def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
def extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
"""Metadata-only yt-dlp extraction, using the source adapter's client sets + options (YouTube
needs the POT/player-client strategy even just to extract)."""
with formats.hiding(url):
@@ -37,11 +42,15 @@ def _extract_meta(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> d
def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict | None:
for clients in adapter.client_sets():
opts = formats.build_ydl_opts(
{}, "%(id)s.%(ext)s", lambda d: None, extra=adapter.ydl_opts_extra(clients)
{},
"%(id)s.%(ext)s",
lambda d: None,
extra=adapter.ydl_opts_extra(clients),
egress=adapter.egress_guarded,
)
opts["skip_download"] = True
try:
with yt_dlp.YoutubeDL(opts) as ydl:
with egress.open_ydl(opts) as ydl:
info = ydl.extract_info(url, download=False)
if info:
return info
@@ -62,8 +71,43 @@ def _extract_meta_scoped(url: str, adapter: adapters.SourceAdapter, asset_id: in
return None
def public_title_info(info: dict, url: str, adapter: adapters.SourceAdapter, asset_id: int) -> dict:
"""`info` to title a download from: itself, unless its title is a source's placeholder with
nothing behind it (a signed-in Facebook read: "Facebook", no description). Then one signed-out,
metadata-only read of the page lends its text — the description, else its title — as the
description the adapter takes a placeholder's title from. Nothing better there: `info` as it was."""
if not adapter.is_placeholder_title(adapter.title(info)):
return info
public = extract_meta(url, adapter, asset_id)
text = public and (public.get("description") or public.get("title"))
return {**info, "description": text} if text else info
def _retitle(db: Session, a: MediaAsset, adapter: adapters.SourceAdapter, url: str) -> bool:
"""Swap a placeholder title kept from an earlier download for the page's text, and the download
names that only repeated it. A name its owner gave (anything else) stays."""
if not adapter.is_placeholder_title(a.title):
return False
info = public_title_info({"title": a.title, "description": None}, url, adapter, a.id)
named = normalize_title(adapter.title(info))
if not named or adapter.is_placeholder_title(named):
return False
# One conditional write, not a read and a compare: a name its owner gives while the page is read
# is not overwritten by what was read before it. The asset first, as every other writer of the
# two locks them (`service.lock_own_job`): the session does not flush by itself.
placeholder = a.title
a.title = named
db.flush()
db.execute(
update(DownloadJob)
.where(DownloadJob.asset_id == a.id, DownloadJob.display_name == placeholder)
.values(display_name=named[:DISPLAY_TEXT_MAX])
)
return True
def run() -> dict:
filled_url = filled_poster = 0
filled_url = filled_poster = retitled = 0
with SessionLocal() as db:
assets = (
db.execute(
@@ -78,7 +122,7 @@ def run() -> dict:
adapter = adapters.resolve_adapter(a.source_kind, a.source_ref)
if a.uploader_url is None and adapter.wants_metadata_backfill():
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
info = _extract_meta(url, adapter, a.id)
info = extract_meta(url, adapter, a.id)
uu = adapter.uploader_url(info) if info else None
if uu:
a.uploader_url = uu
@@ -86,6 +130,14 @@ def run() -> dict:
log.info(
"asset %s uploader_url filled", a.id
) # the URL names what they downloaded
if adapter.wants_metadata_backfill():
url = a.source_webpage_url or service.source_url(a.source_kind, a.source_ref)
if _retitle(db, a, adapter, url):
# Before the poster step: it can cut a frame for 30 s, and these locks held
# across it block a rename and deadlock a delete.
db.commit()
retitled += 1
log.info("asset %s retitled", a.id) # the title names what they downloaded
if a.poster_path is None:
# Record a self-hosted poster for every download (the reliable og:image source):
# ensure_poster returns the existing <base>.jpg thumbnail sidecar, or cuts a frame.
@@ -99,6 +151,7 @@ def run() -> dict:
"assets": len(assets),
"filled_uploader_url": filled_url,
"filled_poster": filled_poster,
"retitled": retitled,
}
log.info("backfill done: %s", result)
return result
+28 -2
View File
@@ -22,6 +22,7 @@ import json
import math
import re
from pathlib import Path
from typing import Literal
from app.downloads.edit import AAC_ARGS, EVEN_FRAME, INPUT_GUARD, X264_ARGS
from app.plex.sync import HDR_TRANSFERS
@@ -57,12 +58,27 @@ _CROP_ASPECTS = (0.25, 4.0)
_EDGE_SLACK = 1e-3
# Every reason a clip is refused; the UI names each one (`clip.errors.<reason>`, held by the frontend's
# backendIds contract, which reads this Literal).
ClipReason = Literal[
"bad_source",
"bad_range",
"too_short",
"too_long",
"bad_audio",
"bad_crop",
"not_found",
"media_missing",
"past_end",
]
class ClipError(Exception):
"""A clip request that cannot be cut; `reason` is an i18n-mappable key, `params` the numbers its
message names (the cap itself, so the client never says a stale one)."""
def __init__(self, reason: str, **params: float):
self.reason = reason
def __init__(self, reason: ClipReason, **params: float):
self.reason: ClipReason = reason
self.params = params
super().__init__(reason)
@@ -185,6 +201,16 @@ def _ratio(v: object) -> float | None:
return n / d if n > 0 and d > 0 else None
def quarter_turned(probe: dict) -> bool | None:
"""Whether the first video stream of an ffprobe `-show_streams` JSON is stored turned a quarter
(ffmpeg stands it upright before any filter, so its filters see width and height swapped). None
when there is no video stream."""
for s in probe.get("streams") or []:
if s.get("codec_type") == "video":
return _quarter_turned(s)
return None
def _quarter_turned(stream: dict) -> bool:
rotation: object = (stream.get("tags") or {}).get("rotate")
for sd in stream.get("side_data_list") or []:
+10
View File
@@ -25,6 +25,7 @@ Security controls enforced here:
import os
from pathlib import Path
from typing import Literal
from sqlalchemy import select
from sqlalchemy.orm import Session
@@ -163,9 +164,18 @@ def get_credential(db: Session, user_id: int, host_key: str) -> DownloadCredenti
).scalar_one_or_none()
# Every reason a pasted cookies.txt is refused; the UI names each one (`downloads.errors.cookies.<reason>`,
# held by the frontend's backendIds contract, which reads this Literal).
CookieReason = Literal["empty", "format"]
class InvalidCookies(ValueError):
"""The pasted cookies aren't a usable Netscape cookies.txt — surfaced as a 400 by the route."""
def __init__(self, reason: CookieReason):
super().__init__(reason)
self.reason: CookieReason = reason
def validate_cookies(raw: str) -> str:
"""Sanity-check a pasted cookies.txt (Netscape format) and return the cleaned text. yt-dlp reads
+48 -5
View File
@@ -11,7 +11,9 @@ with the worker branching on the asset's `source_kind` to run ffmpeg instead of
(the `-ss` seeks first), so a short clip is cheap even from a long source
* crop (± trim) → always re-encode (a filter can't be stream-copied), frame-accurate
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h}, accurate?: bool}`. `accurate` is the
`edit_spec = {trim?: {start_s, end_s}, crop?: {x, y, w, h, frame_w?, frame_h?}, accurate?: bool}`. A
crop with `frame_w`/`frame_h` counts the picture as it is SHOWN, measured on a frame of that size
(`crop_filter`); one without them is in the file's stored pixels (its older form). `accurate` is the
per-edit user choice for a trim-only cut (ignored when a crop forces a re-encode anyway). `split`
is a frontend concept: the
UI fans a split out into N trim jobs, so there is one output file per job here.
@@ -60,6 +62,46 @@ class EditAborted(Exception):
_MAX_SEGMENTS = 100
def _crop_frame(crop: dict) -> dict | None:
"""The picture size a crop was measured on, or None when it has none it could be scaled by (a
spec from before it was sent, which `crop_filter` cuts in the file's own pixels)."""
try:
w, h = int(crop["frame_w"]), int(crop["frame_h"])
# silent-ok: no frame is a crop's older form, and it is still the crop that was asked for. An
# infinite number (JSON `1e400`) is OverflowError, and it is no more a frame than "x" is.
except (KeyError, TypeError, ValueError, OverflowError):
return None
return {"frame_w": w, "frame_h": h} if w > 0 and h > 0 else None
def crop_filter(crop: dict) -> str:
"""The ffmpeg `crop` of an edit. Its numbers count the picture as the browser reports it, which
is its SHOWN size: a video with wide pixels (320 stored, 640 shown) would be cut off to one side
in ffmpeg's stored ones, so they are scaled to `iw` / `ih` by the size they were measured on. A
rotated file needs nothing more: both the browser and ffmpeg's filters see it upright."""
frame_w, frame_h = crop.get("frame_w"), crop.get("frame_h")
if not frame_w or not frame_h:
return f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}"
return (
f"crop=iw*{crop['w']}/{frame_w}:ih*{crop['h']}/{frame_h}"
f":iw*{crop['x']}/{frame_w}:ih*{crop['y']}/{frame_h}"
)
def crop_stored_size(crop: dict, src_w: int | None, src_h: int | None) -> tuple[int, int] | None:
"""The size `crop_filter` cuts out of a `src_w` x `src_h` frame, evened as `EVEN_FRAME` evens it —
the clip's size when ffprobe cannot tell. The frame is the one ffmpeg's filters see: a rotated
file's stood upright, its stored size turned (the caller's to turn). A crop counted in the shown
picture is a fraction of that frame, not its own pixel count; None when the size is unknown."""
if crop.get("frame_w") and crop.get("frame_h"):
if not src_w or not src_h:
return None
w, h = src_w * crop["w"] // crop["frame_w"], src_h * crop["h"] // crop["frame_h"]
else:
w, h = crop["w"], crop["h"]
return w // 2 * 2, h // 2 * 2
def normalize_edit_spec(spec: dict | None) -> dict:
"""Canonicalize a raw edit spec: clamp/round trim, coerce crop to ints, drop empties.
@@ -72,7 +114,7 @@ def normalize_edit_spec(spec: dict | None) -> dict:
if crop:
try:
c = {k: int(round(float(crop[k]))) for k in ("x", "y", "w", "h")}
except (KeyError, TypeError, ValueError) as exc:
except (KeyError, TypeError, ValueError, OverflowError) as exc:
# The clip still renders — uncropped. The user asked for a crop, got a whole frame,
# and nothing anywhere called it a failure; this line is the only trace there is.
log.warning(
@@ -81,7 +123,8 @@ def normalize_edit_spec(spec: dict | None) -> dict:
)
c = None
if c and c["w"] > 0 and c["h"] > 0 and c["x"] >= 0 and c["y"] >= 0:
out["crop"] = c
frame = _crop_frame(crop)
out["crop"] = {**c, **frame} if frame else c
# Multi-segment JOIN (cut-list → one concatenated file). Takes precedence over `trim`.
segs = spec.get("segments")
@@ -268,7 +311,7 @@ def build_edit_cmd(src: Path, dest: Path, spec: dict, out_ext: str) -> list[str]
if reencode:
# Input -ss + re-encode is frame-accurate (decodes from the prior keyframe, discards up to
# the exact cut). A crop adds the filter; an accurate trim only evens the frame.
cropf = f"crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}," if crop else ""
cropf = f"{crop_filter(crop)}," if crop else ""
cmd += ["-vf", cropf + EVEN_FRAME, *X264_ARGS, *AAC_ARGS]
else:
cmd += ["-c", "copy", "-avoid_negative_ts", "make_zero"]
@@ -313,7 +356,7 @@ def build_concat_plan(src: Path, dest: Path, spec: dict, out_ext: str, staging:
]
if needs_reencode(spec):
cropf = f",crop={crop['w']}:{crop['h']}:{crop['x']}:{crop['y']}" if crop else ""
cropf = f",{crop_filter(crop)}" if crop else ""
parts, labels = [], []
for i, s in enumerate(segs):
st, en = s["start_s"], s["end_s"]
+297
View File
@@ -0,0 +1,297 @@
"""The egress gate: every fetch of a user-supplied download goes through here, and this is where the
address is judged — at the moment of the connection, not when the URL was typed (SEC-17).
`urlguard` judges the URL the user typed, at enqueue and again in the worker. What that cannot see is
every request yt-dlp makes after it: a public host answering 3xx with an internal `Location`, a page
whose formats point inside, a name that resolves public for the check and private for the fetch (DNS
rebinding). So a guarded download runs with yt-dlp's `proxy` set to this gate, a small HTTP proxy on
the worker's loopback. For each request it resolves the host ONCE, refuses the connection if any
address the name has is one the server must not fetch from (`urlguard.is_forbidden`), and otherwise
connects to the very address it vetted — there is no second lookup for a rebinding answer to slip into.
Measured (yt-dlp 2026.8.19, 2026-11-06) on the worker image, with a logging proxy: the `proxy` option
reaches every path a download takes — the Requests handler, the CurlCFFI one an impersonated request
uses, the native HLS downloader and an ffmpeg HLS download (segments included). ffmpeg itself also
speaks rtmp, rtsp, ftp, tcp and more, which no HTTP proxy covers, so a guarded download keeps only the
formats it fetches over http(s) (`HttpOnlyFormats`, added by `open_ydl`) and fetches HLS natively.
A refused connection is answered `403` before anything is dialled, so a live internal port and a dead
one read the same; `errors.user_error_text` then stores the one redacted sentence for both.
Not used for YouTube: those downloads talk to Google's fixed hosts and to the PO-token sidecar, which
lives on the internal network on purpose (`adapters.YouTubeAdapter.egress_guarded`).
"""
import ipaddress
import select
import socket
import socketserver
import threading
from collections.abc import Callable
from typing import Any, cast
from urllib.parse import urlsplit
import yt_dlp
from yt_dlp.postprocessor.common import PostProcessor
from app import logs
from app.downloads import urlguard
log = logs.get_logger(__name__)
Resolver = Callable[..., list]
Forbidden = Callable[[ipaddress.IPv4Address | ipaddress.IPv6Address], bool]
_BLOCKED = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
_MAX_HEAD = 64 * 1024
# A tunnel with no traffic either way for this long is closed. Above yt-dlp's own socket_timeout
# (30 s, `formats.build_ydl_opts`), so the gate never ends a transfer yt-dlp would still wait for.
_IDLE_S = 120
# Hop-by-hop headers a proxy must not pass on; `Connection: close` replaces them, so one connection
# carries one request and a reused client connection can never be steered to a second host unvetted.
_HOP = (b"proxy-", b"connection:", b"keep-alive:")
def vetted_address(
host: str, port: int, *, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden
) -> tuple[str, int] | None:
"""The address to connect to for `host:port`, or None when it must not be fetched: the name has
no answer, or ANY of its addresses is forbidden (a round-robin answer is not a coin flip). IPv4
only, as every download is (`source_address` 0.0.0.0 in `formats.build_ydl_opts`)."""
try:
infos = resolve(host, port, socket.AF_INET, socket.SOCK_STREAM)
except (socket.gaierror, UnicodeError, OSError):
# silent-ok: "no answer" is a refusal, and the caller's refusal is the line that is logged.
return None
addrs = [info[4] for info in infos]
if not addrs:
return None
for addr in addrs:
try:
ip = ipaddress.ip_address(addr[0])
except ValueError:
# silent-ok: an answer that is not an address is refused like a forbidden one, and the
# caller logs the refusal.
return None
if forbidden(ip):
return None
return addrs[0][0], addrs[0][1]
def _read_head(sock: socket.socket) -> bytes | None:
buf = b""
while b"\r\n\r\n" not in buf:
if len(buf) > _MAX_HEAD:
return None
chunk = sock.recv(8192)
if not chunk:
return None
buf += chunk
return buf
def _pipe(a: socket.socket, b: socket.socket) -> None:
try:
_relay(a, b)
except OSError:
# silent-ok: one side hung up mid-transfer (yt-dlp closes a connection once it has read a
# file's size, or gives up on a slow host); whichever side cared reports it itself.
return
def _relay(a: socket.socket, b: socket.socket) -> None:
socks = [a, b]
while True:
ready, _, _ = select.select(socks, [], [], _IDLE_S)
if not ready:
return
for s in ready:
data = s.recv(65536)
if not data:
return
(b if s is a else a).sendall(data)
class _Handler(socketserver.BaseRequestHandler):
def handle(self) -> None:
client: socket.socket = self.request
head = _read_head(client)
if head is None:
return
head_part, _, body = head.partition(b"\r\n\r\n")
lines = head_part.split(b"\r\n")
try:
method, target, version = lines[0].decode("latin1").split(" ", 2)
except ValueError:
# silent-ok: the only client is yt-dlp on this loopback; a malformed request is answered
# 400 and reaches its own error, which the job records.
client.sendall(_BAD_REQUEST)
return
if method == "CONNECT":
host, _, port_s = target.rpartition(":")
host = host.strip("[]")
upstream = self._dial(host, port_s)
if upstream is None:
return
with upstream:
client.sendall(b"HTTP/1.1 200 Connection established\r\n\r\n")
if body:
upstream.sendall(body)
_pipe(client, upstream)
return
u = urlsplit(target)
if u.scheme != "http" or not u.hostname:
client.sendall(_BLOCKED)
return
try:
port = u.port or 80
except ValueError:
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
client.sendall(_BAD_REQUEST)
return
upstream = self._dial(u.hostname, str(port))
if upstream is None:
return
path = (u.path or "/") + (f"?{u.query}" if u.query else "")
headers = [h for h in lines[1:] if not h.lower().startswith(_HOP)]
request = b"\r\n".join(
[f"{method} {path} {version}".encode("latin1"), *headers, b"Connection: close"]
)
with upstream:
upstream.sendall(request + b"\r\n\r\n" + body)
_pipe(client, upstream)
def _dial(self, host: str, port_s: str) -> socket.socket | None:
"""Connect to the address `host` was vetted at, or answer the client and return None."""
client: socket.socket = self.request
try:
port = int(port_s)
except ValueError:
# silent-ok: as for a malformed request line above — answered 400, recorded by the job.
client.sendall(_BAD_REQUEST)
return None
gate = cast("_Gate", self.server)
addr = vetted_address(host, port, resolve=gate.resolve, forbidden=gate.forbidden)
if addr is None:
log.warning(
"egress refused a connection to %s:%s",
host, # log-pii-ok: the host a download tried to reach, never its path or query
port,
extra=logs.event("download.egress.refused", port=port),
)
client.sendall(_BLOCKED)
return None
try:
return socket.create_connection(addr, timeout=_IDLE_S)
except OSError:
# silent-ok: a public host that does not answer; yt-dlp reports it as the job's own
# network failure, which the job records.
client.sendall(_BAD_GATEWAY)
return None
class _Gate(socketserver.ThreadingTCPServer):
daemon_threads = True
allow_reuse_address = True
def __init__(self, resolve: Resolver, forbidden: Forbidden):
self.resolve = resolve
self.forbidden = forbidden
super().__init__(("127.0.0.1", 0), _Handler)
@property
def url(self) -> str:
return f"http://127.0.0.1:{self.server_address[1]}"
def handle_error(self, request, client_address) -> None:
# socketserver's default prints a bare traceback to stderr, outside the log; a connection the
# gate could not finish is a line with its code instead.
log.exception("egress gate dropped a connection", extra=logs.event("download.egress.error"))
def start_gate(
*, resolve: Resolver = socket.getaddrinfo, forbidden: Forbidden = urlguard.is_forbidden
) -> _Gate:
"""A gate serving on its own thread. The worker uses the one `proxy_url` starts; a test starts
its own, with a resolver and a judgement it controls."""
gate = _Gate(resolve, forbidden)
threading.Thread(target=gate.serve_forever, name="egress-gate", daemon=True).start()
return gate
# The options key `formats.build_ydl_opts` marks a guarded download with; `open_ydl` reads it. yt-dlp
# keeps unknown keys in its params and never acts on them.
GUARDED = "siftlode_egress_guarded"
def open_ydl(opts: dict) -> yt_dlp.YoutubeDL:
"""The one way this package opens yt-dlp (a contract test holds it): a guarded download also
gets `HttpOnlyFormats`, which yt-dlp takes only as an instance, not through the options dict."""
# yt-dlp's stub types its params as a TypedDict; these are the plain dict `formats.build_ydl_opts`
# assembles, the same every caller handed yt-dlp before.
ydl = yt_dlp.YoutubeDL(opts) # type: ignore[arg-type]
if opts.get(GUARDED):
ydl.add_post_processor(HttpOnlyFormats(), when="pre_process")
return ydl
_gate: _Gate | None = None
_gate_lock = threading.Lock()
def proxy_url() -> str:
"""This process's gate, started on first use (the worker's jobs share it)."""
global _gate
with _gate_lock:
if _gate is None:
_gate = start_gate()
return _gate.url
# The protocols fetched over http(s) through the proxy option — the only ones the gate sees: yt-dlp's
# own fetchers (http, f4m, ism, the DASH segments and their generator, native HLS — which a bare m3u8
# also gets, as a guarded download sets hls_prefer_native) and ffmpeg's http input (a live stream or
# a clip), which takes the proxy too. Everything else (rtmp, rtsp, mms, ws) ffmpeg or another tool
# would dial directly.
_HTTP_PROTOCOLS = frozenset(
{
"http",
"https",
"f4m",
"ism",
"m3u8",
"m3u8_native",
"http_dash_segments",
"http_dash_segments_generator",
}
)
class HttpOnlyFormats(PostProcessor):
"""Drops every format a guarded download would fetch past the gate. Runs as `pre_process`, which
yt-dlp re-reads its formats after ("The pre-processors may have modified the formats"), so the
selector never sees them. With nothing left, yt-dlp reports no formats — a refusal, not a leak."""
def set_downloader(self, downloader):
# Not the job's postprocessor hooks: they would report this filter as a "processing" step
# before a byte is fetched (yt-dlp attaches them here, once per call).
self._downloader = downloader
def run(self, information):
# yt-dlp's stub types the info dict as a TypedDict with optional keys; it is a dict here.
info = cast("dict[str, Any]", information)
if info.get("formats"):
info["formats"] = [f for f in info["formats"] if _over_http(f)]
elif info.get("url") and not _over_http(info):
info["url"] = None
return [], information
def _over_http(fmt: dict) -> bool:
url = fmt.get("manifest_url") or fmt.get("url") or ""
if urlsplit(url).scheme not in ("http", "https"):
return False
protocol = fmt.get("protocol") or urlsplit(fmt.get("url") or "").scheme
return all(p in _HTTP_PROTOCOLS for p in protocol.split("+"))
+13 -2
View File
@@ -24,6 +24,7 @@ kept it — and its `ready_at`, the moment it finished (`stamp_done`).
from datetime import datetime, timedelta, timezone
from enum import StrEnum
from typing import Literal
from urllib.parse import urlsplit
from sqlalchemy import select
@@ -181,12 +182,22 @@ def stamp_done(db: Session, job: DownloadJob) -> None:
job.expiry_warned = False
# Every folder change the rules refuse, answered as a generic refusal; the UI names each one
# (`errors.refused.<reason>`, held by the frontend's backendIds contract, which reads this Literal).
FolderReason = Literal[
"file_folder_gone",
"file_folder_name_taken",
"file_folder_not_its_source",
"file_folder_not_own",
]
class FolderError(Exception):
"""A folder change the rules refuse; `reason` is the refusal the route answers with."""
def __init__(self, reason: str) -> None:
def __init__(self, reason: FolderReason) -> None:
super().__init__(reason)
self.reason = reason
self.reason: FolderReason = reason
def own_folder(db: Session, user_id: int, folder_id: int, *, hold: bool = True) -> FileFolder:
+15 -1
View File
@@ -28,6 +28,7 @@ import urllib.parse
from collections.abc import Iterator
from app import logs
from app.downloads import egress as egress_gate
log = logs.get_logger(__name__)
@@ -294,6 +295,8 @@ def build_ydl_opts(
extra: dict | None = None,
cookiefile: str | None = None,
max_bytes: int | None = None,
*,
egress: bool,
) -> dict:
"""Assemble yt-dlp options for a single download to `outtmpl` (a full path template).
@@ -309,7 +312,11 @@ def build_ydl_opts(
a file announced larger than it rather than filling the disk and only then being caught by the
quota accounting. Its ffmpeg downloader (a `download_ranges` fetch) does not read it, nor does
a fragmented download (YouTube's formats: measured, 718 MB fetched with 1 MB left) — so the
pipeline also holds every produced file to what is left by its real size."""
pipeline also holds every produced file to what is left by its real size.
`egress` (SEC-17) routes every fetch through the worker's egress gate, which refuses a connection
into our own network, and keeps only the formats fetched over http(s). Required, with no default:
a new caller has to say which kind of download it builds (`SourceAdapter.egress_guarded`)."""
s = normalize(spec)
h = s["max_height"]
opts: dict = {
@@ -364,6 +371,13 @@ def build_ydl_opts(
if cookiefile:
opts["cookiefile"] = cookiefile
# SEC-17. AFTER the adapter merge, like the cookie: an adapter must not be able to point a
# guarded download past the gate.
if egress:
opts["proxy"] = egress_gate.proxy_url()
opts["hls_prefer_native"] = True
opts[egress_gate.GUARDED] = True
if s["mode"] == "a":
opts["format"] = "bestaudio/best"
opts["postprocessors"].append(
+23 -2
View File
@@ -127,14 +127,35 @@ def check_grant(token: str, grant: str | None, password_version: int) -> bool:
# --- serialization -------------------------------------------------------------------------
def owner_view(link: DownloadLink, base_url: str = "", token: str | None = None) -> dict:
def opens(db: Session, link: DownloadLink) -> str:
"""What a link opens, in the words its address and its preview use: "folder", "video" or
"file". A file still being made reads as the video it mostly is."""
from app.downloads import mediakind
if link.folder_id is not None:
return "folder"
job = db.get(DownloadJob, link.job_id)
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
kind = asset.media_kind if asset else None
return "file" if kind and kind != mediakind.VIDEO else "video"
def share_path(db: Session, link: DownloadLink, token: str) -> str:
"""The address a link is given out on: its path says what it opens (`opens`). Only a word for the
reader, the token is the whole key: every prefix of `main.SHARE_PREFIXES` opens every link, so
the links already sent on /watch/ work for good."""
prefix = {"folder": "folder", "file": "file", "video": "watch"}[opens(db, link)]
return f"/{prefix}/{token}"
def owner_view(db: Session, link: DownloadLink, token: str | None = None) -> dict:
"""What the file owner sees when managing their links. `token` is the one just made (it is in
hand even where no copy is kept); else it is read back, and `url` is None when it cannot be."""
token = token or recover(link)
return {
"id": link.id,
"token": token,
"url": f"{base_url}/watch/{token}" if token else None,
"url": share_path(db, link, token) if token else None,
"allow_download": link.allow_download,
"has_password": link.password_hash is not None,
"expires_at": link.expires_at.isoformat() if link.expires_at else None,
+3 -1
View File
@@ -234,7 +234,9 @@ class DownloadJob(Base, TimestampMixin, UpdatedAtMixin):
source_asset_id: Mapped[int | None] = mapped_column(
ForeignKey("media_assets.id", ondelete="SET NULL")
)
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h}} in pixels. For clip
# For edit jobs: the trim/crop recipe {trim?:{start_s,end_s}, crop?:{x,y,w,h,frame_w?,frame_h?}}, its
# crop in the SHOWN picture's pixels when it carries the frame it was measured on (`edit.crop_filter`),
# else in the file's stored ones. For clip
# jobs: the clip spec {source, ref, start_s, end_s, audio_track?, crop?:{x,y,w,h,aspect}}, its crop
# in fractions of the box it was drawn in (`downloads.clip.normalize`).
edit_spec: Mapped[dict | None] = mapped_column(JSON)
+12 -7
View File
@@ -1,4 +1,4 @@
"""Server-rendered Open Graph tags for the public /watch/{token} page.
"""Server-rendered Open Graph tags for the public page of a share link (/watch/, /file/, /folder/).
The watch page is a client-side SPA, so a social crawler (facebookexternalhit, Twitterbot, …) that
fetches /watch/{token} would otherwise only see the generic index.html — a blank/greyed link card.
@@ -76,16 +76,21 @@ def _watch_tags(token: str, db: Session) -> str | None:
link = linksmod.find(db, token)
if link is None or linksmod.is_expired(link):
return None
url = f"{settings.app_base}/watch/{token}"
folder = link.folder_id is not None
url = f"{settings.app_base}{linksmod.share_path(db, link, token)}"
# What the card calls it, as its address does.
kind = linksmod.opens(db, link)
folder, video = kind == "folder", kind == "video"
if link.password_hash:
# Password-gated: don't leak the title/thumbnail in the unfurl.
return "\n ".join(
[
"<title>Siftlode</title>",
_tag("og:title", "Password-protected " + ("folder" if folder else "video")),
_tag(
"og:title",
"Password-protected " + ("folder" if folder else "video" if video else "file"),
),
_tag("og:description", "Shared via Siftlode"),
_tag("og:type", "website" if folder else "video.other"),
_tag("og:type", "video.other" if video else "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
]
@@ -96,7 +101,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
asset = db.get(MediaAsset, job.asset_id) if job and job.asset_id else None
if job is None or asset is None or asset.status != "ready":
return None
title = (job.display_name or asset.title or "Video").strip()
title = (job.display_name or asset.title or ("Video" if video else "File")).strip()
channel = (job.display_uploader or asset.uploader or "").strip()
# Pick the link-preview image. Facebook's scraper REJECTS our own poster ("Corrupted Image" in the
# Sharing Debugger): the poster is an ffmpeg-produced JPEG whose non-standard marker order FB's
@@ -118,7 +123,7 @@ def _watch_tags(token: str, db: Session) -> str | None:
f"<title>{html.escape(title)}</title>",
_tag("og:title", title),
_tag("og:description", channel or "Shared via Siftlode"),
_tag("og:type", "video.other"),
_tag("og:type", "video.other" if video else "website"),
_tag("og:url", url),
_tag("og:site_name", "Siftlode"),
_tag("twitter:title", title, attr="name"),
+41 -9
View File
@@ -7,6 +7,7 @@ from datetime import date, datetime, timezone
from pathlib import Path
import yt_dlp
from sqlalchemy import update
from yt_dlp.utils import download_range_func
from app import logs, sysconfig
@@ -16,6 +17,7 @@ from app.downloads import clip as clipmod
from app.downloads import edit as editmod
from app.downloads import errors as dlerrors
from app.downloads import adapters, credentials, formats, mediakind, playable, probe, quota
from app.downloads import backfill, egress
from app.downloads import service, storage
from app.downloads import urlguard
from app.downloads.jobdb import (
@@ -392,6 +394,7 @@ def _extract_with_fallback(
_make_progress_hook(job_id, asset_id),
cookiefile=cookiefile,
max_bytes=max_bytes,
egress=adapter.egress_guarded,
),
)
last_exc: Exception | None = None
@@ -403,6 +406,7 @@ def _extract_with_fallback(
extra=adapter.ydl_opts_extra(clients),
cookiefile=cookiefile,
max_bytes=max_bytes,
egress=adapter.egress_guarded,
)
opts["postprocessor_hooks"] = [_make_pp_hook(job_id)]
if section is not None:
@@ -413,7 +417,7 @@ def _extract_with_fallback(
# (Per-ATTEMPT only; a single extract_info that itself runs past the stall timeout is still
# killed — acceptable, as socket_timeout + extractor_retries bound one attempt well under it.)
try:
with yt_dlp.YoutubeDL(opts) as ydl:
with egress.open_ydl(opts) as ydl:
return ydl.extract_info(target, download=True)
except yt_dlp.utils.DownloadError as exc:
last_exc = exc
@@ -585,9 +589,14 @@ def _download(
# missing duration hid the trim action. Probed before the sidecars so the .nfo runtime has it
# too. Same probe the upload ingest uses.
vcodec, acodec, width, height, duration_s = probe.resolve_media_facts(produced, info)
# A placeholder title with nothing behind it ("Facebook" from a signed-in read) borrows the
# page's text from one signed-out read; the download itself stays the signed-in one. The page
# yt-dlp read, not the link queued: a group post's link is rewritten to its video, and
# signed out the post itself is the login wall.
titled = backfill.public_title_info(info, info.get("webpage_url") or url, adapter, asset_id)
meta = storage.MediaMeta(
video_id=info.get("id") or source_ref,
title=normalize_title(adapter.title(info)) or source_ref,
title=normalize_title(adapter.title(titled)) or source_ref,
uploader=info.get("uploader") or info.get("channel") or "Unknown Channel",
upload_date=_parse_upload_date(info.get("upload_date")),
uniqueid_type=adapter.nfo_uniqueid_type(),
@@ -830,6 +839,18 @@ def _fail_edit(
)
def _filter_frame(
path: Path, width: int | None, height: int | None
) -> tuple[int | None, int | None]:
"""A file's frame as ffmpeg's filters see it: the stored size, turned when the stream is stored
turned a quarter (ffmpeg stands it upright first). Unknown when the file cannot be read."""
data = probe.probe_json(path)
turned = clipmod.quarter_turned(data) if data else None
if turned is None:
return None, None
return (height, width) if turned else (width, height)
def _process_edit(job_id: int, asset_id: int) -> None:
"""Produce a per-user trim/crop clip from the source asset's downloaded file via ffmpeg."""
with SessionLocal() as db:
@@ -898,6 +919,12 @@ def _process_edit(job_id: int, asset_id: int) -> None:
size = final.stat().st_size if final.exists() else None
# The clip's real frame: a re-encode evens an odd crop or source (`editmod.EVEN_FRAME`).
_, _, out_w, out_h, _ = probe.probe_media(final)
# Unsized, a crop's own size stands in, cut from the frame the filters saw.
cut = (
editmod.crop_stored_size(edit_spec["crop"], *_filter_frame(src_path, src_w, src_h))
if crop and not (out_w and out_h)
else None
)
with SessionLocal() as db:
asset = db.get(MediaAsset, asset_id)
@@ -909,11 +936,10 @@ def _process_edit(job_id: int, asset_id: int) -> None:
asset.duration_s = total_s or asset.duration_s
if out_w and out_h:
asset.width, asset.height = out_w, out_h
elif crop:
asset.width = edit_spec["crop"]["w"]
asset.height = edit_spec["crop"]["h"]
elif cut:
asset.width, asset.height = cut
else:
asset.width, asset.height = src_w, src_h
asset.width, asset.height = (None, None) if crop else (src_w, src_h)
asset.error = None
asset.last_access_at = datetime.now(timezone.utc)
job = db.get(DownloadJob, job_id)
@@ -1116,14 +1142,20 @@ def _process_clip(job_id: int, asset_id: int) -> None:
fields: dict = {"status": JobStatus.DONE, "progress": 100, "phase": None}
# A video outside the catalog was queued as "Clip (clip)": the fetch has its title now.
placeholder = service.clip_title(None, None)
named = None
if asset and fetched_title and asset.title == placeholder:
named = service.clip_title(fetched_title, None)
asset.title = named
if job_row := db.get(DownloadJob, job_id):
if job_row.display_name == placeholder:
fields["display_name"] = named[:DISPLAY_TEXT_MAX]
title = asset.title if asset else None
finalized = service.finalize_if_running(db, job_id, **fields)
if finalized and named:
# Only a name that still repeats the placeholder, in one conditional write: one its
# owner gave meanwhile stays.
db.execute(
update(DownloadJob)
.where(DownloadJob.id == job_id, DownloadJob.display_name == placeholder)
.values(display_name=named[:DISPLAY_TEXT_MAX])
)
db.commit()
if finalized:
_notify_download_done(db, user_id, title, job_id, kind="edit")
+12 -5
View File
@@ -16,10 +16,12 @@ That is a per-job ceiling, not an aggregate ledger: concurrent jobs of one user
remainder (in-flight bytes are not `ready`, so `footprint` cannot see them), so a wave of
max_concurrent downloads can still land over the cap — the deliberate trade, since a download's size
is unknown until it finishes. CLIP jobs consult it before their fetch or encode and again with
their output's real size before it is placed. Edit jobs do not consult it at all (tracked as SEC-19).
their output's real size before it is placed. Edit jobs consult it with their output's real size
before it is placed (`test_edit_cap.py`).
"""
from dataclasses import dataclass
from typing import Literal
from sqlalchemy import and_, func, or_, select
from sqlalchemy.orm import Session
@@ -50,11 +52,16 @@ class QuotaLimits:
unlimited: bool
class QuotaExceeded(Exception):
"""Raised by check_enqueue; `reason` is one of max_jobs|max_bytes (i18n key on the client)."""
# Every cap a download can hit; the UI names each one (`downloads.errors.quota.<reason>`, held by the
# frontend's backendIds contract, which reads this Literal).
QuotaReason = Literal["max_jobs", "max_bytes"]
def __init__(self, reason: str, limit: int, current: int):
self.reason = reason
class QuotaExceeded(Exception):
"""Raised by check_enqueue; `reason` names the cap it hit (an i18n key on the client)."""
def __init__(self, reason: QuotaReason, limit: int, current: int):
self.reason: QuotaReason = reason
self.limit = limit
self.current = current
super().__init__(f"download quota exceeded: {reason} ({current}/{limit})")
+12 -6
View File
@@ -10,6 +10,7 @@ handler); the folder, link and quota writes are `folders`, `links` and `quota`.
import uuid
from datetime import datetime, timezone
from typing import Literal
from sqlalchemy import delete as delete_rows
from sqlalchemy import func, select, update
@@ -39,11 +40,16 @@ from app.plex import paths as plex_paths
from app.userscope import owned_by, owned_row
class EditError(Exception):
"""Bad edit request (empty spec / source not ready); `reason` is an i18n-mappable key."""
# Every reason an edit is refused; the UI names each one (`downloads.errors.edit.<reason>`, held by
# the frontend's backendIds contract, which reads this Literal).
EditReason = Literal["source_not_ready", "not_media", "empty_edit"]
def __init__(self, reason: str):
self.reason = reason
class EditError(Exception):
"""Bad edit request (empty spec / source not ready / not media); `reason` names the sentence."""
def __init__(self, reason: EditReason):
self.reason: EditReason = reason
super().__init__(reason)
@@ -147,7 +153,7 @@ def populate_from_catalog(db: Session, asset: MediaAsset) -> bool:
return False
asset.title = video.title
asset.thumbnail_url = video.thumbnail_url
asset.duration_s = video.duration_seconds
asset.duration_s = video.length_seconds
if video.published_at:
asset.upload_date = video.published_at.date()
channel = db.get(Channel, video.channel_id)
@@ -331,7 +337,7 @@ def _clip_source_facts(db: Session, spec: dict) -> dict:
"title": video.title,
"uploader": channel.title if channel else None,
"thumbnail_url": video.thumbnail_url,
"length_s": video.duration_seconds,
"length_s": video.length_seconds,
}
item = plex_lookups.by_rating_key(db, PlexItem, spec["ref"])
if item is None:
+25 -26
View File
@@ -10,22 +10,12 @@ What this closes: a URL whose host resolves to a loopback / private / link-local
is refused, at enqueue AND again in the worker before extraction. Both, because DNS can answer
differently between the two moments, and the worker is where the socket is actually opened.
What it does NOT close, stated plainly rather than implied, because both need the same fix — gating
the connection itself with a custom socket factory inside yt-dlp, a much larger change than this:
* DNS rebinding: a host that passes the worker's check and then answers yt-dlp's own resolution
with an internal address.
* HTTP redirects: a public host that answers 3xx with an internal ``Location``. yt-dlp follows it,
and only the URL the user TYPED was ever judged here. An attacker-controlled host or any public
open redirect reaches it, so this is the cheaper of the two. It does NOT usually yield the
response body — a non-media body fails the mp4 remux and stores nothing — but it does hand back
the readout above: the redaction in ``errors.user_error_text`` fires only for the ``network``
class, so a dead internal port (refused → ``network`` → redacted) is distinguishable from a live
one (an HTTP answer that is not media → ``unsupported_host``/``format_unavailable``, raw text
kept). Tracked as SEC-17.
The two checks here stop the straightforward case (typing ``http://127.0.0.1:6379`` or an internal
hostname into the box), which is the reachable one.
What this alone cannot see is every request yt-dlp makes AFTER the typed URL: a public host
answering 3xx with an internal ``Location``, and DNS rebinding (a name that passes this check and
then resolves inside for yt-dlp's own lookup). Those are closed one layer down, by `egress`: a guarded
download's every connection goes through a gate that judges the address it actually dials, with
this module's `is_forbidden`. The checks here stay as the early, structured refusal — a user typing
``http://127.0.0.1:6379`` hears why at once instead of from a failed job.
Not used for YouTube jobs: those resolve to a fixed public host from an 11-char id, so there is no
user-controlled destination to guard.
@@ -33,27 +23,36 @@ user-controlled destination to guard.
import ipaddress
import socket
from typing import Literal
from urllib.parse import urlparse
# Reason codes travel to the client inside the structured 422 detail, so they are part of the API
# vocabulary (frontend `localizeDetail`) — keep them stable.
REASON_SCHEME = "scheme" # not http(s)
REASON_NO_HOST = "no_host" # URL carries no hostname
REASON_MALFORMED = "malformed" # unparseable authority or port; urlparse raises late, so guard it
REASON_UNRESOLVABLE = "unresolvable" # DNS said nothing (honest: not the same as "blocked")
REASON_PRIVATE = "private" # resolves into a network the server must not fetch from
# vocabulary — keep them stable. The UI names each one (`downloads.errors.urlBlocked.<reason>`, held by
# the frontend's backendIds contract, which reads this Literal).
UrlBlockedReason = Literal["scheme", "no_host", "malformed", "unresolvable", "private"]
REASON_SCHEME: UrlBlockedReason = "scheme" # not http(s)
REASON_NO_HOST: UrlBlockedReason = "no_host" # URL carries no hostname
REASON_MALFORMED: UrlBlockedReason = (
"malformed" # unparseable authority or port; urlparse raises late
)
REASON_UNRESOLVABLE: UrlBlockedReason = (
"unresolvable" # DNS said nothing (not the same as "blocked")
)
REASON_PRIVATE: UrlBlockedReason = (
"private" # resolves into a network the server must not fetch from
)
class BlockedUrl(Exception):
"""A user-supplied URL that must not be fetched server-side. ``reason`` is one of the
REASON_* codes above."""
def __init__(self, reason: str):
def __init__(self, reason: UrlBlockedReason):
super().__init__(reason)
self.reason = reason
self.reason: UrlBlockedReason = reason
def _is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
def is_forbidden(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
"""Every address family we must not open a connection to. Checked explicitly rather than
leaning on ``is_private`` alone, whose coverage differs between IPv4 and IPv6."""
# ::ffff:127.0.0.1 is loopback wearing an IPv6 hat — unwrap before judging it.
@@ -104,5 +103,5 @@ def check_url(raw: str) -> None:
ip = ipaddress.ip_address(addr)
except ValueError: # getaddrinfo returned something unparseable; treat as unusable
raise BlockedUrl(REASON_PRIVATE) from None
if _is_forbidden(ip):
if is_forbidden(ip):
raise BlockedUrl(REASON_PRIVATE)
+118
View File
@@ -0,0 +1,118 @@
"""A weighted draw that stays a surprise: what "Surprise me" deals from.
A title's chance to come first is its weight over the pool's (Efraimidis–Spirakis: order by
-ln(1-u)/w, u uniform per row). The weight is 1 + the title's affinity to what the user watches
(`plex.affinity`, `discover.queries`), so a title nobody leans towards still comes up — only less
often. Two things keep it a surprise rather than a ranking:
- **u is the seed's md5 of the row's key**, the Home surprise's seeded order. With every weight 1
the order is exactly that md5 order, so a user with no history sees the unweighted draw, and one
seed still moves both sources of the Home hero together. A draw that is not seeded (the Plex
Discover rail) takes u from random().
- **One slot of three is blind** (`blend`): dealt from the unweighted order, so the same few
favourites cannot fill the rail.
The YouTube surprise draws over a window of the feed scope rather than all of it (`yt_surprise`):
the window is read in SQL, and `shuffled` / `leaning` order it here with the same u.
Both affinities weigh a watch by `recency` and score against the strongest with `scaled`, so the two
sources lean by one rule.
"""
import hashlib
import math
from collections.abc import Callable, Hashable
from datetime import datetime
from typing import Any
from sqlalchemy import BigInteger, Float, cast, func, literal
from sqlalchemy.dialects.postgresql import BIT
from sqlalchemy.orm import InstrumentedAttribute
from sqlalchemy.sql.elements import ColumnElement
# 2**32: the first 8 hex digits of an md5 as a fraction of one.
_SPAN = 4294967296.0
# A watch with no date (an old import) counts as one this many half-lives ago.
UNDATED_HALF_LIVES = 2
def recency(watched_at: datetime | None, now: datetime, half_life_days: float) -> float:
"""A watch's weight in an affinity: 1 now, half as much every `half_life_days`."""
if watched_at is None:
return 0.5**UNDATED_HALF_LIVES
return 0.5 ** (max(0.0, (now - watched_at).total_seconds() / 86400) / half_life_days)
def scaled(scores: dict[str, float]) -> dict[str, float]:
"""Each score over the strongest, so the strongest is 1; empty when nothing scored."""
top = max(scores.values(), default=0.0)
return {k: v / top for k, v in scores.items()} if top else {}
def seeded_u(
key: ColumnElement[Any] | InstrumentedAttribute[Any], seed: int
) -> ColumnElement[float]:
"""A uniform u in [0, 1) per row, from md5(key || seed) — the order `md5(...)` ascending gives."""
digest = func.md5(func.concat(key, str(seed)))
head = cast(cast(literal("x") + func.substr(digest, 1, 8), BIT(32)), BigInteger)
return cast(head, Float) / _SPAN
def weighted_key(u: ColumnElement[float], weight: ColumnElement[float]) -> ColumnElement[float]:
"""The row's place in a weighted draw, ascending: -ln(1-u)/w. Monotonic in u when every w is 1,
so the unweighted order is unchanged."""
return -func.ln(1.0 - u) / weight
def random_u() -> ColumnElement[float]:
return func.random()
def u_of(key: str, seed: int) -> float:
"""`seeded_u` in Python: the same u for the same key and seed."""
return int(hashlib.md5(f"{key}{seed}".encode()).hexdigest()[:8], 16) / _SPAN
def shuffled(items: list, seed: int, key: Callable[[Any], str]) -> list:
"""`items` in the seed's md5 order — the order `ORDER BY md5(key || seed)` gives."""
return sorted(items, key=lambda it: hashlib.md5(f"{key(it)}{seed}".encode()).hexdigest())
def leaning(
items: list, seed: int, key: Callable[[Any], str], weight: Callable[[Any], float]
) -> list:
"""`items` in the weighted draw's order (`weighted_key` in Python), ties in the md5 order."""
order = {key(it): i for i, it in enumerate(shuffled(items, seed, key))}
return sorted(
items,
key=lambda it: (-math.log(1.0 - u_of(key(it), seed)) / weight(it), order[key(it)]),
)
def blend(
weighted: list, blind: list, offset: int, n: int, key: Callable[[Any], Hashable] = lambda x: x
) -> list:
"""`n` picks: slot i comes from `blind` when (i + offset) % 3 == 0, else from `weighted`, each
skipping what an earlier slot took (`key` tells two picks apart). A side that runs out leaves the
rest to the other."""
out: list = []
taken: set = set()
sources = (iter(weighted), iter(blind))
def nxt(source):
for item in source:
if key(item) not in taken:
return item
return None
for i in range(n):
first, second = (sources[1], sources[0]) if (i + offset) % 3 == 0 else sources
item = nxt(first)
if item is None:
item = nxt(second)
if item is None:
break
taken.add(key(item))
out.append(item)
return out
+71 -12
View File
@@ -13,7 +13,19 @@ import binascii
import json
from datetime import date, datetime, timedelta, timezone
from sqlalchemy import BigInteger, Select, and_, case, cast, false, func, or_, select
from sqlalchemy import (
BigInteger,
Select,
and_,
case,
cast,
false,
func,
literal,
or_,
select,
tuple_,
)
from sqlalchemy.orm import Session, aliased
from app.collation import collated
@@ -68,7 +80,7 @@ def feed_columns(user: User, state) -> list:
Channel.handle.label("channel_handle"),
Video.published_at,
Video.thumbnail_url,
Video.duration_seconds,
Video.length_seconds.label("length_seconds"),
Video.view_count,
Video.is_short,
Video.live_status,
@@ -88,7 +100,7 @@ def serialize(row) -> dict:
"channel_url": _channel_url(row.channel_id, row.channel_handle),
"published_at": iso(row.published_at),
"thumbnail_url": row.thumbnail_url,
"duration_seconds": row.duration_seconds,
"duration_seconds": row.length_seconds,
"view_count": row.view_count,
"is_short": row.is_short,
"live_status": row.live_status,
@@ -260,9 +272,9 @@ def filtered_query(
# OR across the picked channels; every other filter still applies on top.
query = query.where(Video.channel_id.in_(channel_ids))
if min_duration is not None:
query = query.where(Video.duration_seconds >= min_duration)
query = query.where(Video.length_seconds >= min_duration)
if max_duration is not None:
query = query.where(Video.duration_seconds <= max_duration)
query = query.where(Video.length_seconds <= max_duration)
if max_age_days is not None:
cutoff = datetime.now(timezone.utc).timestamp() - max_age_days * 86400
query = query.where(Video.published_at >= datetime.fromtimestamp(cutoff, tz=timezone.utc))
@@ -503,10 +515,10 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
"views": [{"expr": Video.view_count, "asc": False, "nullable": True, "kind": "num"}],
"views_asc": [{"expr": Video.view_count, "asc": True, "nullable": True, "kind": "num"}],
"duration_desc": [
{"expr": Video.duration_seconds, "asc": False, "nullable": True, "kind": "num"}
{"expr": Video.length_seconds, "asc": False, "nullable": True, "kind": "num"}
],
"duration_asc": [
{"expr": Video.duration_seconds, "asc": True, "nullable": True, "kind": "num"}
{"expr": Video.length_seconds, "asc": True, "nullable": True, "kind": "num"}
],
"title": [{"expr": title, "asc": True, "nullable": True, "kind": "str"}],
"title_desc": [{"expr": title, "asc": False, "nullable": True, "kind": "str"}],
@@ -536,15 +548,17 @@ def sort_keys(sort: str, seed: int, rank_expr=None, lang: str | None = None) ->
},
pub,
],
# Deterministic per-seed shuffle: the same seed yields the same total order, so
# the cursor stays valid across pages of one shuffled run.
# The ring of stored random keys from the seed's point (`shuffle_walk`): the keys at or after
# it, then the ones before. The same seed yields the same total order, so the cursor stays
# valid across pages; `feed_page` reads it as two index ranges, never as this sort.
"shuffle": [
{
"expr": func.md5(func.concat(Video.id, str(seed))),
"expr": case((Video.shuffle_key < shuffle_start(seed), 1), else_=0),
"asc": True,
"nullable": False,
"kind": "str",
}
"kind": "num",
},
{"expr": Video.shuffle_key, "asc": True, "nullable": False, "kind": "num"},
],
}
# Relevance (search-result ranking) only exists when there's a query to rank against;
@@ -700,6 +714,43 @@ class BadCursor(ValueError):
"""A `cursor` this module did not write (or one from another sort): the caller's bad input."""
# The golden ratio's fraction: consecutive seeds land far apart on the ring, never next to each other.
_GOLDEN = 0.6180339887498949
def shuffle_start(seed: int) -> float:
"""The point in [0, 1) where a seed's shuffle starts on the ring of `Video.shuffle_key`.
Folded to 32 bits first: an int past a float's range overflowed the product, and one past 2**53
left the product no fraction to land on. A seed the client draws (under a billion) is unchanged."""
return ((seed % 2**32) * _GOLDEN) % 1.0
def shuffle_walk(
db: Session, query: Select, seed: int, n: int, after: tuple[int, float, str] | None = None
) -> list:
"""Up to `n` rows of `query` in the seed's ring order: shuffle_key from the seed's point upwards,
then from 0 up to that point. `after` is a cursor's (phase, key, id): the walk resumes past it.
Each arc is an ascending range read along ix_videos_shuffle_key with the scope's filters checked
row by row, so a draw costs the rows it reads, not a sort of the scope — the md5(id || seed) order
it replaced read and sorted every row the filters let through (~350 ms on prod for ~213k)."""
start = shuffle_start(seed)
phase_from, key, last_id = after if after is not None else (0, None, None)
rows: list = []
for phase in (0, 1):
if phase < phase_from or len(rows) >= n:
continue
arc = Video.shuffle_key >= start if phase == 0 else Video.shuffle_key < start
q = query.where(arc)
if key is not None and phase == phase_from:
q = q.where(
tuple_(Video.shuffle_key, Video.id) > tuple_(literal(key), literal(last_id))
)
rows += db.execute(q.order_by(Video.shuffle_key, Video.id).limit(n - len(rows))).all()
return rows
def feed_page(
db: Session,
user: User,
@@ -734,6 +785,14 @@ def feed_page(
k["value"] = v
if ranked and window is not None:
rows = windowed_relevance_rows(db, query, keys, window, cursor_id, limit + 1)
elif sort == "shuffle":
after = None
if vals is not None and cursor_id is not None:
try:
after = (int(vals[0]), float(vals[1]), str(cursor_id))
except (TypeError, ValueError, OverflowError) as e: # Infinity, a 400-digit key
raise BadCursor("Invalid feed cursor") from e
rows = shuffle_walk(db, query, seed, limit + 1, after)
else:
query = query.order_by(*order_by(keys))
if cursor_id is not None:
+11 -5
View File
@@ -151,7 +151,7 @@ async def _reap_hls_sessions() -> None:
somebody pressed play again or the process restarted.
A FINISHED session is reaped on the short rule here, not kept for the full idle window: an open
player checkpoints every 10 s and that route touches the session, so a paused viewer is no longer
player keeps checkpointing and that route touches the session, so a paused viewer is no longer
invisible to this pass (see `reap_idle`). Keeping every finished session for ten minutes instead
held one whole segmented copy of a film per item on the download volume."""
while True:
@@ -368,6 +368,10 @@ INDEX_HTML = STATIC_DIR / "index.html"
# A public share link's own page: the player alone, without the app's bundle (frontend/watch.html).
WATCH_HTML = STATIC_DIR / "watch.html"
INDEX_HEADERS = {"Cache-Control": "no-cache"}
# The paths a share link opens on, by what it opens (`links.share_path`). Any of them opens any link.
# A new one also goes in frontend/public/robots.txt, the Vite dev entry and WatchPage's token parsing
# (watchEntry.contract.test.ts).
SHARE_PREFIXES = ("watch/", "file/", "folder/")
app.mount(
"/assets",
StaticFiles(directory=STATIC_DIR / "assets", check_dir=False),
@@ -381,8 +385,10 @@ async def index() -> FileResponse:
@app.get("/watch/{token}")
async def watch_page(token: str):
"""Serve the public player page for a share link, with per-video Open Graph tags injected so the
@app.get("/file/{token}")
@app.get("/folder/{token}")
async def share_page(token: str):
"""Serve the public page of a share link, with per-link Open Graph tags injected so the
link unfurls richly (title/channel/thumbnail) in Messenger/social crawlers. A real browser
ignores the extra tags and boots WatchPage; invalid/expired/password links fall back to the
generic card."""
@@ -409,8 +415,8 @@ async def spa_fallback(full_path: str) -> FileResponse:
# Real SPA-root assets (welcome images, favicon, robots.txt) rarely change and have
# stable names, so a moderate cache is safe and satisfies the cache-lifetime audit.
return FileResponse(candidate, headers={"Cache-Control": "public, max-age=604800"})
# A /watch/ path the route above does not match (a trailing segment, no token) is still the
# A share path the route above does not match (a trailing segment, no token) is still the
# player's: WatchPage reads the token off the path and says when a link is gone.
if full_path.startswith("watch/"):
if full_path.startswith(SHARE_PREFIXES):
return FileResponse(WATCH_HTML, headers=INDEX_HEADERS)
return FileResponse(INDEX_HTML, headers=INDEX_HEADERS)
+110
View File
@@ -0,0 +1,110 @@
"""What a user's Plex history leans towards, as a weight for a surprise draw (`app.draw`).
Two signals, both already in the database (no TMDB call):
- **genres** of the titles the user watched, the recent ones weighing more (a half-life of
HALF_LIFE_DAYS), each scored 0..1 against the strongest;
- the **TMDB recommendations** stored on those titles (`recommendations`), each scored by the most
recent title that recommends it.
A title counts once however many of its episodes were watched — a binge of one show would otherwise
outvote every film — and only the RECENT_TITLES latest titles count. A candidate's weight is
1 + GENRE_WEIGHT × (the mean score of its genres) + REC_WEIGHT × (its recommendation score), so a
title with no affinity keeps weight 1 and still comes up.
"""
import json
from dataclasses import dataclass, field
from datetime import datetime, timezone
from sqlalchemy import Float, cast, func, literal, literal_column, select
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Session
from sqlalchemy.sql.elements import ColumnElement
from app import draw
from app.plex.models import PlexItem, PlexShow, PlexState
from app.userscope import owned_by
HALF_LIFE_DAYS = 180
RECENT_TITLES = 200
GENRE_WEIGHT = 2.0
REC_WEIGHT = 3.0
@dataclass(frozen=True)
class Affinity:
genres: dict[str, float] = field(default_factory=dict)
movie_recs: dict[str, float] = field(default_factory=dict)
show_recs: dict[str, float] = field(default_factory=dict)
def __bool__(self) -> bool:
return bool(self.genres or self.movie_recs or self.show_recs)
def weight(self, model: type[PlexItem] | type[PlexShow]) -> ColumnElement[float]:
"""The draw weight of a `model` row (a movie or a show)."""
scores = cast(literal(json.dumps(self.genres)), JSONB)
recs = self.show_recs if model is PlexShow else self.movie_recs
genre = func.jsonb_array_elements_text(model.genres).table_valued("value").alias("g")
genre_aff = (
select(
func.coalesce(
func.avg(func.coalesce(cast(scores.op("->>")(genre.c.value), Float), 0.0)),
0.0,
)
)
.select_from(genre)
.scalar_subquery()
)
rec_aff = func.coalesce(
cast(cast(literal(json.dumps(recs)), JSONB).op("->>")(model.guid), Float), 0.0
)
return literal_column("1.0") + GENRE_WEIGHT * genre_aff + REC_WEIGHT * rec_aff
def for_user(db: Session, uid: int) -> Affinity:
"""The user's affinity from their latest watched titles; empty (`not aff`) with no history."""
# One key per title: a show for an episode, the item for a film. concat() of a NULL show_id is
# just "s", which nullif turns back into NULL so the film's key takes over.
title = func.coalesce(
func.nullif(func.concat("s", PlexItem.show_id), "s"), func.concat("m", PlexItem.id)
).label("title")
latest = (
select(title, PlexState.watched_at, PlexItem.id.label("item_id"), PlexItem.show_id)
.join(PlexItem, PlexItem.id == PlexState.item_id)
.where(owned_by(PlexState.user_id, uid), PlexState.status == "watched")
.distinct(title)
.order_by(title, PlexState.watched_at.desc().nullslast())
.subquery()
)
rows = db.execute(
select(
latest.c.watched_at,
PlexItem.genres,
PlexItem.recommendations,
PlexShow.genres,
PlexShow.recommendations,
)
.select_from(latest)
.join(PlexItem, PlexItem.id == latest.c.item_id)
.outerjoin(PlexShow, PlexShow.id == latest.c.show_id)
.order_by(latest.c.watched_at.desc().nullslast())
.limit(RECENT_TITLES)
).all()
if not rows:
return Affinity()
now = datetime.now(timezone.utc)
genres: dict[str, float] = {}
movie_recs: dict[str, float] = {}
show_recs: dict[str, float] = {}
for watched_at, item_genres, item_recs, show_genres, show_recs_json in rows:
r = draw.recency(watched_at, now, HALF_LIFE_DAYS)
for g in (show_genres if show_genres is not None else item_genres) or []:
genres[g] = genres.get(g, 0.0) + r
for rec in (show_recs_json if show_recs_json is not None else item_recs) or []:
if not rec.get("tmdb_id"):
continue
bucket = show_recs if rec.get("media_type") == "tv" else movie_recs
guid = f"tmdb:{rec['tmdb_id']}"
bucket[guid] = max(bucket.get(guid, 0.0), r)
return Affinity(draw.scaled(genres), draw.scaled(movie_recs), draw.scaled(show_recs))
+21 -13
View File
@@ -22,7 +22,8 @@ from sqlalchemy.orm import Session
from app.collation import sorted_words
from app.discover.queries import RAIL_LIMIT
from app.models import User
from app.plex import rollup
from app import draw
from app.plex import affinity, rollup
from app.plex.cards import SHOW_NEW, display_title, leaf_card, movie_card, show_card
from app.plex.models import PlexItem, PlexShow, PlexState
from app.plex.query import (
@@ -399,22 +400,25 @@ def discover_page(db: Session, user: User, scope: str, lang: str, f: LibraryFilt
}
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str]) -> list:
def _draw(db: Session, stmt, key_col, per: int, exclude: list[str], weight=None) -> list:
"""`per` random rows of `stmt`, the previous draw's (`exclude`) left out while the pool allows: a
reshuffle over a small filtered pool kept dealing the same few titles. What the fresh rows leave
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail."""
fresh = (
db.execute(stmt.where(key_col.not_in(exclude)).order_by(func.random()).limit(per))
.scalars()
.all()
if exclude
else []
)
short is topped up from the excluded ones, so a pool smaller than a draw still fills the rail.
With a `weight` (`plex.affinity`) the draw leans on it, every third slot left blind (`app.draw`)."""
def pick(q, n: int) -> list:
blind = db.execute(q.order_by(func.random()).limit(n)).scalars().all()
if weight is None:
return list(blind)
key = draw.weighted_key(draw.random_u(), weight)
leaning = db.execute(q.order_by(key).limit(n)).scalars().all()
return draw.blend(list(leaning), list(blind), 1, n, key=lambda r: r.rating_key)
fresh = pick(stmt.where(key_col.not_in(exclude)), per) if exclude else []
if len(fresh) == per:
return list(fresh)
return fresh
taken = [key_col.not_in([r.rating_key for r in fresh])] if fresh else []
rest = db.execute(stmt.where(*taken).order_by(func.random()).limit(per - len(fresh))).scalars()
return [*fresh, *rest.all()]
return [*fresh, *pick(stmt.where(*taken), per - len(fresh))]
def _kept(
@@ -474,6 +478,8 @@ def surprise(
if keep:
return {"scope": scope, "items": _kept(db, user.id, movie_ids, show_ids, keep, lang)}
per = min(limit, SURPRISE_PER_KIND)
# What the user's history leans towards; with none, every title weighs the same (a plain draw).
leaning = affinity.for_user(db, user.id)
movies = []
if movie_ids:
# Exclude movies this user has watched.
@@ -501,6 +507,7 @@ def surprise(
PlexItem.rating_key,
per,
exclude,
leaning.weight(PlexItem) if leaning else None,
)
shows = []
if show_ids:
@@ -523,6 +530,7 @@ def surprise(
PlexShow.rating_key,
per,
exclude,
leaning.weight(PlexShow) if leaning else None,
)
mcards = _movie_cards(db, user.id, movies, lang)
scards = _show_cards(db, user.id, shows, lang)
+9 -8
View File
@@ -227,16 +227,17 @@ class PlexItem(Base, TimestampMixin, UpdatedAtMixin):
playable: Mapped[str | None] = mapped_column(String(12), index=True)
probed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Transcode HW-capability facet (the playback-class filter). NULL for non-transcode items; for a
# `playable="transcode"` item: True = the prod iGPU can hardware-transcode it (codec is HW-decodable
# AND the source is 8-bit SDR), False = software libx264 only (AV1/XviD, or a 10-bit/HDR source that
# breaks the VAAPI nv12 pipeline — see plex/stream.py). Derived by `plex.sync.transcode_hw_capable`:
# set codec-only at sync (bit depth unknown from the cheap listing), then refined by the bit-depth
# backfill (`_probe_media_batch`) which downgrades a 10-bit/HDR HEVC to False. The one filterable
# home of the invariant `_want_hw`'s source gate computes at play time — kept in sync via the shared
# classifier so the catalog facet and the runtime encoder choice never disagree by codec.
# `playable="transcode"` item: True = the prod iGPU ENCODES it (a HW-decodable 8-bit SDR source, a
# true-HDR one through the OpenCL tonemap, or an 8-bit source in a codec the CPU decodes for it —
# VC-1, XviD, DivX), False = software libx264 (AV1, or a 10-bit SDR source — see plex/stream.py).
# Derived by `plex.sync.transcode_hw_capable`: set codec-only at sync (bit depth unknown from the
# cheap listing), then refined by the bit-depth backfill (`_probe_media_batch`), which downgrades a
# 10-bit SDR HEVC to False. The filterable twin of the rule `_select_encode` applies at play time;
# the two share their codec sets, not their code, so a change to one tree is a change to both.
transcode_hw: Mapped[bool | None] = mapped_column(Boolean, index=True)
# Video bit depth (8/10/12…) from the Plex media Stream, filled by the bit-depth backfill. NULL =
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw`.
# not yet probed. Surfaced on the info page (e.g. "HEVC · 10-bit") and feeds `transcode_hw` (a
# 10-bit SDR source drops to software; an HDR one keeps the GPU through the tonemap).
video_bit_depth: Mapped[int | None] = mapped_column(Integer)
# Watermark for the bit-depth backfill: when this item's media Stream was last probed for
# bit-depth/HDR. NULL = never (the backfill's work-list). Only HW-decodable transcode items are
+30 -9
View File
@@ -6,9 +6,9 @@ The rest of the mirror's writes stay with the sync (`plex.sync`): a collection's
after a write-back, its local cleanup after a delete.
"""
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import CursorResult, delete, func, select, update
from sqlalchemy import CursorResult, delete, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
@@ -271,6 +271,10 @@ def reorder_playlist(db: Session, pl: PlexPlaylist, rating_keys: list[str]) -> N
# --- a user's watch state --------------------------------------------------------------------
# How long after a watched mark a mid-item position is read as the finishing player's late tick
# (`checkpoint`): the tick is every 10 s, plus the request's own latency and a retry.
_LATE_CHECKPOINT_S = 30
def mark_finished(db: Session, user_id: int, item_id: int, now: datetime) -> bool:
"""Mark the item watched (position cleared) and say whether THIS call made it so. Two near-end
@@ -352,16 +356,33 @@ def checkpoint(
now: datetime,
*,
dirty: bool,
) -> PlexState:
"""Record a resume position. `dirty` marks it not yet mirrored to Plex — set before a push, so
the reconcile re-pushes it if the push fails and the push can settle it once Plex confirms."""
) -> bool:
"""Record a resume position, and say whether it was recorded. `dirty` marks it not yet mirrored
to Plex — set before a push, so the reconcile re-pushes it if the push fails and the push can
settle it once Plex confirms.
Not onto a row marked watched in the last `_LATE_CHECKPOINT_S`: that position is the player's
tick sent just before its own finish (or a mark from another tab), and writing it made a watched
item in progress again — on Plex too, through the resume push it carried. Decided by the write
itself, since the mark can commit between any read here and the write. A rewatch is checkpointed
as before once the mark is that far behind it."""
st = state_row(db, user, item_id, st)
st.position_seconds = position
st.progress_updated_at = now
values: dict[str, object] = {"position_seconds": position, "progress_updated_at": now}
if dirty:
st.synced_to_plex = False
values["synced_to_plex"] = False
not_just_finished = or_(
PlexState.status != "watched",
PlexState.watched_at.is_(None),
PlexState.watched_at <= now - timedelta(seconds=_LATE_CHECKPOINT_S),
)
result = db.execute(
update(PlexState)
.where(PlexState.id == st.id, not_just_finished)
.values(values)
.execution_options(synchronize_session="fetch")
)
db.flush()
return st
return isinstance(result, CursorResult) and result.rowcount > 0
def forget_state(db: Session, st: PlexState) -> None:
+265 -129
View File
@@ -4,9 +4,10 @@ Playback is from the LOCAL physical file. Browser-compatible files (playable="di
raw with HTTP range requests. Everything else that is h264 video (playable="remux") is remuxed on
the fly to HLS with **video stream-copy** (cheap, I/O-bound — no video re-encode) and audio to AAC
when needed. Non-h264 video (playable="transcode": HEVC/VP9/AV1/MPEG-2/XviD…) needs a full re-encode
to H.264: on the prod host that runs on the Intel iGPU via VAAPI (`h264_vaapi`, near-zero CPU), with
software libx264 as the automatic fallback where the render node is absent or the source codec isn't
hardware-decodable. Same HLS session/seek machinery either way — only the video encoder differs.
to H.264: on the prod host that is encoded on the Intel iGPU via VAAPI (`h264_vaapi`) — decoded there
too when it can, else decoded on the CPU and uploaded (VC-1, XviD, DivX) — with software libx264 as
the fallback where the render node is absent, for a source neither GPU route takes (AV1, 10-bit SDR),
and when a GPU session produces nothing. Same HLS session/seek machinery either way.
Seek model (Jellyfin-style): one ffmpeg session per item, started at a given offset via `-ss`
(fast keyframe seek). A seek beyond the generated region restarts the session at the new offset,
@@ -35,7 +36,7 @@ from app.config import settings
from app.downloads.edit import encode_thread_cap
from app.plex.models import PlexItem
from app.plex import paths
from app.plex.sync import HDR_TRANSFERS, HW_DECODABLE_CODECS
from app.plex.sync import CPU_DECODE_CODECS, HDR_TRANSFERS, HW_DECODABLE_CODECS
log = logs.get_logger(__name__)
_HLS_ROOT = (
@@ -66,7 +67,9 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
# restart leaves them behind like any other. `sweep_orphans` deletes ONLY names
# matching one of those two shapes: `_HLS_ROOT` is admin-configurable (PLEX_HLS_DIR), so it may
# legitimately point at a shared scratch path we do not own, and a blanket "delete every directory
# under the root" would take the neighbours with it.
# under the root" would take the neighbours with it. A torn-down session's directory is renamed to
# its own name + `_TOMB` + 8 hex digits before it is deleted (see `_detach`), so a crash between the
# two leaves a name this sweep still recognises.
#
# The KEY is anchored to digits (a Plex rating_key is a number) — that is what keeps foreign names
# out. A `.+` there let `snapshot_2024_backup_+1.00` through, and pairing `.+` with an optional tail
@@ -74,8 +77,10 @@ _SESSION_IDLE_S = 600 # reap a session with no access for this long
# non-numeric rating_key would merely leave its directory unswept, which is the safe direction to
# fail. Covered by test_plex_stream.py — this regex is the only thing standing between the sweep
# and someone else's data.
_TOMB = ".dead-"
_SESSION_DIR_RE = re.compile(
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{2}$"
r"^\d+_\d+(_[^_]+_[+-]\d+\.\d{2})?$"
rf"|^\d+_u\d+_\d+_[^_]+_[+-]\d+\.\d{{2}}({re.escape(_TOMB)}[0-9a-f]{{8}})?$"
)
# When this process started. `sweep_orphans` only deletes directories older than this: everything
# newer belongs to a session THIS run created, and since `start_session` reuses a directory path
@@ -93,7 +98,7 @@ _MTIME_GRACE_S = 2.0
_lock = threading.Lock()
# Keyed by VIEWER and item, (user_id, rating_key). Keyed by the item alone, a second account opening
# the same film `_kill`ed the first one's session: the displaced viewer got a 404, reclaimed at its
# the same film tore down the first one's session: the displaced viewer got a 404, reclaimed at its
# playhead, and so evicted the other in turn — two people could not watch one film at once. (One
# account on two devices still shares a session; that case ends in the player's bounded reclaim.)
_sessions: dict[tuple[int, str], "HlsSession"] = {}
@@ -116,12 +121,13 @@ class HlsSession:
hw: bool = False,
tonemap: bool = False,
user_id: int = 0,
cpu_decode: bool = False,
):
# Unguessable per-session id, carried in the URLs the client is handed. A seek-restart
# replaces the viewer's registered session for the item — and the old player's next request
# for `seg_12.ts`, resolved by (viewer, rating_key) alone, was answered 200 with the NEW session's twelfth segment, produced from a different offset
# under `-copyts`. Wrong bytes, not an error, so the client's reclaim never fired on it.
# A request naming a session that is no longer registered is a 404,
# A request naming a session that is no longer registered is a 410,
# which the player recovers from by restarting at the playhead.
self.id = secrets.token_urlsafe(9)
self.key = key # the item's rating_key (what the logs name)
@@ -131,6 +137,7 @@ class HlsSession:
self.transcode = transcode # video re-encoded (non-h264 source), not stream-copied
self.hw = hw # transcode ran on the VAAPI hardware encoder (else libx264 software)
self.tonemap = tonemap # HW HDR/10-bit path: OpenCL tonemap between VAAPI decode and encode
self.cpu_decode = cpu_decode # HW encode of frames the CPU decoded (a codec the GPU can't)
self.start_s = start_s # the REQUESTED seek offset
# The REAL media start: with `-ss X -c:v copy` ffmpeg can only cut at the nearest earlier
# keyframe K<=X, and hls.js zero-bases the timeline to K (so video.currentTime=0 is K, not X).
@@ -184,8 +191,11 @@ _HLS_TAIL = [
# * VAAPI hardware (`h264_vaapi`) — the prod default once the iGPU is passed in. A full GPU pipeline
# (decode + scale + encode all on-device) offloads the CPU almost entirely, so several concurrent
# 1080p transcodes are fine and the API stays responsive.
# * CPU decode → VAAPI encode (`format=nv12,hwupload` + `h264_vaapi`) — a codec the GPU cannot decode
# but whose 8-bit picture it encodes (`CPU_DECODE_CODECS`: VC-1, XviD, DivX). The encode is the
# expensive half, so this still takes most of the load off the CPU.
# * libx264 software — the automatic fallback where `/dev/dri` is absent (dev box, or prod before the
# iGPU enablement) or the source codec isn't hardware-decodable (AV1/XviD on Gen 9.5). CPU-bound, so
# iGPU enablement) or the source fits neither GPU route (AV1, a 10-bit SDR source). CPU-bound, so
# it reserves a core via `-threads` (see encode_thread_cap).
# Everything else — the `-copyts`/`_HLS_TAIL`/segment/seek-restart model and the AAC audio path — is
# unchanged, so hls.js, subtitles, multi-audio and seeking keep working exactly as for remux.
@@ -194,10 +204,10 @@ _VAAPI_DEVICE = "/dev/dri/renderD128"
# here; its presence is the precondition for the HW HDR→SDR tonemap graph (see _opencl_present).
_OPENCL_ICD_DIR = Path("/etc/OpenCL/vendors")
# Codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for the
# full-GPU VAAPI pipeline. A source outside it (AV1, MPEG-4 ASP/XviD…) falls back to software even with
# the render node present, because the GPU can't decode it to feed its own encoder. THE single source is
# plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't disagree by
# codec; aliased to the old name to keep the call sites below unchanged.
# full-GPU VAAPI pipeline. A source outside it is decoded on the CPU when its codec is in
# `CPU_DECODE_CODECS` (the GPU still encodes), else transcoded in software. THE single source of both
# sets is plex.sync, so the runtime encoder choice here and the catalog `transcode_hw` facet can't
# disagree by codec; aliased to the old name to keep the call sites below unchanged.
_HW_DECODABLE = HW_DECODABLE_CODECS
# Color-transfer values that mark a true-HDR source (PQ / HLG). A HW-decodable HDR source can't take the
# plain scale_vaapi=format=nv12 path (Gen 9.5 VPP refuses it) — it needs the OpenCL tonemap graph
@@ -298,16 +308,22 @@ def _opencl_present() -> bool:
return False
def _hwaccel_in_opts(hw: bool, tonemap: bool = False) -> list[str]:
def _hwaccel_in_opts(hw: bool, tonemap: bool = False, cpu_decode: bool = False) -> list[str]:
"""VAAPI decode options, placed before `-i` so the source is decoded (and kept) on the GPU.
`-hwaccel_output_format vaapi` keeps frames as GPU surfaces all the way to `h264_vaapi` — no
round-trip to system memory. Empty for the software path.
The tonemap variant additionally derives an OpenCL device FROM the VAAPI one (`opencl=ocl@va`) so
tonemap_opencl can map the decoded surfaces zero-copy (see _venc_opts); it needs named devices, so it
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`."""
uses `-init_hw_device` + `-hwaccel_device` rather than the plain path's implicit `-vaapi_device`.
The CPU-decode variant opens the VAAPI device for the FILTER graph only (`hwupload` needs one) and
asks for no `-hwaccel`: the codec is one the GPU refuses to decode. This exact form is the one
measured on prod with the HLS muxer and `-copyts` (and to the end of the file)."""
if not hw:
return []
if cpu_decode:
return ["-init_hw_device", f"vaapi=va:{_VAAPI_DEVICE}", "-filter_hw_device", "va"]
if tonemap:
return [
"-init_hw_device",
@@ -355,6 +371,7 @@ def _venc_opts(
preset: str = "veryfast",
crf: int = 20,
max_height: int = 0,
cpu_decode: bool = False,
) -> list[str]:
"""The video-encoder args that replace `-c:v copy` for a transcode session (returns the copy args
when not transcoding). A re-encode must MINT its own keyframes at the segment boundaries — the copy
@@ -381,6 +398,10 @@ def _venc_opts(
"h264_vaapi",
*force_kf,
]
if hw and cpu_decode:
# The CPU decoded the frames: convert them to the encoder's 8-bit nv12 and upload them to the
# device `-filter_hw_device` named (see _hwaccel_in_opts).
return ["-vf", "format=nv12,hwupload", "-c:v", "h264_vaapi", *force_kf]
if hw:
# scale_vaapi with format=nv12 downconverts a 10-bit (p010) HEVC source to 8-bit nv12 so
# h264_vaapi (8-bit High) can encode it; for an 8-bit source it's a cheap format pass-through.
@@ -418,13 +439,18 @@ def _render_node_present() -> bool:
return Path(_VAAPI_DEVICE).exists()
def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> tuple[bool, bool]:
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap)`:
* `(False, False)` → software libx264 (the fallback: no render node, non-decodable source, or a
10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
* `(True, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR source.
* `(True, True)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG) source, when
the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
def _select_encode(
db: DbSession, item: PlexItem, transcode: bool, src: Path
) -> tuple[bool, bool, bool]:
"""Pick the transcode video pipeline for this session — returns `(hw, tonemap, cpu_decode)`:
* `(False, False, False)` → software libx264 (the fallback: no render node, a source neither GPU
route takes, or a 10-bit/HDR source with no OpenCL runtime to tonemap on the GPU).
* `(True, False, False)` → full-GPU VAAPI (`scale_vaapi=format=nv12` + `h264_vaapi`) — an 8-bit SDR
source the GPU decodes.
* `(True, True, False)` → VAAPI decode → OpenCL tonemap → VAAPI encode — a true-HDR (PQ/HLG)
source, when the OpenCL runtime is present (Gen 9.5 VPP can't tonemap natively).
* `(True, False, True)` → CPU decode → `hwupload` → VAAPI encode — an 8-bit SDR source in a codec
the GPU cannot decode (`CPU_DECODE_CODECS`).
A 10-bit *SDR* source is deliberately NOT a `(True, True)` case: it needs a plain 10→8-bit downconvert,
not the HDR tone curve (which would visibly darken/flatten an SDR picture), so it drops to software.
@@ -434,31 +460,40 @@ def _select_encode(db: DbSession, item: PlexItem, transcode: bool, src: Path) ->
backstops a session that produces nothing — but still routing a true-HDR source through the tonemap
graph so it doesn't hit the scale_vaapi wall); "auto" (default) → HW only when the render node is
present AND the iGPU can hardware-decode the source, with tonemap chosen only for a true-HDR (PQ/HLG)
source that has an OpenCL runtime, while a 10-bit SDR source drops to software."""
source that has an OpenCL runtime, while a 10-bit SDR source drops to software. In both "on" and
"auto" a codec the GPU is known not to decode goes the CPU-decode route: asking the GPU to decode it
only bought the fallback's wasted second before libx264."""
sw = (False, False, False)
if not transcode:
return (False, False)
return sw
mode = sysconfig.get_str(db, "plex_transcode_hw")
if mode == "off" or not _render_node_present():
return (False, False)
return sw
codec = (item.codec_video or "").lower()
if mode == "on":
if codec in CPU_DECODE_CODECS:
return (True, False, True)
# Trust the admin: HW wherever the render node exists, skipping the decodability gate (the runtime
# backstop catches a miss). Only a true-HDR (PQ/HLG) source needs the tonemap graph — a 10-bit SDR
# source takes the plain scale path here, NOT the HDR tone curve. Probe only when OpenCL exists
# (short-circuit), so "on" stays probe-free where tonemap can't apply anyway.
tonemap = _opencl_present() and _probe_video_kind(src) == _VK_HDR
return (True, tonemap)
return (True, tonemap, False)
# auto
if (item.codec_video or "").lower() not in _HW_DECODABLE:
return (False, False)
if codec in CPU_DECODE_CODECS:
# The upload converts to 8-bit nv12: only an 8-bit SDR picture comes through it unchanged.
return (True, False, True) if _probe_video_kind(src) == _VK_HWSAFE else sw
if codec not in _HW_DECODABLE:
return sw
kind = _probe_video_kind(src)
if kind == _VK_HWSAFE:
return (True, False) # 8-bit SDR → full-GPU scale path
return (True, False, False) # 8-bit SDR → full-GPU scale path
if kind == _VK_HDR:
# PQ/HLG: HW OpenCL tonemap when the runtime is present, else software libx264.
return (True, True) if _opencl_present() else (False, False)
return (True, True, False) if _opencl_present() else sw
# _VK_SDR10 — 10-bit SDR: a plain 10→8-bit downconvert, done exactly on the CPU. Never the HDR tone
# curve (it would alter an SDR picture), and off the unverified scale_vaapi 10-bit path — software.
return (False, False)
return sw
def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
@@ -467,16 +502,17 @@ def _want_hw(db: DbSession, item: PlexItem, transcode: bool, src: Path) -> bool:
return _select_encode(db, item, transcode, src)[0]
# CPU-nice increment for the SOFTWARE transcode child. The Plex ffmpeg is a Popen in the long-lived API
# process (which must NOT itself be niced), so — unlike the download worker's _deprioritize (run once in
# its own main()) — we lower priority per-child via preexec_fn in the fork. The GPU path needs none of
# this. Reuses the download child's nice default so all co-located re-encoders share one handicap.
# CPU-nice increment for a transcode child that works the CPU: a software encode, or a CPU decode
# feeding the GPU encoder. The Plex ffmpeg is a Popen in the long-lived API process (which must NOT
# itself be niced), so — unlike the download worker's _deprioritize (run once in its own main()) — we
# lower priority per-child via preexec_fn in the fork. The full-GPU path needs none of this. Reuses
# the download child's nice default so all co-located re-encoders share one handicap.
_SW_TRANSCODE_NICE = settings.download_child_nice
def _sw_child_preexec() -> None:
"""Runs in the forked child (POSIX) before exec: renice this software-transcode ffmpeg down so it
can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
"""Runs in the forked child (POSIX) before exec: renice this CPU-bound transcode ffmpeg down so
it can't starve the co-located API / Postgres / playback. Best-effort — a failed renice must not fail
playback."""
try:
os.nice(_SW_TRANSCODE_NICE)
@@ -536,13 +572,15 @@ def _ffmpeg_cmd(
crf: int = 20,
max_height: int = 0,
loudnorm: bool = False,
cpu_decode: bool = False,
) -> list[str]:
# For a transcode session the video is re-encoded (VAAPI or libx264) instead of stream-copied; the
# hwaccel decode options go before -i, the encoder args replace `-c:v copy` below. Everything else is
# identical to remux. `tonemap` is the HDR/10-bit HW variant (OpenCL tonemap between VAAPI decode and
# encode) — it needs matching decode-side (named devices) and encode-side (the OpenCL filter graph) opts.
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height)
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap)]
# `cpu_decode` is the HW encode of CPU-decoded frames: a device for the filter graph, no -hwaccel.
venc = _venc_opts(transcode, hw, tonemap, preset, crf, max_height, cpu_decode)
args = ["ffmpeg", "-nostdin", "-loglevel", "error", *_hwaccel_in_opts(hw, tonemap, cpu_decode)]
# Keep ORIGINAL timestamps on the output so the first segment carries the true absolute PTS of the
# keyframe ffmpeg actually seeked to. hls.js still zero-bases playback to that PTS; we read it back
# (see _probe_first_pts) to learn the real content offset. Without -copyts the segment PTS is
@@ -641,29 +679,82 @@ def _probe_first_pts(seg: Path) -> float | None:
return None
def _kill(s: HlsSession) -> None:
# Teardown is split in two, because every segment request and checkpoint takes `_lock`: waiting up to
# 3 s for an ffmpeg to exit and deleting a whole segmented film under it stalled every other
# viewer's playback for one viewer's seek or one reap pass. `_detach` (under the lock) signals the
# process and renames its directory aside — so the deletion can never hit a NEW session that reuses
# the path — and `_finish` (after the lock is released) waits and deletes.
#
# What the rename cannot cover is the dying ffmpeg itself: it writes by path, so until it has exited
# it can still put a segment, or its final ENDLIST playlist, into a new session's directory on the
# same path. `_dying` remembers it by that path, and `_spawn` waits it out — under the lock, on
# purpose — before it creates the directory again.
_dying: dict[Path, subprocess.Popen] = {}
def _await_exit(key: str, proc: subprocess.Popen) -> None:
try:
s.proc.terminate()
try:
s.proc.wait(timeout=3)
proc.wait(timeout=3)
# silent-ok: the escalation IS the handling. A transcode that ignores TERM for three
# seconds gets KILL, which is exactly what this branch does.
except subprocess.TimeoutExpired:
s.proc.kill()
proc.kill()
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
# Not nothing: what survives this is an ffmpeg nobody owns any more, holding a VAAPI
# handle and a core, and the transcode cap that is supposed to bound them counts rows in
# `_sessions` rather than processes. One line is the only way that ever surfaces.
log.info(
"could not stop the transcode process",
extra=logs.event("plex.stream.kill_failed", key=key, reason=type(exc).__name__),
)
def _detach(s: HlsSession) -> tuple[HlsSession, Path | None]:
"""Caller holds _lock and has taken `s` out of `_sessions`. Hand the result to `_finish` once the
lock is released."""
try:
s.proc.terminate()
except Exception as exc: # noqa: BLE001 — teardown finishes whatever the process does
log.info(
"could not stop the transcode process",
extra=logs.event("plex.stream.kill_failed", key=s.key, reason=type(exc).__name__),
)
shutil.rmtree(s.dir, ignore_errors=True)
_dying[s.dir] = s.proc
tomb = s.dir.with_name(f"{s.dir.name}{_TOMB}{secrets.token_hex(4)}")
try:
s.dir.rename(tomb)
# silent-ok: nothing on disk (ffmpeg died before writing, or a sweep took it) — nothing to delete.
except FileNotFoundError:
return s, None
except OSError as exc:
# Cannot be moved aside, so it cannot be deleted outside the lock either: the path may be a
# new session's a moment from now. Deleted here instead, the old way.
log.info(
"a session directory could not be moved aside; deleted under the lock",
extra=logs.event("plex.stream.tomb_failed", key=s.key, reason=type(exc).__name__),
)
shutil.rmtree(s.dir, ignore_errors=True)
return s, None
return s, tomb
def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
"""Caller holds _lock. Three bounds, applied after a session is registered, each dropping the
least recently accessed sessions over it:
def _finish(doomed: "list[tuple[HlsSession, Path | None]]") -> None:
"""The slow half of a teardown, called WITHOUT `_lock`."""
for s, tomb in doomed:
_await_exit(s.key, s.proc)
with _lock:
if _dying.get(s.dir) is s.proc:
del _dying[s.dir]
if tomb is not None:
shutil.rmtree(tomb, ignore_errors=True)
def _enforce_caps(
user_id: int, max_transcodes: int, max_sessions: int
) -> "list[tuple[HlsSession, Path | None]]":
"""Caller holds _lock, and `_finish`es what this returns once it has released it. Three bounds,
applied after a session is registered, each dropping the least recently accessed sessions over it:
* this account's own sessions (`_MAX_SESSIONS_PER_USER`) — first, so a single account filling the
box evicts itself before anyone else;
@@ -672,10 +763,12 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
whole segmented copy of the film under `_HLS_ROOT`, written faster than realtime. Counted only
by the transcode cap, N distinct items were N ffmpegs and N film-sized scratch directories.
Somebody's playback stops at each drop. The route turns the missing session into a retryable 404
Somebody's playback stops at each drop. The route turns the missing session into a 410
and the player restarts at the playhead, so from the server nothing failed — and "my film keeps
rebuffering" has no other explanation anywhere, hence the line."""
doomed: list[tuple[HlsSession, Path | None]] = []
def cull(pool: list[tuple[tuple[int, str], HlsSession]], cap: int, bound: str) -> None:
if len(pool) <= cap:
return
@@ -690,12 +783,13 @@ def _enforce_caps(user_id: int, max_transcodes: int, max_sessions: int) -> None:
sessions=len(pool),
),
)
_kill(sess)
_sessions.pop(reg, None)
doomed.append(_detach(sess))
cull([kv for kv in _sessions.items() if kv[0][0] == user_id], _MAX_SESSIONS_PER_USER, "user")
cull([kv for kv in _sessions.items() if kv[1].transcode], max_transcodes, "transcode")
cull(list(_sessions.items()), max_sessions, "total")
return doomed
def start_session(
@@ -740,7 +834,11 @@ def start_session(
return directory / segment_name(i, _VIDEO_VARIANT if variants else None)
def _spawn(
hw: bool, tonemap: bool = False, *, replace_only: "HlsSession | None" = None
hw: bool,
tonemap: bool = False,
cpu_decode: bool = False,
*,
replace_only: "HlsSession | None" = None,
) -> "HlsSession | None":
# `replace_only` makes the supersede identity-aware for the runtime fallback: the whole
# check-and-replace happens under ONE _lock, so a newer same-key session (a concurrent
@@ -755,72 +853,88 @@ def start_session(
preset = sysconfig.get_str(db, "plex_transcode_preset")
crf = sysconfig.get_int(db, "plex_transcode_crf")
max_height = sysconfig.get_int(db, "plex_transcode_max_height")
# Nice the SOFTWARE transcode child (not the HW path, and not remux) so a CPU-bound re-encode
# can't starve the co-located API/playback. preexec_fn runs in the forked child (POSIX); on a
# platform without fork (the API always runs in a Linux container in prod) it's simply omitted.
preexec = _sw_child_preexec if (transcode and not hw and os.name == "posix") else None
with _lock:
if replace_only is not None and _sessions.get(reg) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(reg, None)
if old is not None:
_kill(old)
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src,
start_s,
# Nice the child that works the CPU — a SOFTWARE transcode, or a CPU decode feeding the GPU
# encoder (not the full-GPU path, and not remux) — so it can't starve the co-located
# API/playback. preexec_fn runs in the forked child (POSIX); on a platform without fork (the API
# always runs in a Linux container in prod) it's simply omitted.
cpu_bound = transcode and (not hw or cpu_decode)
preexec = _sw_child_preexec if (cpu_bound and os.name == "posix") else None
doomed: list[tuple[HlsSession, Path | None]] = []
# Finished even when the new session fails to start (no space, no ffmpeg, a DB error): what
# was moved aside under the lock is otherwise left for a restart's sweep.
try:
with _lock:
if replace_only is not None and _sessions.get(reg) is not replace_only:
return None # superseded/culled while we waited out of lock — don't clobber the newer one
old = _sessions.pop(reg, None)
if old is not None:
doomed.append(_detach(old))
# An ffmpeg still exiting from THIS path (the old session, or one stopped a moment ago)
# would write into the directory about to be made; see `_dying`.
prev = _dying.pop(directory, None)
if prev is not None:
_await_exit(key, prev)
shutil.rmtree(directory, ignore_errors=True)
directory.mkdir(parents=True, exist_ok=True)
proc = subprocess.Popen(
_ffmpeg_cmd(
src,
start_s,
directory,
audio_ord,
aoff,
audio_langs if is_multi else None,
transcode=transcode,
hw=hw,
tonemap=tonemap,
preset=preset,
crf=crf,
max_height=max_height,
loudnorm=loudnorm,
cpu_decode=cpu_decode,
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
preexec_fn=preexec,
)
s = HlsSession(
key,
directory,
audio_ord,
aoff,
audio_langs if is_multi else None,
proc,
start_s,
variants,
transcode=transcode,
hw=hw,
tonemap=tonemap,
preset=preset,
crf=crf,
max_height=max_height,
loudnorm=loudnorm,
),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
preexec_fn=preexec,
)
s = HlsSession(
key,
directory,
proc,
start_s,
variants,
transcode=transcode,
hw=hw,
tonemap=tonemap,
user_id=user_id,
)
_sessions[reg] = s
# The one line that says which pipeline a session actually got. Everything downstream
# — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
# now it was only inferable from the ffmpeg command nobody keeps.
log.info(
"an HLS session started",
extra=logs.event(
"plex.stream.session_started",
key=key,
transcode=transcode,
hw=hw,
tonemap=tonemap,
variants=len(variants or ()),
start_s=round(start_s, 1),
),
)
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
# capped to zero.
_enforce_caps(
user_id,
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
max(1, sysconfig.get_int(db, "plex_max_sessions")),
)
user_id=user_id,
cpu_decode=cpu_decode,
)
_sessions[reg] = s
# The one line that says which pipeline a session actually got. Everything downstream
# — the cap, the HW→SW fallback, the subtitle offset — is about this choice, and until
# now it was only inferable from the ffmpeg command nobody keeps.
log.info(
"an HLS session started",
extra=logs.event(
"plex.stream.session_started",
key=key,
transcode=transcode,
hw=hw,
tonemap=tonemap,
cpu_decode=cpu_decode,
variants=len(variants or ()),
start_s=round(start_s, 1),
),
)
# Admin-configurable caps (Configuration → Plex); at least 1 each so playback can't be
# capped to zero.
doomed += _enforce_caps(
user_id,
max(1, sysconfig.get_int(db, "plex_max_transcodes")),
max(1, sysconfig.get_int(db, "plex_max_sessions")),
)
finally:
_finish(doomed)
return s
s = _spawn(*_select_encode(db, item, transcode, src))
@@ -847,7 +961,7 @@ def start_session(
# stale and stops the session it names (`stop_session`), which stopped the playback that won.
# Never None either: the POST /session route turns a None return into a "Media file not found
# on disk" 404 — dishonest for a file that exists. A dead session instead degrades to a
# retryable 404 ("No active playback session" / "Not available yet") the client re-requests.
# 410 ("session_gone") on its URLs, and the client asks again.
return s
s = respawned
ready = wait_for(_seg(0), timeout=25.0, proc=s.proc)
@@ -866,7 +980,8 @@ def start_session(
if not wait_for(_seg(i), timeout=_WARMUP_SEG_TIMEOUT_S, proc=s.proc):
break
log.info(
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s tonemap=%s",
"plex hls session start key=%s req=%.1f real=%.3f multi=%s audio=%s transcode=%s hw=%s"
" tonemap=%s cpu_decode=%s",
key,
start_s,
s.media_start_s,
@@ -875,6 +990,7 @@ def start_session(
transcode,
s.hw,
s.tonemap,
s.cpu_decode,
)
return s
@@ -974,6 +1090,11 @@ def artifact_may_arrive(s: "HlsSession", filename: str) -> bool:
# session, absent for a single-audio one.
if variant not in (s.variants or (None,)):
return False
# And the name must be the one ffmpeg SPELLS for that sequence: `int()` also reads `seg_007.ts`
# and a non-ASCII digit run as a sequence the session may have reached, and neither file is
# ever written — a yes there parked the request for the whole wait.
if filename != segment_name(seq, variant):
return False
if seq == 0:
return True
try:
@@ -1084,9 +1205,13 @@ def sweep_orphans(older_than: float | None = None) -> int:
return dropped
# How long a session whose ffmpeg has FINISHED may go unfetched before a new session request reaps
# it (see reap_idle).
_FINISHED_IDLE_S = 30
# The slowest heartbeat a live player sends: a browser cuts the timers of a hidden page that makes no
# sound to one a minute (measured on Chromium: a paused and a muted player, about a minute after
# hiding; a hidden player with sound keeps its 10 s).
_THROTTLED_HEARTBEAT_S = 60
# How long a session whose ffmpeg has FINISHED may go without a signal before `reap_idle` takes it:
# the beat before the reaper looks, one lost on the way, and the time the next one takes to arrive.
_FINISHED_IDLE_S = 3 * _THROTTLED_HEARTBEAT_S
def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
@@ -1095,29 +1220,39 @@ def reap_idle(finished_idle_s: float = _FINISHED_IDLE_S) -> int:
A finished session is also what a PAUSED viewer has: a remux runs far faster than realtime, so
ffmpeg is done a minute or two in and writes ENDLIST; hls.js then stops asking for the playlist,
and with its forward buffer full a paused player fetches nothing at all. With the 30 s rule that
killed a one-minute pause, and playback died ~30 s after resuming.
and with its forward buffer full a paused player fetches nothing at all.
What makes the short rule safe again is that fetching is no longer the only signal: a player
parked MID-ITEM checkpoints its position every 10 s, and that route calls `touch_session`, so a
viewer who is merely paused says so far more often than this rule asks. The heartbeat
What makes the short rule safe is that fetching is not the only signal: a player parked MID-ITEM
checkpoints its position and that route calls `touch_session`. That beat is every 10 s only while
the page is visible or making sound; a hidden, silent one sends it once a minute, which a 30 s
rule killed whenever the reaper happened to run more than 30 s after the beat (measured: a paused
session gone that way, and one that lived on the same cadence because its phase was luckier).
So the rule is sized to the throttled beat (`_THROTTLED_HEARTBEAT_S`). The heartbeat
stops once the item is finished or before a position is known (`usePlexProgress.flush`) — both
states where reaping is correct, and where the player's 404 → reclaim recovers a viewer who
scrubs back in anyway. Do not restate this as "every 10 s whether or not it is playing":
states where reaping is correct, and where the player's 410 → reclaim recovers a viewer who
scrubs back in anyway. Do not restate this as "whether or not it is playing":
a review found that claim in two comments and not in the code. An earlier blanket
`reap_abandoned` bought the same safety by keeping every finished session for ten minutes — each
one a whole segmented copy of the film on a volume shared with the download center."""
now = time.time()
dropped = 0
doomed: list[tuple[HlsSession, Path | None]] = []
reaped: list[tuple[str, str, int]] = []
with _lock:
for reg, s in list(_sessions.items()):
done = s.proc.poll() is not None
idle = now - s.last_access
if idle > _SESSION_IDLE_S or (done and idle > finished_idle_s):
_kill(s)
_sessions.pop(reg, None)
dropped += 1
return dropped
doomed.append(_detach(s))
reaped.append((s.key, "finished_idle" if done else "idle", int(idle)))
# The viewer of a session wrongly taken sees only a 410 later, so this line is the one record of why.
for key, reason, idle_s in reaped:
log.info(
"an idle session was reaped",
extra=logs.event("plex.stream.session_reaped", key=key, reason=reason, idle_s=idle_s),
)
_finish(doomed)
return len(doomed)
def stop_session(user_id: int, key: str, session_id: str) -> bool:
@@ -1132,8 +1267,9 @@ def stop_session(user_id: int, key: str, session_id: str) -> bool:
s = _sessions.get((user_id, key))
if s is None or s.id != session_id:
return False
_kill(s)
_sessions.pop((user_id, key), None)
doomed = [_detach(s)]
_finish(doomed)
return True
+34 -21
View File
@@ -60,14 +60,20 @@ def classify_playable(vcodec: str | None, acodec: str | None, container: str | N
# Video codecs the prod iGPU (UHD 630 / Quick Sync Gen 9.5) can hardware-DECODE — the precondition for
# the full-GPU VAAPI transcode pipeline. THE single source of this set: plex.stream imports it here so
# the runtime encoder choice (`_want_hw`) and this catalog facet can never disagree by codec. A source
# outside it (AV1, MPEG-4 ASP/XviD…) is software-only even on the iGPU (it can't decode it to feed its
# own encoder).
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "vc1", "mpeg2video"})
# the runtime encoder choice (`_select_encode`) and this catalog facet can never disagree by codec. VC-1
# is NOT in it: measured on prod, the iHD driver answers "No support for codec vc1 profile 3" (VC-1
# Advanced, what a Blu-ray remux carries), so every session that asked the GPU to decode one died and
# fell back to libx264.
HW_DECODABLE_CODECS = frozenset({"h264", "hevc", "h265", "vp9", "mpeg2video"})
# Codecs the iGPU cannot decode but whose picture it can still ENCODE: the CPU decodes, the frames go up
# to the GPU as 8-bit nv12, and h264_vaapi encodes (plex.stream's `cpu_decode` path). Measured on prod:
# three segments in ~5 s for a 1080p VC-1 remux (libx264: ~17 s) and ~1 s for an SD XviD or DivX
# episode. Only codecs that are always 8-bit SDR, and only ones measured; any other codec stays software.
CPU_DECODE_CODECS = frozenset({"vc1", "mpeg4", "msmpeg4v3"})
# Color-transfer values marking an HDR source (PQ / HLG). THE single source of this set: plex.stream
# (the tonemap choice) and downloads.clip (a clip's tonemap) import it here, so a transfer added for
# one is tonemapped by all. A 10-bit/HDR source breaks the VAAPI scale=nv12 path on Gen 9.5 (see
# plex.stream) → software transcode, so it counts as NOT HW-capable.
# one is tonemapped by all. An HDR source breaks the plain VAAPI scale=nv12 path on Gen 9.5, so it takes
# the OpenCL tonemap graph (see plex.stream) — still the GPU encoder.
HDR_TRANSFERS = frozenset({"smpte2084", "arib-std-b67"})
@@ -78,18 +84,24 @@ def transcode_hw_capable(
hdr: bool = False,
) -> bool | None:
"""The stored `PlexItem.transcode_hw` facet: for a `playable="transcode"` item, whether the prod
iGPU can HARDWARE-transcode it (else software libx264 only); None for a non-transcode item. HW needs
a HW-decodable codec AND an 8-bit SDR source — a 10-bit/HDR source (bit_depth>8 or an HDR transfer)
breaks the VAAPI nv12 pipeline and falls back to software. `bit_depth`/`hdr` are unknown at sync
time (the cheap listing carries no Stream), so they default to 8-bit SDR (codec-only) and the
bit-depth backfill refines a 10-bit/HDR HEVC to False later."""
iGPU ENCODES it ("Transcoded on the hardware encoder"), else software libx264; None for a
non-transcode item. The same answer plex.stream's `_select_encode` gives on prod (render node and
OpenCL runtime present, policy "auto"):
* a HW-decodable codec: 8-bit SDR → the full-GPU path; HDR → the OpenCL tonemap, also on the GPU;
10-bit SDR → software (an exact downconvert, never the HDR tone curve);
* a CPU-decode codec (`CPU_DECODE_CODECS`) at 8 bits → CPU decode, GPU encode;
* any other codec → software.
`bit_depth`/`hdr` are unknown at sync time (the cheap listing carries no Stream), so they default to
8-bit SDR and the bit-depth backfill refines a 10-bit SDR HEVC to False later."""
if playable != "transcode":
return None
if (vcodec or "").lower() not in HW_DECODABLE_CODECS:
return False
if bit_depth is not None and bit_depth > 8:
return False
return not hdr
codec = (vcodec or "").lower()
deep = bit_depth is not None and bit_depth > 8
if codec in HW_DECODABLE_CODECS:
return hdr or not deep
if codec in CPU_DECODE_CODECS:
return not deep and not hdr
return False
def _video_stream_props(meta: dict) -> tuple[int | None, bool]:
@@ -266,8 +278,9 @@ def sync(db: Session) -> dict:
# a Plex shape change, a DB error or our own bug, and `%s` told them apart badly.
log.exception("the Plex people-enrich batch failed")
# Probe a bounded batch of HW-decodable transcode items for bit-depth/HDR, so the
# playback-class filter's HW/SW split is precise (a 10-bit/HDR HEVC → software) over
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are SW by codec.
# playback-class filter's HW/SW split is precise (a 10-bit SDR HEVC → software) over
# successive runs. Only HW-decodable-codec transcodes are probed — the rest are classed
# by codec alone.
try:
stats["media_probed"] = _probe_media_batch(db, plex)
except Exception: # best-effort; never fail the sync over a backfill
@@ -294,15 +307,15 @@ def sync(db: Session) -> dict:
# whole library is covered over successive runs (each fetched item is stamped people_enriched_at once).
_ENRICH_BATCH = 60
# Same bounded-per-run pattern for the bit-depth probe. Only HW-decodable-codec transcode items are
# probed (typically a small minority — the bulk of transcodes are old codecs that are SW by codec), so
# probed (typically a small minority — the bulk of transcodes are old codecs classed by codec), so
# a modest batch covers the work-list in a run or two; a stuck item retries next run (watermark unset).
_PROBE_BATCH = 60
def _probe_media_batch(db: Session, plex: PlexClient, limit: int = _PROBE_BATCH) -> int:
"""Rich-fetch up to `limit` not-yet-probed HW-decodable transcode items, read the video Stream's
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit/HDR HEVC →
False/software). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
bit-depth + HDR, and store `video_bit_depth` + the refined `transcode_hw` (a 10-bit SDR HEVC →
False/software; an HDR one stays on the GPU through the tonemap). Stamps `media_probed_at` on every attempt (success OR a Plex miss with no Stream),
so an item is probed once and not re-fetched every sync; a transient fetch FAILURE leaves the
watermark unset to retry. Oldest-id first. Returns how many items were probed this run."""
rows = (
+39 -15
View File
@@ -41,11 +41,25 @@ def video(db: Session, video_id: str) -> Video | None:
# --- tags ------------------------------------------------------------------------------------
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int]]:
"""The tags this user sees (the shared system ones and their own), each with how many channel
links it has, by category then name."""
def tags_with_counts(db: Session, user: User) -> list[tuple[Tag, int, int]]:
"""The tags this user sees (the shared system ones and their own), by category then name, each
with how many channel links it has and how many of the user's own links sit on a channel they
subscribe to and have not hidden — the rows the Channels page's tag filter shows (a hidden
channel has its own tab). A link outlives an unsubscribe and a hide, so the two differ."""
subscribed = select(Subscription.channel_id).where(
owned_by(Subscription.user_id, user), Subscription.hidden.is_(False)
)
on_subscribed = dict(
db.execute(
select(ChannelTag.tag_id, func.count(ChannelTag.id))
.where(owned_by(ChannelTag.user_id, user), ChannelTag.channel_id.in_(subscribed))
.group_by(ChannelTag.tag_id)
)
.tuples()
.all()
)
return [
(tag, count)
(tag, count, on_subscribed.get(tag.id, 0))
for tag, count in db.execute(
select(Tag, func.count(ChannelTag.id))
.outerjoin(ChannelTag, ChannelTag.tag_id == Tag.id)
@@ -132,12 +146,17 @@ def item_counts(db: Session, ids: list[int]) -> dict[int, int]:
)
def durations(db: Session, ids: list[int]) -> dict[int, int]:
"""Each playlist's running time: the sum of its videos' lengths."""
def durations(db: Session, ids: list[int]) -> dict[int, tuple[int, int]]:
"""Each playlist's running time — the sum of its videos' known lengths — and how many videos
it left out for having none (`Video.without_length`)."""
return {
pid: int(total or 0)
for pid, total in db.execute(
select(PlaylistItem.playlist_id, func.coalesce(func.sum(Video.duration_seconds), 0))
pid: (int(total or 0), int(left_out or 0))
for pid, total, left_out in db.execute(
select(
PlaylistItem.playlist_id,
func.coalesce(func.sum(Video.length_seconds), 0),
func.count().filter(Video.without_length),
)
.join(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id.in_(ids))
.group_by(PlaylistItem.playlist_id)
@@ -177,11 +196,16 @@ def covers(db: Session, ids: list[int]) -> dict[int, str | None]:
)
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None]:
"""One playlist's item count, running time and cover. An outer join, so an empty list still
answers (0, 0, None)."""
count, total = db.execute(
select(func.count(PlaylistItem.id), func.coalesce(func.sum(Video.duration_seconds), 0))
def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, int, str | None]:
"""One playlist's item count, running time, the videos the running time left out for having no
length (`Video.without_length`) and cover. An outer join, so an empty list still answers
(0, 0, 0, None)."""
count, total, left_out = db.execute(
select(
func.count(PlaylistItem.id),
func.coalesce(func.sum(Video.length_seconds), 0),
func.count().filter(Video.without_length),
)
.select_from(PlaylistItem)
.outerjoin(Video, Video.id == PlaylistItem.video_id)
.where(PlaylistItem.playlist_id == playlist_id)
@@ -193,7 +217,7 @@ def playlist_totals(db: Session, playlist_id: int) -> tuple[int, int, str | None
.order_by(PlaylistItem.position)
.limit(1)
)
return count or 0, total or 0, cover
return count or 0, total or 0, left_out or 0, cover
def playlist_rows(db: Session, user: User, playlist_id: int) -> list:
+159 -65
View File
@@ -9,7 +9,7 @@ unique key, a row lock taken before a presence check. The comfort shelf's writes
from datetime import datetime, timezone
from typing import cast
from typing import Literal, cast
from sqlalchemy import Table, bindparam, delete, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
@@ -255,17 +255,6 @@ def mark(db: Session, user: User, video_id: str, status: str) -> None:
)
def clear_state(db: Session, user: User, video_id: str) -> None:
"""The video back to pristine for this user: status, resume point and `watched_at` all go.
Playlist membership lives elsewhere and stays."""
lock_takeout(db, user) # as `unmark`
row = queries.video_state(db, user, video_id)
if row is not None:
db.delete(row)
_remember_unmark(db, user, video_id)
db.flush()
def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
"""The player's resume point. 0 clears it: a row holding nothing else goes, one with a status
(watched, hidden) keeps it with the position zeroed. The status itself is never changed here.
@@ -301,12 +290,17 @@ def set_progress(db: Session, user: User, video_id: str, position: int) -> None:
# --- playlists -------------------------------------------------------------------------------
# Every reason a move is refused; the UI names each one (`playlists.moveErrors.<reason>`, held by the
# frontend's backendIds contract, which reads this Literal).
MoveReason = Literal["same_playlist"]
class MoveRefused(Exception):
"""A move between playlists that cannot happen: `reason` is the code the UI translates."""
def __init__(self, reason: str):
def __init__(self, reason: MoveReason):
super().__init__(reason)
self.reason = reason
self.reason: MoveReason = reason
def touch(pl: Playlist) -> None:
@@ -412,30 +406,6 @@ def remove_item(db: Session, pl: Playlist, video_id: str, *, mark_dirty: bool) -
return True
def place_item(db: Session, pl: Playlist, video_id: str, position: int) -> None:
"""Put a video already in the list at `position` (clamped to the end), renumbering the whole
list: positions can have gaps after removals, so "index N" is only meaningful over the
ordered rows."""
rows = queries.items(db, pl.id)
new = next(r for r in rows if r.video_id == video_id)
rows.remove(new)
rows.insert(min(position, len(rows)), new)
for pos, row in enumerate(rows):
row.position = pos
db.flush()
def add_to_playlist(db: Session, pl: Playlist, video_id: str, position: int | None) -> None:
"""Add a video to one of the user's lists, at `position` when given (the list page's "Undo" of
a removal puts the row back where it was in ONE request), else at the end. A video already in
the list stays where it is."""
added = add_item(db, pl, video_id, mark_dirty=False)
if added and position is not None:
place_item(db, pl, video_id, position)
recompute_dirty(db, pl)
db.flush()
def rename_playlist(db: Session, pl: Playlist, name: str) -> None:
"""A new name moves `dirty`; the caller holds the playlist's lock (the scheduled mirror refresh
reads `dirty` under it)."""
@@ -453,39 +423,163 @@ def delete_playlist(db: Session, pl: Playlist) -> None:
db.flush()
def move_item(db: Session, src: Playlist, dst: Playlist, video_id: str) -> None:
"""Move one video from `src` into `dst`, in the caller's one transaction.
def remove_items(db: Session, pl: Playlist, video_ids: list[str]) -> list[str]:
"""Remove these videos from a playlist, in one transaction; a video not in it any more is simply
not among them. Says which went: an Undo must not bring back one this removal never took."""
lock_playlists(db, pl.id)
wanted = set(video_ids)
rows = [it for it in queries.items(db, pl.id) if it.video_id in wanted]
for row in rows:
db.delete(row)
if rows:
recompute_dirty(db, pl)
touch(pl)
db.flush()
return [row.video_id for row in rows]
The client-side shape — add to the target, then remove from the source — leaves the video in
BOTH lists whenever the second call fails. Idempotent: a video already in the target is simply
dropped from the source (moving into a list that has it is a removal, not a duplicate). Raises
`MoveRefused` for the same list twice, or a video a concurrent edit already took out."""
def _insert_at(
db: Session,
pl: Playlist,
placed: list[tuple[int, str]],
leaving: dict[str, PlaylistItem] | None = None,
staying_out: frozenset[str] = frozenset(),
) -> int:
"""Add videos at their indices (ascending, so each index is where the row stood when the rows
before it were back) and renumber the list 0..n-1, as positions can have gaps. A video already in
the list stays where it is, and one the catalog has lost or in `staying_out` is left out, the
ones after it closing up. A video with a row in `leaving` brings that row along, for the reason
`move_items` re-parents. Says how many were added; the caller holds the playlist's lock."""
leaving = leaving or {}
rows = queries.items(db, pl.id)
present = {r.video_id for r in rows}
known = queries.known_video_ids(db, [vid for _, vid in placed])
added = 0
left_out = 0
for at, vid in sorted(placed, key=lambda p: p[0]):
if vid in present or vid not in known or vid in staying_out:
left_out += 1
continue
row = leaving.pop(vid, None)
if row is None:
row = PlaylistItem(playlist_id=pl.id, video_id=vid, position=0)
db.add(row)
else:
row.playlist_id = pl.id
rows.insert(min(at - left_out, len(rows)), row)
present.add(vid)
added += 1
for pos, row in enumerate(rows):
row.position = pos
return added
def restore_items(db: Session, pl: Playlist, video_ids: list[str], positions: list[int]) -> int:
"""Put removed videos back at their places, in ONE request: an append plus a whole-list reorder
leaves them at the end whenever the second call fails. Says how many came back."""
lock_playlists(db, pl.id)
added = _insert_at(db, pl, list(zip(positions, video_ids, strict=True)))
if added:
recompute_dirty(db, pl)
touch(pl)
db.flush()
return added
class ListGone(Exception):
"""A list of a two-list write was deleted between the route's read and the lock."""
def __init__(self, *, source: bool):
super().__init__("source" if source else "target")
self.source = source
def _lock_both(db: Session, src: Playlist, dst: Playlist) -> None:
"""Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise. A list
already gone is said here: a write into it would answer with its foreign key's 500."""
found = lock_playlists(db, src.id, dst.id)
if src.id not in found:
raise ListGone(source=True)
if dst.id not in found:
raise ListGone(source=False)
def move_items(
db: Session, src: Playlist, dst: Playlist, video_ids: list[str]
) -> tuple[list[str], list[str]]:
"""Move these videos from `src` into `dst`, in the caller's one transaction, in the source's
order whatever order they were picked in. Returns the ones that landed in the target and the ones
it already held, which only left the source: the Undo needs to tell them apart. A video a
concurrent edit already took out of the source is in neither. Raises `MoveRefused` for the same
list twice."""
if src.id == dst.id:
raise MoveRefused("same_playlist")
# Both lists, in id order: a move A→B beside a move B→A would otherwise lock crosswise.
lock_playlists(db, src.id, dst.id)
row = queries.item(db, src.id, video_id)
if row is None:
raise MoveRefused("not_in_playlist")
dst_changed = False
if queries.item(db, dst.id, video_id) is None:
# Re-parent the existing row rather than delete+insert: `added_at` ("when I saved this")
# survives the move, and there is no window where the video belongs to neither list. The
# check above holds: both lists are locked, and an insert into the target (another tab's
# add, a YouTube merge) waits for the lock — its foreign key needs the target row.
row.playlist_id = dst.id
row.position = next_item_position(db, dst.id)
dst_changed = True
if not dst_changed:
db.delete(row)
# The source always changed; the target only when the row landed in it. Touching a target
# that already had the video promoted an untouched list in the rail's "recently changed" sort.
recompute_dirty(db, src)
touch(src)
if dst_changed:
_lock_both(db, src, dst)
wanted = set(video_ids)
rows = [it for it in queries.items(db, src.id) if it.video_id in wanted]
held = queries.in_playlist(db, dst.id, [r.video_id for r in rows])
position = next_item_position(db, dst.id)
moved: list[str] = []
already_there: list[str] = []
for row in rows:
if row.video_id in held:
already_there.append(row.video_id)
db.delete(row)
else:
moved.append(row.video_id)
# Re-parented rather than deleted and inserted: `added_at` ("when I saved this") survives
# the move, and there is no window where the video belongs to neither list. `held` stays
# true: both lists are locked, and an insert into the target (another tab's add, a
# YouTube merge) waits for the lock, as its foreign key needs the target row.
row.playlist_id = dst.id
row.position = position
position += 1
if rows:
recompute_dirty(db, src)
touch(src)
# Only when a row landed in it: touching a target that already had the video promoted an untouched
# list in the rail's "recently changed" sort.
if moved:
recompute_dirty(db, dst)
touch(dst)
db.flush()
return moved, already_there
def undo_move(
db: Session,
src: Playlist,
dst: Playlist,
moved: list[str],
already_there: list[str],
positions: list[int],
) -> int:
"""Take a move back in one transaction: the `moved` videos leave `dst` and return to their places
in `src`, keeping their rows, as the move did. The `already_there` ones return too and stay in
`dst`, which held them before. A moved video `dst` no longer holds has gone on from there, and
bringing it back would leave it in lists the reader never put it in, so it stays out, as a
move skips a video a concurrent edit took out. `positions` pairs with `moved` + `already_there`.
Says how many came back. Raises `MoveRefused` for the same list twice, as a move does."""
if src.id == dst.id:
raise MoveRefused("same_playlist")
_lock_both(db, src, dst)
wanted = set(moved)
rows = {it.video_id: it for it in queries.items(db, dst.id) if it.video_id in wanted}
changed = bool(rows)
placed = list(zip(positions, [*moved, *already_there], strict=True))
back = _insert_at(db, src, placed, leaving=rows, staying_out=frozenset(wanted - rows.keys()))
# What `src` holds again by now only leaves `dst`.
for row in rows.values():
db.delete(row)
db.flush()
if changed:
recompute_dirty(db, dst)
touch(dst)
if back:
recompute_dirty(db, src)
touch(src)
db.flush()
return back
def reorder_items(db: Session, pl: Playlist, order: list[str]) -> int:
+20 -10
View File
@@ -1,29 +1,39 @@
"""A request the app refuses, said in a way the UI can put in the reader's language.
Every refusal the API answers carries a structured `detail` — `{code, reason, …params}` — and the client
translates it (`lib/api/core.ts` `localizeDetail`). A bare English string reached the Hungarian UI
translates it (`lib/api/refusalText.ts` `localizeDetail`). A bare English string reached the Hungarian UI
verbatim: the blocking error dialog, and every inline surface that printed `HttpError.detail`, showed
whatever sentence the route happened to hand-write. `refused` is the generic family
(`errors.refused.<reason>`, the params interpolated into the sentence); the older specific codes
(`not_found`, `youtube`, `playlist`, the admin refusals, …) stay, since each is already translated.
A test (`test_http_details.py`) reads every `HTTPException` in the app and fails on a string detail,
and the frontend's contract test requires every `refused(…, "<reason>")` to be named in both
languages — so the rule is enforced, not described.
A test (`test_http_details.py`) reads every place an `HTTPException` or a `JSONResponse` sends a detail
and fails on one the client cannot say: a sentence, a dict without a `code`, a variable, none at all.
`tests/refusal_sites.py` reads every `{"code": …}` detail from the syntax tree, written where it is sent
(or returned by a factory that only ever is), with any `**` spread before its `code` and `reason`, and
follows each reason along a closed list of shapes: a literal, a `Literal` type on the parameter or
attribute it travels in, every call of a helper like `refused` below, a map's values; anything else is
reported, never guessed. The frontend's contract test
(`i18n/backendIds.contract.test.ts`) runs it and requires each code in one record of
`lib/api/refusalCodes.ts`, and each reason's sentence in both languages; a code or reason it cannot
follow fails it, unless no screen shows that code. So the rule is enforced, not described.
"""
from fastapi import HTTPException
from app.sysconfig import InvalidSetting, SettingReason
def refused(status: int, reason: str, **params: object) -> HTTPException:
"""The refusal to raise: `raise refused(409, "view_name_taken", name=name)`. `params` travel in
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see."""
return HTTPException(status_code=status, detail={"code": "refused", "reason": reason, **params})
the detail for the sentence to interpolate (`{{name}}`); keep them to values the reader may see.
They are spread first, so a `code` or `reason` among them could never replace these two."""
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the frontend's
# contract test can read every reason this can produce.
_SETTING_REASONS = {
# `sysconfig.InvalidSetting.reason` → the refusal it answers as. A dict of literals, so the refusal
# scan reads every reason this can produce, and holds its keys to `SettingReason`.
_SETTING_REASONS: dict[SettingReason, str] = {
"not_a_number": "setting_not_a_number",
"too_low": "setting_too_low",
"too_high": "setting_too_high",
@@ -32,6 +42,6 @@ _SETTING_REASONS = {
}
def setting_refused(exc) -> HTTPException:
def setting_refused(exc: InvalidSetting) -> HTTPException:
"""A value the admin entered that `sysconfig.set_value` refused (`exc`: its `InvalidSetting`)."""
return refused(400, _SETTING_REASONS[exc.reason], key=exc.key, **exc.params)
+6 -2
View File
@@ -6,9 +6,9 @@ and "fetch a row this user owns or 404" idioms have one definition, one status c
one existence-leak policy.
"""
from typing import Any, Protocol, TypeVar
from typing import Annotated, Any, Protocol, TypeVar
from fastapi import HTTPException, Request
from fastapi import HTTPException, Query, Request
from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy.orm import Session
@@ -21,6 +21,10 @@ from app.youtube.client import YouTubeError, refusal_reason
M = TypeVar("M", bound=Base)
# A shuffle / surprise seed off the query string. The client draws one under a billion; bounded so a
# hand-typed 400-digit one is a 422, not an int no float or bigint can hold further down.
Seed = Annotated[int, Query(ge=-(2**53), le=2**53)]
# Resume-position thresholds of a player that only keeps a resume point (the YouTube player, the
# library's): positions below this are "didn't really start" and within this of the end are
# "basically finished" — neither is worth storing, so the position is cleared instead (no progress
+19 -10
View File
@@ -976,7 +976,16 @@ def _resend_verify_if_eligible(email: str) -> None:
send_verification_link(db, user)
def _login_refused(status: int, reason: str, user: User | None = None, **extra) -> HTTPException:
# Every refusal a password sign-in answers with; the UI names each one (`welcome.auth.blocked.<reason>`,
# held by the frontend's backendIds contract, which reads this Literal).
LoginRefusal = Literal[
"rate_limited", "invalid", "suspended", "email_unverified", "pending_approval"
]
def _login_refused(
status: int, reason: LoginRefusal, user: User | None = None, **extra
) -> HTTPException:
"""Every refusal password_login can answer with, in the codebase's structured `{code, reason}`
shape so the SPA can say it in the user's own language (the old bare English strings were
unreadable under a bilingual UI, and unusable as anything but a one-line error).
@@ -992,15 +1001,15 @@ def _login_refused(status: int, reason: str, user: User | None = None, **extra)
`email_verified` is false, and vice versa — the SPA renders whichever is actually outstanding
rather than assuming a sequence.
"""
detail: dict = {"code": "login", "reason": reason}
if user is not None:
detail |= {
"email": user.email,
"email_verified": user.email_verified,
"approved": user.is_active,
}
detail |= {k: v for k, v in extra.items() if v is not None}
return HTTPException(status_code=status, detail=detail)
account = (
{"email": user.email, "email_verified": user.email_verified, "approved": user.is_active}
if user is not None
else {}
)
given = {k: v for k, v in extra.items() if v is not None}
return HTTPException(
status_code=status, detail={**account, **given, "code": "login", "reason": reason}
)
@router.post("/password-login")
+4
View File
@@ -76,6 +76,10 @@ def list_channels(user: User = Depends(current_user), db: Session = Depends(get_
"details_synced": ch.details_synced_at is not None,
"recent_synced": ch.recent_synced_at is not None,
"backfill_done": ch.backfill_done,
# When the follow reached this list: written by the subscription import the run that
# first saw it, or by a subscribe made here — not the channel row's own date, which an
# explored or discovered channel had before it was followed.
"added_at": iso(sub.created_at),
}
for sub, ch in rows
]
+3 -3
View File
@@ -23,7 +23,7 @@ from app.discover import pins, rails
from app.discover.budget import Budget, DEFAULT_BUDGET
from app.models import User
from app.plex.cards import norm_lang
from app.routes._common import not_found
from app.routes._common import Seed, not_found
router = APIRouter(prefix="/api/discover", tags=["discover"])
@@ -44,7 +44,7 @@ class ReorderIn(BaseModel):
@router.get("")
def get_discover(
budget: Budget = DEFAULT_BUDGET,
seed: int = 0,
seed: Seed = 0,
lang: str = "en",
user: User = Depends(current_user),
db: Session = Depends(get_db),
@@ -79,7 +79,7 @@ MAX_EXCLUDE = 32
@router.get("/pick")
def get_pick(
budget: Budget = DEFAULT_BUDGET,
seed: int = 0,
seed: Seed = 0,
exclude: str = "",
start: int = 0,
lang: str = "en",
+9 -8
View File
@@ -208,7 +208,7 @@ def set_credential(
try:
cleaned = dlcreds.validate_cookies(body.cookies)
except dlcreds.InvalidCookies as e:
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": str(e)}) from e
raise HTTPException(status_code=400, detail={"code": "cookies", "reason": e.reason}) from e
dlcreds.upsert_credential(db, user.id, host_key, cleaned)
audit.record(
db,
@@ -359,8 +359,9 @@ def enqueue_clip(
job = service.enqueue_clip(db, user.id, spec, display_name=display_name)
except clipmod.ClipError as e:
status = 404 if e.reason == "not_found" else 422
detail = {"code": "clip", "reason": e.reason, **e.params}
raise HTTPException(status_code=status, detail=detail) from e
raise HTTPException(
status_code=status, detail={**e.params, "code": "clip", "reason": e.reason}
) from e
except quota.QuotaExceeded as e:
raise HTTPException(status_code=422, detail=quota_detail(e)) from e
db.commit()
@@ -610,7 +611,7 @@ def list_links(
job_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
own_job(db, user, job_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, job_id=job_id)]
return [linksmod.owner_view(db, link) for link in queries.links_to(db, job_id=job_id)]
class CreateLinkIn(BaseModel):
@@ -652,7 +653,7 @@ def create_link(
raise refused(409, "share_unfinished")
link, token = _new_link(db, user, body, job_id=job_id)
db.commit()
return linksmod.owner_view(link, token=token)
return linksmod.owner_view(db, link, token=token)
def _shareable_job(db: Session, user: User, job_id: int) -> "DownloadJob":
@@ -683,7 +684,7 @@ def list_folder_links(
folder_id: int, user: User = Depends(current_user), db: Session = Depends(get_db)
) -> list[dict]:
own_folder(db, user, folder_id)
return [linksmod.owner_view(link) for link in queries.links_to(db, folder_id=folder_id)]
return [linksmod.owner_view(db, link) for link in queries.links_to(db, folder_id=folder_id)]
@router.post("/folders/{folder_id}/links")
@@ -696,7 +697,7 @@ def create_folder_link(
_shareable_folder(db, user, folder_id)
link, token = _new_link(db, user, body, folder_id=folder_id)
db.commit()
return linksmod.owner_view(link, token=token)
return linksmod.owner_view(db, link, token=token)
class UpdateLinkIn(BaseModel):
@@ -723,7 +724,7 @@ def update_link(
sent=frozenset(body.model_fields_set),
)
db.commit()
return linksmod.owner_view(link)
return linksmod.owner_view(db, link)
@router.delete("/links/{link_id}")
+13 -12
View File
@@ -10,6 +10,7 @@ from sqlalchemy.orm.exc import StaleDataError
from app import jobruns, quota, scheduler
from app.catalog import queries as catalog_queries, service as catalog_service
from app.catalog.models import known_length
from app.auth import current_user
from app.db import get_db, release_read_connection
from app.mailtext import lang_for_user
@@ -19,6 +20,7 @@ from app.models import User
from app.reader import queries, service, takeout
from app.routes._common import (
ProgressIn,
Seed,
capped_body,
drain_body,
kept_position,
@@ -88,7 +90,7 @@ def _feed_params(
def get_feed(
params: dict = Depends(_feed_params),
sort: str = "newest",
seed: int = 0,
seed: Seed = 0,
limit: int = Query(default=60, le=200),
cursor: str | None = None,
user: User = Depends(current_user),
@@ -204,22 +206,21 @@ def set_video_state(
return {"ok": True, "video_id": video_id, "status": status}
@router.delete("/videos/{video_id}/state")
def clear_video_state(
@router.delete("/videos/{video_id}/progress")
def clear_video_progress(
video_id: str,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Reset a video to pristine for this user — drop the whole VideoState row (status,
watch position and watched_at), as if it had never been opened. Saved/playlist membership
lives elsewhere and is untouched."""
"""Forget where the user left a video: its resume point goes, its status (watched, hidden) stays.
Saved/playlist membership lives elsewhere and is untouched."""
try:
service.clear_state(db, user, video_id)
service.set_progress(db, user, video_id, 0)
db.commit()
# silent-ok: the row went under the delete, which is the outcome the delete wanted.
# silent-ok: the row went under the write, which is the outcome the write wanted.
except StaleDataError:
db.rollback()
return {"ok": True, "video_id": video_id, "status": "new", "position_seconds": 0}
return {"ok": True, "video_id": video_id, "position_seconds": 0}
@router.post("/videos/{video_id}/progress")
@@ -288,7 +289,7 @@ def get_video_detail(
"channel_title": v.channel.title if v.channel else None,
"published_at": iso(v.published_at),
"view_count": v.view_count,
"duration_seconds": v.duration_seconds,
"duration_seconds": v.length_seconds,
}
# The DB path above is free; the off-catalog YouTube lookup below spends a quota unit.
@@ -316,8 +317,8 @@ def get_video_detail(
"channel_title": snippet.get("channelTitle"),
"published_at": snippet.get("publishedAt"),
"view_count": int(views) if views is not None else None,
"duration_seconds": parse_iso8601_duration(
items[0].get("contentDetails", {}).get("duration")
"duration_seconds": known_length(
parse_iso8601_duration(items[0].get("contentDetails", {}).get("duration"))
),
}
+1 -1
View File
@@ -139,7 +139,7 @@ def own_job_for_update(db: Session, user: User, job_id: int) -> "DownloadJob":
def quota_detail(e: quota.QuotaExceeded) -> dict:
"""Structured error the trilingual client localizes (reason key + numbers), instead of a
pre-baked English string with a dot-decimal GB value. See lib/api.ts localizeDetail()."""
pre-baked English string with a dot-decimal GB value. See lib/api/refusalText.ts localizeDetail()."""
return {"code": "quota", "reason": e.reason, "limit": e.limit, "current": e.current}
+177 -49
View File
@@ -3,10 +3,10 @@ CRUD (create / rename / delete, add / remove / reorder items) plus YouTube two-w
mirror the user's own YouTube playlists in (`/sync-youtube`) and push/revert local edits back
out (`/{id}/push-plan`, `/{id}/revert-youtube`), behind the optional write scope."""
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Literal
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from pydantic import BaseModel, model_validator
from sqlalchemy.orm import Session
from app import quota
@@ -41,11 +41,16 @@ router = APIRouter(prefix="/api/playlists", tags=["playlists"])
# It guards EVERY route that spends under QuotaAction.PLAYLISTS_PULL, not just the bulk one: a
# scripted loop over the single-playlist re-pull drains the same shared budget, and limiting one of
# two routes with the same quota action is the asymmetric fix this codebase keeps re-learning.
# Deliberately NOT on the push side (`push-plan` / `push`): those run on every push attempt,
# including the no-op "already up to date" path, so a 3-per-10-minute allowance would refuse
# legitimate work. Their exposure is tracked as SEC-18 instead.
_playlist_pull_limiter = RateLimiter(max_events=3, window_seconds=600)
# SEC-18, the push side: `push-plan` reads the live list (1 unit a page) and `push` reads and writes
# it, both against the same instance-wide budget. They get a budget of their own, counted in calls
# and far wider than the pull's: a press in the UI is one plan and, unless nothing changed, one push,
# so no person reaches it, while a scripted loop stops at it. A big push is not this limit's job —
# `push` already refuses one the day's remaining budget cannot cover.
PUSH_ALLOWANCE = 20
_playlist_push_limiter = RateLimiter(max_events=PUSH_ALLOWANCE, window_seconds=600)
def _check_pull_cooldown(user: User) -> None:
"""Shared gate so the two PLAYLISTS_PULL routes cannot drift apart again."""
@@ -53,7 +58,23 @@ def _check_pull_cooldown(user: User) -> None:
raise HTTPException(status_code=429, detail={"code": "sync_cooldown"})
def _refused(status: int, reason: str) -> HTTPException:
# Every playlist action the server refuses; the UI names each one (`playlists.errors.<reason>`, held by
# the frontend's backendIds contract, which reads this Literal).
PlaylistRefusal = Literal[
"name_required",
"not_on_youtube",
"push_cooldown",
"quota_exceeded",
"sync_all_busy",
"sync_busy",
"watch_later_not_pushable",
"watch_later_undeletable",
"youtube_editing_off",
"youtube_read_scope",
]
def _refused(status: int, reason: PlaylistRefusal) -> HTTPException:
"""A playlist action the server refuses, as a reason the UI translates
(`playlists.errors.<reason>`) — a bare English detail reached the Hungarian UI verbatim."""
return HTTPException(status_code=status, detail={"code": "playlist", "reason": reason})
@@ -74,10 +95,13 @@ def _video_or_404(db: Session, video_id: str) -> None:
def _summary(
pl: "Playlist",
count: int,
total_duration: int = 0,
duration: tuple[int, int] = (0, 0),
has_video: bool | None = None,
cover: str | None = None,
) -> dict:
"""`duration` is `queries.durations`'s pair: the known running time, and the videos it left out
for having no length — the "+N without length" beside it."""
total_duration, left_out = duration
out = {
"id": pl.id,
"name": pl.name,
@@ -87,6 +111,7 @@ def _summary(
"dirty": pl.dirty,
"item_count": count,
"total_duration_seconds": total_duration,
"without_length": left_out,
"cover_thumbnail": cover,
# When this list last CHANGED (rename, add/remove/move/reorder) — the rail's
# "recently changed first" sort reads it. See `service.touch` for why it isn't automatic.
@@ -117,7 +142,7 @@ def list_playlists(
_summary(
p,
counts.get(p.id, 0),
durations.get(p.id, 0),
durations.get(p.id, (0, 0)),
(p.id in member) if contains else None,
covers.get(p.id),
)
@@ -229,9 +254,16 @@ def revert_youtube(
raise youtube_refusal(exc, playlist=True) from exc
def _check_push_cooldown(user: User) -> None:
"""Shared gate of the two push routes, so they spend one allowance and cannot drift apart."""
if not _playlist_push_limiter.allow(str(user.id)):
raise _refused(429, "push_cooldown")
def _pushable(db: Session, user: User, playlist_id: int) -> "Playlist":
"""A local playlist the user may push to YouTube (create or update). The built-in
Watch later and read-only YouTube mirrors are never pushable."""
"""A playlist the user may push to YouTube: a local one (created there on its first push) or a
mirror of their own YouTube list (its local edits merged back). Only the built-in Watch later is
never pushable."""
pl = _own_playlist(db, user, playlist_id)
if pl.kind == "watch_later":
raise _refused(400, "watch_later_not_pushable")
@@ -248,6 +280,7 @@ def push_plan(
the quota estimate, so the UI can show a confirm. For a linked playlist this reads the live
YouTube state (cheap, 1 unit/page). Write scope is required — syncing is a YouTube write."""
pl = _pushable(db, user, playlist_id)
_check_push_cooldown(user) # after the refusal: a list that cannot be pushed spends nothing
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
plan = plan_merge(db, user, pl)
@@ -273,6 +306,8 @@ def push(
with sync_claim(db, pl.id) as free:
if not free:
raise _refused(409, "sync_busy")
# Spent only by a push that runs, as Revert's pull: a busy list or Watch later costs nothing.
_check_push_cooldown(user)
try:
with quota.attribute(user.id, quota.QuotaAction.PLAYLISTS_PUSH):
# Read the live playlist ONCE (linked playlists only) and share it with both the
@@ -334,8 +369,8 @@ def rename_playlist(
raise _refused(400, "name_required")
service.rename_playlist(db, pl, name)
db.commit()
count, total, cover = queries.playlist_totals(db, pl.id)
return _summary(pl, count, total, cover=cover)
count, total, left_out, cover = queries.playlist_totals(db, pl.id)
return _summary(pl, count, (total, left_out), cover=cover)
@router.delete("/{playlist_id}")
@@ -371,24 +406,17 @@ def delete_playlist(
return {"deleted": playlist_id}
class AddItemIn(VideoRefIn):
# Where the video lands among the list's items, in their stored order (0 = first). Omitted, it
# is appended. The list page's "Undo" of a removal puts the row back where it was with this —
# ONE request, because append-then-reorder leaves it at the end whenever the second call fails.
position: int | None = Field(default=None, ge=0)
@router.post("/{playlist_id}/items")
def add_item(
playlist_id: int,
body: AddItemIn,
body: VideoRefIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = _own_playlist(db, user, playlist_id, lock=True)
video_id = body.video_id
_video_or_404(db, video_id)
service.add_to_playlist(db, pl, video_id, body.position)
service.add_item(db, pl, video_id, mark_dirty=True)
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
@@ -400,58 +428,158 @@ def remove_item(
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
pl = _own_playlist(db, user, playlist_id)
pl = _own_playlist(db, user, playlist_id, lock=True)
service.remove_item(db, pl, video_id, mark_dirty=True)
db.commit()
return {"ok": True, "playlist_id": pl.id, "video_id": video_id}
class MoveItemIn(BaseModel):
class VideoIdsIn(BaseModel):
# Required: with a default, a misnamed key answered 200 having acted on no video.
video_ids: list[str]
def _paired(video_count: int, positions: list[int]) -> None:
"""One place per video, none before the start and none twice: both undo bodies come back from
the client, and the insert's arithmetic counts on places that only rise."""
if len(positions) != video_count:
raise ValueError("positions must pair one-to-one with the videos")
if any(at < 0 for at in positions):
raise ValueError("positions cannot be negative")
if len(set(positions)) != len(positions):
raise ValueError("positions cannot repeat")
class RestoreItemsIn(VideoIdsIn):
# Where each video goes back: an index over the ordered rows, one per video.
positions: list[int] = []
@model_validator(mode="after")
def _one_position_per_video(self):
_paired(len(self.video_ids), self.positions)
return self
class MoveItemsIn(VideoIdsIn):
target_playlist_id: int
@router.post("/{playlist_id}/items/{video_id}/move")
def move_item(
class UndoMoveIn(BaseModel):
from_playlist_id: int
# Required, as `video_ids` is. An empty one is still a real body: the target held them all.
moved: list[str]
already_there: list[str] = []
# One per video of `moved` + `already_there`, in that order.
positions: list[int]
@model_validator(mode="after")
def _one_position_per_video(self):
_paired(len(self.moved) + len(self.already_there), self.positions)
return self
def _list_gone(e: service.ListGone) -> HTTPException:
"""Each list of a two-list write under the reason the route's own read gives it."""
return not_found("playlist") if e.source else not_found("move_target")
def _move_refused(e: service.MoveRefused) -> HTTPException:
return HTTPException(status_code=400, detail={"code": "playlist_move", "reason": e.reason})
def _other_list(db: Session, user: User, playlist_id: int) -> "Playlist":
"""The other list of a move. Its own reason: "this playlist doesn't exist any more" about a list
that is gone was read as the one on screen, which is still there."""
other = queries.own_playlist(db, user, playlist_id)
if other is None:
raise not_found("move_target")
return other
@router.post("/{playlist_id}/items/remove-bulk")
def remove_items(
playlist_id: int,
video_id: str,
body: MoveItemIn,
body: VideoIdsIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Move one video from a playlist into another of the caller's playlists, atomically.
"""Remove many videos from a playlist at once. Returns the ones that went."""
pl = _own_playlist(db, user, playlist_id, lock=True)
removed = service.remove_items(db, pl, body.video_ids)
db.commit()
return {"ok": True, "playlist_id": pl.id, "removed": removed}
@router.post("/{playlist_id}/items/restore")
def restore_items(
playlist_id: int,
body: RestoreItemsIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The "Undo" of a removal: the videos back at their places. Returns how many came back."""
pl = _own_playlist(db, user, playlist_id, lock=True)
added = service.restore_items(db, pl, body.video_ids, body.positions)
db.commit()
return {"ok": True, "playlist_id": pl.id, "added": added}
@router.post("/{playlist_id}/items/move-bulk")
def move_items(
playlist_id: int,
body: MoveItemsIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""Move videos into another of the caller's playlists, atomically.
ONE transaction on purpose. The obvious client-side shape — add to the target, then remove
from the source — leaves the video in BOTH lists whenever the second call fails, which is
exactly the kind of half-applied state this codebase treats as a bug rather than an edge
case. Idempotent: if the video is already in the target, the source row is simply dropped
(moving into a list that already has it is a removal, not a duplicate)."""
from the source — leaves a video in BOTH lists whenever the second call fails. A video the
target already has only leaves the source, and one a concurrent edit (another tab, a YouTube
re-pull) already took out is skipped. Says which landed there and which it already held, for
the "Undo"."""
src = _own_playlist(db, user, playlist_id)
# Its own reason: "this playlist doesn't exist any more" about a TARGET that is gone was read as
# the list on screen — which is still there.
dst = queries.own_playlist(db, user, body.target_playlist_id)
if dst is None:
raise not_found("move_target")
dst = _other_list(db, user, body.target_playlist_id)
try:
service.move_item(db, src, dst, video_id)
moved, already_there = service.move_items(db, src, dst, body.video_ids)
except service.MoveRefused as e:
# A same-list move is a bad request; a video a concurrent edit (another tab, a YouTube
# re-pull) already took out is gone.
status = 400 if e.reason == "same_playlist" else 404
raise HTTPException(
status_code=status, detail={"code": "playlist_move", "reason": e.reason}
) from None
raise _move_refused(e) from None
except service.ListGone as e:
raise _list_gone(e) from None
db.commit()
return {"ok": True, "from_playlist_id": src.id, "to_playlist_id": dst.id, "video_id": video_id}
return {
"ok": True,
"from_playlist_id": src.id,
"to_playlist_id": dst.id,
"moved": moved,
"already_there": already_there,
}
class ReorderItemsIn(BaseModel):
video_ids: list[str] = []
@router.post("/{playlist_id}/items/undo-move")
def undo_move(
playlist_id: int,
body: UndoMoveIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
"""The "Undo" of a move, back into this playlist in one transaction. Returns how many came back."""
src = _own_playlist(db, user, playlist_id)
dst = _other_list(db, user, body.from_playlist_id)
try:
added = service.undo_move(db, src, dst, body.moved, body.already_there, body.positions)
except service.MoveRefused as e:
raise _move_refused(e) from None
except service.ListGone as e:
raise _list_gone(e) from None
db.commit()
return {"ok": True, "playlist_id": src.id, "added": added}
@router.put("/{playlist_id}/order")
def reorder_items(
playlist_id: int,
body: ReorderItemsIn,
body: VideoIdsIn,
user: User = Depends(current_user),
db: Session = Depends(get_db),
) -> dict:
+1
View File
@@ -432,6 +432,7 @@ def item_detail(
"summary": (it.summary_hu or it.summary) if hu else it.summary,
"year": it.year,
"duration_seconds": it.duration_s,
"size_bytes": it.size_bytes,
"playable": it.playable,
# Playback-class detail: HW/SW transcode split + the codec/bit-depth the info-page chip shows.
"transcode_hw": it.transcode_hw,
+33 -15
View File
@@ -234,6 +234,15 @@ def person_image(u: str, request: Request) -> Response:
return imgcache.response(data, ct)
def _supersede_resume_pushes(user_id: int, item_id: int, now: datetime) -> None:
"""A watched mark supersedes every resume push still in flight for the item: stamping it as the
latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep the map
small) removed the very guard, and an older resume push could then land at Plex AFTER the
scrobble and revive a resume point on a watched item. Eviction bounds the map."""
with _push_throttle_lock:
_last_push_at[(user_id, item_id)] = now
def _push_resume_guarded(
user_id: int,
item_id: int,
@@ -314,13 +323,14 @@ def item_progress(
`_should_push_resume`). `final` (sent on pause / seek / pagehide / unmount) pushes immediately and
resets the throttle window, so Plex sees a settled position the instant the user pauses or seeks."""
it = item_or_404(db, rating_key)
# A player parked mid-item — playing or PAUSED — checkpoints every 10 s, so this is the signal
# A player parked mid-item — playing or PAUSED — keeps checkpointing (every 10 s, once a minute
# in a hidden silent page), so this is the signal
# that such a viewer still wants their HLS session: the one thing `last_access` could not see,
# since it moved only on a file fetch and a paused player fetches nothing (see
# stream.reap_idle). It is NOT every open player: `usePlexProgress.flush` sends nothing once the
# finish latch is up, or before the first position is known. Those are the states in which
# reaping is the right answer anyway, and a viewer who scrubs back into a reaped session gets
# the same 404 → reclaim the player already handles.
# the same 410 → reclaim the player already handles.
plex_stream.touch_session(user.id, str(rating_key))
position = max(0, int(body.position_seconds))
duration = int(body.duration_seconds)
@@ -329,12 +339,7 @@ def item_progress(
near_end = duration > 0 and position > duration - _FINISH_MARGIN_S
if near_end:
flipped = service.mark_finished(db, user.id, it.id, now)
# The finish supersedes every resume push still in flight for this item: stamping it as the
# latest push makes `_resume_push_superseded` drop them. Popping the entry instead (to keep
# the map small) removed the very guard, and an older resume push could then land at Plex
# AFTER the scrobble and revive a resume point on a watched item. Eviction bounds the map.
with _push_throttle_lock:
_last_push_at[(user.id, it.id)] = now
_supersede_resume_pushes(user.id, it.id, now)
db.commit()
# Only the checkpoint that flipped the row scrobbles: a later checkpoint of the same finished
# item — or a second one racing this one — would otherwise bump Plex's viewCount again.
@@ -352,7 +357,11 @@ def item_progress(
push_now = _should_push_resume(user.id, it.id, now, final)
# Marked dirty before the (background) push so the reconcile knows to re-push if it fails, and so
# `push_state_to_plex` can settle synced_to_plex once Plex confirms.
service.checkpoint(db, user, it.id, st, position, now, dirty=push_now)
if not service.checkpoint(db, user, it.id, st, position, now, dirty=push_now):
# Marked watched a moment ago: this is the finishing player's late tick (see
# `service.checkpoint`), and the answer is the same one a near-end checkpoint gets.
db.commit()
return {"ok": True, "status": "watched", "position_seconds": 0}
db.commit()
if push_now:
_push_watch(
@@ -408,6 +417,8 @@ def item_state(
_push_watch(background, db, user.id, it.id, rating_key, "unwatched")
return {"ok": True, "status": "new"}
service.set_status(db, user, it.id, st, status)
if status == "watched":
_supersede_resume_pushes(user.id, it.id, datetime.now(timezone.utc))
db.commit()
if status == "watched" and prev_status != "watched":
_push_watch(background, db, user.id, it.id, rating_key, "watched")
@@ -427,6 +438,10 @@ def _bulk_state(
always reads the committed marks."""
pushable = plex_watch.link_for_push(db, user_id) is not None
to_push = service.bulk_state(db, user_id, eps, watched)
if watched:
now = datetime.now(timezone.utc)
for item_id, _rating_key, _action in to_push:
_supersede_resume_pushes(user_id, item_id, now)
if pushable and to_push:
# One coalesced background task for the whole batch (reuses a single session + Plex client),
# not one task per episode — a 200-ep show mark is 1 task, not 200.
@@ -595,7 +610,9 @@ def stream_hls_file(
is per (key, viewer, offset, tag) — so resolving by the key alone handed the old player the new
session's `seg_12.ts`, produced from a different offset under `-copyts`: wrong
bytes with a 200, which the player has no way to notice. A request for a
session that is no longer the registered one is a 404, and the player reclaims at the playhead.
session that is no longer the registered one is a 410, and the player reclaims at the playhead.
410, not 404: a 404 here means "that segment is not there yet", worth hls.js's retries, and the
player can only read the status (a gone session retried for half a minute behind that 404).
That reclaim is also why a request with no live session is no longer auto-started here: the
auto-start could only ever begin at 0, silently throwing away the viewer's position."""
@@ -611,7 +628,7 @@ def stream_hls_file(
raise not_found("file")
s = plex_stream.current_session(user.id, str(rating_key))
if s is None or s.id != session_id:
raise refused(404, "session_gone")
raise refused(410, "session_gone")
path = s.dir / filename
# Only a name THIS session can actually produce is worth waiting for: its own playlists, and a
# segment of one of its own renditions whose predecessor is already on disk (ffmpeg writes them
@@ -632,14 +649,15 @@ def stream_hls_file(
try:
content = path.read_bytes()
except OSError:
# The session was torn down between the wait and the read — `_kill` rmtrees the whole
# directory, and nothing holds `_lock` across those two steps. It has to present as the same
# 404 a gone session does: the player's recovery is keyed on the STATUS (`code === 404`), so
# The session was torn down between the wait and the read — teardown moves the whole
# directory aside and deletes it, and nothing holds `_lock` across those two steps. It has
# to present as the same 410 a gone session does: the player's recovery is keyed on the
# STATUS, so
# a 500 here skipped the reclaim and ended playback with "couldn't play" instead of
# restarting at the playhead. Ordinary trigger: `loadSession` awaits the new session — which
# kills and rmtrees the old one — before destroying the old hls instance, so the outgoing
# instance's in-flight segment request races the rmtree on every seek-restart.
raise refused(404, "session_gone") from None
raise refused(410, "session_gone") from None
return Response(
content=content,
media_type=_HLS_CT[ext],
+3 -1
View File
@@ -183,7 +183,9 @@ def playlist_remove_item(
class PlexItemKeysIn(BaseModel):
item_rating_keys: list[str] = []
# Required: with a default, a misnamed key answered 200 having acted on no item, and a reorder
# took it for an empty order.
item_rating_keys: list[str]
class PlexItemAddBulkIn(PlexItemKeysIn):
+9 -2
View File
@@ -176,8 +176,15 @@ def watch_meta(token: str, db: Session = Depends(get_db)) -> dict:
returns only `needs_password` until the viewer unlocks it (no title/thumbnail leak)."""
link = _link_or_404(db, token)
if link.password_hash:
# Which kind it opens, for the password prompt's title — never what is in it.
return {"needs_password": True, "kind": "folder" if link.folder_id is not None else "file"}
# Which kind it opens, for the password prompt's words — never what is in it. `video` tells
# a video (watched) from any other file (opened); the address is only a hint, every
# prefix opens every link.
opens = linksmod.opens(db, link)
return {
"needs_password": True,
"kind": "folder" if opens == "folder" else "file",
"video": opens == "video",
}
meta = _opened(db, link, token)
db.commit()
return meta
+2 -1
View File
@@ -128,7 +128,8 @@ def update_view(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "view_name_taken", name=view.name) from None
# The name asked for: the rollback has put the view's old name back on `view`.
raise refused(409, "view_name_taken", name=name) from None
return _serialize(view)
+2 -1
View File
@@ -314,7 +314,8 @@ def search_youtube(
class ClearSearchIn(BaseModel):
video_ids: list[str] = []
# Required: a default read a misnamed key as an empty list, a 200 that cleared nothing.
video_ids: list[str]
@router.post("/clear")
+4 -2
View File
@@ -48,8 +48,9 @@ def list_tags(user: User = Depends(current_user), db: Session = Depends(get_db))
"category": tag.category,
"system": tag.user_id is None,
"channel_count": count,
"subscribed_count": subscribed,
}
for tag, count in queries.tags_with_counts(db, user)
for tag, count, subscribed in queries.tags_with_counts(db, user)
]
@@ -107,7 +108,8 @@ def update_tag(
db.commit()
except IntegrityError:
db.rollback()
raise refused(409, "tag_name_taken", name=tag.name) from None
# The name asked for: the rollback has put the tag's old name back on `tag`.
raise refused(409, "tag_name_taken", name=name) from None
return {
"ok": True,
"id": tag.id,
+7 -2
View File
@@ -443,15 +443,20 @@ def _default(s: ConfigSpec):
return getattr(settings, s.default_attr)
# Every reason `set_value` refuses with. `refusal.setting_refused` looks each one up in its map, which
# the refusal scan holds to exactly these keys: a reason the map lacks would answer a 500.
SettingReason = Literal["not_a_number", "too_low", "too_high", "not_a_choice", "no_encryption_key"]
class InvalidSetting(ValueError):
"""A value `set_value` refuses, as a reason the UI translates (`errors.refused.setting_<reason>`)
plus what the sentence needs — the English message is for logs only. A ValueError still, so a
caller that only needs "refused or not" keeps working."""
def __init__(self, key: str, reason: str, **params: object):
def __init__(self, key: str, reason: SettingReason, **params: object):
super().__init__(f"{key}: {reason} {params}")
self.key = key
self.reason = reason
self.reason: SettingReason = reason
self.params = params
+2 -2
View File
@@ -7,7 +7,7 @@
# `pip install -r requirements.lock && pip check` must pass. In place so far: urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177); yt-dlp 2026.7.4 -> 2026.8.19
# (it marks the clients that need a stream PO token, so the downloader stops fetching into a 403);
# certifi 2026.7.22 -> 2026.10.19 (GHSA-6q48-6c7w-9gw2); brotli 1.2.0 -> 1.2.1 (PYSEC-2026-4322);
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616).
# Authlib 1.7.2 -> 1.9.1 (CVE-2026-96760, CVE-2026-114616); joserfc 1.7.4 -> 1.7.6 (CVE-2026-119283, CVE-2026-119284).
APScheduler==3.11.3
Authlib==1.9.1
Mako==1.4.1
@@ -39,7 +39,7 @@ httptools==0.8.0
httpx==0.28.1
idna==3.18
itsdangerous==2.2.0
joserfc==1.7.4
joserfc==1.7.6
markdown-it-py==4.2.0
mdurl==0.1.2
mutagen==1.48.1
+738
View File
@@ -0,0 +1,738 @@
"""Every refusal the backend can answer, read from its syntax tree: each `{"code": …, "reason": …}` detail,
with every reason its code can be sent with.
The frontend's contract (`i18n/backendIds.contract.test.ts`) runs this and requires a sentence for each
code and reason in both languages. A scan that tries to follow every shape opens a silent gap with each one
it learns, and following how a value flows through Python (variables, scopes, aliases, a dict changed in
place) never closes. So the CODE keeps to a narrow form, and this reads a CLOSED list of it, reporting
everything else, never guessing:
- where a detail stands: where it is sent — `HTTPException`'s second argument or `detail=`, the `"detail"`
value of a `JSONResponse`'s content dict — through the arms of `a if c else b`, or as what a module-level
factory returns when every return of it is a detail and every reference to it is a call sent as one. A
detail held in a variable is not followed;
- the code: one literal. A `**` spread, or a key that is not a literal, stands BEFORE a literal `code` and
`reason`, both of which the detail then carries: the later key wins, so no spread can change them;
- a reason: a string literal (`None` sends none: the site counts as reasonless); both arms of
`a if c else b`; a function's local bound only by plain `x = …`; a module constant reached as `mod.NAME`;
a call's `Literal` return annotation;
- a parameter typed with a `Literal` (basedpyright holds every value passed in), or a `str` one of a
module-level helper (`refused`, `not_found`), followed to every call, each of which must pass it by
position or by name; a parameter its body binds again is unread;
- an attribute: `e.x` of the exception an `except C as e` catches, or of a parameter typed `C`, when a
method of `C` declares `self.x: <Literal>`;
- a map lookup (`_SETTING_REASONS[exc.reason]`): the map's values, and its keys must be exactly what the
index can be.
A name bound twice at module level, inside a module-level `if`/`try`/loop, named by a `global`, or imported
from two places is not looked up; nor is a local a lambda or a comprehension binds, or one a `nonlocal` or
`global` names. A module constant or map that anything stores into, deletes from or (a map) uses but by
`map[k]` or a read method is unread. A reference named like a followed helper that the scan cannot resolve
(`import app.x`, a relative import, an app module that does not bind it) is unread. Anything else is UNREAD,
or an unread code, and the contract fails on it unless no screen shows that code. Stdlib only: it runs on
the host, beside the frontend's tests (`python -I backend/tests/refusal_sites.py` → JSON).
"""
import ast
import json
import pathlib
import sys
from collections.abc import Iterator
from dataclasses import dataclass, field
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
# Files whose `"code"` dicts are data, not refusals: a subtitle track's status payload, TMDB country codes.
NOT_REFUSALS = {
"plex/subtitles.py": "a subtitle track's status payload (its own `code`)",
"plex/tmdb.py": "a production country's ISO code",
}
# What a map may be used by besides `map[k]`: a method that only reads it.
READS = {"get", "items", "keys", "values"}
Node = ast.AST
FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
SCOPES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
COMPREHENSIONS = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
COMPOUND = (
ast.If,
ast.Try,
ast.TryStar,
ast.For,
ast.AsyncFor,
ast.While,
ast.With,
ast.AsyncWith,
ast.Match,
)
@dataclass
class Res:
values: set[str] = field(default_factory=set)
unread: list[str] = field(default_factory=list)
none: bool = False # an arm sends no reason
def __or__(self, other: "Res") -> "Res":
return Res(self.values | other.values, self.unread + other.unread, self.none or other.none)
@dataclass
class Module:
path: str # relative to app/, posix
name: str # dotted, `app.…`
tree: ast.Module
parents: dict[Node, Node]
# local name → (module, symbol or None for the module)
imports: dict[str, tuple[str, str | None]]
top: dict[str, Node] # module-level bindings: def, class, or the assignment
ambiguous: set[
str
] # bound more than once, conditionally, by a `global`, or imported from two places
slots: set[int] # the ids of the nodes sent as a detail
def _own_scope(node: Node) -> Iterator[Node]:
"""`node` and what it holds in its own scope: a nested def, class or lambda is itself, not its body."""
stack = [node]
while stack:
n = stack.pop()
yield n
if n is not node and isinstance(n, SCOPES):
continue
stack.extend(ast.iter_child_nodes(n))
def _stored(target: Node) -> set[str]:
"""The names an assignment target binds, not the ones it reads (`LABELS[k] = v` binds none)."""
return {
n.id for n in ast.walk(target) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)
}
def _binds(node: Node) -> set[str]:
"""The names one node binds in the scope it sits in."""
if isinstance(node, ast.Assign):
return set().union(*(_stored(t) for t in node.targets))
if isinstance(node, (ast.AnnAssign, ast.AugAssign, ast.For, ast.AsyncFor, ast.NamedExpr)):
return _stored(node.target)
if isinstance(node, (ast.With, ast.AsyncWith)):
return set().union(*(_stored(i.optional_vars) for i in node.items if i.optional_vars))
if isinstance(node, (ast.Import, ast.ImportFrom)):
return {(a.asname or a.name).split(".")[0] for a in node.names}
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
return {node.name}
if isinstance(node, (ast.MatchAs, ast.MatchStar, ast.ExceptHandler)) and node.name:
return {node.name}
return set()
def _sender(call: ast.Call) -> str | None:
return getattr(call.func, "id", None) or getattr(call.func, "attr", None)
def detail_slots(tree: Node) -> Iterator[tuple[ast.Call, Node | None]]:
"""Every place a detail is sent from, with what it holds: `HTTPException`'s second argument or
`detail=` (None: sent without one), and the `"detail"` value of a `JSONResponse`'s content dict (a
content without one is an answer, not a refusal)."""
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
kw = {k.arg: k.value for k in node.keywords}
if _sender(node) == "HTTPException":
yield node, kw.get("detail", node.args[1] if len(node.args) > 1 else None)
elif _sender(node) == "JSONResponse":
content = kw.get("content", node.args[0] if node.args else None)
if isinstance(content, ast.Dict):
for k, v in zip(content.keys, content.values, strict=True):
if isinstance(k, ast.Constant) and k.value == "detail":
yield node, v
def _code_dict(node: Node) -> bool:
"""A `{"code": …}` dict, or a `dict(code=…)` call."""
if isinstance(node, ast.Dict):
return any(isinstance(k, ast.Constant) and k.value == "code" for k in node.keys)
return (
isinstance(node, ast.Call)
and getattr(node.func, "id", None) == "dict"
and any(k.arg == "code" for k in node.keywords)
)
def _module_name(path: str) -> str:
parts = path.removesuffix(".py").split("/")
if parts[-1] == "__init__":
parts = parts[:-1]
return ".".join(["app", *parts])
def _load(sources: dict[str, str]) -> dict[str, Module]:
mods: dict[str, Module] = {}
for path, src in sources.items():
tree = ast.parse(src)
parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)}
origins: dict[str, set[tuple[str, str | None]]] = {}
ambiguous: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
for a in node.names:
origins.setdefault(a.asname or a.name, set()).add((node.module, a.name))
elif isinstance(node, ast.Import):
for a in node.names:
if a.asname:
origins.setdefault(a.asname, set()).add((a.name, None))
elif isinstance(node, ast.Global):
ambiguous |= set(node.names)
top: dict[str, Node] = {}
count: dict[str, int] = {}
ambiguous |= {name for name, o in origins.items() if len(o) > 1}
for stmt in tree.body:
if isinstance(stmt, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
top[stmt.name] = stmt
elif isinstance(stmt, ast.Assign):
for t in stmt.targets:
if isinstance(t, ast.Name):
top[t.id] = stmt
elif isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name):
top[stmt.target.id] = stmt
for n in [stmt] if isinstance(stmt, SCOPES) else _own_scope(stmt):
for name in _binds(n):
count[name] = count.get(name, 0) + 1
if isinstance(stmt, COMPOUND):
ambiguous.add(name)
ambiguous |= {name for name, c in count.items() if c > 1}
imports = {name: next(iter(o)) for name, o in origins.items() if name not in ambiguous}
slots = {id(d) for _, d in detail_slots(tree) if d is not None}
name = _module_name(path)
mods[name] = Module(path, name, tree, parents, imports, top, ambiguous, slots)
return mods
class Scan:
def __init__(self, sources: dict[str, str]):
self.mods = _load(sources)
self.errors: list[str] = []
self._following: set[tuple[str, str, str]] = set()
self._followed: dict[tuple[str, str, str], Res] = {}
self._resolving: set[int] = set()
self._factory: dict[int, bool] = {}
self._changed: dict[int, bool] = {}
# --- names ---------------------------------------------------------------------------------------
def lookup(
self, m: Module, name: str, seen: frozenset[tuple[str, str]] = frozenset()
) -> tuple[Module, Node] | tuple[str, None] | None:
"""What a module-level name is: (its module, its binding), or (a module's name, None)."""
if name in m.ambiguous:
return None
if name in m.top:
return m, m.top[name]
if name in m.imports:
mod, sym = m.imports[name]
if sym is None:
return (mod, None) if mod in self.mods else None
if f"{mod}.{sym}" in self.mods:
return f"{mod}.{sym}", None
if mod in self.mods and (mod, sym) not in seen:
return self.lookup(self.mods[mod], sym, seen | {(mod, sym)})
return None
def lookup_expr(self, m: Module, e: Node) -> tuple[Module, Node] | tuple[str, None] | None:
if isinstance(e, ast.Name):
return None if self.binder(m, e) is not None else self.lookup(m, e.id)
if isinstance(e, ast.Attribute):
base = self.lookup_expr(m, e.value)
if base and base[1] is None and isinstance(base[0], str):
return self.lookup(self.mods[base[0]], e.attr)
return None
def binder(self, m: Module, e: ast.Name) -> Node | None:
"""The innermost scope `e` is bound in: a function (a parameter, or a binding in its own body), a
lambda (a parameter), a comprehension (its target); None: a module-level name."""
node = m.parents.get(e)
while node is not None:
if isinstance(node, (*FUNCS, ast.Lambda)):
a = node.args
params = [*a.posonlyargs, *a.args, *a.kwonlyargs, a.vararg, a.kwarg]
if e.id in {p.arg for p in params if p}:
return node
if isinstance(node, FUNCS) and self.assignments(node, e.id):
return node
elif isinstance(node, COMPREHENSIONS) and any(
e.id in _stored(g.target) for g in node.generators
):
return node
node = m.parents.get(node)
return None
def at(self, m: Module, e: Node) -> str:
return f"{m.path}:{getattr(e, 'lineno', '?')}: {ast.unparse(e)}"
def enclosing(self, m: Module, e: Node, kind: type | tuple[type, ...]) -> Node | None:
node = m.parents.get(e)
while node is not None and not isinstance(node, kind):
node = m.parents.get(node)
return node
def assignments(self, fn: Node, name: str) -> list[Node]:
"""What binds `name` in this function's own body (not in a nested one)."""
return [n for n in _own_scope(fn) if n is not fn and name in _binds(n)]
def caught_as(self, m: Module, e: Node, name: str) -> ast.ExceptHandler | None:
node = m.parents.get(e)
while node is not None and not isinstance(node, FUNCS):
if isinstance(node, ast.ExceptHandler) and node.name == name:
return node
node = m.parents.get(node)
return None
def references(
self, m: Module, target: Node
) -> Iterator[tuple[Module, Node, ast.Call | None, bool]]:
"""Every reference to a module-level def or class: the call it makes (None: not a call), and
whether it resolved. A reference named like it that the scan cannot resolve is yielded too."""
name = getattr(target, "name", None)
for cm in self.mods.values():
for node in ast.walk(cm.tree):
if not isinstance(node, (ast.Name, ast.Attribute)) or not isinstance(
node.ctx, ast.Load
):
continue
if (node.id if isinstance(node, ast.Name) else node.attr) != name:
continue
parent = cm.parents.get(node)
if isinstance(parent, ast.Attribute) and parent.value is node:
continue
found = self.lookup_expr(cm, node)
if found:
if not (found[0] is m and found[1] is target):
continue
elif not self.unresolved(cm, node):
continue
call = parent if isinstance(parent, ast.Call) and parent.func is node else None
yield cm, node, call, found is not None
def unresolved(self, m: Module, node: Node) -> bool:
"""A name, or a dotted path, the app could mean that the scan cannot resolve: not a local, nor an
outside module's (an attribute of an object the function holds is not one)."""
root = node
while isinstance(root, ast.Attribute):
root = root.value
if not isinstance(root, ast.Name) or self.binder(m, root) is not None:
return False
if root.id in m.ambiguous or (root.id not in m.top and root.id not in m.imports):
return True
if root.id in m.top:
return False
if root is node: # imported from an app module that does not bind it where the scan can see
mod = m.imports[root.id][0]
return (mod == "app" or mod.startswith("app.")) and self.lookup(m, root.id) is None
found = self.lookup(m, root.id)
return (
found is not None and found[1] is None
) # an app module whose dotted path went nowhere
def changed(self, m: Module, stmt: Node, name: str, *, only_read: bool) -> bool:
"""Something in the app stores into, deletes or rebinds the module-level `name` that `stmt` binds,
or (`only_read`, a map) uses it but by `map[k]` or a method that only reads it."""
key = id(stmt) * 2 + only_read
if key in self._changed:
return self._changed[key]
own = stmt.targets if isinstance(stmt, ast.Assign) else [getattr(stmt, "target", None)]
out = False
for cm in self.mods.values():
for node in ast.walk(cm.tree):
if out:
break
if not (
isinstance(node, ast.Name)
and node.id == name
or isinstance(node, ast.Attribute)
and node.attr == name
) or (cm is m and node in own):
continue
found = self.lookup_expr(cm, node)
if not (found and found[0] is m and found[1] is stmt):
continue
if not isinstance(node.ctx, ast.Load):
out = True
elif only_read:
p = cm.parents.get(node)
out = not (
isinstance(p, ast.Subscript)
and p.value is node
and isinstance(p.ctx, ast.Load)
or isinstance(p, ast.Attribute)
and p.attr in READS
)
self._changed[key] = out
return out
# --- types ---------------------------------------------------------------------------------------
def ann_values(self, m: Module, ann: Node | None, depth: int = 0) -> set[str] | None:
"""The values a `Literal[…]` annotation, or an alias of one, allows; None when it is not one."""
if isinstance(ann, ast.Subscript) and getattr(ann.value, "id", None) == "Literal":
items = ann.slice.elts if isinstance(ann.slice, ast.Tuple) else [ann.slice]
if all(isinstance(i, ast.Constant) and isinstance(i.value, str) for i in items):
return {i.value for i in items}
return None
if isinstance(ann, (ast.Name, ast.Attribute)) and depth < 8:
found = self.lookup_expr(m, ann)
if found and isinstance(found[0], Module) and isinstance(found[1], ast.Assign):
return self.ann_values(found[0], found[1].value, depth + 1)
return None
def classes(self, m: Module, e: Node | None) -> list[tuple[Module, ast.ClassDef]] | None:
found = self.lookup_expr(m, e) if e is not None else None
if found and isinstance(found[0], Module) and isinstance(found[1], ast.ClassDef):
return [(found[0], found[1])]
return None
def typed(self, m: Module, at: Node, name: str) -> list[tuple[Module, ast.ClassDef]] | None:
"""The class `name` holds where `at` sits: the one an `except C as name` catches, or a parameter's
annotation (a parameter its body binds again holds no known class)."""
handler = self.caught_as(m, at, name)
if handler is not None:
return self.classes(m, handler.type)
fn = self.enclosing(m, at, FUNCS)
if fn is None:
return None
params = {a.arg: a for a in [*fn.args.posonlyargs, *fn.args.args, *fn.args.kwonlyargs]}
if name in params and not self.assignments(fn, name):
return self.classes(m, params[name].annotation)
return None
def attr_values(self, cm: Module, cls: ast.ClassDef, attr: str) -> set[str] | None:
"""An attribute's `Literal` values, from `self.attr: T` in a method of the class."""
for node in ast.walk(cls):
if (
isinstance(node, ast.AnnAssign)
and isinstance(node.target, ast.Attribute)
and node.target.attr == attr
and getattr(node.target.value, "id", None) == "self"
):
return self.ann_values(cm, node.annotation)
return None
# --- reasons -------------------------------------------------------------------------------------
def resolve(self, m: Module, e: Node) -> Res:
if isinstance(e, ast.Constant):
if e.value is None:
return Res(none=True)
if isinstance(e.value, str):
return Res({e.value})
if isinstance(e, ast.IfExp):
return self.resolve(m, e.body) | self.resolve(m, e.orelse)
if isinstance(e, ast.Name):
return self.resolve_name(m, e)
if isinstance(e, ast.Attribute):
return self.resolve_attr(m, e)
if isinstance(e, ast.Call):
found = self.lookup_expr(m, e.func)
if found and isinstance(found[0], Module) and isinstance(found[1], FUNCS):
v = self.ann_values(found[0], found[1].returns)
if v is not None:
return Res(v)
if isinstance(e, ast.Subscript):
return self.resolve_lookup(m, e)
return Res(unread=[self.at(m, e)])
def resolve_name(self, m: Module, e: ast.Name) -> Res:
if self.caught_as(m, e, e.id) is not None:
return Res(unread=[self.at(m, e)]) # the exception itself, not a reason
fn = self.binder(m, e)
if fn is None:
return Res(unread=[self.at(m, e)])
if not isinstance(fn, FUNCS):
return Res(unread=[f"{self.at(m, e)} (bound by a lambda or a comprehension)"])
if any(isinstance(n, (ast.Nonlocal, ast.Global)) and e.id in n.names for n in ast.walk(fn)):
return Res(unread=[f"{self.at(m, e)} (named by a nonlocal or a global)"])
a = fn.args
params = {p.arg: p for p in [*a.posonlyargs, *a.args, *a.kwonlyargs]}
assigned = self.assignments(fn, e.id)
if e.id in params:
if assigned:
return Res(unread=[f"{self.at(m, e)} (a parameter its body binds again)"])
return self.resolve_param(m, fn, params[e.id], e)
out = Res(unread=[] if assigned else [self.at(m, e)]) # `*args`, `**kwargs`
for s in assigned:
if not (isinstance(s, ast.Assign) and all(isinstance(t, ast.Name) for t in s.targets)):
out = out | Res(unread=[self.at(m, e)])
elif id(s) in self._resolving:
out = out | Res(unread=[f"{self.at(m, e)} (reached again while it is resolved)"])
else:
self._resolving.add(id(s))
out = out | self.resolve(m, s.value)
self._resolving.discard(id(s))
return out
def resolve_param(self, m: Module, fn: Node, param: ast.arg, e: Node) -> Res:
v = self.ann_values(m, param.annotation)
if v is not None:
return Res(v)
if param.annotation is not None and getattr(param.annotation, "id", None) != "str":
return Res(unread=[self.at(m, e)])
return self.follow(m, fn, param)
def follow(self, m: Module, fn: Node, param: ast.arg) -> Res:
"""A `str` parameter of a helper: what every call of the helper passes it."""
assert isinstance(fn, FUNCS)
key = (m.name, fn.name, param.arg)
if key in self._followed:
return self._followed[key]
where = f"{m.path}:{fn.lineno}: {fn.name}({param.arg})"
if key in self._following:
return Res(unread=[f"{where} reached again while it is followed"])
if m.top.get(fn.name) is not fn or fn.name in m.ambiguous:
return Res(unread=[f"{where} is not a module-level helper"])
self._following.add(key)
positional = [a.arg for a in [*fn.args.posonlyargs, *fn.args.args]]
by_name = param not in fn.args.posonlyargs
out = Res()
for cm, node, call, resolved in self.references(m, fn):
if not resolved:
out = out | Res(
unread=[f"{self.at(cm, node)} (a reference the scan cannot resolve)"]
)
continue
if call is None:
out = out | Res(unread=[f"{self.at(cm, node)} (the helper used as a value)"])
continue
kw = {k.arg: k.value for k in call.keywords}
given = param.arg in positional and positional.index(param.arg) < len(call.args)
if by_name and param.arg in kw:
out = out | self.resolve(cm, kw[param.arg])
elif any(isinstance(a, ast.Starred) for a in call.args):
out = out | Res(unread=[self.at(cm, call)])
elif given:
out = out | self.resolve(cm, call.args[positional.index(param.arg)])
else:
out = out | Res(unread=[f"{self.at(cm, call)} (does not pass {param.arg})"])
self._followed[key] = out # once followed, its key answers from here: never reached again
return out
def resolve_attr(self, m: Module, e: ast.Attribute) -> Res:
base = e.value
if isinstance(base, ast.Name):
typed = self.typed(m, e, base.id)
if typed is not None:
out = Res()
for cm, cls in typed:
v = self.attr_values(cm, cls, e.attr)
out = out | (
Res(v)
if v is not None
else Res(
unread=[f"{self.at(m, e)} ({cls.name}.{e.attr} has no Literal type)"]
)
)
return out
found = self.lookup_expr(m, e)
if (
found
and isinstance(found[0], Module)
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
and isinstance(found[1].value, ast.Constant)
):
if self.changed(found[0], found[1], e.attr, only_read=False):
return Res(unread=[f"{self.at(m, e)} (bound again or deleted elsewhere)"])
return self.resolve(found[0], found[1].value)
return Res(unread=[self.at(m, e)])
def resolve_lookup(self, m: Module, e: ast.Subscript) -> Res:
found = self.lookup_expr(m, e.value)
if not (
found
and isinstance(found[0], Module)
and isinstance(found[1], (ast.Assign, ast.AnnAssign))
):
return Res(unread=[self.at(m, e)])
table = found[1].value
if not isinstance(table, ast.Dict) or not all(
isinstance(k, ast.Constant) and isinstance(k.value, str) for k in table.keys
):
return Res(unread=[self.at(m, e)])
name = e.value.attr if isinstance(e.value, ast.Attribute) else getattr(e.value, "id", "")
if self.changed(found[0], found[1], name, only_read=True):
return Res(unread=[f"{self.at(m, e)} (the map is changed after it is built)"])
out = Res()
for v in table.values:
out = out | self.resolve(found[0], v)
index = self.resolve(m, e.slice)
keys = {k.value for k in table.keys if isinstance(k, ast.Constant)}
if index.unread or index.none or index.values != keys:
self.errors.append(
f"{self.at(m, e)}: the map's keys {sorted(keys)} are not what its index can be "
f"{sorted(index.values)}{' + unread ' + str(index.unread) if index.unread else ''}"
)
return out
# --- sites ---------------------------------------------------------------------------------------
def is_detail(self, node: Node | None) -> bool:
if isinstance(node, ast.IfExp):
return self.is_detail(node.body) and self.is_detail(node.orelse)
return _code_dict(node)
def factory(self, m: Module, fn: Node) -> bool:
"""A module-level function each return of which is a detail, and each reference to which is a
call sent as one (`quota_detail`)."""
assert isinstance(fn, FUNCS)
if id(fn) in self._factory:
return self._factory[id(fn)]
returns = [n for n in _own_scope(fn) if isinstance(n, ast.Return)]
ok = (
m.top.get(fn.name) is fn # a name bound twice: its calls are unresolved below
and bool(returns)
and all(self.is_detail(r.value) for r in returns)
)
for cm, _, call, resolved in self.references(m, fn) if ok else []:
ok = resolved and id(call) in cm.slots # a bare reference is no call: never sent
if not ok:
break
self._factory[id(fn)] = ok
return ok
def placed(self, m: Module, d: Node) -> bool:
"""`d` is sent as a detail: from a sender's slot, through the arms of `a if c else b`, or as what a
factory returns."""
holder, p = d, m.parents.get(d)
while isinstance(p, ast.IfExp) and holder is not p.test:
holder, p = p, m.parents.get(p)
if id(holder) in m.slots:
return True
fn = self.enclosing(m, p, FUNCS) if isinstance(p, ast.Return) else None
return fn is not None and self.factory(m, fn)
def sayable(self, m: Module, d: Node | None) -> bool:
"""What a slot holds is a detail the client can say: a code dict, or a factory's call."""
if isinstance(d, ast.IfExp):
return self.sayable(m, d.body) and self.sayable(m, d.orelse)
if _code_dict(d):
return True
found = self.lookup_expr(m, d.func) if isinstance(d, ast.Call) else None
return bool(
found
and isinstance(found[0], Module)
and isinstance(found[1], FUNCS)
and self.factory(found[0], found[1])
)
def entries(self, d: Node) -> list[tuple[str | None, Node]]:
"""A dict's (key, value) pairs: None for a `**` spread, "?" for a key that is not a literal."""
if isinstance(d, ast.Dict):
return [
(
k.value
if isinstance(k, ast.Constant) and isinstance(k.value, str)
else None
if k is None
else "?",
v,
)
for k, v in zip(d.keys, d.values, strict=True)
]
assert isinstance(d, ast.Call)
return [(None, a) for a in d.args] + [(k.arg, k.value) for k in d.keywords]
def site(self, m: Module, d: Node, out: dict) -> None:
items = self.entries(d)
code = Res()
for v in (v for k, v in items if k == "code"):
code = code | self.resolve(m, v)
problems = [
f"{self.at(m, v)} (a spread in dict())"
for k, v in items
if k is None and isinstance(d, ast.Call)
] + [
f"{self.at(m, v)} (a spread or a computed key the code and the reason do not follow)"
for i, (k, v) in enumerate(items)
if k in (None, "?") and not {"code", "reason"} <= {k2 for k2, _ in items[i + 1 :]}
]
if not self.placed(m, d):
problems.append(f"{self.at(m, d)} (a detail put where the scan cannot follow it)")
if code.unread or code.none or len(code.values) != 1 or problems:
out["unreadCodes"] += problems or [self.at(m, d)]
return
entry = out["codes"].setdefault(
next(iter(code.values)), {"reasons": set(), "unread": [], "sites": 0, "reasonless": []}
)
entry["sites"] += 1
res, has_reason = Res(), False
for k, v in items:
if k == "reason":
res, has_reason = res | self.resolve(m, v), True
if not has_reason or res.none:
entry["reasonless"].append(f"{m.path}:{d.lineno}")
entry["reasons"] |= res.values
entry["unread"] += res.unread
def other_details(self) -> list[tuple[str, int, str | None]]:
"""Each slot that sends what the client cannot say — nothing, or anything but a code dict or a
factory's call: (its file, its line, the function it is sent from)."""
out = []
for m in self.mods.values():
for call, d in detail_slots(m.tree):
if self.sayable(m, d):
continue
fn = self.enclosing(m, call, FUNCS)
out.append((m.path, call.lineno, getattr(fn, "name", None)))
return sorted(out)
def run(self) -> dict:
out: dict = {
"codes": {},
"unreadCodes": [],
"errors": self.errors,
"excluded": {},
"aliases": {},
"notRefusals": sorted(NOT_REFUSALS),
}
for m in self.mods.values():
for name, stmt in m.top.items():
if (
isinstance(stmt, ast.Assign)
and isinstance(stmt.value, ast.Subscript)
and (v := self.ann_values(m, stmt.value))
):
out["aliases"].setdefault(m.path, {})[name] = sorted(v)
for node in ast.walk(m.tree):
if not _code_dict(node):
continue
if m.path in NOT_REFUSALS:
out["excluded"][m.path] = out["excluded"].get(m.path, 0) + 1
continue
self.site(m, node, out)
for entry in out["codes"].values():
entry["reasons"] = sorted(entry["reasons"])
return out
def scan(sources: dict[str, str]) -> dict:
return Scan(sources).run()
def app_sources(app: pathlib.Path = APP) -> dict[str, str]:
return {
p.relative_to(app).as_posix(): p.read_text(encoding="utf-8")
for p in sorted(app.rglob("*.py"))
if "__pycache__" not in p.parts
}
def scan_app(app: pathlib.Path = APP) -> dict:
return scan(app_sources(app))
if __name__ == "__main__":
json.dump(
scan_app(pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else APP),
sys.stdout,
indent=1,
sort_keys=True,
)
+106
View File
@@ -125,3 +125,109 @@ def test_the_single_playlist_repull_shares_the_pull_cooldown(db, as_user, make_u
r = c.post("/api/playlists/sync-youtube")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "sync_cooldown"}
def _pushable_list(db, make_user, monkeypatch, plroutes):
"""A local (never exported) list and a user with the write scope; the merge itself is stubbed —
the limiter is under test, not the YouTube round trip."""
from app import auth
from app.reader.models import Playlist
user = make_user()
pl = Playlist(user_id=user.id, name="Local", source="local")
db.add(pl)
db.commit()
monkeypatch.setattr(auth, "has_write_scope", lambda u: True)
monkeypatch.setattr(plroutes, "plan_merge", lambda d, u, p, live=None: {"units_estimate": 0})
monkeypatch.setattr(plroutes, "merge_playlist", lambda d, u, p, live=None: {"added": 0})
return user, pl
def test_the_push_side_goes_on_cooldown_after_its_allowance(db, as_user, make_user, monkeypatch):
# SEC-18: push-plan reads the live list and push writes it, both against the instance-wide
# YouTube budget. A press in the UI is one plan + one push, so the allowance is generous — but a
# scripted loop stops at it, with a reason the UI can say.
from app.routes import playlists as plroutes
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
c = as_user(user)
allowance = plroutes.PUSH_ALLOWANCE
codes = [c.get(f"/api/playlists/{pl.id}/push-plan").status_code for _ in range(allowance)]
assert codes == [200] * allowance
r = c.get(f"/api/playlists/{pl.id}/push-plan")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
def test_plan_and_push_draw_on_one_allowance(db, as_user, make_user, monkeypatch):
# The asymmetric fix again, avoided up front: a limit on one of the two routes leaves the same
# drain open next door.
from app.routes import playlists as plroutes
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
c = as_user(user)
for i in range(plroutes.PUSH_ALLOWANCE):
if i % 2:
r = c.get(f"/api/playlists/{pl.id}/push-plan")
else:
r = c.post(f"/api/playlists/{pl.id}/push")
assert r.status_code == 200, (i, r.json())
r = c.post(f"/api/playlists/{pl.id}/push")
assert r.status_code == 429
assert r.json()["detail"] == {"code": "playlist", "reason": "push_cooldown"}
def test_a_refused_push_spends_no_allowance(db, as_user, make_user, monkeypatch):
# The allowance counts pushes that run: Watch later (never pushable) and a list another sync
# holds are refused before any YouTube call, so they must not leave the next real press at 429.
from contextlib import contextmanager
from app.reader.models import Playlist
from app.routes import playlists as plroutes
user, pl = _pushable_list(db, make_user, monkeypatch, plroutes)
wl = Playlist(user_id=user.id, name="Watch later", kind="watch_later")
db.add(wl)
db.commit()
c = as_user(user)
over = plroutes.PUSH_ALLOWANCE + 1
assert {c.get(f"/api/playlists/{wl.id}/push-plan").status_code for _ in range(over)} == {400}
assert {c.post(f"/api/playlists/{wl.id}/push").status_code for _ in range(over)} == {400}
@contextmanager
def busy(_db, _id):
yield False
with monkeypatch.context() as m:
m.setattr(plroutes, "sync_claim", busy)
assert {c.post(f"/api/playlists/{pl.id}/push").status_code for _ in range(over)} == {409}
assert c.post(f"/api/playlists/{pl.id}/push").status_code == 200
def test_a_mirrored_youtube_list_is_pushable(db, as_user, make_user, monkeypatch):
# A mirror of the user's own YouTube list takes local edits (`dirty`) and syncs them back; only
# Watch later is refused.
from app.reader.models import Playlist
from app.routes import playlists as plroutes
user, _ = _pushable_list(db, make_user, monkeypatch, plroutes)
mirror = Playlist(
user_id=user.id, name="Mine", source="youtube", yt_playlist_id="PLx", dirty=True
)
db.add(mirror)
db.commit()
assert as_user(user).get(f"/api/playlists/{mirror.id}/push-plan").status_code == 200
def test_the_push_cooldown_is_per_user(db, as_user, make_user, monkeypatch):
from app.routes import playlists as plroutes
a, pa = _pushable_list(db, make_user, monkeypatch, plroutes)
b, pb = _pushable_list(db, make_user, monkeypatch, plroutes)
ca = as_user(a)
codes = [
ca.get(f"/api/playlists/{pa.id}/push-plan").status_code
for _ in range(plroutes.PUSH_ALLOWANCE + 1)
]
assert codes == [200] * plroutes.PUSH_ALLOWANCE + [429]
assert as_user(b).get(f"/api/playlists/{pb.id}/push-plan").status_code == 200
+20
View File
@@ -72,3 +72,23 @@ def test_the_sync_status_counts_the_callers_subscriptions(db, make_user, as_user
assert (body["subscriptions"], body["channels_total"]) == (1, 1)
mine = as_user(user).get("/api/sync/my-status").json()
assert (mine["channels_total"], mine["channels_recent_pending"]) == (1, 1)
def test_a_channel_lists_when_it_entered_this_users_list_not_when_the_channel_did(
db, make_user, as_user
):
from datetime import datetime, timezone
user = make_user()
db.add(Channel(id="UCa", title="A", created_at=datetime(2026, 6, 10, tzinfo=timezone.utc)))
db.flush()
db.add(
Subscription(
user_id=user.id,
channel_id="UCa",
created_at=datetime(2026, 10, 12, 8, 30, tzinfo=timezone.utc),
)
)
db.commit()
listed = as_user(user).get("/api/channels").json()
assert [(c["id"], c["added_at"]) for c in listed] == [("UCa", "2026-10-12T08:30:00+00:00")]
+3
View File
@@ -531,6 +531,9 @@ def test_a_youtube_clip_hands_the_range_to_yt_dlp(monkeypatch, tmp_path):
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *_a):
return False
+209
View File
@@ -0,0 +1,209 @@
"""The editor's crop lands on the part of the picture the frame was drawn on, also when a video's
pixels are not square. The browser reports such a video by its shown size (320 stored, 640 shown),
and ffmpeg's `crop` counts stored pixels: measured on a real clip, a frame over the middle half kept
half the frame's red surroundings. So a crop carries the size it was measured on.
Real ffmpeg: the pixels of the output are the oracle.
"""
import shutil
import subprocess
from pathlib import Path
import pytest
from app.config import settings
from app.downloads import edit, pipeline, service, storage
from app.downloads import upload as uploadmod
from app.downloads.models import DownloadJob, MediaAsset
pytestmark = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not available")
def _source(path: Path, size: str, box: str, sar: int, rotate: int = 0) -> None:
"""Red, with a green box: a crop of the box holds no red. `rotate` tags the file to be shown
turned by that many degrees counter-clockwise, as a phone records a portrait video."""
tag = ["-display_rotation:v:0", str(rotate)] if rotate else []
subprocess.run(
[
"ffmpeg", "-y", "-loglevel", "error",
"-f", "lavfi", "-i", f"color=c=red:s={size}:d=2:r=10",
"-f", "lavfi", "-i", "sine=duration=2",
"-vf", f"drawbox={box}:c=lime:t=fill,setsar={sar}",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", str(path),
],
check=True,
) # fmt: skip
if tag:
tagged = path.with_name("tagged-" + path.name)
subprocess.run(
["ffmpeg", "-y", "-loglevel", "error", *tag, "-i", str(path), "-c", "copy", str(tagged)],
check=True,
) # fmt: skip
tagged.replace(path)
def _red_and_green(path: Path) -> tuple[int, int]:
raw = subprocess.run(
["ffmpeg", "-loglevel", "error", "-i", str(path), "-frames:v", "1",
"-f", "rawvideo", "-pix_fmt", "rgb24", "-"],
check=True,
capture_output=True,
).stdout # fmt: skip
red = green = 0
for i in range(0, len(raw) - 2, 3):
if raw[i] > 180 and raw[i + 1] < 80:
red += 1
elif raw[i + 1] > 180 and raw[i] < 80:
green += 1
return red, green
def _cut_asset(db, make_user, src: Path, spec: dict, then=lambda: None) -> MediaAsset:
user = make_user()
source = uploadmod.ingest_upload(
db, user.id, filename="clip.mp4", src_path=src, size=src.stat().st_size
)
assert source.status == "done"
then()
job = service.enqueue_edit(db, user.id, source, spec)
asset = db.get(MediaAsset, job.asset_id)
job.status, asset.status = "running", "downloading" # the worker's claim
db.commit()
pipeline._process_edit(job.id, asset.id)
db.expire_all()
job, asset = db.get(DownloadJob, job.id), db.get(MediaAsset, job.asset_id)
assert (job.status, job.error, asset.status) == ("done", None, "ready")
return asset
def _cut(db, make_user, tmp_path, src: Path, crop: dict, **spec) -> Path:
asset = _cut_asset(db, make_user, src, {"crop": crop, **spec})
return storage.abs_path(str(tmp_path), asset.rel_path)
def test_a_crop_of_a_video_with_wide_pixels_keeps_what_the_frame_showed(
db, make_user, tmp_path, monkeypatch
):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
# What the editor sends for the middle half of this video: the browser shows it at 640 x 240.
final = _cut(
db, make_user, tmp_path, src,
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
) # fmt: skip
red, green = _red_and_green(final)
assert (red, green) == (0, 160 * 120)
def test_a_crop_of_square_pixels_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_source(src, "480x640", "x=120:y=160:w=240:h=320", sar=1)
final = _cut(
db, make_user, tmp_path, src,
{"x": 120, "y": 160, "w": 240, "h": 320, "frame_w": 480, "frame_h": 640},
) # fmt: skip
assert _red_and_green(final) == (0, 240 * 320)
def test_a_crop_stored_before_it_carried_its_frame_is_cut_as_it_always_was():
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50}})
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
cmd = edit.build_edit_cmd(Path("in.mp4"), Path("out.mp4"), spec, "mp4")
assert cmd[cmd.index("-vf") + 1].startswith("crop=100:50:4:6,")
@pytest.mark.parametrize("frame", [{"frame_w": 0, "frame_h": 240}, {"frame_w": "x", "frame_h": 2}])
def test_a_frame_that_cannot_be_read_is_not_divided_by(frame):
spec = edit.normalize_edit_spec({"crop": {"x": 4, "y": 6, "w": 100, "h": 50, **frame}})
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
def test_a_crop_of_a_join_keeps_what_the_frame_showed(db, make_user, tmp_path, monkeypatch):
"""A cut-list joins through `filter_complex`, a crop path of its own."""
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
final = _cut(
db, make_user, tmp_path, src,
{"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240},
segments=[{"start_s": 0, "end_s": 0.5}, {"start_s": 1, "end_s": 1.5}], accurate=True,
) # fmt: skip
assert _red_and_green(final) == (0, 160 * 120)
def test_a_crop_of_a_rotated_video_is_cut_where_it_was_drawn(db, make_user, tmp_path, monkeypatch):
"""The browser shows a rotated file upright, so the frame was drawn on the upright picture."""
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
# Stored 320 x 240, green on its left half; shown turned a quarter, 240 x 320.
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
# Turned counter-clockwise, the stored left half is the shown bottom half.
final = _cut(
db, make_user, tmp_path, src,
{"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320},
) # fmt: skip
assert _red_and_green(final) == (0, 240 * 160)
def test_a_clip_ffprobe_cannot_size_gets_the_size_its_crop_cut(
db, make_user, tmp_path, monkeypatch
):
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_source(src, "320x240", "x=80:y=60:w=160:h=120", sar=2)
def probe_fails():
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
asset = _cut_asset(
db, make_user, src,
{"crop": {"x": 160, "y": 60, "w": 320, "h": 120, "frame_w": 640, "frame_h": 240}},
then=probe_fails,
) # fmt: skip
# Half the shown width of a 320-wide file: 160 stored, not the 320 the frame counted.
assert (asset.width, asset.height) == (160, 120)
def test_a_rotated_clip_ffprobe_cannot_size_gets_the_upright_size_its_crop_cut(
db, make_user, tmp_path, monkeypatch
):
"""The frame was drawn on the upright picture, which is the stored one turned: so is its cut."""
monkeypatch.setattr(settings, "files_root", str(tmp_path))
src = tmp_path / "in.mp4"
_source(src, "320x240", "x=0:y=0:w=160:h=240", sar=1, rotate=90)
def probe_fails():
monkeypatch.setattr(pipeline.probe, "probe_media", lambda _p: (None,) * 5)
asset = _cut_asset(
db, make_user, src,
{"crop": {"x": 0, "y": 160, "w": 240, "h": 160, "frame_w": 240, "frame_h": 320}},
then=probe_fails,
) # fmt: skip
# What ffmpeg cut (the rotated test above): 240 x 160, not the stored 320 x 120.
assert (asset.width, asset.height) == (240, 160)
@pytest.mark.parametrize("key", ["frame_w", "x"])
def test_an_infinite_number_in_a_crop_is_dropped_not_a_crash(key):
crop = {"x": 4, "y": 6, "w": 100, "h": 50, "frame_w": 640, "frame_h": 240}
spec = edit.normalize_edit_spec({"crop": {**crop, key: float("inf")}})
if key == "frame_w":
assert spec["crop"] == {"x": 4, "y": 6, "w": 100, "h": 50}
else:
assert "crop" not in spec
+412
View File
@@ -0,0 +1,412 @@
"""SEC-17: the egress gate judges the address a guarded download actually dials.
Two loopback addresses stand in for the two sides: 127.0.0.2 plays a public host, 127.0.0.1 our own
network. The gate under test refuses exactly 127.0.0.1, so every assertion is about the judgement and
the wiring, with real sockets, a real yt-dlp and a real ffmpeg — no mocked transport.
"""
import contextlib
import http.server
import ipaddress
import socket
import socketserver
import struct
import threading
import time
from pathlib import Path
import pytest
import yt_dlp
from yt_dlp.utils import download_range_func
from app.downloads import adapters, egress, errors, formats
PUBLIC = "127.0.0.2"
INTERNAL = "127.0.0.1"
def _forbid_internal(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
return str(ip) == INTERNAL
@pytest.fixture
def gate():
g = egress.start_gate(forbidden=_forbid_internal)
yield g
g.shutdown()
g.server_close()
class _Listener:
"""A TCP listener that only counts who connected — the internal service under attack."""
def __init__(self, host: str = INTERNAL, port: int = 0):
self.hits = 0
self.sock = socket.socket()
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((host, port))
self.sock.listen()
self.port = self.sock.getsockname()[1]
threading.Thread(target=self._serve, daemon=True).start()
def _serve(self):
while True:
try:
conn, _ = self.sock.accept()
except OSError:
return
self.hits += 1
with contextlib.suppress(OSError):
conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
conn.close()
def close(self):
self.sock.close()
def _public_server(routes: dict, port: int = 0, directory: Path | None = None):
"""An HTTP server on the 'public' address: `routes` maps a path to (status, headers); anything
else is served from `directory`."""
class H(http.server.SimpleHTTPRequestHandler):
def __init__(self, *a, **k):
super().__init__(*a, directory=str(directory or "."), **k)
def log_message(self, *a):
pass
def do_GET(self):
if self.path in routes:
status, headers, body = routes[self.path]
self.send_response(status)
for k, v in headers.items():
self.send_header(k, v)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
return
super().do_GET()
srv = socketserver.ThreadingTCPServer((PUBLIC, port), H)
srv.daemon_threads = True
threading.Thread(target=srv.serve_forever, daemon=True).start()
return srv
def _guarded_opts(tmp_path: Path, gate_url: str, **extra) -> dict:
opts = formats.build_ydl_opts({}, str(tmp_path / "%(id)s.%(ext)s"), lambda d: None, egress=True)
opts.update(retries=0, fragment_retries=0, extractor_retries=0, socket_timeout=5)
opts["proxy"] = gate_url # the test's gate (it lets 127.0.0.2 through), not the worker's
opts.update(extra)
return opts
def _download_error(opts: dict, url: str) -> str:
with pytest.raises(yt_dlp.utils.DownloadError) as exc, egress.open_ydl(opts) as ydl:
ydl.extract_info(url, download=True)
return str(exc.value)
# --- the judgement ---------------------------------------------------------------------------------
def test_a_name_with_any_forbidden_answer_is_refused():
def resolve(answers):
return lambda host, port, *a: [(2, 1, 6, "", (ip, port)) for ip in answers]
assert egress.vetted_address(
"x", 80, resolve=resolve([PUBLIC]), forbidden=_forbid_internal
) == (
PUBLIC,
80,
)
assert (
egress.vetted_address(
"x", 80, resolve=resolve([PUBLIC, INTERNAL]), forbidden=_forbid_internal
)
is None
)
def no_answer(*a):
raise socket.gaierror("no such name")
assert egress.vetted_address("x", 80, resolve=no_answer, forbidden=_forbid_internal) is None
def test_the_real_judgement_refuses_our_own_networks():
# The worker's gate uses urlguard's judgement unchanged — the same one the enqueue check uses.
for addr in ("127.0.0.1", "10.1.2.3", "172.18.0.5", "192.168.1.1", "169.254.169.254"):
assert egress.vetted_address(addr, 80, forbidden=egress.urlguard.is_forbidden) is None, addr
assert egress.vetted_address("93.184.215.14", 80, forbidden=egress.urlguard.is_forbidden)
def test_the_gate_dials_the_address_it_vetted_not_a_second_answer(gate):
# DNS rebinding: a name that answers public for the check and private for the fetch. The gate
# resolves once per connection and dials that answer.
public = _public_server({"/": (200, {}, b"public")})
internal = _Listener(INTERNAL, public.server_address[1])
calls = []
def rebinding(host, port, *a):
calls.append(host)
ip = PUBLIC if len(calls) == 1 else INTERNAL
return [(2, 1, 6, "", (ip, port))]
gate.resolve = rebinding
try:
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
s.sendall(
f"GET http://rebind.test:{public.server_address[1]}/ HTTP/1.1\r\n"
f"Host: rebind.test\r\n\r\n".encode()
)
answer = b"".join(iter(lambda: s.recv(4096), b""))
assert answer.endswith(b"public")
assert calls == ["rebind.test"]
assert internal.hits == 0
finally:
public.shutdown()
internal.close()
def test_a_tunnel_into_the_network_is_refused_before_it_is_dialled(gate):
internal = _Listener()
try:
with socket.create_connection(("127.0.0.1", gate.server_address[1])) as s:
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
answer = s.recv(4096)
assert answer.startswith(b"HTTP/1.1 403")
assert internal.hits == 0
finally:
internal.close()
def test_a_client_that_hangs_up_mid_transfer_is_not_an_error(gate, monkeypatch):
# yt-dlp closes a connection once it has read what it needed; seen live as a traceback the gate
# printed for every such download.
errors_seen = []
monkeypatch.setattr(gate, "handle_error", lambda *a: errors_seen.append(a))
public = _public_server({"/big": (200, {}, b"x" * 4_000_000)})
try:
s = socket.create_connection(("127.0.0.1", gate.server_address[1]))
s.sendall(f"GET http://{PUBLIC}:{public.server_address[1]}/big HTTP/1.1\r\n\r\n".encode())
s.recv(1024)
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close() # a reset, not a polite close
time.sleep(0.5)
assert errors_seen == []
finally:
public.shutdown()
# --- a real yt-dlp through it ----------------------------------------------------------------------
def test_a_redirect_into_the_network_reaches_it_unguarded_and_not_through_the_gate(tmp_path, gate):
internal = _Listener()
public = _public_server(
{"/v.mp4": (302, {"Location": f"http://{INTERNAL}:{internal.port}/v.mp4"}, b"")}
)
url = f"http://{PUBLIC}:{public.server_address[1]}/v.mp4"
try:
# The hole, as it was: yt-dlp follows the 3xx into our network.
bare = {"outtmpl": str(tmp_path / "bare" / "%(id)s.%(ext)s"), "quiet": True, "retries": 0}
with contextlib.suppress(yt_dlp.utils.DownloadError), yt_dlp.YoutubeDL(bare) as ydl:
ydl.extract_info(url, download=True)
assert internal.hits >= 1
before = internal.hits
_download_error(_guarded_opts(tmp_path, gate.url), url)
assert internal.hits == before
finally:
public.shutdown()
internal.close()
def test_a_live_and_a_dead_internal_port_leave_the_same_text(tmp_path, gate):
live = _Listener()
dead = _Listener()
dead_port = dead.port
dead.close()
public = _public_server(
{
"/live.mp4": (302, {"Location": f"http://{INTERNAL}:{live.port}/x"}, b""),
"/dead.mp4": (302, {"Location": f"http://{INTERNAL}:{dead_port}/x"}, b""),
}
)
base = f"http://{PUBLIC}:{public.server_address[1]}"
try:
stored = []
for name in ("live", "dead"):
msg = _download_error(_guarded_opts(tmp_path / name, gate.url), f"{base}/{name}.mp4")
stored.append(errors.user_error_text("url", msg, errors.classify(msg)))
assert stored[0] == stored[1]
assert live.hits == 0
finally:
public.shutdown()
live.close()
def test_a_format_fetched_past_the_gate_is_never_chosen(tmp_path):
# ffmpeg speaks rtmp, rtsp, tcp… directly, where no HTTP proxy is asked. A guarded download keeps
# only the formats fetched over http(s).
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9")
info = {
"id": "x",
"title": "x",
"extractor": "generic",
"extractor_key": "Generic",
"webpage_url": f"http://{PUBLIC}/page",
"formats": [
{"format_id": "rtmp", "url": f"rtmp://{INTERNAL}/live", "ext": "flv", "height": 1080},
{"format_id": "web", "url": f"http://{PUBLIC}/a.mp4", "ext": "mp4", "height": 360},
],
}
with egress.open_ydl(opts) as ydl:
chosen = ydl.process_ie_result(dict(info), download=False)
assert chosen["format_id"] == "web"
only_rtmp = dict(info, formats=[info["formats"][0]])
with (
pytest.raises(yt_dlp.utils.YoutubeDLError, match="Requested format is not available"),
egress.open_ydl(opts) as ydl,
):
ydl.process_ie_result(only_rtmp, download=False)
def test_a_format_yt_dlp_fetches_itself_is_kept():
# f4m, ism and the DASH generator fetch through yt-dlp's own urlopen; a bare m3u8 goes to the
# native HLS fetcher under hls_prefer_native (set for every guarded download). All ask the gate.
kept = [
{"format_id": p, "url": f"https://{PUBLIC}/{p}", "protocol": p}
for p in ("f4m", "ism", "m3u8", "m3u8_native", "http_dash_segments_generator", "https")
]
raw = {"format_id": "rtmp", "url": f"rtmp://{PUBLIC}/live", "protocol": "rtmp"}
_, info = egress.HttpOnlyFormats().run({"formats": [*kept, raw]})
assert [f["format_id"] for f in info["formats"]] == [f["format_id"] for f in kept]
def test_a_single_url_result_past_the_gate_loses_its_url():
_, info = egress.HttpOnlyFormats().run({"url": f"rtmp://{INTERNAL}/live"})
assert info["url"] is None
_, info = egress.HttpOnlyFormats().run({"url": f"https://{PUBLIC}/a.mp4"})
assert info["url"] == f"https://{PUBLIC}/a.mp4"
def test_the_format_filter_says_no_phase_of_its_own(tmp_path):
# It is a post-processor only to run before the format choice; the job's postprocessor hook
# must not see it (it painted every guarded job "processing" before a byte was fetched).
calls: list[dict] = []
opts = _guarded_opts(tmp_path, "http://127.0.0.1:9", postprocessor_hooks=[calls.append])
with egress.open_ydl(opts) as ydl:
ydl.run_all_pps("pre_process", {"id": "x", "formats": []})
assert calls == []
def test_an_allowed_tunnel_carries_bytes_both_ways(gate):
# The CONNECT path a site served over https takes: a 200, then the bytes the client sent after
# the head (in the same packet here) reach the far end, and its answer comes back.
far = socket.socket()
far.bind((PUBLIC, 0))
far.listen()
def echo():
conn, _ = far.accept()
with conn:
conn.sendall(b"PONG:" + conn.recv(64))
threading.Thread(target=echo, daemon=True).start()
try:
with socket.create_connection(("127.0.0.1", gate.server_address[1]), timeout=5) as s:
s.sendall(f"CONNECT {PUBLIC}:{far.getsockname()[1]} HTTP/1.1\r\n\r\nPING".encode())
got = b""
while b"PONG:PING" not in got:
chunk = s.recv(4096)
if not chunk:
break
got += chunk
assert got.startswith(b"HTTP/1.1 200")
assert got.endswith(b"PONG:PING")
finally:
far.close()
def test_the_workers_own_gate_refuses_our_network():
# Every other gate here is built with a test rule; this is the one `proxy_url()` starts.
internal = _Listener()
try:
port = int(egress.proxy_url().rsplit(":", 1)[1])
with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
s.sendall(f"CONNECT {INTERNAL}:{internal.port} HTTP/1.1\r\n\r\n".encode())
assert s.recv(4096).startswith(b"HTTP/1.1 403")
assert internal.hits == 0
finally:
internal.close()
def _clip_hits(tmp_path: Path, segment: str, proxy: str) -> int:
"""How many connections the internal listener takes while a clip (`download_ranges`, which
yt-dlp hands to ffmpeg even for an HLS format) is cut from a playlist whose one segment is
`segment` — formatted with the listener's address."""
internal = _Listener()
playlist = (
"#EXTM3U\n#EXT-X-VERSION:3\n#EXT-X-TARGETDURATION:2\n#EXTINF:2.0,\n"
+ segment.format(host=INTERNAL, port=internal.port)
+ "\n#EXT-X-ENDLIST\n"
).encode()
public = _public_server(
{"/p.m3u8": (200, {"Content-Type": "application/vnd.apple.mpegurl"}, playlist)}
)
try:
opts = _guarded_opts(tmp_path, proxy, download_ranges=download_range_func([], [(0, 1)]))
with contextlib.suppress(yt_dlp.utils.DownloadError), egress.open_ydl(opts) as ydl:
ydl.extract_info(f"http://{PUBLIC}:{public.server_address[1]}/p.m3u8", download=True)
return internal.hits
finally:
public.shutdown()
internal.close()
def test_a_clip_cut_by_ffmpeg_reaches_a_segment_only_through_the_gate(tmp_path, gate):
# Without the gate, ffmpeg fetches a hostile playlist's segment from inside our network — the
# control that shows this path is real. Through the gate it is never dialled.
segment = "http://{host}:{port}/s.ts"
assert _clip_hits(tmp_path / "bare", segment, proxy="") >= 1
assert _clip_hits(tmp_path / "gated", segment, proxy=gate.url) == 0
def test_ffmpeg_itself_refuses_a_raw_tcp_segment(tmp_path):
# A boundary, not our code: the HTTP proxy cannot see a tcp:// segment, and measured, ffmpeg's
# HLS reader does not open one from an http playlist. This fails if an ffmpeg upgrade starts to.
assert _clip_hits(tmp_path, "tcp://{host}:{port}/", proxy="") == 0
# --- the wiring ------------------------------------------------------------------------------------
def test_only_youtube_downloads_skip_the_gate():
assert adapters.SourceAdapter.egress_guarded is True
assert adapters.resolve_adapter("url", "https://www.facebook.com/x").egress_guarded is True
assert adapters.resolve_adapter("url", "https://vimeo.com/1").egress_guarded is True
assert adapters.resolve_adapter("youtube", "dQw4w9WgXcQ").egress_guarded is False
def test_a_guarded_download_goes_through_the_gate_whatever_its_adapter_adds():
opts = formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, extra={"proxy": "", "hls_prefer_native": False}, egress=True
)
assert opts["proxy"] == egress.proxy_url()
assert opts["hls_prefer_native"] is True
assert opts[egress.GUARDED] is True
assert "proxy" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None, egress=False)
def test_every_yt_dlp_this_app_opens_goes_through_open_ydl():
app = Path(egress.__file__).resolve().parents[1]
offenders = [
f"{p.relative_to(app)}:{n}"
for p in sorted(app.rglob("*.py"))
if p.name != "egress.py"
for n, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1)
if "YoutubeDL(" in line
]
assert offenders == []
+14 -1
View File
@@ -18,7 +18,7 @@ import yt_dlp
from yt_dlp.networking.exceptions import TransportError
from yt_dlp.networking.impersonate import ImpersonateTarget
from app.downloads import adapters, pipeline
from app.downloads import adapters, egress, pipeline
from app.downloads import errors as dlerrors
_URL = "https://www.facebook.com/reel/806379462330494/?fs=e&fs=e"
@@ -73,6 +73,9 @@ def ydl(monkeypatch):
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *exc):
return False
@@ -157,6 +160,16 @@ def test_a_cookie_group_post_downloads_the_posts_own_video(ydl, tmp_path):
# Never the post link: that is where the feed video comes from.
assert web["extracted"] == [watch, watch]
assert [o.get("impersonate") for o in seen[1:]] == [None, _CHROME]
# The post page read and both attempts all go through the egress gate.
assert all(o.get(egress.GUARDED) and o["proxy"] == egress.proxy_url() for o in seen), seen
def test_a_youtube_download_skips_the_gate(ydl, tmp_path):
seen, outcomes, _ = ydl
outcomes += [{"id": "dQw4w9WgXcQ"}]
yt = adapters.resolve_adapter("youtube", "dQw4w9WgXcQ")
pipeline._extract_with_fallback(1, 1, "https://youtu.be/dQw4w9WgXcQ", {}, tmp_path, yt)
assert seen and not any(o.get(egress.GUARDED) or o.get("proxy") for o in seen)
def test_a_group_post_without_a_single_video_fails_saying_so(ydl, tmp_path):
+303
View File
@@ -0,0 +1,303 @@
"""A Facebook download's title when the signed-in read gives only a placeholder.
Measured on prod (2026-11-01): every Facebook download since 2026-10-28 is titled "Facebook" — the
signed-in read (the owner's cookie) now gives that placeholder and NO description, so the adapter had
no post text to fall back on, and the download name copied the placeholder too. A signed-out read of
the same reel (yt-dlp 2026.08.19, as on prod) still carries the post's text in its description.
"""
from sqlalchemy import select, update
from app.config import settings
from app.db import SessionLocal
from app.downloads import adapters, backfill, egress, pipeline
from app.downloads.models import DownloadJob, MediaAsset
_URL = "https://www.facebook.com/reel/1860682468422402/"
# The page yt-dlp says it read for it (`webpage_url`); a group post's link resolves the same way.
_PAGE = "https://www.facebook.com/watch/?v=1860682468422402"
# The measured signed-out read of that reel.
_PUBLIC = {
"title": "163K reactions · 14K shares | sorry not helping #workhumor #officelife | Work dot exe",
"description": "sorry not helping\n\n#workhumor #officelife",
"uploader": "Work dot exe",
}
def _fake_ydl(monkeypatch, outcome):
"""yt-dlp faked at its `YoutubeDL` boundary: records each read (its options and its `url`),
answers `outcome`."""
seen: list[dict] = []
class FakeYDL:
def __init__(self, opts):
self.opts = opts
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *exc):
return False
def extract_info(self, url, download):
seen.append({**self.opts, "url": url})
return outcome
monkeypatch.setattr("yt_dlp.YoutubeDL", FakeYDL)
return seen
def test_a_placeholder_with_nothing_behind_it_takes_the_signed_out_pages_text(monkeypatch):
seen = _fake_ydl(monkeypatch, _PUBLIC)
fb = adapters.FacebookAdapter()
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
assert fb.title(info) == "sorry not helping"
# One metadata-only read, and signed out: no cookie goes with it.
assert len(seen) == 1
assert seen[0]["skip_download"] is True
assert not seen[0].get("cookiefile")
assert seen[0][egress.GUARDED] is True and seen[0]["proxy"] == egress.proxy_url()
def test_a_real_title_reads_nothing_more(monkeypatch):
seen = _fake_ydl(monkeypatch, _PUBLIC)
info = {"title": "Funny Dark Humor Dad Joke!", "description": None}
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
assert seen == []
def test_a_placeholder_with_the_posts_text_behind_it_reads_nothing_more(monkeypatch):
seen = _fake_ydl(monkeypatch, _PUBLIC)
info = {"title": "Video", "description": "Office jokes, part two"}
assert backfill.public_title_info(info, _URL, adapters.FacebookAdapter(), 7) is info
assert seen == []
def test_a_placeholder_the_signed_out_page_cannot_better_stays(monkeypatch):
_fake_ydl(monkeypatch, {"title": "Facebook", "description": None})
fb = adapters.FacebookAdapter()
info = backfill.public_title_info({"title": "Facebook", "description": None}, _URL, fb, 7)
assert fb.title(info) == "Facebook"
def test_another_source_is_never_read_again(monkeypatch):
seen = _fake_ydl(monkeypatch, _PUBLIC)
info = {"title": "Video", "description": None}
assert backfill.public_title_info(info, _URL, adapters.SourceAdapter(), 7) is info
assert seen == []
def test_the_backfill_retitles_a_placeholder_download_and_the_names_that_repeated_it(
db, make_user, monkeypatch
):
seen = _fake_ydl(monkeypatch, _PUBLIC)
owner, other = make_user(), make_user()
asset = MediaAsset(
source_kind="url",
source_ref=_URL,
format_sig="fb",
status="ready",
rel_path="facebook/x.mp4",
poster_path="facebook/x.jpg",
uploader_url="https://www.facebook.com/1",
title="Facebook",
ref_count=2,
)
db.add(asset)
db.flush()
copied = DownloadJob(
user_id=owner.id,
asset_id=asset.id,
source_kind="url",
source_ref=_URL,
status="done",
queue_pos=0,
profile_snapshot={},
display_name="Facebook",
)
renamed = DownloadJob(
user_id=other.id,
asset_id=asset.id,
source_kind="url",
source_ref=_URL,
status="done",
queue_pos=0,
profile_snapshot={},
display_name="Office jokes",
)
db.add_all([copied, renamed])
db.commit()
assert backfill.run()["retitled"] == 1
assert len(seen) == 1
db.expire_all()
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
assert db.get(DownloadJob, copied.id).display_name == "sorry not helping"
assert db.get(DownloadJob, renamed.id).display_name == "Office jokes"
# Run again: nothing left to retitle, and no page read for it.
assert backfill.run()["retitled"] == 0
assert len(seen) == 1
def test_a_name_given_while_the_backfill_runs_stays(db, make_user, monkeypatch):
# The owner renames the download while the backfill reads the page: the owner's name wins, it is
# not overwritten by what was read before it.
_fake_ydl(monkeypatch, _PUBLIC)
owner = make_user()
asset = MediaAsset(
source_kind="url",
source_ref=_URL,
format_sig="fb",
status="ready",
rel_path="facebook/x.mp4",
uploader_url="https://www.facebook.com/1",
title="Facebook",
ref_count=1,
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=owner.id,
asset_id=asset.id,
source_kind="url",
source_ref=_URL,
status="done",
queue_pos=0,
profile_snapshot={},
display_name="Facebook",
)
db.add(job)
db.commit()
read_page = backfill.public_title_info
def renamed_while_reading(*args):
with SessionLocal() as other:
other.execute(
update(DownloadJob)
.where(DownloadJob.id == job.id)
.values(display_name="Office jokes")
)
other.commit()
return read_page(*args)
monkeypatch.setattr(backfill, "public_title_info", renamed_while_reading)
monkeypatch.setattr(backfill.editmod, "ensure_poster", lambda a, root: None)
assert backfill.run()["retitled"] == 1
db.expire_all()
assert db.get(DownloadJob, job.id).display_name == "Office jokes"
def test_the_poster_step_holds_no_row_lock(db, make_user, monkeypatch):
# It can cut a frame for 30 s. A delete locks the asset and then the job: with the job held here
# and the asset still to write, that is a deadlock, and a rename waits out the whole cut.
_fake_ydl(monkeypatch, _PUBLIC)
owner = make_user()
asset = MediaAsset(
source_kind="url",
source_ref=_URL,
format_sig="fb",
status="ready",
rel_path="facebook/x.mp4",
uploader_url="https://www.facebook.com/1",
title="Facebook",
ref_count=1,
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=owner.id,
asset_id=asset.id,
source_kind="url",
source_ref=_URL,
status="done",
queue_pos=0,
profile_snapshot={},
display_name="Facebook",
)
db.add(job)
db.commit()
asset_id, job_id = asset.id, job.id
def poster_finds_rows_free(a, root):
with SessionLocal() as other:
other.execute(
select(MediaAsset).where(MediaAsset.id == asset_id).with_for_update(nowait=True)
)
other.execute(
select(DownloadJob).where(DownloadJob.id == job_id).with_for_update(nowait=True)
)
other.rollback()
return None
monkeypatch.setattr(backfill.editmod, "ensure_poster", poster_finds_rows_free)
assert backfill.run()["retitled"] == 1
def test_a_download_titled_facebook_is_named_from_the_signed_out_page(
db, make_user, monkeypatch, tmp_path
):
# The whole download path with yt-dlp and the media tools faked: the signed-in read gives only
# the placeholder, and the file and the download's name get the post's text.
seen = _fake_ydl(monkeypatch, _PUBLIC)
monkeypatch.setattr(settings, "files_root", str(tmp_path))
monkeypatch.setattr(pipeline.urlguard, "check_url", lambda url: None)
monkeypatch.setattr(pipeline, "_ensure_phone_safe", lambda _j, produced, *_a: produced)
monkeypatch.setattr(
pipeline.probe, "resolve_media_facts", lambda produced, info: ("h264", "aac", 720, 1280, 9)
)
monkeypatch.setattr(pipeline.editmod, "thumbnail_to_jpg", lambda *a: None)
monkeypatch.setattr(pipeline.editmod, "ensure_poster", lambda *a: None)
owner = make_user()
asset = MediaAsset(
source_kind="url", source_ref=_URL, format_sig="fb", status="downloading", ref_count=1
)
db.add(asset)
db.flush()
job = DownloadJob(
user_id=owner.id,
asset_id=asset.id,
source_kind="url",
source_ref=_URL,
status="running",
queue_pos=0,
profile_snapshot={},
)
db.add(job)
db.commit()
def signed_in_read(job_id, asset_id, url, spec, staging, adapter, **kw):
f = staging / "1860682468422402.mp4"
f.write_bytes(b"v" * 100)
return {
"id": "1860682468422402",
"title": "Facebook",
"uploader": "Work dot exe",
"webpage_url": _PAGE,
"requested_downloads": [{"filepath": str(f)}],
}
monkeypatch.setattr(pipeline, "_extract_with_fallback", signed_in_read)
pipeline._download(job.id, asset.id, "url", _URL, {}, owner.id)
# The signed-out read goes to the page yt-dlp read, not the link queued: a group post's link
# resolves to its video, and signed out the post itself is the login wall.
assert [r["url"] for r in seen] == [_PAGE]
db.expire_all()
assert db.get(DownloadJob, job.id).status == "done"
assert db.get(MediaAsset, asset.id).title == "sorry not helping"
assert db.get(DownloadJob, job.id).display_name == "sorry not helping"
+14 -4
View File
@@ -359,6 +359,9 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *exc):
return False
@@ -367,8 +370,8 @@ def test_the_metadata_backfill_names_the_asset_and_not_the_download(sink, monkey
f"ERROR: [youtube] dQw4w9WgXcQ: Video unavailable ({target})"
)
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
assert backfill._extract_meta(url, adapters.YouTubeAdapter(), 11) is None
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
assert backfill.extract_meta(url, adapters.YouTubeAdapter(), 11) is None
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
assert lines and all("dQw4w9WgXcQ" not in json.dumps(o) for o in lines)
@@ -386,14 +389,17 @@ def test_the_metadata_backfill_takes_out_a_generic_name_whatever_it_holds(sink,
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *exc):
return False
def extract_info(self, target, download):
raise yt_dlp.utils.DownloadError("ERROR: [generic] Our Trip: Day 1: Unable to download")
monkeypatch.setattr(backfill.yt_dlp, "YoutubeDL", _Refusing)
assert backfill._extract_meta(url, adapters.SourceAdapter(), 11) is None
monkeypatch.setattr("yt_dlp.YoutubeDL", _Refusing)
assert backfill.extract_meta(url, adapters.SourceAdapter(), 11) is None
lines = [o for o in sink.objects() if "meta extract failed" in o["msg"]]
assert lines and all("Day 1" not in json.dumps(o) for o in lines)
@@ -449,6 +455,9 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
def __enter__(self):
return self
def add_post_processor(self, pp, when="post_process"):
pass # the egress format filter; nothing here to filter
def __exit__(self, *exc):
return False
@@ -463,6 +472,7 @@ def test_a_retried_attempt_names_the_host_and_not_the_video(sink, monkeypatch, t
resolve_url=lambda target, opts: target,
ydl_opts_extra=lambda clients: {},
retriable=lambda text: True,
egress_guarded=False,
)
monkeypatch.setattr(pipeline.yt_dlp, "YoutubeDL", _YDL)
monkeypatch.setattr(pipeline, "_touch_job", lambda job_id: None)
+2 -1
View File
@@ -150,6 +150,7 @@ def test_a_password_folder_link_shows_nothing_until_unlocked(shelf, client):
assert client.get(f"/api/public/watch/{token}").json() == {
"needs_password": True,
"kind": "folder",
"video": False,
}
bare = f"/api/public/watch/{token}/files/{shelf['inside'][0]}/file"
assert client.get(bare).status_code == 403
@@ -312,7 +313,7 @@ def test_a_password_file_link_says_it_opens_a_file(shelf, client):
.json()
)
meta = client.get(f"/api/public/watch/{made['token']}").json()
assert meta == {"needs_password": True, "kind": "file"}
assert meta == {"needs_password": True, "kind": "file", "video": True}
def test_a_file_whose_bytes_are_gone_is_not_offered(shelf, client, db):
+16 -7
View File
@@ -15,16 +15,20 @@ from app.downloads import formats
def _opts(spec):
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None)
return formats.build_ydl_opts(spec, "/tmp/%(id)s.%(ext)s", lambda d: None, egress=False)
def test_adapter_extra_opts_are_merged_in():
# The SourceAdapter contributes source-specific opts (e.g. YouTube's extractor_args) via `extra`.
extra = {"extractor_args": {"youtube": {"player_client": ["web"]}}}
opts = formats.build_ydl_opts({}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra)
opts = formats.build_ydl_opts(
{}, "/tmp/%(id)s.%(ext)s", lambda d: None, extra=extra, egress=False
)
assert opts["extractor_args"]["youtube"]["player_client"] == ["web"]
# No extra → no extractor_args (a generic source contributes nothing).
assert "extractor_args" not in formats.build_ydl_opts({}, "/tmp/x", lambda d: None)
assert "extractor_args" not in formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, egress=False
)
def test_mp4_default_selects_avc1_with_progressive_fallback():
@@ -113,7 +117,7 @@ def test_a_playlist_url_can_only_yield_one_item():
def test_the_remaining_quota_becomes_a_hard_size_ceiling():
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000)
opts = formats.build_ydl_opts({}, "/tmp/x", lambda d: None, max_bytes=5_000_000, egress=False)
assert opts["max_filesize"] == 5_000_000
@@ -122,7 +126,7 @@ def test_no_budget_means_no_size_filter():
# enqueue, not an impossible yt-dlp filter to express here.
for budget in (None, 0, -1):
assert "max_filesize" not in formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, max_bytes=budget
{}, "/tmp/x", lambda d: None, max_bytes=budget, egress=False
)
@@ -130,7 +134,12 @@ def test_a_source_adapter_cannot_raise_the_size_ceiling():
# The adapter merge happens after max_filesize is set, so this pins that no adapter option can
# widen a storage cap — the mirror of the cookiefile rule, in the opposite direction.
opts = formats.build_ydl_opts(
{}, "/tmp/x", lambda d: None, extra={"max_filesize": 999_999_999}, max_bytes=5_000_000
{},
"/tmp/x",
lambda d: None,
extra={"max_filesize": 999_999_999},
max_bytes=5_000_000,
egress=False,
)
assert opts["max_filesize"] == 5_000_000
@@ -197,7 +206,7 @@ def test_yt_dlp_refuses_a_file_over_the_cap_without_raising_and_the_watch_sees_i
for cap, refused in ((1_000, True), (10_000_000, False)):
out = tmp_path / f"out-{cap}"
opts = formats.build_ydl_opts(
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap
{}, str(out / "%(id)s.%(ext)s"), lambda d: None, max_bytes=cap, egress=False
)
with formats.watching_size_cap() as over, yt_dlp.YoutubeDL(opts) as ydl:
ydl.extract_info(url, download=True)
+53 -7
View File
@@ -5,11 +5,11 @@ POST /session: after the last playback, an abandoned session (its ffmpeg still w
film, its directory a whole segmented copy of it) lived until somebody pressed play again.
And it must not take a PAUSED viewer's session: a remux finishes long before the film does, and a
paused player with a full buffer fetches nothing, so "finished and unfetched for 30 s" does not mean
paused player with a full buffer fetches nothing, so "finished and unfetched for a while" does not mean
"abandoned". An earlier fix bought that safety by keeping EVERY finished session for the full
ten-minute idle window — one segmented copy of a film per item, on the volume the download center
shares. Its successor replaced the blanket retention with the signal that already existed: a player
parked mid-item checkpoints its position every 10 s, paused or not, and that route says "still
parked mid-item keeps checkpointing its position, paused or not, and that route says "still
here" (it stops once the item is finished, where reaping is correct).
Both call sites are pinned here, because the bug each time was that only one of them moved: the
@@ -81,8 +81,12 @@ class _Proc:
return 0
_PAST_THE_RULE = stream._FINISHED_IDLE_S + 5
class _Session:
def __init__(self, tmp_path, name: str, finished: bool, idle_s: float):
self.key = name
self.dir = tmp_path / name
self.dir.mkdir()
self.proc = _Proc(finished)
@@ -97,11 +101,29 @@ def sessions(monkeypatch):
def test_the_periodic_pass_reaps_a_finished_session_nobody_has_claimed(tmp_path, sessions):
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=45)
sessions["12345"] = _Session(tmp_path, "done", finished=True, idle_s=_PAST_THE_RULE)
assert stream.reap_idle() == 1
assert sessions == {}
@pytest.mark.parametrize("idle_s", [65, 125])
def test_a_heartbeat_throttled_to_once_a_minute_keeps_a_finished_session(
tmp_path, sessions, idle_s
):
"""A hidden page that makes no sound gets its timers cut to one a minute (measured on a paused and
on a muted player, about a minute after hiding), so that is the viewer's heartbeat there: the
reaper can see a beat over a minute old, and two if one is lost on the way."""
s = _Session(tmp_path, "hidden", finished=True, idle_s=idle_s)
sessions[(1, "12345")] = s
assert stream.reap_idle() == 0
assert (1, "12345") in sessions and s.dir.exists() and not s.proc.killed
def test_an_abandoned_finished_session_is_still_gone_within_minutes(tmp_path, sessions):
sessions[(1, "12345")] = _Session(tmp_path, "gone", finished=True, idle_s=185)
assert stream.reap_idle() == 1
def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limit(
tmp_path, sessions
):
@@ -115,8 +137,32 @@ def test_the_periodic_pass_still_reaps_what_nobody_has_touched_for_the_idle_limi
assert sessions == {}
def test_every_reap_says_which_rule_took_the_session(tmp_path, sessions, caplog):
"""A session that vanishes leaves the viewer a 410 and nothing else: with no line here, a live
stall could only ever be blamed on the reaper by inference."""
sessions["1"] = _Session(
tmp_path, "gone_done", finished=True, idle_s=stream._SESSION_IDLE_S - 5
)
sessions["2"] = _Session(
tmp_path, "gone_live", finished=False, idle_s=stream._SESSION_IDLE_S + 5
)
sessions["3"] = _Session(tmp_path, "kept", finished=False, idle_s=5)
with caplog.at_level("INFO"):
assert stream.reap_idle() == 2
reaped = {
r.attrs["key"]: r.attrs
for r in caplog.records
if getattr(r, "event", "") == "plex.stream.session_reaped"
}
assert set(reaped) == {"gone_done", "gone_live"}
assert reaped["gone_done"]["reason"] == "finished_idle"
assert reaped["gone_live"]["reason"] == "idle"
assert reaped["gone_done"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S - 5, abs=2)
assert reaped["gone_live"]["idle_s"] == pytest.approx(stream._SESSION_IDLE_S + 5, abs=2)
def test_touching_a_session_keeps_it_through_the_short_rule(tmp_path, sessions):
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
sessions[(1, "12345")] = s
stream.touch_session(1, "12345")
assert stream.reap_idle() == 0
@@ -159,7 +205,7 @@ def test_a_paused_viewers_checkpoint_keeps_their_session_alive(
user = User(email="paused@example.com")
db.add(user)
db.commit()
s = _Session(tmp_path, "paused", finished=True, idle_s=90)
s = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
sessions[(user.id, "paused1")] = s
resp = _client(db, user, override_dep).post(
f"/api/plex/item/{it.rating_key}/progress",
@@ -174,7 +220,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
db, monkeypatch, tmp_path, sessions, override_dep
):
"""The opportunistic sweep this route used to run was not scoped to the key being started: it
walked the WHOLE registry with the 30 s finished rule, so pressing play on any item killed a
walked the WHOLE registry with the finished rule, so pressing play on any item killed a
paused viewer's session — the very case the retention fix set out to protect."""
monkeypatch.setattr(state, "_configured_cache", True)
monkeypatch.setattr(plex_paths, "local_media_path", lambda db_, fp: "/local/a.mkv")
@@ -182,7 +228,7 @@ def test_starting_a_session_leaves_another_viewers_paused_one_alone(
user = User(email="starter@example.com")
db.add(user)
db.commit()
paused = _Session(tmp_path, "paused", finished=True, idle_s=90)
paused = _Session(tmp_path, "paused", finished=True, idle_s=_PAST_THE_RULE)
sessions[(999, "someone-else")] = paused
def fake_start(db_, item, user_id, start, audio=None, aoff=0.0, multi=False, transcode=False):
+158
View File
@@ -0,0 +1,158 @@
"""Tearing a session down does not stall everybody else's playback.
Every segment request (`current_session`) and every checkpoint (`touch_session`) takes the one
registry `_lock`. Stopping a session waits up to 3 s for its ffmpeg to exit and then deletes a whole
segmented film, ~1200 files — and that used to happen under the same lock, so one viewer's seek, a
cap cull or a reap pass held up every other viewer's HLS requests for seconds, each one a parked
threadpool worker and its pooled DB connection.
What still has to be atomic is pinned too: a session directory is reused for the same item, offset
and tag, so a new session on that path must not start before the old ffmpeg — which writes by path
— has exited.
"""
import threading
import time
import pytest
from app.plex import stream
class _SlowProc:
"""An ffmpeg that takes a while to exit after TERM: `wait` blocks until `exited` is set."""
def __init__(self, order: list[str], name: str):
self.order = order
self.name = name
self.waiting = threading.Event()
self.exited = threading.Event()
def poll(self):
return 0 if self.exited.is_set() else None
def terminate(self):
pass
def kill(self):
self.exited.set()
def wait(self, timeout=None):
self.waiting.set()
if not self.exited.wait(timeout):
raise stream.subprocess.TimeoutExpired("ffmpeg", timeout)
self.order.append(f"{self.name} exited")
return 0
class _Session:
def __init__(self, directory, proc, key: str):
self.id = f"sid-{key}"
self.key = key
self.dir = directory
self.proc = proc
self.last_access = time.time()
self.transcode = False
@pytest.fixture
def sessions(monkeypatch):
live: dict = {}
monkeypatch.setattr(stream, "_sessions", live)
monkeypatch.setattr(stream, "_dying", {}, raising=False)
return live
def test_stopping_a_session_does_not_hold_up_another_viewers_requests(tmp_path, sessions):
order: list[str] = []
slow = _SlowProc(order, "a")
d = tmp_path / "1_u1_0_0_+0.00"
d.mkdir()
(d / "seg_0.ts").write_bytes(b"x")
sessions[(1, "a")] = _Session(d, slow, "a")
sessions[(2, "b")] = _Session(tmp_path / "2_u2_0_0_+0.00", _SlowProc(order, "b"), "b")
stopper = threading.Thread(target=stream.stop_session, args=(1, "a", "sid-a"))
stopper.start()
try:
assert slow.waiting.wait(2.0), "the stop never waited for its ffmpeg"
got: list = []
reader = threading.Thread(target=lambda: got.append(stream.current_session(2, "b")))
reader.start()
reader.join(1.0)
assert not reader.is_alive(), "a segment request waited on another viewer's teardown"
assert got and got[0] is sessions[(2, "b")]
finally:
slow.exited.set()
stopper.join(5.0)
# The teardown still finishes: the session is gone and nothing of its directory is left behind.
assert (1, "a") not in sessions
assert sorted(p.name for p in tmp_path.iterdir()) == []
def test_a_reap_pass_does_not_hold_up_another_viewers_requests(tmp_path, sessions):
order: list[str] = []
slow = _SlowProc(order, "a")
d = tmp_path / "1_u1_0_0_+0.00"
d.mkdir()
gone = _Session(d, slow, "a")
gone.last_access = time.time() - stream._SESSION_IDLE_S - 5
sessions[(1, "a")] = gone
sessions[(2, "b")] = _Session(tmp_path / "2_u2_0_0_+0.00", _SlowProc(order, "b"), "b")
reaper = threading.Thread(target=stream.reap_idle)
reaper.start()
try:
assert slow.waiting.wait(2.0), "the reap never waited for its ffmpeg"
toucher = threading.Thread(target=stream.touch_session, args=(2, "b"))
toucher.start()
toucher.join(1.0)
assert not toucher.is_alive(), "a checkpoint waited on another viewer's teardown"
finally:
slow.exited.set()
reaper.join(5.0)
assert list(sessions) == [(2, "b")]
assert not d.exists()
class _FakeItem:
rating_key = "12345"
file_path = "/media/movie.mkv"
playable = "remux"
codec_video = "h264"
def test_a_new_session_on_a_dying_ones_path_starts_only_after_it_exited(
tmp_path, sessions, monkeypatch
):
"""The same item at the same second gets the same directory. The old ffmpeg writes by PATH, so
while it is still winding down after TERM it can write a segment — or, on a clean exit, a final
playlist with ENDLIST — into the NEW session's directory."""
order: list[str] = []
procs = iter([_SlowProc(order, "old"), _SlowProc(order, "new")])
def popen(*a, **k):
order.append("popen")
return next(procs)
monkeypatch.setattr(stream, "_HLS_ROOT", tmp_path)
src = tmp_path / "movie.mkv"
src.write_bytes(b"x")
monkeypatch.setattr(stream.paths, "local_media_path", lambda db, fp: src)
monkeypatch.setattr(stream.sysconfig, "get_int", lambda db, k: 2)
monkeypatch.setattr(stream.sysconfig, "get_bool", lambda db, k: False)
monkeypatch.setattr(stream.subprocess, "Popen", popen)
monkeypatch.setattr(stream, "wait_for", lambda *a, **k: True)
first = stream.start_session(db=None, user_id=7, item=_FakeItem(), start_s=0.0)
assert first is not None
stopper = threading.Thread(target=stream.stop_session, args=(7, "12345", first.id))
stopper.start()
assert first.proc.waiting.wait(2.0)
threading.Timer(0.3, first.proc.exited.set).start()
second = stream.start_session(db=None, user_id=7, item=_FakeItem(), start_s=0.0)
stopper.join(5.0)
assert second is not None and second.dir == first.dir
assert order.index("old exited") < order.index("popen", 1)
assert second.dir.is_dir(), "the old teardown deleted the new session's directory"
second.proc.exited.set()
+72 -37
View File
@@ -1,53 +1,88 @@
"""No refusal answers in a hand-written English sentence.
"""No refusal answers in a way the client cannot say.
The UI is bilingual and translates a structured `detail` (`{code, reason, …}`), but a string detail it
can only print as it came — so ~150 of them reached the Hungarian UI verbatim, in the blocking error
dialog and on every inline surface that shows `HttpError.detail`. `app/refusal.py` is the family they
became; this test is what keeps a new one from being written the old way. Pure: it reads the sources.
became; this test is what keeps a new one from being written the old way. It reads the same slots the
refusal scan does (`refusal_sites.detail_slots`), so the two cannot disagree on where a detail is sent.
Pure: it reads the sources.
"""
import ast
import pathlib
import textwrap
APP = pathlib.Path(__file__).resolve().parents[1] / "app"
from tests.refusal_sites import Scan, app_sources
# Answers that are no refusal of the app's, by the function they are sent from: the client shows a
# detail only when it is an object (`lib/api/core.ts`), so these read as its generic line, and no screen
# needs more from them.
NO_REFUSAL = {
("main.py", "setup_gate"): "the 503 before setup: the setup screen is what the user sees",
("main.py", "spa_fallback"): "a bare 404 for an api path no route matches: no screen calls one",
("main.py", "_validation_error"): "FastAPI's own 422 list: a body the client itself got wrong",
("routes/diagnostics.py", "_parsed_events"): "the client-events beacon's 422: nothing shows it",
}
def _is_text(node: ast.AST) -> bool:
"""A detail that is a sentence: a literal, an f-string, `str(exc)`, or `"…" + x`."""
if isinstance(node, ast.Constant) and isinstance(node.value, str):
return True
if isinstance(node, ast.JoinedStr):
return True
if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "str":
return True
if isinstance(node, ast.BoolOp):
return any(_is_text(v) for v in node.values)
if isinstance(node, ast.BinOp):
return _is_text(node.left) or _is_text(node.right)
return False
def _offenders(sources: dict[str, str]) -> list[str]:
return [
f"{path}:{line}"
for path, line, fn in Scan(sources).other_details()
if (path, fn) not in NO_REFUSAL
]
def test_no_http_exception_carries_a_sentence():
offenders = []
for path in sorted(APP.rglob("*.py")):
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
if not (
isinstance(node, ast.Call)
and getattr(node.func, "id", getattr(node.func, "attr", "")) == "HTTPException"
):
continue
kw = {k.arg: k.value for k in node.keywords}
detail = kw.get("detail", node.args[1] if len(node.args) > 1 else None)
if detail is not None and _is_text(detail):
offenders.append(f"{path.relative_to(APP).as_posix()}:{node.lineno}")
def test_every_detail_is_one_the_client_can_say():
offenders = _offenders(app_sources())
assert offenders == [], (
"a refusal with an English sentence for a detail — raise `refused(status, reason)` "
f"(app/refusal.py) and name the reason in both locales: {offenders}"
"a refusal the client can only answer with its generic line — send `{code, reason}` "
f"(`refused(status, reason)`), or list it in NO_REFUSAL with why: {offenders}"
)
def test_the_scan_sees_a_sentence_when_there_is_one():
def test_the_detail_scan_sees_each_shape_the_client_cannot_say():
"""A guard that matches nothing is green over nothing."""
sample = ast.parse('HTTPException(status_code=400, detail=f"bad {x}")').body[0].value
assert _is_text(sample.keywords[1].value)
assert not _is_text(ast.parse('{"code": "refused", "reason": "x"}').body[0].value)
src = textwrap.dedent(
"""
def f(x):
raise HTTPException(409, detail={"reason": "view_name_taken"})
def g(x):
raise HTTPException(409, dict(reason="taken"))
def h(x):
raise HTTPException(status_code=404)
def i(x):
return JSONResponse({"detail": "This instance isn't set up yet."}, status_code=503)
def j(x):
return JSONResponse(content={"detail": {"reason": "x"}}, status_code=409)
def k(x):
raise HTTPException(400, detail=f"bad {x}")
def l(x):
raise HTTPException(404, None)
def n(x, msg):
raise HTTPException(400, detail=msg)
def o(x):
raise HTTPException(400, detail=["a sentence"])
def p(x):
raise HTTPException(400, detail=describe(x))
def q(x):
raise HTTPException(400, detail={"code": "a"} if x else "text")
def r(x):
raise HTTPException(400, detail=nothing(x))
def nothing(x):
pass
def ok(x):
raise HTTPException(409, {"code": "refused", "reason": "x"})
raise HTTPException(409, detail=quota(x))
raise HTTPException(409, detail={"code": "a"} if x else {"code": "b"})
return JSONResponse({"detail": {"code": "too_large"}}, status_code=413)
return JSONResponse({"ok": True})
def quota(x):
return {"code": "quota", "reason": "files"}
def setup_gate(x):
return JSONResponse({"detail": "listed"}, status_code=503)
"""
)
lines = (3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25)
assert _offenders({"main.py": src}) == [f"main.py:{n}" for n in lines]
# A file whose code dicts are data still answers its refusals in a way the client can say.
data = 'def f():\n raise HTTPException(400, "text")\n'
assert _offenders({"plex/tmdb.py": data}) == ["plex/tmdb.py:2"]
+3
View File
@@ -45,6 +45,9 @@ def test_an_interval_override_moves_into_sysconfig_and_back(db):
}
assert conn.execute(text("SELECT to_regclass('scheduler_settings')")).scalar() is None
assert scheduler.load_intervals(db)["rss_poll"] == 25
# The way down rebuilds an index CONCURRENTLY (0117), which waits for every open transaction —
# the read above left this session's open.
db.rollback()
command.downgrade(cfg, "0109_job_runs")
try:
+70
View File
@@ -0,0 +1,70 @@
"""Migration 0119's data step, DB-lane: the stored `transcode_hw` facet follows the rule that it says
whether the GPU does the encode, and a 10-bit decodable item is probed again so its HDR flag decides.
The schema half is checked by test_migrations.py; a facet value left stale is invisible there.
"""
from alembic import command
from alembic.config import Config
from sqlalchemy import text
from app.db import engine
def test_the_stored_facet_is_rewritten_by_the_gpu_encode_rule(db):
cfg = Config("alembic.ini")
command.downgrade(cfg, "0118_videos_shuffle_key")
try:
with engine.begin() as conn:
lib = conn.execute(
text(
"INSERT INTO plex_libraries (plex_key, title, kind, enabled)"
" VALUES ('m1', 'Movies', 'movie', true) RETURNING id"
)
).scalar_one()
def item(key, playable, codec, hw, depth=None, probed=False):
conn.execute(
text(
"INSERT INTO plex_items (rating_key, library_id, kind, title, playable,"
" codec_video, transcode_hw, video_bit_depth, media_probed_at)"
" VALUES (:k, :l, 'movie', :k, :p, :c, :hw, :d,"
" CASE WHEN :pr THEN now() END)"
),
{
"k": key,
"l": lib,
"p": playable,
"c": codec,
"hw": hw,
"d": depth,
"pr": probed,
},
)
item("vc1", "transcode", "VC1", True, 8, probed=True) # the old list said decodable
item("xvid", "transcode", "mpeg4", False)
item("divx", "transcode", "msmpeg4v3", False)
item("av1", "transcode", "av1", False)
item("h8", "transcode", "hevc", True, 8, probed=True)
item("h10", "transcode", "hevc", False, 10, probed=True) # SDR or HDR: unknown here
item("remux", "remux", "h264", None)
finally:
command.upgrade(cfg, "head")
with engine.connect() as conn:
rows = {
k: (hw, probed)
for k, hw, probed in conn.execute(
text("SELECT rating_key, transcode_hw, media_probed_at IS NOT NULL FROM plex_items")
).tuples()
}
assert rows == {
"vc1": (True, True),
"xvid": (True, False),
"divx": (True, False),
"av1": (False, False),
"h8": (True, True),
"h10": (False, False), # re-queued: the probe reads its HDR flag again
"remux": (None, False),
}
+46
View File
@@ -0,0 +1,46 @@
"""A rename onto a name already taken says WHICH name is taken: the one asked for, not the row's
old name (the rollback that follows the conflict puts the old name back on the row). DB lane.
"""
from fastapi.testclient import TestClient
import app.state as state
from app.auth import current_user
from app.db import get_db
from app.main import app
from app.models import User
def _client(db, monkeypatch, override_dep):
u = User(email="taken@example.com")
db.add(u)
db.commit()
monkeypatch.setattr(
state, "_configured_cache", True
) # the request gate 503s an unconfigured app
override_dep(get_db, db.info["request"])
override_dep(current_user, lambda: u)
return TestClient(app)
def test_renaming_a_view_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
c = _client(db, monkeypatch, override_dep)
gaming = c.post("/api/saved-views", json={"name": "gaming", "filters": {}, "source": "feed"})
assert gaming.status_code == 200
hh = c.post("/api/saved-views", json={"name": "hh", "filters": {}, "source": "feed"}).json()
r = c.patch(f"/api/saved-views/{hh['id']}", json={"name": "gaming"})
assert r.status_code == 409
assert r.json()["detail"] == {"code": "refused", "reason": "view_name_taken", "name": "gaming"}
def test_renaming_a_tag_onto_a_taken_name_names_the_taken_one(db, monkeypatch, override_dep):
c = _client(db, monkeypatch, override_dep)
assert c.post("/api/tags", json={"name": "music"}).status_code == 200
news = c.post("/api/tags", json={"name": "news"}).json()
r = c.patch(f"/api/tags/{news['id']}", json={"name": "music"})
assert r.status_code == 409
assert r.json()["detail"] == {"code": "refused", "reason": "tag_name_taken", "name": "music"}
+1 -1
View File
@@ -181,7 +181,7 @@ def test_clearing_a_video_leaves_another_users_state(db, make_user, as_user):
_videos(db, "v1")
db.add(VideoState(user_id=other.id, video_id="v1", status="hidden", position_seconds=90))
db.commit()
assert as_user(me).delete("/api/videos/v1/state").status_code == 200
assert as_user(me).delete("/api/videos/v1/progress").status_code == 200
assert _video_state(db, other) == ("hidden", 90)
+16 -12
View File
@@ -1,4 +1,4 @@
"""Adding a video AT a position: the list page's "Undo" of a removal.
"""Putting a video back AT a position: the list page's "Undo" of a removal.
The row has to come back where it was in ONE request. The two-call shape (append, then send the
whole order) leaves it at the end of the list whenever the second call fails, and the reorder would
@@ -38,14 +38,17 @@ def _ids(db, pl_id):
]
def _add(client, pl_id, body):
return client.post(f"/api/playlists/{pl_id}/items", json=body)
def _restore(client, pl_id, video_id, position):
return client.post(
f"/api/playlists/{pl_id}/items/restore",
json={"video_ids": [video_id], "positions": [position]},
)
def test_a_position_puts_the_video_back_in_its_place(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b", "c"])
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": 1})
r = _restore(as_user(user), pl.id, "vX", 1)
assert r.status_code == 200
assert _ids(db, pl.id) == ["a", "vX", "b", "c"]
@@ -54,36 +57,37 @@ def test_the_position_counts_rows_not_stored_numbers(db, make_user, as_user):
# A removal leaves a gap (0, 5, 9): index 2 is before the THIRD row, not "position 2".
user = make_user()
pl = _seed(db, user, ["a", "b", "c"], positions=[0, 5, 9])
_add(as_user(user), pl.id, {"video_id": "vX", "position": 2})
_restore(as_user(user), pl.id, "vX", 2)
assert _ids(db, pl.id) == ["a", "b", "vX", "c"]
def test_a_position_past_the_end_appends(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b"])
_add(as_user(user), pl.id, {"video_id": "vX", "position": 40})
_restore(as_user(user), pl.id, "vX", 40)
assert _ids(db, pl.id) == ["a", "b", "vX"]
def test_no_position_still_appends(db, make_user, as_user):
def test_an_add_appends(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b"])
_add(as_user(user), pl.id, {"video_id": "vX"})
r = as_user(user).post(f"/api/playlists/{pl.id}/items", json={"video_id": "vX"})
assert r.status_code == 200
assert _ids(db, pl.id) == ["a", "b", "vX"]
def test_a_video_already_in_the_list_does_not_move(db, make_user, as_user):
# The add is idempotent; a position must not turn it into a reorder.
# A restore is idempotent; a position must not turn it into a reorder.
user = make_user()
pl = _seed(db, user, ["a", "b", "c"])
_add(as_user(user), pl.id, {"video_id": "c", "position": 0})
_restore(as_user(user), pl.id, "c", 0)
assert _ids(db, pl.id) == ["a", "b", "c"]
def test_a_negative_position_is_refused(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a"])
r = _add(as_user(user), pl.id, {"video_id": "vX", "position": -1})
r = _restore(as_user(user), pl.id, "vX", -1)
assert r.status_code == 422
assert _ids(db, pl.id) == ["a"]
@@ -101,6 +105,6 @@ def test_putting_a_removed_video_back_leaves_a_linked_list_clean(db, make_user,
assert client.delete(f"/api/playlists/{pl.id}/items/b").status_code == 200
db.expire_all()
assert db.get(Playlist, pl.id).dirty is True
_add(client, pl.id, {"video_id": "b", "position": 1})
_restore(client, pl.id, "b", 1)
db.expire_all()
assert db.get(Playlist, pl.id).dirty is False
+640
View File
@@ -0,0 +1,640 @@
"""Many videos at once: remove, move, and the Undo of each. Every test reads both lists after the call,
as `test_playlist_move` does: a half-applied move is the failure these routes exist to rule out."""
from datetime import datetime, timedelta, timezone
import pytest
from sqlalchemy import select
from app.catalog.models import Channel, Video
from app.reader import queries, service
from app.reader.models import Playlist, PlaylistItem
from app.sync.playlists import current_fingerprint
def _seed_videos(db, ids):
db.add(Channel(id="UCtest", title="Test"))
now = datetime.now(timezone.utc)
for v in ids:
db.add(Video(id=v, channel_id="UCtest", published_at=now))
db.commit()
def _playlist(db, user, name, video_ids, **kw):
pl = Playlist(user_id=user.id, name=name, **kw)
db.add(pl)
db.flush()
for i, vid in enumerate(video_ids):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
db.commit()
return pl
def _ids(db, pl_id):
db.expire_all()
return list(
db.execute(
select(PlaylistItem.video_id)
.where(PlaylistItem.playlist_id == pl_id)
.order_by(PlaylistItem.position)
).scalars()
)
def _positions(db, pl_id):
db.expire_all()
return list(
db.execute(
select(PlaylistItem.position)
.where(PlaylistItem.playlist_id == pl_id)
.order_by(PlaylistItem.position)
).scalars()
)
def test_remove_bulk_takes_out_the_named_videos_and_counts_what_was_there(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
pl = _playlist(db, user, "L", ["a", "b", "c", "d"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["b", "d", "gone"]}
)
assert r.status_code == 200
assert r.json() == {"ok": True, "playlist_id": pl.id, "removed": ["b", "d"]}
assert _ids(db, pl.id) == ["a", "c"]
def test_remove_bulk_of_nothing_leaves_the_list_untouched(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a"])
pl = _playlist(db, user, "L", ["a"])
stamp = datetime.now(timezone.utc) - timedelta(days=3)
pl.updated_at = stamp
db.commit()
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["gone"]}
)
assert r.json()["removed"] == []
db.expire_all()
assert db.get(Playlist, pl.id).updated_at == stamp
def test_restore_puts_the_videos_back_at_their_places(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
pl = _playlist(db, user, "L", ["b", "d"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/restore",
json={"video_ids": ["c", "a"], "positions": [2, 0]},
)
assert r.json() == {"ok": True, "playlist_id": pl.id, "added": 2}
assert _ids(db, pl.id) == ["a", "b", "c", "d"]
assert _positions(db, pl.id) == [0, 1, 2, 3]
def test_restore_leaves_a_video_already_back_where_it_is_and_skips_one_the_catalog_lost(
db, make_user, as_user
):
user = make_user()
_seed_videos(db, ["a", "b"])
pl = _playlist(db, user, "L", ["b", "a"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/restore",
json={"video_ids": ["a", "lost"], "positions": [0, 1]},
)
assert r.json()["added"] == 0
assert _ids(db, pl.id) == ["b", "a"]
@pytest.mark.parametrize(
"body",
[
{"video_ids": ["a"], "positions": []},
{"video_ids": ["a"], "positions": [0, 1]},
{"video_ids": ["a"], "positions": [-1]},
],
)
def test_restore_refuses_positions_that_do_not_pair(db, make_user, as_user, body):
user = make_user()
pl = _playlist(db, user, "L", [])
assert as_user(user).post(f"/api/playlists/{pl.id}/items/restore", json=body).status_code == 422
def test_move_bulk_lands_in_the_sources_order_whatever_order_they_were_picked_in(
db, make_user, as_user
):
user = make_user()
_seed_videos(db, ["a", "b", "c", "x"])
src = _playlist(db, user, "Source", ["a", "b", "c"])
dst = _playlist(db, user, "Target", ["x"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["c", "a"], "target_playlist_id": dst.id},
)
assert r.status_code == 200
assert r.json() == {
"ok": True,
"from_playlist_id": src.id,
"to_playlist_id": dst.id,
"moved": ["a", "c"],
"already_there": [],
}
assert _ids(db, src.id) == ["b"]
assert _ids(db, dst.id) == ["x", "a", "c"]
def test_move_bulk_says_which_the_target_already_held_and_skips_one_already_gone(
db, make_user, as_user
):
user = make_user()
_seed_videos(db, ["a", "b", "c"])
src = _playlist(db, user, "Source", ["a", "b"])
dst = _playlist(db, user, "Target", ["b"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a", "b", "c"], "target_playlist_id": dst.id},
)
assert r.json()["moved"] == ["a"]
assert r.json()["already_there"] == ["b"]
assert _ids(db, src.id) == []
assert _ids(db, dst.id) == ["b", "a"]
def test_move_bulk_keeps_when_each_video_was_saved(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", ["a"])
dst = _playlist(db, user, "Target", [])
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
row.added_at = saved
db.commit()
r = as_user(user).post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a"], "target_playlist_id": dst.id},
)
assert r.status_code == 200
db.expire_all()
moved = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
assert moved.added_at == saved
def test_a_target_that_gained_nothing_does_not_rise_in_the_recently_changed_sort(
db, make_user, as_user
):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", ["a"])
dst = _playlist(db, user, "Target", ["a"])
stamp = datetime.now(timezone.utc) - timedelta(days=3)
dst.updated_at = stamp
db.commit()
r = as_user(user).post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a"], "target_playlist_id": dst.id},
)
assert r.status_code == 200
db.expire_all()
assert db.get(Playlist, dst.id).updated_at == stamp
def test_move_bulk_into_the_same_list_is_refused(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a"])
pl = _playlist(db, user, "L", ["a"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/move-bulk",
json={"video_ids": ["a"], "target_playlist_id": pl.id},
)
assert r.status_code == 400
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
assert _ids(db, pl.id) == ["a"]
def test_move_bulk_marks_a_linked_list_on_both_sides_as_unsynced(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", ["a"], yt_playlist_id="PLs")
dst = _playlist(db, user, "Target", [], yt_playlist_id="PLd")
r = as_user(user).post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a"], "target_playlist_id": dst.id},
)
assert r.status_code == 200
db.expire_all()
assert db.get(Playlist, src.id).dirty is True
assert db.get(Playlist, dst.id).dirty is True
def test_undo_move_restores_both_lists_as_they_stood(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
src = _playlist(db, user, "Source", ["a", "b", "c", "d"])
dst = _playlist(db, user, "Target", ["c"])
client = as_user(user)
moved = client.post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a", "c"], "target_playlist_id": dst.id},
).json()
assert _ids(db, src.id) == ["b", "d"]
assert _ids(db, dst.id) == ["c", "a"]
r = client.post(
f"/api/playlists/{src.id}/items/undo-move",
json={
"from_playlist_id": dst.id,
"moved": moved["moved"],
"already_there": moved["already_there"],
"positions": [0, 2],
},
)
assert r.json() == {"ok": True, "playlist_id": src.id, "added": 2}
assert _ids(db, src.id) == ["a", "b", "c", "d"]
# The one the target held before the move stays there.
assert _ids(db, dst.id) == ["c"]
def test_undo_move_keeps_when_each_video_was_saved(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", [])
dst = _playlist(db, user, "Target", ["a"])
saved = datetime(2020, 1, 2, tzinfo=timezone.utc)
row = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == dst.id))
row.added_at = saved
db.commit()
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
)
assert r.json()["added"] == 1
db.expire_all()
back = db.scalar(select(PlaylistItem).where(PlaylistItem.playlist_id == src.id))
assert back.added_at == saved
def test_undo_move_leaves_a_video_that_moved_on_where_it_is(db, make_user, as_user):
"""A to B, then B to C, then the first Undo: the video is not in the list the Undo takes it from,
and coming back would leave it in two lists the reader never put it in."""
user = make_user()
_seed_videos(db, ["a", "b"])
src = _playlist(db, user, "Source", [])
dst = _playlist(db, user, "Target", ["b"])
third = _playlist(db, user, "Third", ["a"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
)
assert r.json()["added"] == 1
assert _ids(db, src.id) == ["b"]
assert _ids(db, dst.id) == []
assert _ids(db, third.id) == ["a"]
def test_an_undo_closes_up_the_places_of_a_video_that_stayed_out(db, make_user, as_user):
"""Each index counts the rows put back before it, so one that stays out moves the rest up."""
user = make_user()
_seed_videos(db, ["a", "b", "c", "d"])
src = _playlist(db, user, "Source", ["c", "d"])
dst = _playlist(db, user, "Target", ["b"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a", "b"], "positions": [0, 1]},
)
assert r.json()["added"] == 1
assert _ids(db, src.id) == ["b", "c", "d"]
def test_a_restore_closes_up_the_places_of_a_video_the_catalog_lost(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["b", "c", "d"])
pl = _playlist(db, user, "L", ["c", "d"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/restore",
json={"video_ids": ["lost", "b"], "positions": [0, 1]},
)
assert r.json()["added"] == 1
assert _ids(db, pl.id) == ["b", "c", "d"]
def test_undo_move_takes_a_video_out_of_the_target_when_the_source_has_it_again(
db, make_user, as_user
):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", ["a"])
dst = _playlist(db, user, "Target", ["a"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
)
assert r.json()["added"] == 0
assert _ids(db, src.id) == ["a"]
assert _ids(db, dst.id) == []
@pytest.mark.parametrize(
("path", "body", "gone", "reason"),
[
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "dst", "move_target"),
("move-bulk", {"video_ids": ["a"], "target_playlist_id": "dst"}, "src", "playlist"),
(
"undo-move",
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
"src",
"playlist",
),
(
"undo-move",
{"from_playlist_id": "dst", "moved": ["a"], "positions": [0]},
"dst",
"move_target",
),
],
)
def test_a_list_deleted_before_the_lock_is_not_found(
db, make_user, as_user, monkeypatch, path, body, gone, reason
):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", ["a"])
dst = _playlist(db, user, "Target", ["a"])
ids = {"src": src.id, "dst": dst.id}
body = {k: (dst.id if v == "dst" else v) for k, v in body.items()}
# Another tab's delete committed between the route's read and this lock.
monkeypatch.setattr(service, "lock_playlists", lambda _db, *got: set(got) - {ids[gone]})
r = as_user(user).post(f"/api/playlists/{src.id}/items/{path}", json=body)
assert r.status_code == 404
assert reason in str(r.json()["detail"])
@pytest.mark.parametrize(
("method", "path", "body"),
[
("post", "items", {"video_id": "a"}),
("delete", "items/a", None),
("post", "items/remove-bulk", {"video_ids": ["a"]}),
("post", "items/restore", {"video_ids": ["a"], "positions": [0]}),
],
)
def test_a_one_list_write_finds_its_list_under_the_lock(
db, make_user, as_user, monkeypatch, method, path, body
):
"""Read with its lock, so a delete that won the row answers 404 rather than a foreign key's 500."""
user = make_user()
_seed_videos(db, ["a"])
pl = _playlist(db, user, "L", [])
real = queries.own_playlist
def locked_read_finds_nothing(db_, user_, playlist_id, *, lock=False):
return None if lock else real(db_, user_, playlist_id)
monkeypatch.setattr(queries, "own_playlist", locked_read_finds_nothing)
kw = {"json": body} if body is not None else {}
r = getattr(as_user(user), method)(f"/api/playlists/{pl.id}/{path}", **kw)
assert r.status_code == 404
@pytest.mark.parametrize(
("path", "body"),
[
("items/remove-bulk", {"ids": ["a"]}),
("items/move-bulk", {"ids": ["a"], "target_playlist_id": 1}),
("items/undo-move", {"ids": ["a"], "from_playlist_id": 1}),
],
)
def test_a_body_without_its_videos_is_refused_rather_than_acting_on_none(
db, make_user, as_user, path, body
):
user = make_user()
pl = _playlist(db, user, "L", [])
assert as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body).status_code == 422
def test_an_undo_of_only_videos_the_target_held_names_no_moved_ones(db, make_user, as_user):
"""An empty `moved` is a real body: every video of the move was already in the target."""
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", [])
dst = _playlist(db, user, "Target", ["a"])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": [], "already_there": ["a"], "positions": [0]},
)
assert r.json()["added"] == 1
assert _ids(db, src.id) == ["a"]
assert _ids(db, dst.id) == ["a"]
def test_an_undo_within_one_list_is_refused_and_keeps_its_videos(db, make_user, as_user):
"""A move refuses the same list twice, so no Undo of one exists; a row must not go for asking."""
user = make_user()
_seed_videos(db, ["a", "b"])
pl = _playlist(db, user, "L", ["a", "b"])
r = as_user(user).post(
f"/api/playlists/{pl.id}/items/undo-move",
json={"from_playlist_id": pl.id, "moved": ["a"], "positions": [0]},
)
assert r.status_code == 400
assert r.json()["detail"] == {"code": "playlist_move", "reason": "same_playlist"}
assert _ids(db, pl.id) == ["a", "b"]
@pytest.mark.parametrize(
("path", "key", "extra"),
[
("items/restore", "video_ids", {}),
("items/undo-move", "moved", {"from_playlist_id": None}),
],
)
def test_two_videos_cannot_ask_for_one_place(db, make_user, as_user, path, key, extra):
user = make_user()
_seed_videos(db, ["a", "b", "c"])
pl = _playlist(db, user, "L", ["c"])
other = _playlist(db, user, "Other", ["a", "b"])
body = {key: ["a", "b"], "positions": [0, 0]}
body.update({k: other.id for k in extra})
r = as_user(user).post(f"/api/playlists/{pl.id}/{path}", json=body)
assert r.status_code == 422
assert _ids(db, pl.id) == ["c"]
def test_a_reorder_without_its_videos_is_refused(db, make_user, as_user):
user = make_user()
pl = _playlist(db, user, "L", [])
assert as_user(user).put(f"/api/playlists/{pl.id}/order", json={"ids": []}).status_code == 422
def _synced(db, pl):
pl.synced_fingerprint = current_fingerprint(db, pl)
pl.dirty = False
db.commit()
def test_a_bulk_remove_and_its_undo_leave_a_linked_list_as_it_was(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a", "b"])
pl = _playlist(db, user, "L", ["a", "b"], yt_playlist_id="PLx")
_synced(db, pl)
client = as_user(user)
r = client.post(f"/api/playlists/{pl.id}/items/remove-bulk", json={"video_ids": ["a"]})
assert r.status_code == 200
db.expire_all()
assert db.get(Playlist, pl.id).dirty is True
r = client.post(
f"/api/playlists/{pl.id}/items/restore", json={"video_ids": ["a"], "positions": [0]}
)
assert r.status_code == 200
db.expire_all()
assert db.get(Playlist, pl.id).dirty is False
def test_a_move_and_its_undo_leave_both_linked_lists_as_they_were(db, make_user, as_user):
user = make_user()
_seed_videos(db, ["a", "b"])
src = _playlist(db, user, "Source", ["a", "b"], yt_playlist_id="PLs")
dst = _playlist(db, user, "Target", ["b"], yt_playlist_id="PLd")
_synced(db, src)
_synced(db, dst)
client = as_user(user)
moved = client.post(
f"/api/playlists/{src.id}/items/move-bulk",
json={"video_ids": ["a"], "target_playlist_id": dst.id},
).json()
assert moved["moved"] == ["a"]
db.expire_all()
assert db.get(Playlist, src.id).dirty is True
assert db.get(Playlist, dst.id).dirty is True
r = client.post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a"], "positions": [0]},
)
assert r.status_code == 200
db.expire_all()
assert db.get(Playlist, src.id).dirty is False
assert db.get(Playlist, dst.id).dirty is False
def test_undo_move_is_one_transaction(db, make_user, as_user, monkeypatch):
user = make_user()
_seed_videos(db, ["a"])
src = _playlist(db, user, "Source", [])
dst = _playlist(db, user, "Target", ["a"])
def boom(*a, **kw):
raise RuntimeError("second step failed")
monkeypatch.setattr(service, "_insert_at", boom)
client = as_user(user)
body = {"from_playlist_id": dst.id, "moved": ["a"], "already_there": [], "positions": [0]}
with pytest.raises(RuntimeError):
_ = client.post(f"/api/playlists/{src.id}/items/undo-move", json=body)
db.rollback()
assert _ids(db, dst.id) == ["a"]
assert _ids(db, src.id) == []
def test_undo_move_refuses_positions_that_do_not_pair(db, make_user, as_user):
user = make_user()
src = _playlist(db, user, "Source", [])
dst = _playlist(db, user, "Target", [])
r = as_user(user).post(
f"/api/playlists/{src.id}/items/undo-move",
json={"from_playlist_id": dst.id, "moved": ["a"], "already_there": ["b"], "positions": [0]},
)
assert r.status_code == 422
@pytest.mark.parametrize(
("path", "body"),
[
("remove-bulk", {"video_ids": ["a"]}),
("restore", {"video_ids": ["a"], "positions": [0]}),
("move-bulk", {"video_ids": ["a"], "target_playlist_id": 0}),
("undo-move", {"from_playlist_id": 0, "moved": ["a"], "positions": [0]}),
],
)
def test_another_accounts_list_is_not_found(db, make_user, as_user, path, body):
owner, other = make_user(), make_user()
_seed_videos(db, ["a"])
theirs = _playlist(db, owner, "Theirs", ["a"])
mine = _playlist(db, other, "Mine", [])
body = {
k: (mine.id if k in ("target_playlist_id", "from_playlist_id") else v)
for k, v in body.items()
}
r = as_user(other).post(f"/api/playlists/{theirs.id}/items/{path}", json=body)
assert r.status_code == 404
assert _ids(db, theirs.id) == ["a"]
@pytest.mark.parametrize(
("path", "key"), [("move-bulk", "target_playlist_id"), ("undo-move", "from_playlist_id")]
)
def test_the_other_list_of_a_move_must_be_the_callers_too(db, make_user, as_user, path, key):
owner, other = make_user(), make_user()
_seed_videos(db, ["a"])
theirs = _playlist(db, owner, "Theirs", ["a"])
mine = _playlist(db, other, "Mine", ["a"])
body = {"video_ids": ["a"], "moved": ["a"], "positions": [0], key: theirs.id}
r = as_user(other).post(f"/api/playlists/{mine.id}/items/{path}", json=body)
assert r.status_code == 404
assert "move_target" in str(r.json()["detail"])
assert _ids(db, theirs.id) == ["a"]
assert _ids(db, mine.id) == ["a"]
+7 -4
View File
@@ -51,7 +51,8 @@ def _ids(db, pl_id):
def _move(client, src, video_id, dst):
return client.post(
f"/api/playlists/{src}/items/{video_id}/move", json={"target_playlist_id": dst}
f"/api/playlists/{src}/items/move-bulk",
json={"video_ids": [video_id], "target_playlist_id": dst},
)
@@ -67,7 +68,8 @@ def test_move_leaves_the_video_in_exactly_one_list(db, make_user, as_user):
"ok": True,
"from_playlist_id": src.id,
"to_playlist_id": dst.id,
"video_id": "v1",
"moved": ["v1"],
"already_there": [],
}
db.expire_all()
assert _ids(db, src.id) == ["v2"]
@@ -116,8 +118,9 @@ def test_move_of_a_video_that_left_the_source_reports_it(db, make_user, as_user)
dst = _playlist(db, user, "Target", [])
r = _move(as_user(user), src.id, "v1", dst.id)
assert r.status_code == 404
assert r.json()["detail"] == {"code": "playlist_move", "reason": "not_in_playlist"}
assert r.status_code == 200
assert (r.json()["moved"], r.json()["already_there"]) == ([], [])
assert _ids(db, dst.id) == []
def test_move_into_someone_elses_list_404s_and_changes_nothing(db, make_user, as_user):
+49 -15
View File
@@ -102,16 +102,24 @@ PLEX = PlexPlaylist.__table__.name
"call",
[
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v2"}),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
),
lambda c, pid, other: c.post("/api/playlists/watch-later", json={"video_id": "v2"}),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
f"/api/playlists/{pid}/items/move-bulk",
json={"video_ids": ["v1"], "target_playlist_id": other},
),
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/undo-move",
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
),
],
ids=["add", "add-at", "watch-later", "move", "reorder"],
ids=["add", "watch-later", "move", "reorder", "remove-bulk", "restore", "undo-move"],
)
def test_a_youtube_playlist_write_locks_the_playlist(db, make_user, as_user, locks, call):
user = make_user()
@@ -168,18 +176,35 @@ PLEX_ITEMS = PlexPlaylistItem.__table__.name
"call",
[
lambda c, pid, other: c.post(f"/api/playlists/{pid}/items", json={"video_id": "v1"}),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items", json={"video_id": "v2", "position": 0}
),
lambda c, pid, other: c.delete(f"/api/playlists/{pid}/items/v1"),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
f"/api/playlists/{pid}/items/move-bulk",
json={"video_ids": ["v1"], "target_playlist_id": other},
),
lambda c, pid, other: c.put(f"/api/playlists/{pid}/order", json={"video_ids": ["v1"]}),
# the ORM cascade deletes the items before the list row
lambda c, pid, other: c.delete(f"/api/playlists/{pid}"),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/remove-bulk", json={"video_ids": ["v1"]}
),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/restore", json={"video_ids": ["v2"], "positions": [0]}
),
lambda c, pid, other: c.post(
f"/api/playlists/{pid}/items/undo-move",
json={"from_playlist_id": other, "moved": ["v2"], "positions": [0]},
),
],
ids=[
"add-present",
"remove",
"move",
"reorder",
"delete",
"remove-bulk",
"restore",
"undo-move",
],
ids=["add-present", "add-at", "remove", "move", "reorder", "delete"],
)
def test_a_youtube_item_write_locks_the_playlist_before_any_item(
db, make_user, as_user, statements, call
@@ -193,14 +218,23 @@ def test_a_youtube_item_write_locks_the_playlist_before_any_item(
assert _locked_before_items(statements, YT, YT_ITEMS), statements
def test_a_move_locks_both_lists_in_one_ordered_statement(db, make_user, as_user, statements):
"""Two moves A→B and B→A each locking "its" list first would lock crosswise."""
@pytest.mark.parametrize(
("path", "body"),
[
("move-bulk", lambda other: {"video_ids": ["v1"], "target_playlist_id": other}),
("undo-move", lambda other: {"from_playlist_id": other, "moved": ["v2"], "positions": [0]}),
],
ids=["move", "undo"],
)
def test_a_move_locks_both_lists_in_one_ordered_statement(
db, make_user, as_user, statements, path, body
):
"""Two moves A→B and B→A each locking "its" list first would lock crosswise, and an Undo runs the
other way round from a move by its nature."""
user = make_user()
pid, other = _yt(db, user)
statements.clear()
r = as_user(user).post(
f"/api/playlists/{pid}/items/v1/move", json={"target_playlist_id": other}
)
r = as_user(user).post(f"/api/playlists/{pid}/items/{path}", json=body(other))
assert r.status_code == 200, r.text
lock = _locked_before_items(statements, YT, YT_ITEMS)
assert lock and "ORDER BY" in lock, lock
+2
View File
@@ -219,6 +219,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
library_id=lib.id,
kind="movie",
title="The Godfather",
size_bytes=15_000_000_000,
genres=["Crime", "Drama"],
directors=["Francis Ford Coppola"],
studio="Paramount",
@@ -238,6 +239,7 @@ def test_item_detail_movie_serves_from_row_despite_live_outage(db, monkeypatch,
resp = _get(db, user, monkeypatch, override_dep, "/api/plex/item/m1")
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["size_bytes"] == 15_000_000_000
assert body["imdb_rating"] == 8.7
assert body["imdb_id"] == "tt0068646"
assert body["imdb_url"] == "https://www.imdb.com/title/tt0068646/"
+148 -1
View File
@@ -13,7 +13,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app.db import SessionLocal
from app.plex.models import PlexItem, PlexLibrary, PlexState
from app.plex.models import PlexItem, PlexLibrary, PlexSeason, PlexShow, PlexState
from app.plex import service as plex_service
from app.routes.plex import playback
@@ -81,3 +81,150 @@ def test_the_finish_supersedes_a_resume_push_still_in_flight(db, as_user, make_u
)
assert r.json() == {"ok": True, "status": "watched", "position_seconds": 0}
assert playback._resume_push_superseded(user.id, item.id, earlier) is True
def test_a_checkpoint_sent_before_the_finish_does_not_land_on_the_watched_row(
db, as_user, make_user, monkeypatch
):
"""The 10 s tick posted at 2699 of 3000 s (not near the end) races the credits auto-skip's
"watched". When the mark commits first, the late checkpoint used to write its position onto the
watched row and schedule a resume push after the scrobble: Plex showed the item in progress, and
a Plex-authoritative pull imported it back."""
user = make_user()
item = _item(db)
monkeypatch.setattr(playback.plex_stream, "touch_session", lambda uid, key: None)
client = as_user(user)
assert client.post("/api/plex/item/m1/state", json={"status": "watched"}).status_code == 200
r = client.post(
"/api/plex/item/m1/progress", json={"position_seconds": 2699, "duration_seconds": 3000}
)
assert r.status_code == 200, r.text
assert r.json() == {"ok": True, "status": "watched", "position_seconds": 0}
db.expire_all()
st = db.execute(select(PlexState).where(PlexState.item_id == item.id)).scalars().one()
assert (st.status, st.position_seconds) == ("watched", 0)
def test_a_rewatch_records_its_position_once_the_finish_is_behind_it(
db, as_user, make_user, monkeypatch
):
"""The other half: a watched item played again is still checkpointed — only a position arriving
right on the heels of the mark is read as that mark's late tick."""
user = make_user()
item = _item(db)
long_ago = datetime.now(timezone.utc) - timedelta(days=3)
db.add(PlexState(user_id=user.id, item_id=item.id, status="watched", watched_at=long_ago))
db.commit()
monkeypatch.setattr(playback.plex_stream, "touch_session", lambda uid, key: None)
r = as_user(user).post(
"/api/plex/item/m1/progress", json={"position_seconds": 600, "duration_seconds": 3000}
)
assert r.json() == {"ok": True, "position_seconds": 600}
db.expire_all()
st = db.execute(select(PlexState).where(PlexState.item_id == item.id)).scalars().one()
assert (st.status, st.position_seconds) == ("watched", 600)
def test_marking_watched_supersedes_a_resume_push_still_in_flight(
db, as_user, make_user, monkeypatch
):
"""The same guard the near-end finish has: a resume push scheduled before the mark must not reach
Plex after its scrobble."""
user = make_user()
item = _item(db)
earlier = datetime.now(timezone.utc) - timedelta(seconds=5)
assert playback._should_push_resume(user.id, item.id, earlier, final=True)
r = as_user(user).post("/api/plex/item/m1/state", json={"status": "watched"})
assert r.status_code == 200, r.text
assert playback._resume_push_superseded(user.id, item.id, earlier) is True
def _state(db, user, item):
db.expire_all()
st = (
db.execute(
select(PlexState).where(PlexState.user_id == user.id, PlexState.item_id == item.id)
)
.scalars()
.one()
)
return st.status, st.position_seconds
def test_a_mark_committed_after_the_checkpoints_read_still_refuses_it(db, make_user):
"""The race itself, not its sequential shadow: the route read the row while it was still in
progress, and the finish committed from another session before the checkpoint wrote. Only a write
conditional on the row as it is NOW refuses that position."""
user = make_user()
item = _item(db)
db.add(PlexState(user_id=user.id, item_id=item.id, status="new", position_seconds=2690))
db.commit()
now = datetime.now(timezone.utc)
read = plex_service.state_row(db, user, item.id) # what the checkpoint's route read: "new"
assert read.status == "new"
other = SessionLocal()
try:
assert plex_service.mark_finished(other, user.id, item.id, now) is True
other.commit()
finally:
other.close()
assert plex_service.checkpoint(db, user, item.id, read, 2699, now, dirty=True) is False
db.commit()
assert _state(db, user, item) == ("watched", 0)
def test_a_tick_seconds_behind_the_mark_is_refused_a_rewatch_a_minute_later_is_not(db, make_user):
"""A position right behind the mark is that mark's late tick; one a minute later is a rewatch.
Literal times on purpose: a window widened to an hour must fail here, not move with it."""
user = make_user()
item = _item(db)
now = datetime.now(timezone.utc)
st = PlexState(user_id=user.id, item_id=item.id, status="watched", watched_at=now)
db.add(st)
db.commit()
st.watched_at = now - timedelta(seconds=20)
db.commit()
assert plex_service.checkpoint(db, user, item.id, st, 600, now, dirty=True) is False
db.commit()
st.watched_at = now - timedelta(seconds=60)
db.commit()
assert plex_service.checkpoint(db, user, item.id, st, 600, now, dirty=True) is True
db.commit()
assert _state(db, user, item) == ("watched", 600)
def test_marking_a_season_watched_supersedes_its_episodes_resume_pushes(db, as_user, make_user):
"""The bulk mark has the same guard as one item's: an episode's resume push scheduled before
the season was marked must not reach Plex after the scrobble."""
user = make_user()
lib = PlexLibrary(plex_key="2", title="Shows", kind="show", enabled=True)
db.add(lib)
db.flush()
sh = PlexShow(rating_key="sh1", library_id=lib.id, title="Show")
db.add(sh)
db.flush()
se = PlexSeason(rating_key="se1", show_id=sh.id, season_number=1)
db.add(se)
db.flush()
ep = PlexItem(
rating_key="e1",
library_id=lib.id,
kind="episode",
title="E1",
show_id=sh.id,
season_id=se.id,
season_number=1,
episode_number=1,
)
db.add(ep)
db.commit()
earlier = datetime.now(timezone.utc) - timedelta(seconds=5)
assert playback._should_push_resume(user.id, ep.id, earlier, final=True)
r = as_user(user).post("/api/plex/season/se1/state", json={"watched": True})
assert r.json() == {"ok": True, "changed": 1, "watched": True}
assert playback._resume_push_superseded(user.id, ep.id, earlier) is True
+36 -11
View File
@@ -5,7 +5,7 @@ seek-restart — or a second viewer of the same item — replaces the registered
viewer's next request for `seg_12.ts`, resolved by rating_key, was answered 200 with the NEW
session's twelfth segment: bytes produced from a different `-ss` offset under `-copyts`, i.e. a
timeline the client is not on. Wrong content with a success status is the one failure a player cannot
notice, and the client's reclaim-on-404 could never fire on it.
notice, and the client's reclaim could never fire on it.
No `db` fixture: the route reads nothing from the database (its own `Depends(get_db)` went with the
removed auto-start, and `current_user` is overridden wholesale), so these must not sit in the DB
@@ -88,10 +88,10 @@ def test_another_viewers_session_of_the_same_item_is_not_this_viewers(client, tm
(d / "index.m3u8").write_bytes(b"#EXTM3U")
sessions[(7, "m1")] = _Session("sid-a", d)
r = client.get("/api/plex/stream/m1/hls/sid-a/index.m3u8")
assert r.status_code == 404, r.text
assert r.status_code == 410, r.text
def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
def test_a_segment_of_a_replaced_session_is_gone_not_the_new_sessions_bytes(
client, tmp_path, sessions
):
"""Viewer A is at 0:00; viewer B seeks to 25:00 and their session takes the registry slot. A's
@@ -102,15 +102,15 @@ def test_a_segment_of_a_replaced_session_is_a_404_not_the_new_sessions_bytes(
(theirs / "seg_12.ts").write_bytes(b"theirs")
sessions[(VIEWER, "m1")] = _Session("sid-b", theirs)
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
assert r.status_code == 404, r.text
assert r.status_code == 410, r.text
assert b"theirs" not in r.content
def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
def test_a_playlist_with_no_live_session_is_gone_not_a_silent_restart_at_zero(
client, monkeypatch, sessions
):
"""The entry playlist used to auto-start a session when none was registered — always at 0.0,
silently throwing away the viewer's position. A 404 sends the player through its own reclaim,
silently throwing away the viewer's position. A 410 sends the player through its own reclaim,
which restarts at the playhead."""
started: list[float] = []
monkeypatch.setattr(
@@ -119,7 +119,7 @@ def test_a_playlist_with_no_live_session_is_a_404_not_a_silent_restart_at_zero(
lambda *a, **k: started.append(a[3] if len(a) > 3 else 0.0),
)
r = client.get("/api/plex/stream/m1/hls/sid-gone/index.m3u8")
assert r.status_code == 404, r.text
assert r.status_code == 410, r.text
assert started == []
@@ -198,6 +198,31 @@ def test_a_name_the_session_cannot_produce_fails_at_once(client, tmp_path, sessi
assert elapsed < 2.0, f"waited {elapsed:.1f}s for a playlist this session never writes"
@pytest.mark.parametrize(
("name", "audio_langs"),
[
("seg_00.ts", None), # `int()` reads it as segment 0, which needs no predecessor
("seg_007.ts", None), # …as segment 7, whose predecessor seg_6.ts is on disk
("seg_٧.ts", None), # an Arabic-Indic 7: `\d` and `int()` both take it
("seg_a0_07.ts", ["hu", "en"]),
("seg_0_00.ts", ["hu", "en"]),
],
)
def test_a_segment_name_ffmpeg_never_spells_is_refused(tmp_path, name, audio_langs):
"""ffmpeg writes `seg_%d.ts`: no padding, ASCII digits. A name that only PARSES to a sequence the
session has reached is still a file that will never appear, and saying yes to it parked the
request for the full 20 s wait — the parking this guard exists to end."""
d = tmp_path / "m1_0_0_+0.00"
d.mkdir()
for v in plex_stream.session_variants(audio_langs) or (None,):
(d / plex_stream.segment_name(6, v)).write_bytes(b"x")
sess = _Session("sid-a", d, running=True, audio_langs=audio_langs)
assert not plex_stream.artifact_may_arrive(sess, name)
# The canonical spelling of the same next segment is still accepted.
v = (plex_stream.session_variants(audio_langs) or (None,))[0]
assert plex_stream.artifact_may_arrive(sess, plex_stream.segment_name(7, v))
@pytest.mark.parametrize("digits", [100, 5000])
def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions, digits):
"""`int()` on an unbounded digit run raises ValueError past 4300 digits, and only OSError was
@@ -210,12 +235,12 @@ def test_an_absurd_segment_index_is_a_404_not_a_500(client, tmp_path, sessions,
assert r.status_code == 404, r.text
def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
def test_a_session_torn_down_during_the_read_is_gone_not_a_500(
client, monkeypatch, tmp_path, sessions
):
"""`_kill` rmtrees the whole directory and nothing holds `_lock` across the wait and the read,
"""Teardown moves the whole directory aside and nothing holds `_lock` across the wait and the read,
so the file can vanish in between — on every seek-restart, since `loadSession` awaits the new
session before destroying the old hls instance. The player's recovery is keyed on the 404
session before destroying the old hls instance. The player's recovery is keyed on the
STATUS, so a 500 here skipped the reclaim and ended playback with "couldn't play"."""
d = tmp_path / "m1_0_0_+0.00"
d.mkdir()
@@ -224,7 +249,7 @@ def test_a_session_torn_down_during_the_read_is_a_404_not_a_500(
# The file existed when the wait checked, and is gone by the time the route reads it.
monkeypatch.setattr(plex_stream, "wait_for", lambda *a, **k: True)
r = client.get("/api/plex/stream/m1/hls/sid-a/seg_12.ts")
assert r.status_code == 404, r.text
assert r.status_code == 410, r.text
def test_a_dead_producer_is_not_waited_out(client, tmp_path, sessions):
+56 -20
View File
@@ -32,15 +32,25 @@ class TestTranscodeHwCapable:
assert sync.transcode_hw_capable("transcode", "hevc") is True
assert sync.transcode_hw_capable("transcode", "vp9", bit_depth=8) is True
def test_non_decodable_codec_is_sw(self):
# AV1 / MPEG-4 ASP (XviD) can't be HW-decoded on Gen 9.5 → software regardless of bit depth.
def test_a_codec_neither_path_knows_is_sw(self):
assert sync.transcode_hw_capable("transcode", "av1") is False
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=8) is False
assert sync.transcode_hw_capable("transcode", "av1", bit_depth=8) is False
def test_10bit_or_hdr_hevc_is_sw(self):
# A HW-decodable codec that is 10-bit or HDR still breaks the VAAPI nv12 pipeline → software.
def test_a_cpu_decoded_8bit_codec_is_encoded_on_the_gpu(self):
# VC-1 (the iGPU has no VC-1 Advanced decode), XviD, DivX: the CPU decodes, the GPU encodes.
for codec in ("vc1", "mpeg4", "msmpeg4v3"):
assert sync.transcode_hw_capable("transcode", codec) is True
assert sync.transcode_hw_capable("transcode", "mpeg4", bit_depth=10) is False
def test_10bit_sdr_hevc_is_sw(self):
# 10-bit SDR: an exact downconvert on the CPU (libx264), never the HDR tone curve.
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10) is False
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is False
def test_hdr_hevc_is_encoded_on_the_gpu(self):
# HDR goes through the OpenCL tonemap between VAAPI decode and encode (the prod image ships the
# runtime); the facet said software for it while every such session ran on the GPU.
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=10, hdr=True) is True
assert sync.transcode_hw_capable("transcode", "hevc", bit_depth=8, hdr=True) is True
class TestVideoStreamProps:
@@ -87,6 +97,9 @@ class TestSyncDerivation:
row2 = PlexItem(rating_key="y", kind="movie")
sync._apply_item(row2, lib_id=1, kind="movie", meta=self._meta("av1"))
assert row2.transcode_hw is False # av1 not HW-decodable
row3 = PlexItem(rating_key="z", kind="movie")
sync._apply_item(row3, lib_id=1, kind="movie", meta=self._meta("vc1"))
assert row3.transcode_hw is True # CPU decode, GPU encode
def test_probed_value_survives_resync_when_codec_unchanged(self):
row = PlexItem(rating_key="x", kind="movie")
@@ -124,21 +137,24 @@ class TestSyncDerivation:
assert row.video_bit_depth is None and row.media_probed_at is None # re-probe scheduled
def test_migration_codec_set_matches_hw_decodable_codecs():
# Migration 0077 hardcodes the HW-decodable codec set in raw SQL and its comment mandates it stay
# identical to plex.sync.HW_DECODABLE_CODECS. A comment isn't enforcement (house rule) — pin the
# equality so adding/removing a codec from the frozenset without editing the migration fails here
# (else already-migrated prod rows keep a wrong transcode_hw the runtime derivation wouldn't catch).
def test_migration_codec_sets_match_the_sync_sets():
# The newest migration that rewrites `transcode_hw` hardcodes both codec sets in raw SQL. Pin them
# to plex.sync, so a codec added to or removed from a frozenset without a migration that rewrites
# the stored rows fails here (else prod rows keep a facet the runtime choice disagrees with).
mig = (
Path(__file__).resolve().parent.parent
/ "alembic"
/ "versions"
/ "0077_plex_playback_class.py"
/ "0119_plex_transcode_gpu.py"
)
m = re.search(r"lower\(codec_video\)\s+IN\s+\(([^)]*)\)", mig.read_text())
assert m, "could not find the codec IN(...) list in migration 0077"
codecs = {c.strip().strip("'\"").lower() for c in m.group(1).split(",")}
assert codecs == {c.lower() for c in sync.HW_DECODABLE_CODECS}
src = mig.read_text()
sets = {}
for name in ("HW_DECODABLE_CODECS", "CPU_DECODE_CODECS"):
m = re.search(name + r"\s*=\s*\(([^)]*)\)", src)
assert m, f"could not find {name} in migration 0119"
sets[name] = {c.strip().strip("'\"").lower() for c in m.group(1).split(",") if c.strip()}
assert sets["HW_DECODABLE_CODECS"] == {c.lower() for c in sync.HW_DECODABLE_CODECS}
assert sets["CPU_DECODE_CODECS"] == {c.lower() for c in sync.CPU_DECODE_CODECS}
class _FakePlex:
@@ -182,22 +198,42 @@ def test_probe_media_batch_refines_hw_to_sw_for_10bit(db):
codec_video="mpeg4",
transcode_hw=False,
)
db.add_all([hevc8, hevc10, xvid])
hdr10 = PlexItem(
rating_key="hdr",
library_id=lib.id,
kind="movie",
title="hdr",
playable="transcode",
codec_video="hevc",
transcode_hw=True,
)
db.add_all([hevc8, hevc10, xvid, hdr10])
db.commit()
plex = _FakePlex(
{
"h8": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 8}]}]}]},
"h10": {"Media": [{"Part": [{"Stream": [{"streamType": 1, "bitDepth": 10}]}]}]},
"hdr": {
"Media": [
{
"Part": [
{"Stream": [{"streamType": 1, "bitDepth": 10, "colorTrc": "smpte2084"}]}
]
}
]
},
}
)
probed = sync._probe_media_batch(db, plex)
db.refresh(hevc8)
db.refresh(hevc10)
db.refresh(xvid)
# Only the two HW-decodable HEVC items are probed; the XviD is SW-by-codec and never fetched.
assert probed == 2
db.refresh(hdr10)
# Only the HW-decodable HEVC items are probed; the XviD's class follows from its codec alone.
assert probed == 3
assert hevc8.video_bit_depth == 8 and hevc8.transcode_hw is True and hevc8.media_probed_at
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # downgraded to software
assert hevc10.video_bit_depth == 10 and hevc10.transcode_hw is False # 10-bit SDR: software
assert hdr10.video_bit_depth == 10 and hdr10.transcode_hw is True # HDR: the GPU tonemap
assert xvid.media_probed_at is None # not in the work-list
# A second run has nothing left to do (watermarks set).
assert sync._probe_media_batch(db, plex) == 0
@@ -6,6 +6,8 @@ rows, since positions have gaps after a removal. One request: an append plus a s
leaves the rows at the end whenever that call fails, and would overwrite another tab's changes.
"""
import pytest
from app.plex.models import PlexItem, PlexLibrary, PlexPlaylist, PlexPlaylistItem
@@ -101,6 +103,22 @@ def test_positions_must_pair_with_the_keys(db, make_user, as_user):
assert _order(db, pl.id)[0] == ["a"]
@pytest.mark.parametrize(
("method", "path"),
[("post", "items/bulk"), ("post", "items/remove-bulk"), ("put", "order")],
)
def test_a_body_without_its_keys_is_refused_rather_than_acting_on_none(
db, make_user, as_user, method, path
):
user = make_user()
pl = _seed(db, user, ["a"], extra=())
r = getattr(as_user(user), method)(f"/api/plex/playlists/{pl.id}/{path}", json={"keys": ["a"]})
assert r.status_code == 422
assert _order(db, pl.id)[0] == ["a"]
def test_remove_then_put_back_is_no_change(db, make_user, as_user):
user = make_user()
pl = _seed(db, user, ["a", "b", "c", "d"], extra=())
+4 -2
View File
@@ -348,7 +348,8 @@ def test_an_early_checkpoint_drops_a_bare_state_and_zeroes_a_marked_one(db, make
user = make_user()
lib = _lib(db)
bare, hid = _movie(db, lib, "bare"), _movie(db, lib, "hid")
st = service.checkpoint(db, user, bare.id, None, 120, NOW, dirty=True)
assert service.checkpoint(db, user, bare.id, None, 120, NOW, dirty=True)
st = queries.state(db, user, bare.id)
assert (st.position_seconds, st.synced_to_plex) == (120, False)
_state(db, user, hid, status="hidden", position_seconds=40)
service.clear_progress(db, queries.state(db, user, bare.id), NOW)
@@ -429,7 +430,8 @@ def test_a_first_write_that_lost_the_race_updates_the_winners_row(db, make_user)
user = make_user()
it = _movie(db, _lib(db), "m-race")
_state(db, user, it, position_seconds=10) # the request that won, committed
st = service.checkpoint(db, user, it.id, None, 42, NOW, dirty=True) # read "none" before it
assert service.checkpoint(db, user, it.id, None, 42, NOW, dirty=True) # read "none" before it
st = queries.state(db, user, it.id)
assert (st.position_seconds, st.synced_to_plex) == (42, False)
st = service.set_status(db, user, it.id, None, "hidden")
assert st.status == "hidden"
+1
View File
@@ -39,6 +39,7 @@ class _Proc:
class _Session:
def __init__(self, session_id: str, directory):
self.id = session_id
self.key = "m1"
self.dir = directory
self.proc = _Proc()
self.last_access = time.time()
+152 -15
View File
@@ -26,6 +26,9 @@ from app.plex.stream import _SESSION_DIR_RE
"12345_0_2_+0.00", # current shape, explicit audio_ord
"12345_842_multi_-1.25", # multi-audio, negative offset
"12345_842_multi_+12.50",
# a torn-down session's directory, moved aside to be deleted outside the lock — left behind
# only if the process died between the rename and the delete
"12345_u7_842_multi_+12.50.dead-0123abcd",
],
)
def test_our_session_directories_are_swept(name):
@@ -47,6 +50,9 @@ def test_our_session_directories_are_swept(name):
"snapshot_2024_backup_+1.00", # session-ish tail, but the key isn't a rating_key
"backup_2024_multi_+0.00",
".config",
"12345_u7_0_0_+0.00.dead-", # a tombstone suffix with no id
"12345_u7_0_0_+0.00.dead-notours!",
"backup_u7_0_0_+0.00.dead-0123abcd",
],
)
def test_foreign_directories_are_left_alone(name):
@@ -159,7 +165,7 @@ class TestSweepOrphans:
class _FakeProc:
"""Stand-in for the ffmpeg Popen — start_session only stores it (and _kill/reap poll it later)."""
"""Stand-in for the ffmpeg Popen — start_session only stores it (and teardown/reap poll it later)."""
def poll(self):
return None
@@ -278,6 +284,24 @@ class TestTranscodeCommand:
)
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # not a software scale
def test_cpu_decode_uploads_the_frames_to_the_gpu_encoder(self):
# A source the GPU cannot decode (VC-1 Advanced on Gen 9.5): the CPU decodes, the frames go to
# nv12 and up to the GPU, and h264_vaapi encodes. No -hwaccel: asking the GPU to decode is what
# failed.
cmd = stream._ffmpeg_cmd(
self.SRC, 0.0, self.OUT, None, transcode=True, hw=True, cpu_decode=True
)
assert "-hwaccel" not in cmd
assert cmd[cmd.index("-init_hw_device") + 1] == f"vaapi=va:{stream._VAAPI_DEVICE}"
assert cmd[cmd.index("-filter_hw_device") + 1] == "va"
assert cmd.index("-init_hw_device") < cmd.index("-i")
assert cmd[cmd.index("-vf") + 1] == "format=nv12,hwupload"
assert _venc(cmd) == "h264_vaapi"
assert "libx264" not in cmd
assert (
cmd[cmd.index("-force_key_frames") + 1] == f"expr:gte(t,n_forced*{stream._SEG_SECONDS})"
)
def test_hdr_tonemap_uses_the_opencl_graph(self):
# The HDR/10-bit HW path: VAAPI decode → OpenCL tonemap → VAAPI encode. Gen 9.5 can't tonemap in
# the VAAPI VPP, so this is the only HW route for a 10-bit/HDR source.
@@ -359,19 +383,48 @@ class TestWantHw:
is True
)
def test_auto_falls_back_to_software_for_a_non_decodable_source(self, monkeypatch):
# AV1 / MPEG-4 ASP (XviD) aren't decodable on Gen 9.5 → the GPU can't feed its own encoder.
def test_auto_falls_back_to_software_for_a_codec_neither_path_knows(self, monkeypatch):
# AV1 is neither GPU-decodable on Gen 9.5 nor on the CPU-decode list (it can be 10-bit HDR).
self._policy(monkeypatch, "auto")
self._hwsafe(monkeypatch, True) # decodability gate must fire BEFORE the probe
self._hwsafe(monkeypatch, True) # the codec gate must fire BEFORE the probe
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert (
stream._want_hw(None, _FakeItem(codec_video="av1"), transcode=True, src=self.SRC)
is False
)
assert (
stream._want_hw(None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC)
is False
)
@pytest.mark.parametrize("codec", ["vc1", "mpeg4", "msmpeg4v3"])
def test_auto_cpu_decodes_what_the_gpu_cannot_and_still_encodes_on_it(self, monkeypatch, codec):
# Measured on prod (UHD 630, iHD 25.2.3): "No support for codec vc1 profile 3", so every VC-1
# HW session died and fell back to libx264 (~19 s to start). Decoding on the CPU and encoding on
# the GPU produced three segments in ~5 s (VC-1 1080p) and ~1 s (XviD, DivX SD).
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_HWSAFE)
self._opencl(monkeypatch, True)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video=codec), transcode=True, src=self.SRC
) == (True, False, True)
def test_auto_a_cpu_decoded_codec_that_is_not_8bit_sdr_stays_software(self, monkeypatch):
# The upload path converts to nv12 for the encoder; only an 8-bit SDR picture is known to come
# through it unchanged.
self._policy(monkeypatch, "auto")
self._kind(monkeypatch, stream._VK_SDR10)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="mpeg4"), transcode=True, src=self.SRC
) == (False, False, False)
def test_on_cpu_decodes_a_codec_the_gpu_cannot(self, monkeypatch):
# "on" trusts the GPU, but asking it to DECODE a codec it is known not to decode only buys the
# fallback's wasted second; the upload path is the GPU route for such a source.
self._policy(monkeypatch, "on")
self._opencl(monkeypatch, False)
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="vc1"), transcode=True, src=self.SRC
) == (True, False, True)
def test_auto_hdr_source_falls_back_to_software_without_opencl(self, monkeypatch):
# A decodable HEVC that is true-HDR (PQ/HLG) can't use scale_vaapi=format=nv12 (Gen 9.5), and with
@@ -382,7 +435,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (False, False)
) == (False, False, False)
def test_auto_hdr_source_uses_hw_tonemap_with_opencl(self, monkeypatch):
# With the OpenCL runtime present, a true-HDR HEVC tone-maps on the GPU (VAAPI decode → OpenCL
@@ -393,7 +446,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, True)
) == (True, True, False)
def test_auto_10bit_SDR_source_uses_software_not_tonemap(self, monkeypatch):
# F1 regression: a 10-bit *SDR* (bt709) source is NOT HDR, so it must NOT get the hable HDR tone
@@ -405,7 +458,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (False, False)
) == (False, False, False)
def test_auto_8bit_source_uses_plain_hw_not_tonemap(self, monkeypatch):
# An 8-bit SDR source takes the plain full-GPU scale path — never the tonemap graph — even where
@@ -416,7 +469,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
) == (True, False, False)
def test_off_forces_software_even_for_a_decodable_source(self, monkeypatch):
self._policy(monkeypatch, "off")
@@ -454,7 +507,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, True)
) == (True, True, False)
def test_on_10bit_SDR_uses_plain_hw_not_tonemap(self, monkeypatch):
# F1 regression under "on": a 10-bit SDR source is NOT HDR, so even with OpenCL present "on" must
@@ -465,7 +518,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
) == (True, False, False)
def test_on_without_opencl_is_plain_hw(self, monkeypatch):
# "on" with no OpenCL runtime → plain HW, tonemap=False, regardless of source class (the runtime
@@ -477,7 +530,7 @@ class TestWantHw:
monkeypatch.setattr(stream, "_render_node_present", lambda: True)
assert stream._select_encode(
None, _FakeItem(codec_video="hevc"), transcode=True, src=self.SRC
) == (True, False)
) == (True, False, False)
class TestColdStartWarmup:
@@ -614,6 +667,30 @@ class TestTranscodeStartSession:
assert cmd[cmd.index("-vf") + 1] == "scale_vaapi=format=nv12" # no SW cap
assert kwargs.get("preexec_fn") is None # GPU encode is cheap — never niced
def test_a_cpu_decode_session_encodes_on_the_gpu_and_falls_back_to_software(
self, capturing_env, monkeypatch
):
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (True, False, True)
)
seen = {"n": 0}
def flaky_wait(path, timeout=20.0, proc=None):
seen["n"] += 1
return seen["n"] > 1 # the upload session yields nothing; the software respawn does
monkeypatch.setattr(stream, "wait_for", flaky_wait)
s = stream.start_session(
db=None, user_id=7, item=_FakeItem(codec_video="vc1"), start_s=0.0, transcode=True
)
(up_cmd, *_), up_kwargs = _CapturingProc.calls[0]
assert "format=nv12,hwupload" in up_cmd and _venc(up_cmd) == "h264_vaapi"
if os.name == "posix": # the decode is CPU work: niced like a software transcode
assert up_kwargs.get("preexec_fn") is stream._sw_child_preexec
(sw_cmd, *_), _ = _CapturingProc.calls[1]
assert _venc(sw_cmd) == "libx264" and "format=nv12,hwupload" not in sw_cmd
assert s is not None and s.hw is False and s.cpu_decode is False
def test_hdr_tonemap_transcode_threads_the_opencl_graph_and_flags_the_session(
self, capturing_env, monkeypatch
):
@@ -1052,3 +1129,63 @@ class TestSessionsPerViewer:
self._start(monkeypatch, user_id=uid)
assert len(stream._sessions) == 3 and (1, "12345") not in stream._sessions
assert oldest.proc.killed and not oldest.transcode
class TestTeardownFinishes:
"""A session taken down by a cap or replaced by its own viewer is only renamed aside under the
lock; it is deleted after. The rename alone already moves the directory and `terminate()` alone
already marks the proc killed — so what is pinned here is the tomb: a whole segmented copy of the
film that only a restart's sweep would otherwise reclaim."""
capturing_env = TestTranscodeStartSession.capturing_env
@staticmethod
def _left_behind(root):
return sorted(p.name for p in root.iterdir() if stream._TOMB in p.name) + sorted(
str(p) for p in stream._dying
)
def test_a_culled_session_is_deleted_not_only_moved_aside(self, capturing_env, monkeypatch):
monkeypatch.setattr(stream, "_dying", {})
first = TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="1")
first.last_access -= 60
TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="2")
TestSessionsPerViewer._start(monkeypatch, user_id=1, rating_key="3")
assert first.proc.killed and not first.dir.exists()
assert self._left_behind(capturing_env) == []
def test_a_viewers_restart_deletes_the_session_it_replaces(self, capturing_env, monkeypatch):
monkeypatch.setattr(stream, "_dying", {})
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
)
monkeypatch.setattr(
stream, "_probe_first_pts", lambda seg: None
) # no real segment to probe
old = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=0.0)
new = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=600.0)
assert old.proc.killed and new.dir.exists() and not old.dir.exists()
assert self._left_behind(capturing_env) == []
def test_a_restart_that_fails_to_start_still_deletes_what_it_replaced(
self, capturing_env, monkeypatch
):
# The new ffmpeg cannot start (a full disk, no ffmpeg, no memory): the old session is already
# moved aside, and nothing else in this process would ever delete it.
monkeypatch.setattr(stream, "_dying", {})
monkeypatch.setattr(
stream, "_select_encode", lambda db, item, transcode, src: (False, False)
)
monkeypatch.setattr(
stream, "_probe_first_pts", lambda seg: None
) # no real segment to probe
old = stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=0.0)
def no_ffmpeg(*args, **kwargs):
raise OSError(28, "No space left on device")
monkeypatch.setattr(stream.subprocess, "Popen", no_ffmpeg)
with pytest.raises(OSError):
stream.start_session(db=None, user_id=1, item=_FakeItem(), start_s=600.0)
assert old.proc.killed and not old.dir.exists()
assert self._left_behind(capturing_env) == []
+43 -16
View File
@@ -14,7 +14,7 @@ from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from app.discover import pins
from app.catalog.models import Channel, Video
from app.catalog.models import Channel, Subscription, Video
from app.reader import queries, service
from app.reader.models import ChannelTag, ComfortPin, Playlist, PlaylistItem, Tag, VideoState
@@ -60,12 +60,44 @@ def test_a_system_tag_is_nobodys_to_edit(db, make_user):
service.create_tag(db, other, "Theirs", "other", None) # another user's tag stays theirs
db.commit()
assert queries.own_tag(db, user, system.id) is None
assert [t.id for t, _ in queries.tags_with_counts(db, user)] == [system.id]
assert [t.id for t, *_ in queries.tags_with_counts(db, user)] == [system.id]
# --- tags ------------------------------------------------------------------------------------
def _music_on_four(db, make_user):
"""A tag on four channels: two followed, one followed but hidden, one no longer followed."""
user, other = make_user(), make_user()
tag = service.create_tag(db, user, "Music", "other", None)
for cid in ("UCa", "UCb", "UChid", "UCgone"):
db.add(Channel(id=cid, title=cid))
db.flush()
for cid in ("UCa", "UCb"):
db.add(Subscription(user_id=user.id, channel_id=cid))
db.add(Subscription(user_id=user.id, channel_id="UChid", hidden=True))
db.add(Subscription(user_id=other.id, channel_id="UCgone"))
for cid in ("UCa", "UCb", "UChid", "UCgone"):
db.add(ChannelTag(channel_id=cid, tag_id=tag.id, user_id=user.id, source="user"))
db.commit()
return user, tag
def test_a_tag_counts_apart_the_links_on_channels_the_user_still_follows(db, make_user):
"""An unsubscribe and a hide keep the channel's tag links; the Channels page's filter lists only
the followed channels not hidden, so its count must not hold the others."""
user, tag = _music_on_four(db, make_user)
[(_, count, subscribed)] = [r for r in queries.tags_with_counts(db, user) if r[0].id == tag.id]
assert (count, subscribed) == (4, 2)
def test_the_tag_list_sends_each_count_under_its_own_name(db, make_user, as_user):
"""Both counts are ints, so a swap would type-check: the answer itself pins which is which."""
user, tag = _music_on_four(db, make_user)
[row] = [t for t in as_user(user).get("/api/tags").json() if t["id"] == tag.id]
assert (row["channel_count"], row["subscribed_count"]) == (4, 2)
def test_a_tag_is_made_once_per_name_and_an_unknown_category_is_other(db, make_user):
user = make_user()
tag = service.create_tag(db, user, "Later", "nonsense", "#fff")
@@ -190,9 +222,6 @@ def test_a_cleared_resume_point_drops_a_bare_row_and_zeroes_one_with_a_status(db
db.commit()
assert _state(db, user, "v1") is None
assert (_state(db, user, "v2").status, _state(db, user, "v2").position_seconds) == ("hidden", 0)
service.clear_state(db, user, "v2")
db.commit()
assert _state(db, user, "v2") is None
# --- playlists -------------------------------------------------------------------------------
@@ -213,9 +242,8 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
pl = service.create_playlist(db, user, "L")
assert service.add_item(db, pl, "a", mark_dirty=False) is True
assert service.add_item(db, pl, "a", mark_dirty=False) is False
service.add_to_playlist(db, pl, "b", None)
service.add_to_playlist(db, pl, "c", 0)
service.add_to_playlist(db, pl, "a", 2) # already there: stays put
assert service.add_item(db, pl, "b", mark_dirty=True) is True
assert service.restore_items(db, pl, ["c", "a"], [0, 2]) == 1 # "a" is there: it stays put
assert _uncommitted(db)
db.commit()
assert _ids(db, pl) == ["c", "a", "b"]
@@ -224,7 +252,7 @@ def test_an_add_is_idempotent_and_lands_where_asked(db, make_user):
db.commit()
assert _ids(db, pl) == ["c", "b"]
assert queries.item_counts(db, [pl.id]) == {pl.id: 2}
assert queries.durations(db, [pl.id]) == {pl.id: 120}
assert queries.durations(db, [pl.id]) == {pl.id: (120, 0)}
assert queries.containing(db, [pl.id], "b") == {pl.id}
@@ -238,14 +266,13 @@ def test_a_move_never_leaves_a_video_in_two_lists(db, make_user):
service.add_item(db, dst, "b", mark_dirty=False)
db.commit()
with pytest.raises(service.MoveRefused, match="same_playlist"):
service.move_item(db, src, src, "a")
service.move_item(db, src, dst, "a")
service.move_item(db, src, dst, "b") # already in the target: dropped from the source
service.move_items(db, src, src, ["a"])
assert service.move_items(db, src, dst, ["a"]) == (["a"], [])
# already in the target: dropped from the source
assert service.move_items(db, src, dst, ["b"]) == ([], ["b"])
db.commit()
assert (_ids(db, src), _ids(db, dst)) == ([], ["b", "a"])
with pytest.raises(service.MoveRefused, match="not_in_playlist"):
service.move_item(db, src, dst, "a")
db.rollback()
assert service.move_items(db, src, dst, ["a"]) == ([], [])
def test_a_reorder_keeps_what_it_left_out_after_the_listed(db, make_user):
@@ -267,7 +294,7 @@ def test_a_rename_and_a_delete(db, make_user):
service.add_item(db, pl, "a", mark_dirty=False)
service.rename_playlist(db, pl, "New")
db.commit()
assert queries.playlist_totals(db, pl.id) == (1, 60, None)
assert queries.playlist_totals(db, pl.id) == (1, 60, 0, None)
assert pl.name == "New"
service.delete_playlist(db, pl)
db.commit()
+770
View File
@@ -0,0 +1,770 @@
"""The refusal scan (`refusal_sites.py`) on small sources: each shape on its closed list is followed to its
values, and each exit from that list is reported, unread or as an unread code. A scan that silently drops a
shape is a guard green over nothing, so every exit is pinned here, not only the paths the app takes. Pure:
no app import, no database."""
import textwrap
from tests.refusal_sites import scan, scan_app
def _scan(**files: str) -> dict:
return scan(
{f"{name.replace('__', '/')}.py": textwrap.dedent(src) for name, src in files.items()}
)
def _code(result: dict, code: str) -> dict:
return result["codes"][code]
def _unread(result: dict, code: str) -> list[str]:
return [u.split(": ", 1)[1] for u in _code(result, code)["unread"]]
REFUSAL = """
from fastapi import HTTPException
def refused(status: int, reason: str, **params: object) -> HTTPException:
return HTTPException(status_code=status, detail={**params, "code": "refused", "reason": reason})
"""
def test_a_literal_wherever_it_sits_in_the_dict():
r = _scan(
a="""
def f(x):
raise HTTPException(409, {"code": "clip", "limit_s": 5, "reason": "too_long"})
def g(x):
raise HTTPException(409, {"reason": "before", "code": "clip"})
def h(x):
raise HTTPException(409, {
"code": "clip", # a comment between
"reason": "after_comment",
})
def i(x):
raise HTTPException(409, detail=dict(code="clip", reason="by_call"))
"""
)
assert _code(r, "clip")["reasons"] == ["after_comment", "before", "by_call", "too_long"]
assert _code(r, "clip")["unread"] == []
def test_both_arms_of_a_conditional_and_a_none_arm_makes_the_site_reasonless():
r = _scan(
a="""
def f(x):
raise HTTPException(409, {"code": "edit", "reason": "a" if x else "b"})
def g(x):
raise HTTPException(409, {"code": "playlist", "reason": "lit" if x else None})
def h(x):
raise HTTPException(409, {"code": "busy", "reason": None})
"""
)
assert _code(r, "edit")["reasons"] == ["a", "b"] and _code(r, "edit")["reasonless"] == []
assert _code(r, "playlist")["reasons"] == ["lit"]
assert _code(r, "playlist")["reasonless"] == ["a.py:5"]
assert _code(r, "busy")["reasonless"] == ["a.py:7"]
def test_what_the_list_does_not_hold_is_unread():
r = _scan(
a="""
def f(x, svc):
raise HTTPException(409, {"code": "edit", "reason": x.why or "c"})
def g(svc):
raise HTTPException(409, {"code": "edit", "reason": svc.reason()})
def h():
raise HTTPException(409, {"code": "edit", "reason": 5})
def i(e):
raise HTTPException(409, {"code": "edit", "reason": f"x{e}"})
"""
)
assert _unread(r, "edit") == ["x.why or 'c'", "svc.reason()", "5", "f'x{e}'"]
assert _code(r, "edit")["reasons"] == []
def test_a_helper_is_followed_to_every_call_however_it_is_written():
r = _scan(
refusal=REFUSAL,
routes__a="""
from http import HTTPStatus
from app import refusal
from app.refusal import refused
def f(status):
raise refused(status, "by_name_status")
def g():
raise refused(HTTPStatus.CONFLICT, reason="by_keyword")
def h():
raise refusal.refused(409, "qualified")
def i(x):
raise refused(409, "a" if x else "second_arm")
""",
routes__b="""
from other.lib import refused
def f():
refused("an outside library's, not the app's")
""",
)
assert _code(r, "refused")["reasons"] == [
"a",
"by_keyword",
"by_name_status",
"qualified",
"second_arm",
]
assert _code(r, "refused")["unread"] == []
def test_a_call_the_helper_cannot_be_followed_through_is_unread():
r = _scan(
refusal=REFUSAL,
routes__a="""
from app.refusal import refused
def f(args, kw):
pick = refused
raise refused(*args)
def g(kw):
raise refused(409, **kw)
def h():
raise refused(409)
""",
routes__b="""
import app.refusal
def f():
raise app.refusal.refused(409, "dotted")
""",
routes__c="""
from ..refusal import refused
def f():
raise refused(409, "relative")
""",
)
unread = _unread(r, "refused")
assert len(unread) == 6
assert sum("the helper used as a value" in u for u in unread) == 1
assert sum("a reference the scan cannot resolve" in u for u in unread) == 2
assert sum("does not pass reason" in u for u in unread) == 2 # `**kw`, and nothing at all
assert any(u == "refused(*args)" for u in unread)
assert _code(r, "refused")["reasons"] == []
def test_a_default_is_not_followed_a_call_must_pass_the_reason():
r = _scan(
routes___common="""
def not_found(what: str = "generic"):
return HTTPException(404, {"code": "not_found", "reason": what})
def get_or_404(db, id_, *, what: str = "generic"):
raise not_found(what)
""",
routes__a="""
from app.routes._common import get_or_404, not_found
def f(db):
get_or_404(db, 1, what="user")
get_or_404(db, 2)
raise not_found("debug_session")
def g():
raise not_found()
""",
)
assert _code(r, "not_found")["reasons"] == ["debug_session", "user"]
assert [u.endswith("(does not pass what)") for u in _unread(r, "not_found")] == [True, True]
def test_a_positional_only_reason_passed_by_name_is_not_bound_to_it():
r = _scan(
a="""
def deny(reason: str, /, **params):
return HTTPException(409, {"code": "playlist", "reason": reason})
def f():
raise deny("by_position")
def g():
raise deny(reason="lands_in_params")
"""
)
assert _code(r, "playlist")["reasons"] == ["by_position"]
assert _unread(r, "playlist") == ["deny(reason='lands_in_params') (does not pass reason)"]
def test_a_helper_reaching_two_sites_gives_both_its_reasons_and_one_reached_again_is_unread():
r = _scan(
a="""
def deny(reason: str, admin):
if admin:
return HTTPException(409, {"code": "activate", "reason": reason})
return HTTPException(409, {"code": "playlist", "reason": reason})
def again(reason: str, n):
if n:
return again(reason, n - 1)
return HTTPException(409, {"code": "edit", "reason": reason})
def f():
raise deny("foo", True)
def g():
raise again("bar", 3)
"""
)
assert _code(r, "activate")["reasons"] == ["foo"]
assert _code(r, "playlist")["reasons"] == ["foo"]
assert _code(r, "edit")["reasons"] == ["bar"]
assert any("reached again while it is followed" in u for u in _unread(r, "edit"))
def test_a_parameter_its_body_binds_again_is_unread():
r = _scan(
a="""
from typing import Literal
def refused(status, reason: str = ""):
reason = reason or "fallback"
return HTTPException(status, {"code": "refused", "reason": reason})
def typed(reason: Literal["a"]):
if not reason:
reason = "b"
return HTTPException(409, {"code": "edit", "reason": reason})
def f():
raise refused(409, "x")
"""
)
assert _code(r, "refused")["reasons"] == []
assert _unread(r, "refused") == ["reason (a parameter its body binds again)"]
assert _unread(r, "edit") == ["reason (a parameter its body binds again)"]
def test_a_local_is_followed_only_through_plain_assignments():
r = _scan(
a="""
from typing import Literal
def f(x):
why = "first"
if x:
why = "second" if x > 1 else "third"
raise HTTPException(409, {"code": "edit", "reason": why})
def g(svc):
hit, why = svc.fetch()
raise HTTPException(409, {"code": "playlist", "reason": why})
def h(items):
for why in items:
pass
raise HTTPException(409, {"code": "playlist", "reason": why})
def i():
why: Literal["typed"] = pick()
raise HTTPException(409, {"code": "playlist", "reason": why})
def j(items):
LABELS[why] = "NotAReason"
raise HTTPException(409, {"code": "playlist", "reason": why})
def k(*why):
raise HTTPException(409, {"code": "playlist", "reason": why})
"""
)
assert _code(r, "edit")["reasons"] == ["first", "second", "third"]
assert _code(r, "playlist")["reasons"] == []
assert _unread(r, "playlist") == [
"why",
"why",
"why",
"why",
"why",
] # unpacked, a loop, annotated, unbound, `*args`
def test_a_call_answers_with_its_literal_return_type():
r = _scan(
a="""
from typing import Literal
def why(x) -> Literal["quota", "gone"]:
return "quota"
def how(x) -> str:
return "x"
def f(x):
raise HTTPException(409, {"code": "youtube", "reason": why(x)})
def g(x):
raise HTTPException(409, {"code": "youtube", "reason": how(x)})
"""
)
assert _code(r, "youtube")["reasons"] == ["gone", "quota"]
assert _unread(r, "youtube") == ["how(x)"]
def test_a_module_constant_is_read_only_as_mod_name_and_only_when_bound_once():
r = _scan(
urlguard="""
REASON_MALFORMED = "malformed"
TWICE = "a"
TWICE = "b"
if FLAG:
LATER = "c"
""",
routes__a="""
from app import urlguard
BARE = "bare"
def f():
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.REASON_MALFORMED})
def g():
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.TWICE})
def h():
raise HTTPException(409, {"code": "url_blocked", "reason": urlguard.LATER})
def i():
raise HTTPException(409, {"code": "url_blocked", "reason": BARE})
""",
)
assert _code(r, "url_blocked")["reasons"] == ["malformed"]
assert _unread(r, "url_blocked") == ["urlguard.TWICE", "urlguard.LATER", "BARE"]
def test_a_name_imported_from_two_places_or_under_an_if_is_not_looked_up():
r = _scan(
m1='R = Literal["one"]',
m2='R = Literal["two"]',
a="""
def f():
from app.m1 import R
def g():
from app.m2 import R
def h(reason: R):
return HTTPException(409, {"code": "edit", "reason": reason})
""",
b="""
if FLAG:
from app.m1 import R
def h(reason: R):
return HTTPException(409, {"code": "playlist", "reason": reason})
""",
c="""
from app.m1 import R
def h(reason: R):
return HTTPException(409, {"code": "clip", "reason": reason})
""",
)
assert _code(r, "edit")["reasons"] == [] and _unread(r, "edit") == ["reason"]
assert _code(r, "playlist")["reasons"] == [] and _unread(r, "playlist") == ["reason"]
assert _code(r, "clip")["reasons"] == ["one"]
def test_a_caught_exception_answers_with_the_literal_its_class_declares_on_self():
r = _scan(
errors="""
from typing import Literal
ClipReason = Literal["bad_range", "too_short"]
EditReason = Literal["not_media"]
class ClipError(Exception):
def __init__(self, reason: ClipReason, **params: float):
self.reason: ClipReason = reason
self.params = params
class EditError(Exception):
reason: EditReason
class Untyped(Exception):
def __init__(self, reason: str):
self.reason = reason
class Narrower(ClipError):
pass
""",
routes__a="""
from app import errors
def f():
try:
pass
except errors.ClipError as e:
raise HTTPException(422, {"code": "clip", "reason": e.reason})
except (errors.ClipError, errors.EditError) as e:
raise HTTPException(422, {"code": "clip", "reason": e.reason})
except errors.EditError as e:
raise HTTPException(422, {"code": "clip", "reason": e.reason})
except errors.Untyped as e:
raise HTTPException(422, {"code": "clip", "reason": e.reason})
except errors.Narrower as e:
raise HTTPException(422, {"code": "clip", "reason": e.reason})
except errors.ClipError as e:
raise HTTPException(422, {"code": "clip", "reason": e})
""",
)
assert _code(r, "clip")["reasons"] == ["bad_range", "too_short"]
assert _unread(r, "clip") == [
"e.reason", # a tuple of classes
"e.reason (EditError.reason has no Literal type)", # declared on the class body only
"e.reason (Untyped.reason has no Literal type)",
"e.reason (Narrower.reason has no Literal type)", # a subclass: not looked up the bases
"e", # the exception itself
]
def test_a_parameter_answers_with_its_literal_or_its_class_and_any_other_type_is_unread():
r = _scan(
a="""
from typing import Literal
Why = Literal["one", "two"]
class Err(Exception):
def __init__(self, reason: Literal["three"]):
self.reason: Literal["three"] = reason
def typed(reason: Why):
return HTTPException(409, {"code": "playlist", "reason": reason})
def inline(reason: Literal["four"]):
return HTTPException(409, {"code": "playlist", "reason": reason})
def by_class(exc: Err):
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
def rebound(exc: Err):
exc = other()
return HTTPException(409, {"code": "playlist", "reason": exc.reason})
def optional(reason: Literal["five"] | None = None):
return HTTPException(409, {"code": "playlist", "reason": reason})
def quoted(reason: "Why"):
return HTTPException(409, {"code": "playlist", "reason": reason})
def number(reason: int):
return HTTPException(409, {"code": "playlist", "reason": reason})
class C:
def method(self, reason: str):
return HTTPException(409, {"code": "playlist", "reason": reason})
"""
)
assert _code(r, "playlist")["reasons"] == ["four", "one", "three", "two"]
unread = _unread(r, "playlist")
# a parameter bound again, `| None`, a quoted annotation, `int`
assert unread[:4] == ["exc.reason", "reason", "reason", "reason"]
assert "method(reason) is not a module-level helper" in unread[4]
assert len(unread) == 5
def test_a_str_helper_nobody_calls_sends_nothing():
r = _scan(
a="""
def helper(reason: str):
return HTTPException(409, {"code": "playlist", "reason": reason})
"""
)
assert _code(r, "playlist") == {"reasons": [], "unread": [], "sites": 1, "reasonless": []}
def test_a_map_lookup_is_its_values_and_its_keys_must_be_exactly_what_the_index_can_be():
src = """
from typing import Literal
SettingReason = Literal["too_low", "too_high"]
class InvalidSetting(ValueError):
def __init__(self, key: str, reason: SettingReason, **params: object):
self.reason: SettingReason = reason
self.params = params
_MAP: dict[SettingReason, str] = {{{keys}}}
def refused(status: int, reason: str, **params: object):
return HTTPException(status, {{**params, "code": "refused", "reason": reason}})
def setting_refused(exc: InvalidSetting):
return refused(400, _MAP[{index}], **exc.params)
"""
both = '"too_low": "setting_too_low", "too_high": "setting_too_high"'
whole = _scan(a=src.format(keys=both, index="exc.reason"))
assert _code(whole, "refused")["reasons"] == ["setting_too_high", "setting_too_low"]
assert whole["errors"] == [] and whole["unreadCodes"] == []
for keys, index in [
('"too_low": "setting_too_low"', "exc.reason"), # a key short
(both + ', "gone": "setting_gone"', "exc.reason"), # a key no index can be
(both, "exc.key"), # an index the scan cannot read
]:
r = _scan(a=src.format(keys=keys, index=index))
assert len(r["errors"]) == 1 and "not what its index can be" in r["errors"][0]
def test_a_lookup_in_anything_but_a_module_dict_of_literal_keys_is_unread():
r = _scan(
a="""
BUILT = make()
NUMBERED = {1: "one"}
def f(table, k):
raise HTTPException(409, {"code": "edit", "reason": table[k]})
def g(k):
raise HTTPException(409, {"code": "edit", "reason": BUILT[k]})
def h(k):
raise HTTPException(409, {"code": "edit", "reason": NUMBERED[k]})
"""
)
assert _unread(r, "edit") == ["table[k]", "BUILT[k]", "NUMBERED[k]"]
assert r["errors"] == []
def test_a_code_must_be_one_literal():
r = _scan(
a="""
BUSY = "busy"
def f(code):
raise HTTPException(409, {"code": "busy"})
def g(code):
raise HTTPException(409, {"code": code})
def h():
raise HTTPException(409, {"code": BUSY})
def i(x):
raise HTTPException(409, {"code": "a" if x else "b"})
def j(x):
raise HTTPException(409, {"code": "a" if x else None})
def k(key):
raise HTTPException(409, {"code": "edit", key: "late"})
"""
)
assert r["codes"]["busy"]["reasonless"] == ["a.py:4"]
assert set(r["codes"]) == {"busy"}
assert len(r["unreadCodes"]) == 5
assert r["unreadCodes"][-1].endswith(
"(a spread or a computed key the code and the reason do not follow)"
)
def test_a_spread_or_a_computed_key_stands_before_both_the_code_and_the_reason():
def unread_codes(detail: str) -> list[str]:
return _scan(a=f"def f(x, more, key):\n raise HTTPException(409, {detail})\n")[
"unreadCodes"
]
assert unread_codes('{**more, "code": "refused", "reason": "a"}') == []
assert unread_codes('{key: 1, **more, "reason": "a", "code": "refused"}') == []
for late in [
'{"code": "refused", "reason": "a", **more}', # the spread may replace both
'{"code": "refused", **more, "reason": "a"}', # … the code
'{"reason": "a", **more, "code": "refused"}', # … the reason
'{**more, "code": "refused"}', # no reason after it: the spread may carry one
'{"code": "refused", "reason": "a", key: "x"}', # a key that may be either
'dict(**more, code="refused", reason="a")', # dict() raises on a repeated key
'dict(more, code="refused", reason="a")',
]:
assert unread_codes(late), late
def test_a_detail_stands_only_where_it_is_sent():
def result(body: str, extra: str = "") -> dict:
return _scan(a=f"def f(x, cached, fix):\n{textwrap.indent(body, ' ')}\n{extra}")
for body in [
'detail = {"code": "login", "reason": "a"}\nraise HTTPException(401, detail)',
'if (detail := {"code": "login", "reason": "a"}):\n raise HTTPException(401, detail)',
'raise HTTPException(401, {"code": "login", "reason": "a"}.copy())',
'raise HTTPException(401, fix({"code": "login", "reason": "a"}))',
'raise HTTPException(401, cached or {"code": "login", "reason": "a"})',
'return JSONResponse({"code": "login", "reason": "a"}, status_code=401)', # the whole body
'raise HTTPException(401, None, {"code": "login", "reason": "a"})', # the headers
'raise HTTPException(401, headers={"code": "login", "reason": "a"})',
'raise HTTPException(401, "a" if {"code": "login", "reason": "a"} else "b")',
]:
r = result(body)
assert r["unreadCodes"] and "login" not in r["codes"], body
assert "(a detail put where the scan cannot follow it)" in r["unreadCodes"][0], body
held = _scan(a='class C:\n detail = {"code": "login"}\n')
assert "(a detail put where the scan cannot follow it)" in held["unreadCodes"][0]
sent = result(
'raise HTTPException(401, {"code": "login", "reason": "a"} if x else {"code": "login"})\n'
'raise HTTPException(401, detail=dict(code="login", reason="b"))\n'
'return JSONResponse(content={"detail": {"code": "login", "reason": "c"}})'
)
assert sent["unreadCodes"] == [] and _code(sent, "login")["reasons"] == ["a", "b", "c"]
def test_a_factory_is_a_detail_only_when_each_return_is_one_and_each_call_is_sent():
factory = """
def quota(e) -> dict:
return {"code": "quota", "reason": "files"}
"""
sent = "def f(e):\n raise HTTPException(422, detail=quota(e))\n"
assert _scan(a=factory + sent)["codes"]["quota"]["reasons"] == ["files"]
for change in [
"def g(e):\n d = quota(e)\n d['code'] = 'other'\n raise HTTPException(422, d)\n",
"def g(e):\n raise HTTPException(422, detail=quota(e).copy())\n",
"pick = quota\n",
]:
r = _scan(a=factory + sent + change)
assert r["unreadCodes"] and "quota" not in r["codes"], change
twice = _scan(a="quota = None\n" + factory + sent) # the def is the last of two bindings
assert twice["unreadCodes"] and "quota" not in twice["codes"]
for other in [
" if e:\n return None\n",
" if e:\n return cached\n",
" if e:\n return {'code': 'quota', 'reason': 'x'} if e > 1 else None\n",
" log({'code': 'quota', 'reason': 'logged'})\n", # a detail it does not return
]:
bent = factory.replace("-> dict:\n", "-> dict:\n" + other)
assert _scan(a=bent + sent)["unreadCodes"], other
method = "class C:\n def quota(self):\n return {'code': 'quota', 'reason': 'm'}\n"
assert _scan(a=method)["unreadCodes"]
dotted = _scan(
a=factory,
routes__b="import app.a\n\ndef g(e):\n raise HTTPException(422, detail=app.a.quota(e))\n",
)
assert dotted["unreadCodes"] and "quota" not in dotted["codes"]
def test_a_name_a_lambda_a_comprehension_a_nonlocal_or_a_global_binds_is_unread():
r = _scan(
a="""
def f(svc):
why = "invalid"
make = lambda why: HTTPException(401, {"code": "login", "reason": why})
raise make(svc.state)
def g(svc):
why = "invalid"
raise [HTTPException(401, {"code": "login", "reason": why}) for why in svc.all][0]
def h(svc):
why = "invalid"
def inner():
nonlocal why
why = svc.reason
inner()
raise HTTPException(401, {"code": "login", "reason": why})
def i(svc):
global why
why = "invalid"
raise HTTPException(401, {"code": "login", "reason": why})
"""
)
assert _code(r, "login")["reasons"] == []
assert sorted(_unread(r, "login")) == [
"why (bound by a lambda or a comprehension)",
"why (bound by a lambda or a comprehension)",
"why (named by a nonlocal or a global)",
"why (named by a nonlocal or a global)",
]
def test_a_local_that_reaches_itself_is_unread():
r = _scan(
a="""
def f(x):
why = "a"
if x:
why = why if x > 1 else "b"
raise HTTPException(409, {"code": "edit", "reason": why})
def g(x):
a = "z"
b = a
a = b
raise HTTPException(409, {"code": "playlist", "reason": a})
def h(x):
why = "c"
if x:
raise HTTPException(409, {"code": "busy", "reason": why})
raise HTTPException(409, {"code": "busy", "reason": why})
"""
)
assert _code(r, "busy")["reasons"] == ["c"] and _code(r, "busy")["unread"] == []
assert _code(r, "edit")["reasons"] == ["a", "b"]
assert _unread(r, "edit") == ["why (reached again while it is resolved)"]
assert _code(r, "playlist")["reasons"] == ["z"]
assert any("reached again while it is resolved" in u for u in _unread(r, "playlist"))
def test_a_module_constant_or_a_map_changed_elsewhere_is_unread():
def reasons(urlguard: str, route: str = "", **more: str) -> tuple[list[str], list[str]]:
r = _scan(
**more,
urlguard="REASON = 'malformed'\n_MAP = {'a': 'map_a'}\n" + urlguard,
routes__a="from app import urlguard\n"
+ route
+ "\ndef f(k: Literal['a']):\n"
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
+ " raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard._MAP[k]})\n",
)
return _code(r, "url_blocked")["reasons"], _unread(r, "url_blocked")
assert reasons("def get(k):\n return _MAP.get(k)\n") == (["malformed", "map_a"], [])
other = "from app import other\ndef g():\n other.REASON = 'x'\n" # another module's REASON
assert reasons("", other, other="REASON = 'y'\n") == (["malformed", "map_a"], [])
shadow = (
"def h(*urlguard):\n"
" raise HTTPException(409, {'code': 'url_blocked', 'reason': urlguard.REASON})\n"
)
assert reasons("", shadow)[1] == ["urlguard.REASON"]
for urlguard, route in [
("def reset():\n global REASON\n REASON = 'x'\n", ""),
("", "def g():\n urlguard.REASON = 'x'\n"),
("", "def g():\n del urlguard.REASON\n"),
]:
assert reasons(urlguard, route)[0] == ["map_a"], (urlguard, route)
for urlguard in [
"_MAP['a'] = compute()\n",
"def init(x):\n _MAP['a'] = x\n",
"_MAP.update(a='other')\n",
"def drop():\n del _MAP['a']\n",
"register(_MAP)\n",
]:
got, unread = reasons(urlguard)
assert got == ["malformed"] and "(the map is changed after it is built)" in unread[0]
def test_a_helper_bound_twice_is_not_followed():
r = _scan(
refusal="refused = None\n" + textwrap.dedent(REFUSAL),
routes__a="from app.refusal import refused\n\ndef f():\n raise refused(409, 'lost')\n",
)
assert _code(r, "refused")["reasons"] == []
assert _unread(r, "refused")[0].endswith("refused(reason) is not a module-level helper")
def test_a_dotted_path_through_an_app_module_that_goes_nowhere_is_unread():
r = scan(
{
"__init__.py": "",
"refusal.py": textwrap.dedent(REFUSAL),
"routes/a.py": "import app as a\n\ndef f():\n raise a.refusal.refused(409, 'dotted')\n",
}
)
assert _unread(r, "refused") == ["a.refusal.refused (a reference the scan cannot resolve)"]
def test_a_helper_reached_through_an_app_module_that_does_not_bind_it_is_unread():
for shim in [
"from app.refusal import *\n",
"try:\n from app.refusal import refused\nexcept ImportError:\n refused = None\n",
]:
r = _scan(
refusal=REFUSAL,
shim=shim,
routes__a="from app.shim import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
)
assert _unread(r, "refused") == ["refused (a reference the scan cannot resolve)"], shim
package = scan(
{
"pkg/refusal.py": textwrap.dedent(REFUSAL),
"pkg/__init__.py": "from .refusal import refused\n",
"routes/a.py": "from app.pkg import refused\n\ndef f():\n raise refused(409, 'hidden')\n",
}
)
assert _unread(package, "refused") == ["refused (a reference the scan cannot resolve)"]
def test_the_data_files_are_excluded_by_name_only():
r = _scan(
plex__tmdb='def f():\n return {"code": "HU", "name": "Hungary"}',
plex__other='def f():\n return {"code": "HU"}',
)
assert r["excluded"] == {"plex/tmdb.py": 1}
assert "HU" in r["codes"]
def test_the_app_scans():
"""The real tree, on the interpreter the backend runs: every refusal it holds is read (the
frontend's contract judges what is read)."""
result = scan_app()
assert {"refused", "not_found", "login", "clip"} <= set(result["codes"])
assert result["errors"] == [] and result["unreadCodes"] == []
+12 -2
View File
@@ -101,8 +101,10 @@ def test_a_video_state_cleared_through_the_route_is_gone(db, make_user, as_user)
_video(db)
db.add(VideoState(user_id=user.id, video_id="v1", status="hidden", position_seconds=90))
db.commit()
assert as_user(user).delete("/api/videos/v1/state").status_code == 200
assert _state(db, user) is None
assert as_user(user).delete("/api/videos/v1/progress").status_code == 200
db.expire_all()
row = _state(db, user)
assert (row.status, row.position_seconds) == ("hidden", 0)
# --- saved views ------------------------------------------------------------------------------
@@ -308,6 +310,14 @@ def test_clearing_leaves_a_video_the_caller_never_found(db, make_user, as_user):
assert r.json() == {"ok": True, "removed": 0}
def test_clearing_refuses_a_body_without_its_list(db, make_user, as_user):
# A misnamed key read as an empty list: 200, and nothing cleared.
user = make_user()
r = as_user(user).post("/api/search/clear", json={"ids": ["s1"]})
assert r.status_code == 422
assert as_user(user).post("/api/search/clear", json={"video_ids": []}).status_code == 200
# --- Plex show and season marks ---------------------------------------------------------------
+148
View File
@@ -0,0 +1,148 @@
"""A share link's address says what it opens: /watch/ a video, /file/ anything else, /folder/ a folder.
Only a word for the reader: the token is the key, so every path opens every link, and the links
already sent out on /watch/ keep working for good. DB-lane.
"""
import pytest
from cryptography.fernet import Fernet
from app import main, security
from app.config import settings
from app.downloads import og
from app.downloads import probe as probemod
_PAGE = """<!doctype html><html><head>
<!--OG:START--><meta property="og:title" content="Siftlode" /><!--OG:END-->
</head><body><script src="/assets/{entry}.js"></script></body></html>"""
_VIDEO = {
"streams": [
{"codec_type": "video", "codec_name": "h264", "pix_fmt": "yuv420p"},
{"codec_type": "audio", "codec_name": "aac"},
],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
_AUDIO = {
"streams": [{"codec_type": "audio", "codec_name": "aac"}],
"format": {"format_name": "mov,mp4", "duration": "5"},
}
@pytest.fixture
def pages(tmp_path, monkeypatch):
index, watch = tmp_path / "index.html", tmp_path / "watch.html"
index.write_text(_PAGE.format(entry="app"), encoding="utf-8")
watch.write_text(_PAGE.format(entry="watch"), encoding="utf-8")
monkeypatch.setattr(main, "INDEX_HTML", index)
monkeypatch.setattr(main, "WATCH_HTML", watch)
@pytest.fixture
def key(monkeypatch):
"""A link's address is read back from its encrypted token."""
monkeypatch.setattr(security, "_fernet", Fernet(Fernet.generate_key()))
@pytest.fixture
def shared(as_user, make_user, tmp_path, monkeypatch, upload_headers):
"""Uploads a file ffprobe reads as `data` (None: it cannot read it) and shares it."""
monkeypatch.setattr(settings, "files_root", str(tmp_path / "media"))
owner = as_user(make_user())
def share(name: str, data: dict | None, **link: object) -> dict:
monkeypatch.setattr(probemod, "probe_json", lambda _p: data)
job = owner.post(
"/api/files/upload",
content=b"some bytes",
headers=upload_headers(name, display_name=name),
)
assert job.status_code == 200, job.text
made = owner.post(f"/api/downloads/{job.json()['id']}/links", json=link)
assert made.status_code == 200, made.text
return made.json()
share.owner = owner # type: ignore[attr-defined]
return share
@pytest.mark.parametrize(
("name", "data", "prefix"),
[
("clip.mp4", _VIDEO, "/watch/"),
("song.m4a", _AUDIO, "/file/"),
("backup.tar.gz", None, "/file/"),
],
)
def test_a_link_is_given_out_on_the_path_of_what_it_opens(shared, name, data, prefix):
made = shared(name, data)
assert made["url"] == f"{prefix}{made['token']}"
def test_a_folder_link_is_given_out_on_folder(shared, key):
owner = shared.owner
folder = owner.post("/api/files/folders", json={"name": "Friends"}).json()
made = owner.post(f"/api/downloads/folders/{folder['id']}/links", json={}).json()
assert made["url"] == f"/folder/{made['token']}"
listed = owner.get(f"/api/downloads/folders/{folder['id']}/links").json()
assert [entry["url"] for entry in listed] == [made["url"]]
def test_the_list_gives_the_same_address_as_the_making(shared, key):
made = shared("backup.tar.gz", None)
owner = shared.owner
files = owner.get("/api/files").json()
job_id = next(f["id"] for f in files if f["display_name"] == "backup.tar.gz")
listed = owner.get(f"/api/downloads/{job_id}/links").json()
assert [entry["url"] for entry in listed] == [made["url"]]
@pytest.mark.parametrize("path", ["watch", "file", "folder"])
@pytest.mark.parametrize(
("name", "data"), [("clip.mp4", _VIDEO), ("backup.tar.gz", None)], ids=["video", "file"]
)
def test_every_path_opens_every_link(pages, shared, client, path, name, data):
"""The links already sent on /watch/ open whatever they open, for good."""
token = shared(name, data)["token"]
r = client.get(f"/{path}/{token}")
assert r.status_code == 200
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
assert f'content="{name}"' in r.text
@pytest.mark.parametrize("path", ["/file/no-such-token", "/folder/no-such-token", "/file/a/b"])
def test_a_new_path_without_a_live_link_still_gets_the_player(pages, client, path):
r = client.get(path)
assert r.status_code == 200
assert "/assets/watch.js" in r.text and "/assets/app.js" not in r.text
def test_a_card_names_a_file_as_a_file(shared, db):
made = shared("backup.tar.gz", None)
tags = og._watch_tags(made["token"], db)
assert f'{made["url"]}"' in tags
assert "video.other" not in tags
locked = shared("notes.tar.gz", None, password="pw")
hidden = og._watch_tags(locked["token"], db)
assert "Password-protected file" in hidden and "notes" not in hidden
assert "video.other" not in hidden
def test_a_card_of_a_video_stays_a_video(shared, db):
made = shared("clip.mp4", _VIDEO)
tags = og._watch_tags(made["token"], db)
assert f"/watch/{made['token']}" in tags and "video.other" in tags
locked = shared("other.mp4", _VIDEO, password="pw")
assert "Password-protected video" in og._watch_tags(locked["token"], db)
@pytest.mark.parametrize(
("name", "data", "video"),
[("clip.mp4", _VIDEO, True), ("song.m4a", _AUDIO, False), ("backup.tar.gz", None, False)],
)
def test_a_password_link_says_whether_it_opens_a_video(shared, client, name, data, video):
"""The lock screen asks to watch a video and to open anything else, whichever path it is on."""
token = shared(name, data, password="pw")["token"]
meta = client.get(f"/api/public/watch/{token}").json()
assert meta == {"needs_password": True, "kind": "file", "video": video}
+223
View File
@@ -0,0 +1,223 @@
"""A seeded shuffle walks a stored random key instead of sorting the whole scope. DB lane.
Every video carries a random number drawn once (`Video.shuffle_key`). A seed picks a point on that
ring; the feed's Shuffle sort pages along the ring from there and wraps, and the Home surprise reads
a window of it and shuffles that window by the seed (`discover.queries.yt_surprise`). Ordering the
scope by md5(id || seed) cost ~350 ms per call on prod and grew with every Shuffle page.
"""
import base64
import random
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from sqlalchemy import event, insert, text
from app import feedquery
from app.catalog.models import Channel, Subscription, Video
from app.discover import queries
from app.discover.queries import _FEED_BASE
NOW = datetime.now(timezone.utc)
KEYS = {"k05": 0.05, "k20": 0.20, "k40": 0.40, "k60": 0.60, "k80": 0.80, "k95": 0.95}
def _seed(db, make_user, keys=KEYS):
user = make_user()
db.add(Channel(id="UCring", title="Ring"))
db.flush()
db.add(Subscription(user_id=user.id, channel_id="UCring"))
db.add_all(
[
Video(
id=vid,
channel_id="UCring",
title=vid,
duration_seconds=20 * 60,
published_at=NOW - timedelta(days=i + 1),
shuffle_key=key,
)
for i, (vid, key) in enumerate(keys.items())
]
)
db.commit()
return user
def _ring(seed, keys=KEYS):
"""The ring order from the seed's point: the keys at or after it ascending, then the rest."""
r = feedquery.shuffle_start(seed)
return sorted(keys, key=lambda v: (keys[v] < r, keys[v]))
def _seed_landing_between(lo, hi):
return next(s for s in range(1000) if lo < feedquery.shuffle_start(s) < hi)
def test_the_walk_starts_at_the_seeds_point_and_wraps(db, make_user):
user = _seed(db, make_user)
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
seed = _seed_landing_between(0.45, 0.55) # starts at k60, wraps to k05 after k95
got = [r.id for r in feedquery.shuffle_walk(db, query, seed, 6)]
assert got == ["k60", "k80", "k95", "k05", "k20", "k40"]
assert [r.id for r in feedquery.shuffle_walk(db, query, seed, 4)] == got[:4]
def test_the_feed_shuffle_pages_along_the_ring(db, make_user, as_user):
user = _seed(db, make_user)
client = as_user(user)
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.85, 0.9)):
walked, cursor = [], None
for _ in range(5):
url = f"/api/feed?sort=shuffle&seed={seed}&limit=2" + (
f"&cursor={cursor}" if cursor else ""
)
body = client.get(url).json()
walked += [it["id"] for it in body["items"]]
cursor = body["next_cursor"]
if not body["has_more"]:
break
assert walked == _ring(seed), seed
def test_a_new_seed_opens_the_shuffle_elsewhere(db, make_user, as_user):
user = _seed(db, make_user)
client = as_user(user)
first = {
seed: client.get(f"/api/feed?sort=shuffle&seed={seed}&limit=2").json()["items"][0]["id"]
for seed in (_seed_landing_between(0.1, 0.15), _seed_landing_between(0.65, 0.7))
}
assert sorted(first.values()) == ["k20", "k80"]
@contextmanager
def _statements(db):
seen: list[str] = []
def grab(conn, cursor, statement, *args):
seen.append(statement)
engine = db.get_bind()
event.listen(engine, "before_cursor_execute", grab)
try:
yield seen
finally:
event.remove(engine, "before_cursor_execute", grab)
def _draws_of_videos(seen):
return [s for s in seen if "FROM videos" in s and "LIMIT" in s]
def test_neither_draw_orders_the_scope_by_a_hash(db, make_user):
# Each draw reads the videos as an ascending range of the key (what ix_videos_shuffle_key serves),
# never ordered by a hash or by the ring's CASE (which no index can serve).
user = _seed(db, make_user)
with _statements(db) as surprise:
queries.yt_surprise(db, user, "any", 7, 3)
with _statements(db) as page:
feedquery.feed_page(
db, user, dict(_FEED_BASE), sort="shuffle", seed=7, limit=2, cursor=None, lang="en"
)
for seen in (surprise, page):
draws = _draws_of_videos(seen)
assert draws
for s in draws:
assert "md5(" not in s.lower()
assert "ORDER BY videos.shuffle_key, videos.id" in s.replace("\n", " "), s
def test_the_walk_reads_the_ring_index_and_sorts_nothing(db, make_user):
# The text of the query is not the plan: each arc must be served by ix_videos_shuffle_key in key
# order (an index on another shape, or none, leaves a Sort of the whole scope). A scope of a few
# thousand videos, analysed, so the planner weighs it as it does a real one.
user = _seed(db, make_user)
rng = random.Random(7)
db.execute(
insert(Video),
[
{
"id": f"bulk{i:05}",
"channel_id": "UCring",
"title": "b",
"duration_seconds": 600,
"published_at": NOW - timedelta(minutes=i),
"shuffle_key": rng.random(),
}
for i in range(4000)
],
)
db.commit()
db.execute(text("ANALYZE videos"))
query, _ = feedquery.filtered_query(db, user, **_FEED_BASE)
captured: list[tuple[str, object]] = []
def grab(conn, cursor, statement, params, *args):
if "FROM videos" in statement and "LIMIT" in statement:
captured.append((statement, params))
engine = db.get_bind()
event.listen(engine, "before_cursor_execute", grab)
try:
seed = _seed_landing_between(0.45, 0.55)
feedquery.shuffle_walk(db, query, seed, 6) # the first arc, from the seed's point
feedquery.shuffle_walk(db, query, seed, 6, (1, 0.3, "bulk00001")) # the wrap, resumed
finally:
event.remove(engine, "before_cursor_execute", grab)
assert len(captured) == 2
conn = db.connection()
for statement, params in captured:
plan = "\n".join(r[0] for r in conn.exec_driver_sql("EXPLAIN " + statement, params))
assert "ix_videos_shuffle_key" in plan, plan
assert "Sort Key: videos.shuffle_key" not in plan, plan
def test_the_surprise_reads_a_window_of_the_ring(db, make_user, monkeypatch):
user = _seed(db, make_user)
monkeypatch.setattr(queries, "SURPRISE_WINDOW", 3)
for seed in range(20):
drawn = {r["id"] for r in queries.yt_surprise(db, user, "any", seed, 3)}
assert drawn == set(_ring(seed)[:3]), seed
def test_a_wide_gap_on_the_ring_does_not_make_a_favourite(db, make_user):
# One video after an empty half of the ring, twenty packed into the other half. Taking the first
# video after the seed's point would deal the lonely one about half the time; the window is
# shuffled by the seed, so it leads about as often as any other.
keys = {"lonely": 0.5} | {f"p{i:02}": 0.51 + i * 0.02 for i in range(20)}
user = _seed(db, make_user, keys)
leads = [queries.yt_surprise(db, user, "any", s, 1)[0]["id"] for s in range(90)]
assert leads.count("lonely") < 15, leads.count("lonely")
def test_any_seed_lands_on_the_ring():
# An int too large for a float overflowed the multiplication: a 500 for a hand-typed seed.
for seed in (10**400, -(10**400), 2**53 + 1, -7):
assert 0 <= feedquery.shuffle_start(seed) < 1, seed
def test_a_seed_out_of_range_is_refused_not_a_500(db, make_user, as_user):
client = as_user(_seed(db, make_user))
huge = "9" * 400
for url in (
f"/api/feed?sort=shuffle&seed={huge}",
f"/api/discover?seed={huge}",
f"/api/discover/pick?seed={huge}",
):
assert client.get(url).status_code == 422, url
def test_a_shuffle_cursor_past_a_float_is_a_bad_cursor(db, make_user, as_user):
client = as_user(_seed(db, make_user))
for k in ("[Infinity, 0.5]", f"[0, {'9' * 400}]"):
raw = '{"k": %s, "id": "k20"}' % k
cursor = base64.urlsafe_b64encode(raw.encode()).decode()
assert client.get(f"/api/feed?sort=shuffle&seed=1&cursor={cursor}").status_code == 400, k
def test_a_new_video_gets_its_own_key(db, make_user):
_seed(db, make_user, {})
db.add_all([Video(id=f"n{i}", channel_id="UCring", title="n") for i in range(3)])
db.commit()
keys = [db.get(Video, f"n{i}").shuffle_key for i in range(3)]
assert len(set(keys)) == 3 and all(0 <= k < 1 for k in keys)
+349
View File
@@ -0,0 +1,349 @@
"""Surprise me leans on what the user watches, and stays a surprise.
The Home hero's surprise tier and the Plex Discover rail were uniform draws over the unseen. They are
now weighted draws (`app.draw`): a title's weight is 1 + its affinity — for Plex the genres of the
films and shows the user watched (recent ones weigh more) and the TMDB recommendations stored on them;
for YouTube the channels the user watches. One slot of three stays a blind draw, and with no watch
history the order is exactly the old one. DB lane.
"""
import hashlib
from datetime import datetime, timedelta, timezone
from sqlalchemy import select, text
from app import draw
from app.catalog.models import Channel, Subscription, Video
from app.discover import queries
from app.plex import affinity, discover as plex_discover
from app.plex.models import PlexItem, PlexLibrary, PlexShow, PlexState
from app.plex.query import LibraryFilters
from app.reader.models import VideoState
NOW = datetime.now(timezone.utc)
SEEDS = range(90)
# ── the draw itself ──────────────────────────────────────────────────────────────────────────────
def test_every_third_slot_is_blind_and_nothing_repeats():
weighted = ["a", "b", "c", "d", "e", "f"]
blind = ["f", "e", "d", "c", "b", "a"]
# offset 0: slots 0 and 3 are blind
assert draw.blend(weighted, blind, 0, 6) == ["f", "a", "b", "e", "c", "d"]
# offset 1: slot 2 and 5
assert draw.blend(weighted, blind, 1, 6) == ["a", "b", "f", "c", "d", "e"]
assert draw.blend(weighted, blind, 2, 3) == ["a", "f", "b"]
def test_a_short_side_is_topped_up_from_the_other():
assert draw.blend(["a"], ["a", "b", "c"], 0, 3) == ["a", "b", "c"]
assert draw.blend(["a", "b", "c"], [], 0, 3) == ["a", "b", "c"]
def test_a_watch_weighs_by_its_age_an_undated_one_two_half_lives_and_a_future_one_now():
# Both affinities weigh every watch through this one rule; an old import has no date, and a
# clock ahead of the server's must not weigh more than a watch made now.
assert draw.recency(NOW, NOW, 30) == 1.0
assert draw.recency(NOW - timedelta(days=30), NOW, 30) == 0.5
assert draw.recency(None, NOW, 30) == 0.25
assert draw.recency(NOW + timedelta(days=5), NOW, 30) == 1.0
def test_scores_are_scaled_to_the_strongest_and_nothing_scored_is_empty():
assert draw.scaled({"a": 4.0, "b": 1.0}) == {"a": 1.0, "b": 0.25}
assert draw.scaled({}) == {}
# ── YouTube: the channels the user watches ───────────────────────────────────────────────────────
def _md5_order(ids, seed):
return sorted(ids, key=lambda i: hashlib.md5(f"{i}{seed}".encode()).hexdigest())
def _seed_yt(db, make_user, *, watched_in_fav: int):
user = make_user()
fav, other = Channel(id="UCfav", title="Fav"), Channel(id="UCother", title="Other")
db.add_all([fav, other])
db.flush()
db.add_all(
[
Subscription(user_id=user.id, channel_id="UCfav"),
Subscription(user_id=user.id, channel_id="UCother"),
]
)
def video(vid, ch, days):
return Video(
id=vid,
channel_id=ch,
title=vid,
duration_seconds=20 * 60,
published_at=NOW - timedelta(days=days),
)
# One unseen video per channel, plus the user's watched history on the favourite.
db.add_all([video("fav_new", "UCfav", 1), video("other_new", "UCother", 1)])
for i in range(watched_in_fav):
db.add(video(f"fav_seen{i}", "UCfav", 10 + i))
db.flush()
for i in range(watched_in_fav):
db.add(
VideoState(
user_id=user.id,
video_id=f"fav_seen{i}",
status="watched",
watched_at=NOW - timedelta(days=i),
)
)
db.commit()
return user
def _first(rows):
return rows[0]["id"] if rows else None
def test_a_watched_channel_wins_most_draws_but_not_all(db, make_user):
user = _seed_yt(db, make_user, watched_in_fav=12)
leads = [_first(queries.yt_surprise(db, user, "any", s, 2)) for s in SEEDS]
fav = leads.count("fav_new") / len(leads)
# weight 1 + 3 against 1: 0.8 in a weighted slot, 0.5 in the blind one (a third of the seeds)
assert 0.6 < fav < 0.9, fav
def test_the_blind_slot_deals_the_plain_shuffle_however_strong_the_lean(db, make_user):
user = _seed_yt(db, make_user, watched_in_fav=12)
for s in range(0, 30, 3): # (0 + seed) % 3 == 0: the first slot is the blind one
got = _first(queries.yt_surprise(db, user, "any", s, 2))
assert got == _md5_order(["fav_new", "other_new"], s)[0]
def test_with_no_history_the_youtube_order_is_the_old_shuffle(db, make_user):
user = _seed_yt(db, make_user, watched_in_fav=0)
for s in range(12):
got = [r["id"] for r in queries.yt_surprise(db, user, "any", s, 2)]
assert got == _md5_order(["fav_new", "other_new"], s)
def test_a_watched_video_is_never_drawn(db, make_user):
user = _seed_yt(db, make_user, watched_in_fav=12)
drawn = {r["id"] for s in SEEDS for r in queries.yt_surprise(db, user, "any", s, 5)}
assert drawn == {"fav_new", "other_new"}
# ── Plex: genres and recommendations of what the user watched ────────────────────────────────────
def _seed_plex(db, make_user, *, history: bool):
user = make_user()
lib = PlexLibrary(plex_key="77", title="Movies", kind="movie", enabled=True)
db.add(lib)
db.flush()
tmdb_ids = iter(range(5000, 6000))
def movie(key, genres, recs=None):
m = PlexItem(
rating_key=key,
library_id=lib.id,
kind="movie",
title=key,
duration_s=100 * 60,
genres=genres,
guid=f"tmdb:{next(tmdb_ids)}",
recommendations=recs,
)
db.add(m)
return m
horror = movie("horror_new", ["Horror"])
comedy = movie("comedy_new", ["Comedy"])
tipped = movie("tipped_new", ["Western"])
db.flush()
tip = [{"tmdb_id": int(tipped.guid.split(":")[1]), "media_type": "movie", "title": "t"}]
seen = [movie(f"horror_seen{i}", ["Horror"], tip if i == 0 else None) for i in range(3)]
db.flush()
if history:
for i, m in enumerate(seen):
db.add(
PlexState(
user_id=user.id,
item_id=m.id,
status="watched",
watched_at=NOW - timedelta(days=i),
)
)
db.commit()
from app import sysconfig
sysconfig.set_value(db, "plex_enabled", True)
db.commit()
return user, horror, comedy, tipped
def test_the_genre_and_the_recommendation_of_what_was_watched_win_most_draws(db, make_user):
user, *_ = _seed_plex(db, make_user, history=True)
leads = [_first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en")) for s in SEEDS]
# Weights: Horror 1 + 2 (its genre), the recommended Western 1 + 3, the Comedy nobody chose 1. In
# a weighted slot the Comedy leads 1 time in 8, in the blind third 1 in 3: about 0.19 against
# 0.81 for the other two together (a uniform draw: 1 in 3 each). It still leads now and then —
# a surprise, not a ranking.
comedy = leads.count("comedy_new")
assert comedy > 0
assert leads.count("horror_new") > 1.5 * comedy and leads.count("tipped_new") > 1.5 * comedy
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * comedy
def test_with_no_history_the_plex_order_is_the_old_shuffle(db, make_user):
user, *_ = _seed_plex(db, make_user, history=False)
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
for s in range(12):
got = [c["id"] for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")]
assert got == _md5_order(keys, s)
def test_a_watched_film_is_never_drawn(db, make_user):
user, *_ = _seed_plex(db, make_user, history=True)
drawn = {
c["id"] for s in SEEDS for c in queries.plex_surprise(db, user.id, "any", s, 6, lang="en")
}
assert drawn == {"horror_new", "comedy_new", "tipped_new"}
def test_the_plex_blind_slot_deals_the_plain_shuffle(db, make_user):
user, *_ = _seed_plex(db, make_user, history=True)
unseen = ["horror_new", "comedy_new", "tipped_new"]
for s in range(0, 30, 3):
got = _first(queries.plex_surprise(db, user.id, "any", s, 3, lang="en"))
assert got == _md5_order(unseen, s)[0]
def test_a_binged_show_counts_once_and_recent_watches_weigh_more(db, make_user):
user = make_user()
lib = PlexLibrary(plex_key="78", title="Shows", kind="show", enabled=True)
mlib = PlexLibrary(plex_key="79", title="Movies", kind="movie", enabled=True)
db.add_all([lib, mlib])
db.flush()
show = PlexShow(rating_key="sitcom", library_id=lib.id, title="Sitcom", genres=["Comedy"])
db.add(show)
db.flush()
eps = [
PlexItem(
rating_key=f"ep{i}",
library_id=lib.id,
kind="episode",
title=f"e{i}",
show_id=show.id,
season_number=1,
episode_number=i,
)
for i in range(10)
]
films = [
PlexItem(
rating_key=f"h{i}", library_id=mlib.id, kind="movie", title=f"h{i}", genres=["Horror"]
)
for i in range(2)
]
old = PlexItem(
rating_key="w0", library_id=mlib.id, kind="movie", title="w0", genres=["Western"]
)
db.add_all([*eps, *films, old])
db.flush()
for it in [*eps, *films]:
db.add(PlexState(user_id=user.id, item_id=it.id, status="watched", watched_at=NOW))
db.add(
PlexState(
user_id=user.id, item_id=old.id, status="watched", watched_at=NOW - timedelta(days=720)
)
)
db.commit()
genres = affinity.for_user(db, user.id).genres
# Ten episodes of one show are one title: two horror films outweigh it.
assert genres["Horror"] == 1.0
assert genres["Comedy"] == 0.5
# Four half-lives ago weighs a sixteenth of today.
assert abs(genres["Western"] - 0.5 / 16) < 0.01
def test_the_discover_rail_leans_the_same_way(db, make_user):
# The rail draws with random(), not a seed; setseed makes this session's draws repeatable.
user, *_ = _seed_plex(db, make_user, history=True)
db.execute(text("SELECT setseed(0.42)"))
leads = [
plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"][0][
"id"
]
for _ in SEEDS
]
assert leads.count("comedy_new") > 0
assert leads.count("horror_new") + leads.count("tipped_new") > 3 * leads.count("comedy_new")
# ── whose history: only the user's own ───────────────────────────────────────────────────────────
def test_another_users_youtube_history_does_not_lean_my_draw(db, make_user):
watcher = _seed_yt(db, make_user, watched_in_fav=12)
me = make_user()
db.add_all(
[
Subscription(user_id=me.id, channel_id="UCfav"),
Subscription(user_id=me.id, channel_id="UCother"),
]
)
db.commit()
assert queries._channel_affinity(db, watcher)
assert queries._channel_affinity(db, me) == {}
def test_another_users_plex_history_does_not_lean_my_draw(db, make_user):
_seed_plex(db, make_user, history=True)
me = make_user()
assert not affinity.for_user(db, me.id)
keys = ["horror_new", "comedy_new", "tipped_new"] + [f"horror_seen{i}" for i in range(3)]
for s in range(12):
got = [c["id"] for c in queries.plex_surprise(db, me.id, "any", s, 6, lang="en")]
assert got == _md5_order(keys, s)
# ── the rail: its blind slot, and a show's own recommendations ───────────────────────────────────
def test_the_rails_third_slot_is_the_blind_draw(db, make_user, monkeypatch):
# The rail draws with random(), so the blend is watched rather than its odds: every leaning draw
# goes through it, with the blind order beside the weighted one and the third slot blind.
user, *_ = _seed_plex(db, make_user, history=True)
calls = []
real = draw.blend
def spy(weighted, blind, offset, n, key):
calls.append((offset, n, sorted(map(key, weighted)), sorted(map(key, blind))))
return real(weighted, blind, offset, n, key=key)
monkeypatch.setattr(draw, "blend", spy)
got = plex_discover.surprise(db, user, "movie", "en", 3, [], [], LibraryFilters())["items"]
assert calls and all(offset == 1 and n == 3 for offset, n, *_ in calls)
pool = sorted(["horror_new", "comedy_new", "tipped_new"])
assert calls[0][2] == pool and calls[0][3] == pool
assert len(got) == 3
def test_a_show_leans_on_the_shows_recommended_not_the_films(db, make_user):
lib = PlexLibrary(plex_key="80", title="Shows", kind="show", enabled=True)
db.add(lib)
db.flush()
tipped = PlexShow(rating_key="tv1", library_id=lib.id, title="t", guid="tmdb:1", genres=[])
db.add(tipped)
db.commit()
shows = affinity.Affinity(show_recs={"tmdb:1": 1.0})
films = affinity.Affinity(movie_recs={"tmdb:1": 1.0})
def weight(aff):
return db.execute(select(aff.weight(PlexShow)).where(PlexShow.id == tipped.id)).scalar_one()
assert weight(shows) == 1 + affinity.REC_WEIGHT
assert weight(films) == 1
+5 -7
View File
@@ -292,17 +292,16 @@ def test_a_known_video_is_marked_watched_at_its_takeout_time_and_the_rest_wait_o
assert (states["older000000"].status, states["older000000"].position_seconds) == ("watched", 40)
@pytest.mark.parametrize("undo", ["unmark", "clear_state"])
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user, undo):
"""Un-marking (or resetting) a video leaves no state row, and the import inserted a watched one
over the user's own choice. A view on YouTube after the un-mark is a new view, and counts."""
user = make_user(f"{undo}@example.com")
def test_a_video_the_user_un_marked_here_is_not_marked_again_by_an_older_view(db, make_user):
"""Un-marking a video leaves no state row, and the import inserted a watched one over the user's
own choice. A view on YouTube after the un-mark is a new view, and counts."""
user = make_user("unmark@example.com")
for vid in ("undone00000", "again000000"):
_video(db, vid)
db.commit()
for vid in ("undone00000", "again000000"):
service.mark(db, user, vid, "watched")
getattr(service, undo)(db, user, vid)
service.unmark(db, user, vid)
db.commit()
assert _states(db, user) == {}
later = datetime.now(timezone.utc) + timedelta(minutes=5)
@@ -603,7 +602,6 @@ def test_staging_and_settling_take_the_users_one_lock(db, make_user):
),
# Between the import's read of the un-marks and its marking, one would be overwritten.
"unmark": lambda s, u: service.unmark(s, u, "aaaaaaaaaaa"),
"reset": lambda s, u: service.clear_state(s, u, "aaaaaaaaaaa"),
}
with SessionLocal() as holder:
service.lock_takeout(holder, holder.get(User, user.id))
+2 -1
View File
@@ -579,7 +579,7 @@ def _display_write_sites() -> dict[tuple[str, str], set[str]]:
# enqueue_clip slices the source's title to the same width.
# files.ingest_spooled passes through `spool_upload`'s value, bounded there.
# routes enqueue_clip the `display_name=` keyword passes on the name bound above it.
# pipeline._download / _process_clip / worker._finish_from_cache
# pipeline._download / _process_clip / worker._finish_from_cache / backfill._retitle
# slice a fetched or stored title to the column's width.
#
# Held as EXACT EQUALITY rather than as a list of offenders. The offender form could only fail
@@ -596,6 +596,7 @@ DISPLAY_WRITE_SITES = {
("downloads/service.py", "enqueue_clip"): {"display_name or asset.title[:DISPLAY_TEXT_MAX]"},
("downloads/pipeline.py", "_download"): {"meta.title[:DISPLAY_TEXT_MAX]"},
("downloads/pipeline.py", "_process_clip"): {"named[:DISPLAY_TEXT_MAX]"},
("downloads/backfill.py", "_retitle"): {"named[:DISPLAY_TEXT_MAX]"},
("worker.py", "_finish_from_cache"): {"asset.title[:DISPLAY_TEXT_MAX]"},
("downloads/upload.py", "ingest_upload"): {"display_text(display_name)"},
("routes/auth.py", "callback"): {"display_text(userinfo.get('name'))"},
+218
View File
@@ -0,0 +1,218 @@
"""A YouTube length past a day is unknown, everywhere a length is used (`catalog.models.LENGTH_MAX_S`).
Measured on prod (2026-11-06): two 24/7 streams' archives report 122,962,914 s and 34,762,459 s, and the
owner's Watch later total read "44,073 hr 55 min". Such a video now has no length on any API answer, the
totals leave it out (a playlist's counts it as `without_length`), and the duration filter and sort pass it over
like a video whose length is not fetched yet. An AST check holds who may still read the raw column.
DB lane (the AST check needs none).
"""
import ast
from datetime import datetime, timezone
from pathlib import Path
from sqlalchemy.dialects import postgresql
from app.catalog.models import LENGTH_MAX_S, Channel, Subscription, Video, known_length
from app.reader.models import Playlist, PlaylistItem
YEARS = 122_962_914 # the measured archive
HOUR = 3600
def _seed(db, user):
db.add(Channel(id="UClen", title="Len"))
db.flush()
db.add(Subscription(user_id=user.id, channel_id="UClen", priority=0, hidden=False))
rows = [
("short", 600, "none"),
("long", 2 * HOUR, "none"),
("day", LENGTH_MAX_S, "was_live"), # exactly a day still is a length
("archive", YEARS, "was_live"),
("onair", None, "live"),
("unfetched", None, "none"), # not live, but left out of the sum: counted all the same
]
for i, (vid, seconds, live) in enumerate(rows):
db.add(
Video(
id=vid,
channel_id="UClen",
published_at=datetime(2024, 1, i + 1, tzinfo=timezone.utc),
title=vid,
duration_seconds=seconds,
live_status=live,
is_short=False,
via_search=False,
via_explore=False,
)
)
pl = Playlist(user_id=user.id, name="Later")
db.add(pl)
db.flush()
for i, (vid, _s, _l) in enumerate(rows):
db.add(PlaylistItem(playlist_id=pl.id, video_id=vid, position=i))
db.commit()
return pl
KNOWN_SUM = 600 + 2 * HOUR + LENGTH_MAX_S
def test_the_rule():
assert known_length(None) is None
assert known_length(LENGTH_MAX_S) == LENGTH_MAX_S
assert known_length(LENGTH_MAX_S + 1) is None
assert known_length(YEARS) is None
def test_the_sql_length_carries_its_limit_as_a_literal():
# The duration sort indexes (migration 0117) are built on this expression with 86400 in it. A
# bound parameter matches no index once the driver reuses a prepared, generic plan, and the
# sort-index test cannot see that: it compiles with literal binds.
compiled = Video.length_seconds.expression.compile(dialect=postgresql.dialect())
assert compiled.params == {}
assert f"> {LENGTH_MAX_S}" in str(compiled)
def test_a_years_long_archive_has_no_length_in_the_feed_or_a_playlist(db, make_user, as_user):
user = make_user()
pl = _seed(db, user)
client = as_user(user)
feed = client.get("/api/feed?show=all&include_live=true&limit=50")
assert feed.status_code == 200
by_id = {v["id"]: v["duration_seconds"] for v in feed.json()["items"]}
assert by_id["archive"] is None
assert by_id["day"] == LENGTH_MAX_S
detail = client.get(f"/api/playlists/{pl.id}")
assert detail.status_code == 200
items = {v["id"]: v["duration_seconds"] for v in detail.json()["items"]}
assert items["archive"] is None
one = client.get("/api/videos/archive")
assert one.status_code == 200
assert one.json()["duration_seconds"] is None
def test_an_off_catalog_archive_has_no_length_either(db, make_user, as_user, monkeypatch):
# A video not in the catalog is looked up on YouTube (videos.list); its length is judged the same.
from app.routes import feed as feed_routes
class Yt:
def __init__(self, *_):
pass
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def get_videos(self, ids):
hours = {"offyears": YEARS // HOUR, "offshort": 1}[ids[0]]
return [
{"snippet": {}, "statistics": {}, "contentDetails": {"duration": f"PT{hours}H"}}
]
monkeypatch.setattr(feed_routes, "YouTubeClient", Yt)
client = as_user(make_user())
assert client.get("/api/videos/offyears").json()["duration_seconds"] is None
assert client.get("/api/videos/offshort").json()["duration_seconds"] == HOUR
def test_the_totals_leave_it_out_and_count_what_they_left_out(db, make_user, as_user):
user = make_user()
pl = _seed(db, user)
client = as_user(user)
listed = client.get("/api/playlists")
assert listed.status_code == 200
(summary,) = [p for p in listed.json() if p["id"] == pl.id]
assert summary["total_duration_seconds"] == KNOWN_SUM
assert summary["without_length"] == 3 # the archive, the one on air, the unfetched one
renamed = client.patch(f"/api/playlists/{pl.id}", json={"name": "Later 2"})
assert renamed.status_code == 200
assert renamed.json()["total_duration_seconds"] == KNOWN_SUM
assert renamed.json()["without_length"] == 3
channels = client.get("/api/channels")
assert channels.status_code == 200
(ch,) = channels.json()
assert ch["total_duration_seconds"] == KNOWN_SUM
# The channel table shows the sum alone (the count did not fit its column), so it is not sent.
assert "without_length" not in ch and "live_without_length" not in ch
def test_the_duration_filter_and_sort_pass_it_over(db, make_user, as_user):
user = make_user()
_seed(db, user)
client = as_user(user)
base = "/api/feed?show=all&include_live=true&limit=50"
longer = client.get(f"{base}&min_duration={HOUR}")
assert longer.status_code == 200
assert {v["id"] for v in longer.json()["items"]} == {"long", "day"}
longest = client.get(f"{base}&sort=duration_desc")
assert longest.status_code == 200
order = [v["id"] for v in longest.json()["items"]]
assert order[:3] == ["day", "long", "short"]
assert set(order[3:]) == {"archive", "onair", "unfetched"}
# Who may read `duration_seconds` itself, by (file, function): what each one asks is not "how long".
RAW_READERS = {
# The column is defined and turned into a length here.
("catalog/models.py", "length_seconds"),
("catalog/models.py", "_length_seconds_expression"),
# Ingest writes it; the live refresh asks whether YouTube has answered for an archive yet; the
# Shorts classification needs only "longer than a Short", which a years-long value proves.
("sync/videos.py", "apply_video_details"),
("sync/videos.py", "refresh_live"),
("sync/videos.py", "run_shorts_classification"),
("catalog/search.py", "classify_shorts"),
("catalog/queries.py", "work_queue"),
# Not a video's column: the player's own report, and the watch history's looked-up lengths, which
# count a watched video as more than a glance (a watched archive is that too).
("routes/_common.py", "kept_position"),
("routes/plex/playback.py", "item_progress"),
("reader/queries.py", "_counted_channels"),
("reader/queries.py", "watched_channels"),
("reader/queries.py", "watched_channel_videos"),
}
def _raw_reads() -> set[tuple[str, str]]:
root = Path(__file__).resolve().parents[1] / "app"
found: set[tuple[str, str]] = set()
for path in root.rglob("*.py"):
if "alembic" in path.parts:
continue
rel = path.relative_to(root).as_posix()
tree = ast.parse(path.read_text(encoding="utf-8"))
in_a_function: set[int] = set()
for fn in ast.walk(tree):
if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
for node in ast.walk(fn):
if _reads_raw(node):
found.add((rel, fn.name))
in_a_function.add(id(node))
# A read outside every def (module or class level, a lambda there) is named `<module>`.
if any(_reads_raw(n) and id(n) not in in_a_function for n in ast.walk(tree)):
found.add((rel, "<module>"))
return found
def _reads_raw(node: ast.AST) -> bool:
return isinstance(node, ast.Attribute) and node.attr == "duration_seconds"
def test_only_the_listed_readers_take_the_raw_column():
found = _raw_reads()
stray = sorted(found - RAW_READERS)
assert not stray, f"read `length_seconds` (a length) instead of the raw column: {stray}"
# A listed reader that stopped reading it leaves the list.
assert not RAW_READERS - found, sorted(RAW_READERS - found)
+2 -2
View File
@@ -216,7 +216,7 @@ def test_a_line_with_no_url_is_left_alone(sink):
def test_the_download_options_carry_the_logger():
opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None)
opts = formats.build_ydl_opts({}, "%(id)s.%(ext)s", lambda d: None, egress=False)
assert opts["logger"] is formats._YDL_LOG
@@ -227,7 +227,7 @@ def test_a_source_adapter_cannot_silence_it():
adapter must not be able to take away the one account of what went wrong that a failed job
leaves behind."""
opts = formats.build_ydl_opts(
{}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}
{}, "%(id)s.%(ext)s", lambda d: None, extra={"logger": None, "quiet": False}, egress=False
)
assert opts["logger"] is formats._YDL_LOG
+188 -134
View File
@@ -256,7 +256,7 @@ Also in force:
- **A glyph means one thing.** Play (▷) starts playback, or resumes a paused transfer; a retry is the
refresh arrow (`RefreshCw`), and so is doing a download again; a reset or revert is the
counter-clockwise arrow (`RotateCcw`), and so is playing from the beginning, on the card and in its
menu alike; clearing a video's progress and status is the eraser (`Eraser`). A failed download's Retry
menu alike; clearing a video's watch progress is the eraser (`Eraser`). A failed download's Retry
had worn the Play glyph, which promised a video. The skip arrows (⏮ ⏭) are Previous /
Next; the auto-advance mode beside them wears list glyphs (`ListEnd` next, `ListStart` previous,
`Shuffle` random; off is `ListEnd`, not lit) — a mode in the skip arrows put two identical ⏭ on the
@@ -301,7 +301,8 @@ Also in force:
`fade-below` utility (`fade-below-panel` / `fade-above-panel` on a panel); the page band and a
scrolling panel under its header fade by the scroller's own edge mask (`useScrollFade`), because what
they sit on is the backdrop, not a colour. The chip rows fold their overflow into "+N" rather than
scroll, so no row has clipped ends; the carousels fade theirs. The dialog edges come with C4's pinned
scroll, so no row has clipped ends, and a filter row puts its picked chips first so the fold never
holds the only sign that a filter is on; the carousels fade theirs. The dialog edges come with C4's pinned
footer (S3f).
- **Viewport height:** `dvh`, never `vh` / `h-screen`: `100vh` is the viewport with a mobile browser's
toolbar retracted, so an `h-screen` shell runs its last rows under the toolbar (contract-tested).
@@ -378,7 +379,11 @@ Which actions are visible and which live in a menu.
managing the shares, re-download, the copies, cancel / delete / remove
behind their confirms); a link inside a row keeps the browser's own right-click menu. A notification
row keeps its link and its unread mark, shows its dismiss on hover (in the "…" on touch), and its
"…" holds find / undo for an activity notice. The notification center's band keeps Mark all read
"…" holds find / undo for an activity notice. The notification center lists what asks for the
reader first ("Needs you · n": an unread row, a notice waiting on an action — one read during the
visit keeps its place there until Mark all read), then one stream of both sources, newest first,
and folds read rows older than a week under "Earlier (n)", a heading-style disclosure, closed — Back to a row
that folded away since opens it on that row. Its band keeps Mark all read
and puts Clear everything, now behind a confirm, in its "…"; the Scheduler's strip keeps Run all
and Pause and puts Purge discovery in its "…". A list whose row objects are rebuilt each render or replaced by a
poll passes `useContextMenu({ keyOf })` (its id), so the "…" still closes its own menu. A menu's
@@ -532,6 +537,26 @@ of a full-width panel and nothing on the right.
shows them as four buttons with the item's place beside the selected view; a side panel in edit mode
gives each group an Up / Down pair (`ui/OrderButtons`, `compact`). A move that would go nowhere is
shown off, not hidden. A grip also picks up from the keyboard (space, the arrows, space).
- **Selecting rows:** a list whose rows take bulk actions shows a box first in every row, always, and one
over the column for the rows shown. Shift+click selects a range, `x` toggles the row under the keyboard,
Esc clears. A selected row wears the accent tint as well as its tick. An action covers the selected rows
shown; a row a filter hides stays selected and comes back ticked; another list starts over. The row's own
menu still acts on that row alone. While anything is selected the toolbar's second row is the action band:
the count (and how many a filter hides), the last move's target as one click, Move to, Remove, Clear
selection. An action clears the selection, undoes by its toast and, when it fails, ticks its rows again.
An Undo brings back what is still there to bring: a video the move put in the target and the target no
longer holds stays where it went, and a warning says how many came back. One the target held already only
left the source, so it returns there.
Delete takes the selection when there is one, else the row under the keyboard, with no confirm. With every
tick hidden by a filter it removes nothing and says so. An Esc during a drag puts the drag down, and only that.
Below `xl` the band's buttons are glyphs named in their tooltips, and a filter's hidden count is "+N" with
its sentence in the tooltip. The row the band replaces keeps its height, invisible, so a tick moves no row.
On a phone the count is its number (the words read out, and in the tooltip) and the undo pair stays out.
A row's grip also carries it onto a list in the rail, under a sort or a filter too (the grip then says
so, and no key lifts it); a selected row carries every selected one, named on the drag image below the
pointer. The target row lights up (`ring-accent`); the open list is none; an unpinned rail peeks by the
drag's own hover and scrolls at its edges. Dropped inside the list, one row takes its new place and many
stay. A key never leaves the list and moves its own row: the band and the row's menu are the other way.
- **Motion:** 150–300ms micro-interactions, transform/opacity only; `prefers-reduced-motion` is respected.
[WCAG 2.3.3]
@@ -543,6 +568,9 @@ of a full-width panel and nothing on the right.
the first tap however long the video played).
- **The bars stay up while the pointer rests on them** (F22, owner); leaving starts a fresh 3 s. YouTube and
Plex Web do the same: before, the row faded out from under a mouse parked on Next.
- **Text over the picture sits on a dark panel** (2026-11-07): loading, stalls, errors and the notices are
readable on any frame. A stall that lasts 5 s on a link measured slower than the stream says "Slow
connection" with both rates.
- **Volume is the native `Slider`** (A6); the hand-drawn bar had no hover, focus, keyboard or screen-reader
state. A pointer drag gives the focus back, so Space and the arrows keep driving the player.
- **The seek bar:** the track stays thin inside a 2.75rem (44px) hit area with `touch-action: none`, so a finger
@@ -693,134 +721,160 @@ of a full-width panel and nothing on the right.
## Decision log
| Date | Decision | Chosen | Over | Why |
| ---------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
| 2026-10-24 | Related videos (owner U1–U3 ★, D1 A, D3) | On demand: a tab beside the description (wide), a drawer (stacked); the narrow thumbnailed row, its menu alone, a followed check; plays in place and the list stays the opened video's; an on-topic / Hungarian-trending switch kept on the account | Always shown; the end-of-video grid alone; a dialog; a big card or a text row; a list that follows the playing video | The modal is full on a small screen; a picture is what a choice among strangers rests on; a list that stays put is tried one by one, and spends no more of the day's cap |
| 2026-10-25 | Related: one played; off (owner D1 a, D2 a) | Played from the list, it is the player's item: its title, controls and progress, marked watched at its end, the queue left where it was; a daily limit of 0 takes the tab and the button away | A linked video's in-place play (no controls, nothing kept); a tab that only shows a refusal | What is watched should count as watched, wherever it was started from; a control that can only be refused is not offered |
| 2026-10-25 | Player panel glyphs (owner 1A, 2A, 3) | In the side panel the queue row is glyph-only on one line, and Mark watched is a glyph toggle among the actions (✓ / ✓✓); the docked bar keeps its labels; the related switch at the default size, "Trending" | Labelled buttons wrapping onto three lines; glyphs on the docked bar too; Mark watched on its own line | The panel is narrow and the list controls are the same few every time; each glyph keeps its full name and shortcut in its tooltip; the docked bar has the width for words |
| 2026-10-25 | Auto-advance glyphs (owner E A, H a) | Auto-advance wears list glyphs (`ListEnd` next and off, `ListStart` previous, `Shuffle` random), in the panel and the docked bar; the auto-advance and loop cycles are `lit`, not `pressed` | The skip arrows for the mode; an X'd list for off; `aria-pressed` on a cycle | Beside Previous / Next the skip arrows read as a second Next; a cycle of three or four modes is no two-state switch, and its name already says which |
| 2026-10-25 | Restart and Reset glyphs (owner L A) | A menu's Play from the beginning wears `RotateCcw`, as the card's own Restart does; Reset (clear progress and status) wears `Eraser`; Redownload wears `RefreshCw`, the retry arrow | The skip arrow ⏮ on the menu's Restart; `RotateCcw` on both Restart and Reset; `TimerReset` for Restart | ⏮ is Previous beside the player; one action wore two glyphs, and two actions in one menu would have worn one |
| Date | Decision | Chosen | Over | Why |
| ---------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-24 | Floating button | Back to top: a round 48 glass FAB; "round" = floats over something (media or the page) | A Material 3 rounded square; a labelled pill; the old 64 circle | The common web convention for back to top; round tells it apart from every toolbar button; 64 was oversized |
| 2026-09-24 | Device contract | Tablet portrait + landscape and up; phones out | A 1024px floor with a "too small" message | Portrait tablet use is real, and the app already renders at 820 portrait |
| 2026-09-24 | Card actions on touch | Fewer visible, the rest in a "…" menu | 44px buttons that no longer fit the card | Six 44px targets need more width than a card has |
| 2026-09-24 | Action placement | One menu with several triggers, never gesture-only, a desktop background menu that mirrors the toolbar | Visible buttons for everything | Fewer non-critical buttons on screen, without hiding any function |
| 2026-09-24 | Text selection | Text stays selectable; a click that ends a selection does nothing | Click-to-play on titles | The user must be able to copy any text the app shows |
| 2026-09-24 | A1 Button shape | r8 + a rung per context; glass is a Button variant | r12; a pill | Nested corners (r8 in r12 in r16); the pill already means "a filter or a choice" |
| 2026-09-24 | A2 Icon buttons | Two sizes, r8; a round button only on media | One size; all round | A lone toolbar icon at the row size looks lost; round rows read as tags |
| 2026-09-24 | A3 Touch size | 44 for every control primitive | 36 + gaps; Button only | One rule (HIG 44) instead of two sizes nobody remembers |
| 2026-09-24 | A4 Input | Bordered, a 55% fg-into-bg edge (≥ 3:1) | Filled; today's look as a primitive | The only option that clears WCAG 1.4.11 without a heavy fill on glass |
| 2026-09-24 | A5 Dropdown trigger | Pill in a toolbar, r8 in a form | Pill everywhere; r8 everywhere | Matches the neighbours in each context |
| 2026-09-24 | A6 Checkbox, switch, slider | Native, accent-tinted, the row is the target | Drawn primitives | Free keyboard / screen-reader / forced-colours support; only 3 sites |
| 2026-09-24 | B1 Density | Compact 32 / 28 / 24 by explicit height | Standard 38 / 32 / 26 | The owner's read of the lab; touch is covered separately (A3), and one toolbar row lines up at 28 |
| 2026-09-24 | B2 Scheme set | Midnight, Forest→cyan, Slate→fuchsia, Matrix→lime | Today's six | The accent collided with danger (YouTube), warn (Slate) and success (Forest, Matrix); Midnight and Starship were one scheme twice; 4 of 6 light accents failed 4.5:1 |
| 2026-09-24 | B3 State layers | Text colour at 10% / 16% over anything | Material's 8 / 12%; today's opacity + bg-card | Visible in every scheme and mode (≥ 1.2:1); today's row hover was 1.00:1 on a card |
| 2026-09-24 | B4 Page chrome | One sticky band + fade; filters fold when short | A floating glass bar; everything scrolls | Tabs and filters always in reach, in one place; a keyboard-up tablet keeps its content |
| 2026-09-24 | B5 Surfaces | One class + radius per role | Today's per-site mix | Nested corners and no per-site alpha drift |
| 2026-09-24 | B6 /watch | The app's surfaces, native video controls | Flat and minimal | The page strangers see should look like the product; native controls work on any device |
| 2026-09-24 | B7 Nav rail | Soft tint + edge bar, optional labels (a user setting), slim | Solid accent (recommended); a Material rail | The owner's combination: calmer than a solid fill, labels for who wants them, no width cost |
| 2026-09-24 | B8 Loading | Skeleton by shape, spinner in a control | A spinner everywhere | Nothing jumps when data lands; a long wait doesn't look stuck |
| 2026-09-24 | B9 Fade | Toolbars / headers, panel tabs, dialog edges, chip rows | Thumbnails; buttons | The north-star depth where content scrolls under chrome; not on content or small controls |
| 2026-09-24 | C1 Action classification | The table (lab C1): state + primary at rest, everyday toggles on hover, "…" on touch, destructive into the menu | Visible buttons per site, as today | One action list per object, nothing gesture-only, fewer buttons without losing a function |
| 2026-09-24 | C2 Side panels | A tap opens an overlay drawer; pin is a separate button ≥ 1024 | A tap pins and the content reflows | Below 768 the filters were unreachable; on touch a tap pinned and pushed the grid |
| 2026-09-24 | C3 Players on touch | Tap = show the controls; 44px seek hit area; 44 buttons; the YT description behind ⓘ | Today (tap also pauses; 7.5px seek bar) | Measured live: you couldn't reach the seek bar without pausing, and the hover-only description never opened on a tap |
| 2026-09-24 | C4 Dialogs | Focus by purpose; a pinned footer | Focus on Close; the footer scrolls | A form starts in its first field, and Save stays on screen with the keyboard up |
| 2026-09-24 | C5 Plex player controls | The owner's lean set: no Stop / Restart / ±10 s buttons, Back by position, Next only with a next item, Resume + From the start on the card, touch double-tap ±10 s | Every control in one row (it ran off-screen below 1024); Back by a press counter; ±10 s keyboard-only | The row fits a portrait tablet; the decision to restart is made on the card; a tablet keeps the 10 s skip |
| 2026-09-24 | Glass tuner | Retired; the schemes are the only colour choice | Keep it with ranges clamped to the rules | Free tuning is the drift the constitution exists to stop (owner) |
| 2026-09-24 | Matrix toned down | Moss #86a05a at Midnight's lightness, near-neutral greys; every white-on-accent part → `--accent-fg` | The B2 lime #a3e635 | The owner found the lime neon and tiring; it was the loudest accent (L 0.85, C 0.21), and fixed white parts sat on it at 1.5:1 |
| 2026-09-24 | Field primitives | `Input` / `Textarea` / `InnerInput` + `fieldClass`, `SelectTrigger`, the native controls; the lint rule "no raw `<input>`" | `inputCls` and ~40 hand-styled boxes | Six edges, four radii and two focus looks for one control; the ≥ 3:1 edge held nowhere |
| 2026-09-25 | Surface roles | `SURFACE` roles in `ui/surfaces.ts`; the lint rule "no bare glass class"; menus p-1 with r8 rows | Three tiers paired ad hoc with five radii and three menu paddings | ~140 sites drifted; nested corners only hold if the radius comes with the role |
| 2026-09-25 | Static labels | Flat tonal `Badge`, no edge; a filtering tag is an edged chip | A glass pill with an edge, like the chips | Edged = pressable at a glance; a static tag that looks like a chip gets clicked and does nothing |
| 2026-09-25 | Header search | A 40 glass pill with the field edge (`lg`); what sits inside it is round | 44, r16, a glass box with a faint edge | The one rounded box among pills, and its edge under 3:1; the hero search is the YouTube/Gmail pill |
| 2026-09-25 | Switch row (F12) | The whole settings row toggles (`SwitchRow`; a hand-laid row points a label at the switch) | Only the 36×24 switch toggles | The iOS / Material settings row; the text was a dead click beside the control it names |
| 2026-09-25 | Edges and hovers (F14) | The ≥ 3:1 edge on the field family (fields, the form trigger); labelled pills keep the faint edge and the accent-edge hover | The ≥ 3:1 edge and a state-layer hover on every control | 1.4.11 asks an edge only where nothing else identifies the control; a heavy edge on every chip and tab weighs the whole UI down |
| 2026-09-25 | Rating and live (F16) | Own tokens: `--rating` (theme-scoped gold), `--live`/`--live-fg`, `--rating-media`, `--media-chrome`; palette classes lint-refused | Map them to `--warn` / `--danger` | A gold star and a red LIVE are what the eye expects; as warn/danger they would read as a status. The old amber score was 2:1 on light |
| 2026-09-25 | Fjord trial (B2 exception) | A fifth scheme, temporary: Midnight's neutrals, Playnite's steel accent (#337599 lifted to #5aa1cb) and steel rules; exit = fold into Midnight or remove | Retune Midnight straight away; borrow only the qualities | The owner wants to live with the calmer palette app-wide before it replaces the default; it breaks "own hue family" on purpose, and only until that call |
| 2026-09-25 | Whole-card row (owner) | A card row's B3 layer is the card's `::before` under its content; the open button's hit layer spans the card, the actions sit above it | The name button's own hover, ending at the first badge or button | A row that half-answers the pointer reads as broken; a badge click did nothing |
| 2026-09-25 | Badge tones (F29) | `Badge tone` neutral / accent / warn at 6%; light `--warn` #b45309 → #a14a07; edged label spans lint-refused | Keep edged status pills (a private Badge in Users, Settings "Granted", the suggestions count) | Edged = pressable at a glance; amber-700 was 4.58:1 as text and fell under 4.5 on its own tint |
| 2026-09-26 | Page band built (B4) | Controls only in the band (intros scroll); the fold shows the same row in place; one `h1` per page (`PageHeading` steps the hidden one aside); `SectionLabel` headings; Skip to content | A popover of the folded filters; the page title kept visible | A folded row is still the same controls, nothing duplicated; the rail and the tab title already name the page |
| 2026-09-26 | Fade built (B9) | `fade-below` on sticky toolbars and table headers; the scroller mask where the band sits on the backdrop | A painted band over the backdrop | The backdrop is the colour under the band, so fading the content into it is the same edge |
| 2026-09-26 | Viewport units | `dvh` everywhere (the Plex player stage with S3f) | `vh` / `h-screen` | `100vh` hides the last rows under a tablet browser's toolbar |
| 2026-09-26 | Video actions built (C1) | One list per video (menu, right-click, long-press, "…"); the card keeps watched / save / add to playlist on its picture on hover; download, details, copy, hide, reset in the menu | The six-button row under the card and the hover ⓘ | The menu is the one door every device has; the card loses a line of height and the touch card its row of 28px buttons |
| 2026-09-26 | Plex actions built (C1) | One list per Plex title (play/open, watched, add to playlist, media info, copy title, pin); the poster keeps watched / add / "…" on its picture on hover; seasons get open / watched / add / copy | The hover ⓘ, the pin-only menu, an episode's add button at rest, a collection's "Remove" X on the card | The same door as a video's; the collection grid goes to two columns on a phone so the touch "…" leaves the title its line |
| 2026-09-26 | Playlist actions built (C1) | A playlist row's "…" (play, pin, copy title / link, move, order, remove), Delete + an undo toast; one list per playlist (rename, pin, YouTube open / push / revert, delete) from its rail row and its header's "…" | The row's X column and its confirm; the header's pencil, YouTube link, push, revert and trash buttons | Remove is reversible, so it needs no confirm; the header keeps Play all; a narrow table drops "#" and Channel so the title keeps a width |
| 2026-09-26 | Channel + Plex list actions (C1) | A channel row's "…" (open, pin, YouTube, copy, full history, hide + undo, unsubscribe); a Plex playlist's title / show / season "…", Delete + undo; its header's "…" (rename, delete); an action toast lasts ≥ 8 s | The channel row's full-history, hide and unsubscribe buttons; the Plex list's red X on every row, show and season; the header's pencil and trash | Hiding and removing are reversible, so they need no confirm and the undo needs time; unsubscribe and deleting a playlist keep theirs |
| 2026-09-26 | Download / notification / Scheduler actions (C1) | A download row's "…" (play, share, save, edit, details, re-download, manage sharing, copies, cancel / delete / remove); a library row's Share stays in sight, ahead of save; a notification row's "…" and hover dismiss; the inbox band's and the Scheduler strip's "…" | The download row's five rare buttons and its cancel / delete / remove; the source link's copy button; an activity notice's Find / Undo links; the band's Clear everything; the strip's Purge discovery | The destructive rows keep their confirm, and Clear everything gains one; a row link keeps the browser's own menu; Find in feed drops the tag / search / age filters that could hide its video |
| 2026-09-26 | Channel menu: Tags + Priority (C1) | Tags A–Z as in-place checks; Priority as fixed steps 10 · 5 · 3 · 2 · 1 · 0, the current one ticked; a pick re-sorts at once and the page + focus follow the row; one drawn `MenuCheck` for every checkable row that cannot be an `<input>` | Relative steps (+1 / +5); straight to the top; the stepper only; seven hand-drawn check marks (three box sizes, a tinted box, a × for "on", a trailing tick) | Bubbling a channel up is one pick; fixed levels keep the scale the reader already uses; the list moving where the reader can see it is the feedback |
| 2026-09-26 | Reorder without a drag (WCAG 2.5.7) | Row menus: Top / Up / Down / Bottom (YouTube + Plex playlists); saved views: four buttons with the place beside the selected view (owner: A); side panels in edit mode: an Up / Down pair per group (owner: A), the name wrapping there | A "…" on every saved-view row; a "…" on every panel group; drag only | A visible, one-click path where the list is edited; the edge moves shown off so the buttons never shift |
| 2026-09-26 | Titles in click surfaces | The surface stays a button, flagged `selectable`: its text selects, and the click that ends a selection does nothing; every plain `Pressable` pinned with its reason | Rebuilding each card and row so the title sits outside the button; exceptions for master-detail lists | Chrome selects inside a button once it is `user-select: text` (measured), so one flag on the primitive keeps the keyboard, focus and list walking every surface already has |
| 2026-09-26 | Right-click on a selection | The browser's own menu opens on selected words; beside them the object's menu | A "Copy selection" row in the app menu | The reader selected the words to do something with them; the native menu copies, searches and translates, and changes nothing else |
| 2026-09-26 | Keyboard ring on a flush row | Inside the row (a table row by its tag, a log line, a show header in its clipping card, with the card's corners) | The ring outside, as on a card | Outside it ran under the table head and past the table, and a clipping card cut it away entirely (invisible focus) |
| 2026-09-26 | Fade at rest | The sticky header's fade appears with the first 1rem of scroll (a scroll-driven animation; no fade where unsupported) | A fade that is always drawn | At rest it dimmed the first row and hid its ring's top edge; nothing is under the header yet |
| 2026-09-26 | Narrow playlist table | "#" and Channel hide by the table's own width (under 42rem), not the window's | Hiding below the `sm` window width | Beside the playlist rail a desktop window left the table 432 px and the title 0 px |
| 2026-09-26 | Background menu built | Five list pages (feed, Plex grid, channels, playlist, notifications); the toolbar's sort / view / filters and the band's actions; no destructive row | Every page with a toolbar; the "…" menus copied whole, delete included | The tab-only pages have nothing a menu would add; a blank-area right-click must not be one click from delete |
| 2026-09-26 | Media card anatomy (C) | One frame; the meta line pinned to its bottom; the length a badge on the picture; the Plex card on the same primitive as the video card | A: the year as an overline over the title; B: the year inline after it; a fixed two-line title slot | A one-line title left a hole the eye reads as a missing line; C puts nothing under a variable-height title |
| 2026-09-26 | One Play | `MediaPlayGlyph` (the media button variant) and only where the click plays | The Plex card's hand-painted accent circle; a playable dot on every card | One Play for every card; a dot that only a colour and a tooltip explain says nothing |
| 2026-09-26 | Filtering values on the info page | A value that filters (year, certificate, studio, director) is an edged chip; a fact is flat text | Everything flat, the clickable ones only hover-tinted | The S3d rule: edged = pressable, so the reader sees which values do something |
| 2026-09-26 | Shelf density | On any strip: one title line over one meta line | The grid's two lines and full meta on a strip | The tallest card set the strip's height and left holes in the cards with less to say |
| 2026-09-26 | Compact list row (B) | The title leads; a faint "…" at the end; the toggles float over the date on hover | A leading 128px action column | That column sat empty at rest, a hole at the start of every row |
| 2026-09-26 | Media-info layout (B) | A main column (collections first, crew, cast) beside a facts sidebar; an empty column is not drawn | Every section full width, stacked | The collection is the most used; the keywords' column left a void beside a short details box |
| 2026-09-26 | Mixed Home shelf (A) | Plex titles as wide art beside videos | Posters, with the videos not stretched | Stretching a video card to a poster's height left an ~80px gap |
| 2026-09-27 | Home hero | A card (hover row + Play on the picture, the whole menu, a touch "…"; owner: A); three quarters wide and centred on a wide window; one fact line; the summary at the measure | The hero's pin row alone; full width; an 18rem picture column | The one answer stands in focus, and it offers what the same card offers anywhere |
| 2026-09-27 | Home shelf tiles | A channel / playlist tile: Play, ↗ open, "…" with Open; a show acts on its next episode, ⓘ opens it; the shelf slot 232px | The tile's pin row alone; 200px | A tile without its object's actions read as broken; at 200 the Play touched the hover row |
| 2026-09-27 | Synopsis preview (C) | A ~0.9 s mouse rest opens a passive panel beside the card | The synopsis as the picture's `title` tooltip; an ⓘ to hover; none | One long tooltip line fired anywhere on the art and clashed with the cut title's own tooltip |
| 2026-09-27 | Reading measure | Prose capped at ~65 characters a line (`measure`) | The container's full width | At 110–130 characters a line the eye loses the next line's start |
| 2026-09-24 | Chip family | One `Chip` (filter / token / tag / suggest / ghost) + `DismissibleChip`; nothing under 24; the lint rule "no raw `<button>` outside ui/" | Five chip geometries in five files | The applied-filter row, the tag cells, the year picker and the saved-view editor each drew their own chip; a one-letter tag was 21px wide |
| 2026-09-24 | Button families | `Button` / `IconButton` / `Pressable` / chips / `DragHandle`; `lg` 40 for a hero CTA; pills and a 48 Play on media (from C5) | Every button hand-painted | 268 hand-rolled buttons in 79 files drifted into as many looks; the ladder only holds if every button stands on it |
| 2026-09-27 | C4 as built | `initialFocus` required on every `Modal`; the commit row in `ModalFooter`, portalled under the body from any depth; the fades driven by the body's scroll timeline; `interactive-widget=resizes-content` | An optional prop defaulting to the ✕; a `footer` prop (the issue composer's buttons live three components down); always-on fades | A default is how 21 dialogs came to open on the ✕; lifting the composer's state to reach a prop is the larger change; a resting fade dimmed the first line |
| 2026-09-27 | C3 + C5 as built | A tap toggles the bar through `pointerType` from `pointerdown`; a double-tap takes back its first tap's toggle; the card's pair is the video card's `MediaResumeButtons` (Continue / Restart), a sibling over the picture; the YT description is an ⓘ click popover | A delayed single tap (wait out the double-tap window); new "Resume / From the start" words for Plex; the pair inside the picture's button | A delayed tap feels dead; one media card says one pair of words; a button inside a button is invalid and the hover row already sits beside the picture |
| 2026-09-27 | C2 as built | The side panel's drawer is `ui/Modal` `drawer-start`; drawer = a coarse pointer or below 768; the pin offered in its header ≥ 1024 | A drawer of its own inside `SidePanel` | Esc, Back, the scrim, the trap and the scroll lock are the dialog's; a second implementation is how two shells drift |
| 2026-09-30 | Composition and the reader (E30 S6) | A chapter and a Conformance step: the space is used, one focal point, worst real data, where am I, a click answers at once, zoom and text size first-class; the person page as two columns (facts and bio left, "Known for" right) | Rules for pieces only; the bio in a full-width panel; a narrower panel | The person page passed every piece rule with half its panel empty; the owner reads zoomed in, so zoom is a requirement, not a nicety |
| 2026-09-30 | Hungarian words (E30 S5) | The D3 list: Feed, Shorts, Admin, Token, Port stay; stream, Prio, Job, Audit, Demo, Auto, Info, Frontend/Backend, Backfill, transcode get the Hungarian word; one name per thing (kérelem, behúzás, one "Test connection") | Keep every loanword; translate Feed/Shorts too | The owner chose the list (D3); the final sweep found the same feature under two names in three places |
| 2026-09-29 | Numbers and order in the UI language (E30 S3) | Every number through `lib/format.ts` in the UI language (decimal mark, grouping, compact, units); A→Z by the shown name in the UI language's alphabet, client and server alike; the language comes off `useTranslation`, never the singleton | `toFixed`, hand-built "1.2K", `localeCompare` in the browser's locale, byte-order server sorts | A Hungarian reader saw "8.3", "1.2K", "Életrajzi" after "Z"; the singleton read also left values in the old language after a switch (React Compiler) |
| 2026-09-27 | Narrow rows by container | The download row, the playlist table and the Channels tag row stack by a container query; a fact line never wraps | Breakpoints on the window | The iPad zoomed in, split screen and a column beside a rail are narrow while the window is not; the zoomed download row broke "18.9 / MB" apart |
| 2026-09-27 | Phones deferred | A phone-only layout (the hamburger nav, a folded feed band) waits for its own epic; E28 stays desktop + tablet | Fix the phone views inside E28 | Principle 2; the first phone attempts made the feed worse, and the epic's review grows with every sprint |
| 2026-09-27 | Conformance check (S3g) | A four-step check before a UI change is done: primitive at a named rung, the chapter re-read, the rendered page measured by kind and look, every state seen | A reviewer's eye per sprint | The S1 probe found 69 button looks; after S3 every control kind is drawn by its primitive, and the check keeps a new look from arriving unnoticed |
| 2026-09-27 | Segmented rung | `SegmentedToggle` defaults to the toolbar rung (28, `text-xs`); 24 only for a panel's captioned match toggle | The small default | Two toolbars had inherited 10px type from the default beside 28px buttons |
| 2026-09-27 | Copy rule (S3g) | One name per feature (the rail may shorten it); four verbs for four meanings; sentence case with the app's places capitalised; "…" when a command asks first; one-sentence messages without a stop; curly quotes; dates relative in lists with the exact minute on hover, three absolute forms; units from Intl, short | Word-for-word names everywhere; the verbs as they were; all lower case; no "…"; always a stop; always absolute dates; long or compact units | The copy survey found 7 EN and 6 HU double names, 22 Delete against 27 Remove, five date shapes and three spellings of a minute; the owner picked every recommended option |
| 2026-09-27 | One glyph, one meaning (S4) | Play only for playback or resuming a paused transfer; Retry = `RefreshCw`; reset / revert = `RotateCcw` | Retry drawn with Play | The first UI-review run (Downloads) found a failed row whose Retry showed ▷ in the row and its menu |
| 2026-09-27 | Fjord kept | Fjord stays a fifth scheme beside Midnight; the trial wording goes | Fold it into Midnight; remove it | The owner lived with it and likes it; the vivid Midnight stays for those who want it. B2's own-hue rule still binds any new scheme |
| 2026-09-27 | Tablet columns | A table hides `tabletHidden` columns below 64rem of its own width; Channels drops Prio, About, Length | Hide About only; give the name a bigger share | At 1024 the channel name had 60 px; the owner chose to drop all three there |
| 2026-09-27 | Glyph pager | First / previous / next / last as glyph buttons round the page box, names as tooltips | Words beside the arrows (hidden only on a phone) | The owner's call: the words widened the row; four arrows are the shared convention |
| 2026-09-27 | Match switch on the panel | `ui/MatchToggle` above the picker from two picks, everywhere; one wording (Any / All) | The picker's popover footer (Feed, Plex collection) | A closed picker hid an active "all"; six pickers had drawn it two ways |
| 2026-09-27 | Missing picture, bare link | A glyph of the kind for no picture; a download with no name is titled by its short link, once | An empty box; the URL as title, channel and link | The owner's picks 4A and 3A: an empty box read as loading, the URL showed three times |
| 2026-09-27 | Copy on a selection | Right-click on selected words keeps the browser's menu; no app Copy row | An app menu with Copy first | A real browser's menu has copy, search and translate; only the desktop pane's is thin |
| 2026-10-01 | Zoom target 125–133% | Screens are held at 125% and 133% browser zoom (1536 × 729, 1444 × 685 on a 1080p window) and the largest text size | 200% browser zoom | The owner: 200% was a guess; 125–133% is what they read Siftlode and most sites at, so 200% would be an upper bound nobody uses |
| 2026-10-01 | E31 Nav rail at short heights | One "Admin" rail item opening a panel of the admin pages | A denser rail when short; a visible "+4" hint | The rail fits at 133–150% without a hidden scroll; the admin pages are rare and admin-only |
| 2026-10-01 | E31 Configuration sections | A vertical, grouped section list, as Settings has | Group tabs + sub-tabs; one long page with a contents list | Sixteen tabs wrapped to three rows at 133%; one pattern for both settings pages |
| 2026-10-01 | E31 Statistics overview | Number tiles, and the per-operation counts as horizontal bars | Three narrow cards; a 30-day chart | The label sits by its value; what spends the quota is seen at a glance |
| 2026-10-01 | E31 Channel page header | The full banner, folding to one row once the grid scrolls | Always compact; no banner | The header is fixed and took 40% of a 685px window; the channel keeps its face at first sight |
| 2026-10-01 | E31 Channel page watch state | A fresh channel page shows the unwatched videos; a picked state is remembered | Show all, forgotten on reload | The owner: open a channel to see what is new; a choice made there should stick |
| 2026-10-01 | E31 Audit log runs | A job's successful runs fold into one row that opens, whatever ran between; only that job's failure (never folded, danger colour) breaks it | A filter defaulting to admin actions; both | Nothing hidden, the noise collapsed, the failure seen |
| 2026-10-01 | E31 Users row | Role and state as badges, the actions in the row's one menu | An inline role toggle; icon-only buttons | Action placement: the row shows state, rare and destructive actions live in the menu |
| 2026-10-01 | E31 YouTube player | Video beside an info panel at 1280px and wider (the video as large as the height allows); stacked below | Today's stack with a one-line bar; full-window like Plex | The owner's eyesight: about 1460px of video on 1080p instead of 1008, the actions still visible |
| 2026-10-01 | E31 Plex info and show pages | A second column for the facts, keywords and crew, as the person page; a collection's hero sized to its content | Full-width text; a larger poster and fanart | The right ~40% was empty; one composition for every media page |
| 2026-10-01 | E31 Nav rail rows are links | Each page row is a real link: a middle-click or Ctrl-click opens the page in a new tab | A button | The owner: open a page in a tab of its own, as on any site |
| 2026-10-02 | E31 Plex detail second column (refined) | The facts over the key crew beside the text; the keywords, chips that filter like the genres, under the text; the show page gets all three too | All three in the column, as the mockup drew it | At 20 keywords and 12 crew the column ran to twice the text's height (849 vs ~400 px at 133%): a hole under the text instead of beside it |
| 2026-10-02 | E31 Plex detail poster and crew | The movie and show poster keeps its 2:3 and grows a rung in a wide hero (w-56 at a 64rem panel); the crew shows its first 6 names and "Show N more" (a list of 8 or fewer shows whole) | Poster stretched to the row; a "More" on the keywords | The row is as tall as the crew column, so the poster drew ~175×590 and cropped; the crew, not the keywords, set the hero's height |
| 2026-10-04 | E32 `field` focus skips a mode switch | A form opens on its first field; a control above it that is not a field (the composer's Edit / Preview) is passed over by marking the field `data-initial-focus` | The first text field everywhere | That moved the focus in the download dialog off its profile choice and in the upload dialog off its drop zone; the mark changes only the form that needs it |
| 2026-10-04 | E35 Lint for z-levels and edged labels | Both rules stay; the two eslint selectors (`Z_RAW`, `EDGED_LABEL`) that refused them go | Keep the selectors | 0 hits since the E28 sweep that fixed the sites: a check that never fires costs a probe test and a snapshot each, and the UI review holds both rules |
| 2026-10-06 | `player` focus | A dialog that is a player opens focused on its panel (the YouTube player's rule, now `Modal`'s too); on the media element itself, Space and the arrows stay the element's own | `close`, or the media element | On the ✕ Space closed the player; on the element the browser's own controls acted on the same key as the player's (one Space played and paused, measured in Chrome) |
| 2026-10-08 | Captions on touch | A `Tooltip` opens for a mouse and a key's focus only, never for a finger | A caption on tap; a long-press caption | On an iPad the tap that showed the caption was taken as a hover and lost its click (every rail button took two taps), and no finger sends the mouseleave that hides it; the rail's labels setting names the buttons on touch |
| 2026-10-08 | Header search on touch | No key cap on a touch screen; the pill grows to 52 px so a ring shows round the 44 px Go; Go drops to its icon by the column's width | Keep the key cap; a 44 px Go overhanging a 40 px pill | On a tablet the 44 px key cap and Go sat on the pill's edge and cut the placeholder; an overhanging Go breaks the nested radii |
| 2026-10-08 | Plex subtitles and the control bar | The player draws the cues itself; while the bar shows they rise above it (200 ms), then settle back | The browser's own cues, moved by `line` | On an iPad the bar covered the text; a native cue cannot be animated or measured, and our layer behaves the same on every engine |
| 2026-10-08 | Hover on a button over a picture | The `media` variant's hover is a 2px accent ring over a darker body (Plex + YT cards, rail arrows; option B) | An accent fill (reads as "selected"); a white wash (weak on a bright poster) | The 60 → 75% black step was invisible, and over a card a rail arrow could not be told from the card's own Play under it |
| 2026-10-08 | Plex "Add to collection" | A movie or show card's menu row opens a panel beside the card: its library's manual collections tick in place, a name + Enter creates one with the title in it (admin; option A) | An Add submenu with a "New…" dialog; the per-title editor modal from the menu | A collection took a trip to the Collections page, a name, a title search, then a second search per title; the panel is the video card's "Add to playlist" |
| 2026-10-08 | YouTube modal chrome | No prev/next strips beside the card; Close an icon (the panel's top right; stacked, the title row's end); the description clamped to three lines + More; the panel shuts to a 2.75rem strip of close, expand, prev/next (option D1) | Keep the strips; a "Close" text button; the full description in a scroller; a panel that hides outright | The strips cost the video 7rem of width for a second copy of the panel's buttons; a long description pushed the channel and the actions out of sight; a hidden panel left no way to step the queue without the keyboard |
| 2026-10-08 | The wheel over the YouTube modal | Over the video: volume. Over a panel with something to scroll: scroll only, never volume, not even at its end. Over a panel with nothing to scroll: volume (owner) | Volume everywhere; a scroll that hands over to the volume at its end | Reading a long description turned the volume down once it hit the end |
| 2026-10-09 | A Plex show card plays | Its picture plays the show's next episode everywhere (an episode with a resume point → Continue + Restart, else the first unwatched, else S1·E1), resolved on the click; the preview names it once up; title and ⓘ open the show (option A + hover ask) | The picture opens the show page (as before); every card carrying `next` | A show card on Discover had no Play while Home's had one; carrying `next` costs a grid page up to 180 queries for the few cards anyone plays |
| 2026-10-10 | Clip mode in the players (E38) | Scissors in the control row + `C`, `[` `]`; the panel in the player with the watched video as preview; overview + zoomed strip on `ui/RangeTrack` (ends are sliders); a toast + the Ready notification after Cut | A ⋯ menu entry; permanent In / Out buttons; a big dialog with its own preview (a second Plex transcode / YouTube iframe) | Owner's picks A, P1, a; a film is hours and a clip seconds, so only a zoomed strip gives a hand the precision. The Downloads editor's cuts moved onto the same track (keyboard-steppable now) |
| 2026-10-10 | Crop in clip mode (E38) | A Crop toggle in the clip panel: a frame over the playing picture (`ui/CropFrame`, the editor's crop too), Free / 16:9 / 9:16 / 1:1, Whole picture; maximised YouTube keeps the panel in the bar's place; the wide side panel shuts while clipping | A frame always on the picture in clip mode (K2); a free shape only | Owner's K1 + a: a crop is rare, so it never covers the picture unasked; a phone or square clip needs a held shape. The iframe hides its picture's size: drawn on the 16:9 box, the cut fits it to the real picture |
| 2026-10-10 | Marks on the player chrome | `--media-mark` (amber) and `--media-danger` (light red), the same in both themes; `media-marks` points the focus ring (`--focus`) and the chosen mark (`--chosen`: a track's tick, a tab's underline; the label stays white) at the amber on the player and every chrome surface; `media-scope` (a form) also the errors at the red | The theme's accent ring, chosen text, warn and danger | On the chrome (#171717) a light theme's accent ring (Slate 2.84:1), warn (2.99:1) and danger (2.77:1) all fall under 3:1; the accent as chosen text under 4.5:1 |
| 2026-10-11 | Files page (one library) | List rows by default, the media grid a toggle remembered per folder; a sidebar of automatic source folders, own folders, Shared with me and the storage meter; the paste-a-link box in the toolbar | A media grid by default | The library holds videos AND plain files: rows read both, show more per screen and scan by size and date |
| 2026-10-11 | Library list columns | Every kind of value in its own column (name, source, phone-safe, retention, size, added, actions), one grid with a subgrid per row so they line up; an action that does not apply keeps its slot empty; a right-click menu on every row (the ⋯ opens the same menu) | Badges flowing after the title; icons that shift when one is missing | Owner: values of one kind must fall in one vertical line; a row's every action reachable by right click |
| 2026-10-11 | Running downloads | An app-wide tray, bottom right, shown only while something runs or failed (reason + retry), foldable to a pill | A rail icon with a count and a popover | Downloading became an action you start anywhere; a running one must be visible without going to look |
| 2026-10-11 | Download done | The notification says whether it plays on a phone and carries Link másolása + Megnyitás (the folder, the item focused); never an automatic jump | A plain "ready" notice; auto-navigation to the item | The daily path is paste → done → copy the link; a jump would pull the user out of what they were doing |
| 2026-10-11 | Library row quick icons (owner A) | On a wide list only the ⋯ holds a place; Copy link, Share and Save come up over the date while the row is hovered or holds the keyboard; a card (phone) keeps them in its own line; Keeps narrower | Three icons always in sight (the name kept ~150px at 1200); dropping Added below 64rem | Owner: the name is what a row is read by; every action stays in the ⋯ and the right-click menu |
| 2026-10-12 | Files preview (owner ★) | A file that is not media opens from its name in the dialog shell, wide: an image, a PDF in the browser's own viewer, text as numbered plain text (HTML and SVG too) with a wrap toggle, else a no-preview card with Save; ← / → page the list's such files in its order; an image under 8 MB is its own thumbnail | A side panel beside the list; a new browser tab; pdf.js; syntax highlighting; a click that saves a file without a preview | Owner: one way to open anything, paged like the list; the browser's PDF viewer is free and isolated; a page is never rendered on our origin |
| 2026-10-12 | A running download's step (owner ★) | The tray row names the step beside its bar ("Downloading audio · 12%", "Merging" with a pulse); a paused-only tray shows a pause mark, never the done check | A bare bar and % | A download fetches video and audio as two streams: unnamed, the second fill reads as a restart; the tray is the only place a download is followed |
| 2026-10-14 | API tokens (owner A/A/A) | Settings → API tokens: a row per token in three lines (name + prefix, its folders and rights, last use and expiry), revoked from its ⋯; a new token asks the kind first (Hub exchange or Custom: rights as chips per folder), expiry 1 year by default; the token shows once in the same dialog, with Copy and a curl line | A table; one form with a preset link; the token in a banner above the list | A table truncates the rights at this width; the common token never shows a matrix to get wrong; closing the dialog is plainly when the token is gone |
| 2026-10-15 | Folder links (owner A/A/A) | An own folder's menu has Copy link and Share with link… as a file row does (the dialog is the link half of the file's); the link opens a list of the folder's files as it is now, Play opening the watch player above it with the playing row lit, a document only downloading; the rail marks a folder with a live link beside its count | Share with link… alone; cards; a download-only list; no mark | Mostly videos from a phone, read on a phone: a list gives a long name its line and the next one is a tap away; the folder link is the one place where moving a file in publishes it, so where it is must show |
| 2026-10-17 | Copy-only changes (owner ★) | Step 4 of the Conformance check alone (not 1–3 or 5): every state the changed text renders in, seen in both languages | The whole five-step check; no screenshot at all (DOM test + key parity) | Steps 1–3 and 5 judge controls, patterns and composition, which a text change does not touch; the one risk left is a longer Hungarian string wrapping or clipping, and only a screenshot shows that |
| 2026-10-17 | Text on glass (owner ★) | Light `--muted` 15% toward `--fg` in every scheme (≥ 6:1 on the bare page); the item count, the badges and the current Files row on opaque tints mixed into the surface | Only the two failing components; the item count's number in `--fg` | Pixel-measured over every scheme × mode × backdrop, the count fell to 3.8:1 (dark too: thumbnails scroll under the toolbar's glass) and Channels' tabs to 4.1:1; an `--fg` count outshouts the filters beside it |
| 2026-10-18 | Scheduler run history (owner D4, E1, E2 ★) | A row's last three runs as dots, newest first (interrupted: a hollow warn ring, set apart from skipped and failed), and a history button that opens the job's runs on the Activity page; a user's ad-hoc runs in their own block under the jobs, with no interval and no run button | An inline expandable history per row; a dark-grey interrupted dot; ad-hoc runs only on the Activity page | Three dots answer "did it run, did it work" at a glance; the whole history needs filters, which the Activity page has |
| 2026-10-18 | Activity page (owner D1, D2, G1–G3 ★) | One page, "Napló" / "Activity": four tabs (Runs, Admin actions, Server log, Quota) under one filter bar (text, a period of 1 h · 24 h · 7 d · 30 d · 90 d defaulting to 7 days, an account), the view in the URL; runs and admin actions as dense rows that open in place as a log line does, filtered and paged on the server; a run's or a request's log lines one pivot away, as a history entry | One merged timeline; a page per source; a default period per tab; card rows; the audit log's client-side table | Each source keeps its own table and retention, read in one place; one bar keeps what you narrowed across tabs; about 1.8k runs a day need a log's density and a server-side filter |
| 2026-10-24 | Related videos (owner U1–U3 ★, D1 A, D3) | On demand: a tab beside the description (wide), a drawer (stacked); the narrow thumbnailed row, its menu alone, a followed check; plays in place and the list stays the opened video's; an on-topic / Hungarian-trending switch kept on the account | Always shown; the end-of-video grid alone; a dialog; a big card or a text row; a list that follows the playing video | The modal is full on a small screen; a picture is what a choice among strangers rests on; a list that stays put is tried one by one, and spends no more of the day's cap |
| 2026-10-25 | Related: one played; off (owner D1 a, D2 a) | Played from the list, it is the player's item: its title, controls and progress, marked watched at its end, the queue left where it was; a daily limit of 0 takes the tab and the button away | A linked video's in-place play (no controls, nothing kept); a tab that only shows a refusal | What is watched should count as watched, wherever it was started from; a control that can only be refused is not offered |
| 2026-10-25 | Player panel glyphs (owner 1A, 2A, 3) | In the side panel the queue row is glyph-only on one line, and Mark watched is a glyph toggle among the actions (✓ / ✓✓); the docked bar keeps its labels; the related switch at the default size, "Trending" | Labelled buttons wrapping onto three lines; glyphs on the docked bar too; Mark watched on its own line | The panel is narrow and the list controls are the same few every time; each glyph keeps its full name and shortcut in its tooltip; the docked bar has the width for words |
| 2026-10-25 | Auto-advance glyphs (owner E A, H a) | Auto-advance wears list glyphs (`ListEnd` next and off, `ListStart` previous, `Shuffle` random), in the panel and the docked bar; the auto-advance and loop cycles are `lit`, not `pressed` | The skip arrows for the mode; an X'd list for off; `aria-pressed` on a cycle | Beside Previous / Next the skip arrows read as a second Next; a cycle of three or four modes is no two-state switch, and its name already says which |
| 2026-10-25 | Restart and Reset glyphs (owner L A) | A menu's Play from the beginning wears `RotateCcw`, as the card's own Restart does; Reset (clear progress and status) wears `Eraser`; Redownload wears `RefreshCw`, the retry arrow | The skip arrow ⏮ on the menu's Restart; `RotateCcw` on both Restart and Reset; `TimerReset` for Restart | ⏮ is Previous beside the player; one action wore two glyphs, and two actions in one menu would have worn one |
| 2026-10-28 | Notification order (owner A-D1 a, A-D2 a, ★ a) | What asks for the reader first, from either source ("Needs you · n"), then one stream newest first; read rows older than a week fold under "Earlier (n)", closed, its label in the section-heading style with a chevron | One stream with no band; this browser's activity above the server's (the two fixed sections); a server-side retention that deletes old rows | An admin's access requests sat under up to a hundred server rows; week-old read notices crowded what is new; folding keeps the rows (nothing is deleted) and an unread row never folds, whatever its age |
| 2026-11-01 | Docked bar glyphs (owner D8 a) | The maximised view's docked bar is glyph-only too (tooltips carry the names and keys): transport, auto-advance, loop, Mark watched (✓ / ✓✓); its controls scroll on a narrow window while Fullscreen and Exit stay pinned at its right end | Labels from `xl` up | On a tablet the labelled bar ran past the right edge and took Exit with it; the side panel had already lost its labels |
| 2026-11-01 | Toolbar group glyphs (owner D9 a, D10 a, D11 a) | The feed's, channel page's and Plex Browse's filter pills name their group by a glyph (`Eye` watch state, `Shapes` content types, `ArrowDownUp` sort), the word in the tooltip and the accessible name; Sort + View end the toolbar's row, the chip row shows only with a chip; a channel page has no count pill | The value alone; Sort + View on a row of their own | "Watch state:" and "Sort:" cost a tablet row its room; the value says enough once the glyph names the group; the channel header already counts the videos |
| 2026-11-01 | Channel header on a tablet (owner E2) | Below `xl` the action row drops its two words: "Prio" (the stepper's tooltip names it) and "Subscribed" (a ✓ glyph; its name in the aria-label, Unsubscribe in the tooltip); the tags keep the row's rest, folding into "+N" | The tag lane on a full line of its own below ~12rem; the tags under the channel name | On an iPad (820 px, 44 px targets) the lane got 29 px, a sliver; now 130 px, and every tag shows at 1180 |
| 2026-11-01 | Clear watch progress (owner E a) | A video menu's Reset reads "Clear watch progress" in English (HU "Megtekintés visszaállítása" stays); "Reset view" is only a page's view reset (sort, filters, grouping) | "Reset view" for both; the playlist page's view reset renamed instead | On a playlist page the two stood one right-click apart under one name, and one of them erases the video's progress with no undo |
| 2026-11-01 | Clear watch progress clears only that (owner A) | The row shows only on a video with a resume point and clears that alone: a hidden video stays hidden, a watched one watched. Un-hiding and un-watching have rows of their own | The old Reset, which also un-hid and un-watched | A hidden video stays hidden: what the row's name says is all it does |
| 2026-11-01 | Selecting a list's rows (owner D2 a) | An always visible box first in the row and one over the column for the rows shown; Shift+click a range, `x` the row under the keyboard, Esc clears; the selected row wears `accent/12` with its tick. Rows a filter hides stay selected, an action covers those shown; Watch later offers no selecting. | A "Select" button that shows the boxes on demand | Found at once and works on touch, with no mode to leave; costs about 2 rem a row |
| 2026-11-01 | Selection band (owner D1 a, D3 a, D4 a) | The toolbar's second row while rows are selected, no floating bar: move (the last target as one click) and remove, no copy; both undo back to the places, a video the target held before stays there; a single row's move undoes too. | A floating bar at the bottom; copying to a list too; no undo on a move | It covers no row, and the player and the notices live at the bottom; twenty videos sent to the wrong list need a way back to their places |
| 2026-11-01 | Grip on a phone (owner A) | Below a 24rem table a playlist row has no grip: the title keeps its width, and the row menu keeps the moves. | The grip and its drag kept, the title at 44 px | The title gets its width back, and the rail is shut on a phone, so a drag had nowhere to go |
| 2026-11-01 | Selection band on a tablet (owner A) | Below `xl` the band is glyphs on one row, the names in tooltip and aria-label, as on the channel header; the replaced row stays in the same grid cell, invisible, so the first tick moves no row under the second tap. | The buttons in words, wrapping to a second row | The first tick pushed the list from under the second tap |
| 2026-11-01 | Playlist heading on a narrow window (owner A) | The list's name had no width left: below `xl` Unsynced is its dot (the word read out, the hint in the tooltip); on a phone the cover is gone and Play all is its glyph, named in the tooltip. | Every word kept, the list's name at no width | A heading without its name says nothing |
| 2026-11-02 | Dragging onto a list (owner D5 a, D6 a) | The grip carries a row, or every selected row with a selected one, onto a list in the rail; the toast undoes. Many dropped inside the list stay; a coarse pointer's drawer never peeks, so its rail is a target only when pinned. | Only the grabbed row going along; the grip off under a view | One row moving out of three lit ones misleads, and a filter is how the rows to move are found |
| 2026-11-02 | Delete with every tick hidden (owner a) | It removes nothing and a warning says the filter hides them | Taking the row under the keyboard | While the band shows a selection, Delete means that selection: a row no one ticked must not go |
| 2026-11-02 | Undo of a video gone further (owner a) | A video the move put in the target comes back only if the target still holds it, and a warning says how many | Everything back, whatever it did since | A move skips a video that left the source; the same rule back keeps a video out of lists the reader never put it in |
| 2026-11-04 | Tag chip counts (owner D5a) | A Channels tag chip shows how many followed channels it holds: a muted number after the name, no badge | The number in brackets after the name | The owner's pick; the number is a detail of the name, so it takes the name's own text, muted, rather than a `CountBadge` |
| 2026-11-05 | Picked tag chips lead the row (owner c) | On the Channels tag row the picked chips come first, in the tags' order, then the rest | Keep the order and fold the unpicked first; mark the "+N" when it holds a pick | The row folds from its end, so a picked tag late in the list sat only inside "+N" with nothing on the row saying a filter was on |
| 2026-11-05 | A toggled tag chip keeps the focus (owner a) | A chip that moves when toggled takes the keyboard focus with it: to its new place, or to "+N" when it went into the closed fold | Keep a picked chip in the "+N" popover until it closes | The row always shows the filters as they are; the focus, not the order, was what got lost |
| 2026-11-06 | Tag chip count after a dot (owner a) | The count keeps the chip's own colour, set off from the name by a "·" | The number dimmed (it fell under the 4.5:1 text floor, resting and pressed); the number in a tinted inner badge | Contrast holds in every scheme without a new token, and the dot still reads the number as a detail of the name |
| 2026-11-06 | A pick in the open fold closes it (owner a) | Toggling a chip inside the "+N" popover closes the popover; the focus goes with the chip | Keep the popover open and focus the chip now in the picked one's place | The focus never sits outside a popover left open, and a picked chip leads the row anyway |
| 2026-11-06 | An expired share link offers no Copy (owner a) | An expired row in the share dialog has a warn badge "expired <date>" and its address struck through; Open and Revoke stay, Copy goes | The warn badge alone, Copy kept | An expired link opens nothing, so it is no link to hand out; Open still lets the owner check it |
| 2026-11-06 | The shared page's unlock button is a noun in Hungarian (owner a) | "Megtekintés" for a video, "Megnyitás" for a file or folder | Both verbs, "Megnézem" / "Megnyitom" | The app's Hungarian buttons are nouns; one button should not change grammatical form by link kind |
| 2026-11-07 | A slow link says so, with both rates (owner b1) | After 5 s of stalling on a link measured slower than the stream: "Slow connection" and "This file needs about X, about Y is arriving" under the spinner; no number measured, no claim | The line without numbers; a pill at the bottom | The viewer can tell their connection from a broken player, and by how much it falls short |
| 2026-11-07 | Text over the picture sits on a dark panel (owner a) | The player's loading, stall and error text, the subtitle notice and the track OSD sit on one dark panel (`OVER_VIDEO`, black 75 %, a white 15 % edge) | A shadow behind the letters; bare text as before | Bare white or red text was unreadable over a bright frame (seen live); the panel is the style the notices already had |
| 2026-11-07 | The panel over the picture is darker, its error the media red (owner a) | `OVER_VIDEO` is black 85 % under `media-scope`: an error on it is the media red in both themes | Only the red changed (3.75:1 on a white frame); black 90 % | A light theme's red was 1.6:1 on the 75 % panel over a white frame; now 5.47:1 at worst, white text too |
| 2026-11-07 | A total's left-out videos are "without length" (owner b) | A summed length names what it left out as "+N without length" / "+N hossz nélkül" | "+N live" as before; "+N stream" | The count holds archives and upcoming streams too, which a card calls a stream or upcoming: "live" there named only one of them (One name per feature) |
| 2026-11-08 | The Channels table's Length is the sum alone (owner) | The table's Length cell shows the summed hours only; the "+N without length" stays on a playlist's total | The line wrapped in the column; the column widened to 6.5 rem | The 4.5 rem column is sized for the sum: the longer line spilled into Tags (88 px in 56), and the owner wants no such line there |
+2 -1
View File
@@ -144,7 +144,8 @@ npm run typecheck # tsc x3 (app / node / e2e)
npm run lint # eslint (typed rules included; warnings fail)
npm run format:check # prettier
npm run knip # unused files/exports/deps
npm run test # vitest (incl. the compiler and gate-register contracts)
npm run test # vitest (incl. the compiler and gate-register contracts; the refusal contract runs
# the host's `python` on backend/tests/refusal_sites.py, stdlib only)
npx vite build # the production bundle (devOnlyGuard fails it on dev-only code, chunkGuard on a lazy page loaded at boot)
npm audit --omit=dev --audit-level=high # what ships to the browser, at high or above
```
+2 -2
View File
@@ -108,8 +108,8 @@ project checks needs a row.
| Scope | Narrowed | Why | Measured |
| --- | --- | --- | --- |
| check | e2e (Playwright) not in `check` | It needs the :8080 stack built from the tree (minutes). It runs in `check-full`, and `release` rebuilds the stack and requires it (E32 D2) — nothing ships without it. | 9 specs: 6 × 3 engines, 3 on chromium only (the row below) |
| e2e | `a11y.spec.ts` `member.spec.ts` `devices.spec.ts` run on chromium only (`CHROMIUM_ONLY`, the webkit and ipad projects' `testIgnore`) | What they hold does not change with the engine: the axe scan reads the DOM and the accessibility tree, which the engines build alike; the member spec checks which pages and requests a role gets; the devices spec opens its own desktop and iPad contexts. Run three times they would triple their time and catch nothing more. What an engine does change (layout, events) the other specs hold on all three. | 3 of 9 specs |
| check | e2e (Playwright) not in `check` | It needs the :8080 stack built from the tree (minutes). It runs in `check-full`, and `release` rebuilds the stack and requires it (E32 D2) — nothing ships without it. | 12 specs: 9 × 3 engines, 3 on chromium only (the row below) |
| e2e | `a11y.spec.ts` `member.spec.ts` `devices.spec.ts` run on chromium only (`CHROMIUM_ONLY`, the webkit and ipad projects' `testIgnore`) | What they hold does not change with the engine: the axe scan reads the DOM and the accessibility tree, which the engines build alike; the member spec checks which pages and requests a role gets; the devices spec opens its own desktop and iPad contexts. Run three times they would triple their time and catch nothing more. What an engine does change (layout, events) the other specs hold on all three. | 3 of 12 specs |
| pushes | no server-side gate (no CI) | Owner decision 2026-10-04. The gate (`scripts/check.sh`) runs on the dev machine before work is handed over, and `release` runs it, then e2e, before anything is merged and Trivy before the image is pushed — the CI only repeated that, after the merge. Its runner shared the git host's container, and one run there starved Gitea of memory until it stopped answering. What it added — a fresh `npm ci` on Linux — the image build does at publish. | — |
| coverage | not in the gate; no threshold | Owner decision E32 D4: never enforced; "every regression test seen red" is the bar, and a percentage invites tests written to move it. Out of the lanes since E35: read once, by hand, it found 5 bugs; as a lane it was a number nobody acted on, and the slowest vitest run (the one that flaked). On demand: docs/development.md. | frontend 80% / backend 78% of lines (E32 S1) |
| CVE gate | HIGH/CRITICAL with a fix available only | `--ignore-unfixed` + `--severity HIGH,CRITICAL`: a finding with no fix cannot be acted on by an upgrade; it shows in the scan output. A fixable one fails the publish, and the only way past it is `.trivyignore.yaml`. | 0 open (E32 S1) |
+4
View File
@@ -24,6 +24,10 @@ outlives an `expect` timeout); Node does not, so a request from the test side fa
that one error again and passes every other failure straight on. Don't run two e2e suites on the
same machine at once.
**A page load the machine had no buffer for** — on the same host a page load can fail at once with
`net::ERR_NO_BUFFER_SPACE`. Every page load goes through `gotoRetrying` (same file), which tries that
one error again; a test that calls `page.goto` itself fails `src/lib/e2eRetryConnect.test.ts`.
**Browsers** — every flow runs in three projects: `chromium` (Desktop Chrome), `webkit` (Desktop
Safari) and `ipad` (iPad gen 7 landscape: 1080×810, touch, mobile) — except `a11y`, `member` and
`devices`, which run on chromium only (`CHROMIUM_ONLY`; why is in the `docs/gates.md` row). Install the engines once per
+1 -1
View File
@@ -196,7 +196,7 @@ test.describe("a bookmark reopens the view it was taken of", () => {
test("a channel page's own tab, over the page it was opened from", async ({ page, browser }) => {
await openApp(page, "/?page=feed");
await expect(page.getByTestId("video-card").first()).toBeVisible();
expect(await clickVisibleChannelLink(page)).toBe(true);
await clickVisibleChannelLink(page);
const title = await page.getByTestId("channel-title").textContent();
await tab(page, "About").click();
const fresh = await bookmarked(browser, page);

Some files were not shown because too many files have changed in this diff Show More